Making Computations and Publications Reproducible with Vistrails

R eproducible R e s e a r c h f o r S c i e n t i f i c C o m p u t i n g

Making Computations and Publications Reproducible with VisTrails

The VisTrails system supports the creation of reproducible experiments. VisTrails integrates data acquisition, derivation, analysis, and visualization as executable components throughout the scientific exploration process, and through systematic provenance capture, it makes it easier to generate and share reproducible results. Using VisTrails, authors can link results to their provenance, reviewers can assess the experiment’s validity, and readers can repeat and utilize the computations.

mportant scientific results give insight and www.vpf.ethz.ch/services/researchethics/Broschure. lead to practical progress. The ability to pdf), funding agencies, conferences (www. test these results is crucial for science to be sigmod2011.org/calls_papers_sigmod_research_ self-correcting, and the ability to reuse and repeatability.shtml), and journals (www. Iextend the results enables science to move for- signalprocessingsociety.org/publications/periodicals/ ward. In natural science, long tradition requires tsp) have started to encourage (or require) authors that results be reproducible, and in math, results to include reproducible results in their publications. must be accompanied by formal, verifiable proofs. However, a major barrier to the wider adoption However, the same standard hasn’t been applied of reproducibility is the fact that it’s hard for au- for the results of computational experiments. thors to derive a compendium that encapsulates Most computational experiments are specified all the components (for example, the data, code, only informally in papers, where experimental parameter settings, and environment) needed to results are briefly described in figure captions, reproduce a result; and even when a compendium and the code that produced the results is seldom is available, it’s often hard for reviewers to verify available. The lack of reproducibility for computa- results. tional results currently reported in the literature As a step toward simplifying the creation and has raised questions about their reliability1 and led review of reproducible results, and motivated by to a widespread discussion on the importance of the needs of computational scientists, we built computational reproducibility. an infrastructure that supports the life cycle of Academic institutions such as the Swiss Federal computational experiments. A key component Institute of Technology, Zurich (ETH, Zurich; of this infrastructure is a provenance management system that systematically and transparently

1521-9615/12/$31.00 © 2012 IEEE captures the metadata necessary to reproduce Copublished by the IEEE CS and the AIP experiments, including the specifications of the computations, input and output data, source code, Juliana Freire and Claudio T. Silva and library versions. We also developed a set of Polytechnic Institute of New York University solutions to address practical aspects related to

2 This article has been peer-reviewed. Computing in Science & Engineering

CISE-14-4-Freire.indd 2 6/8/12 10:41 AM Figure 1. Anatomy of a real reproducible paper that investigates Galois conjugates of quantum double models.3 Figures in the paper are accompanied by their provenance, and users and reviewers can execute and examine the interactive results on the Web.

reproducibility, including methods to link results This provenance information allows all of the to their provenance, explore parameter spaces, paper’s results to be reproduced. In the paper’s PDF wrap command-line tools, interact with results version (available at http://arxiv.org/abs/1106.3267), through a Web-based interface, and upgrade the the figures are active and, when clicked, their cor- specification of computational experiments to responding workflow is loaded into VisTrails and work in different environments and with newer executed on the reader’s machine. The reader can versions of software. This infrastructure has been then modify the workflow, change parameter val- implemented and released as part of VisTrails ues, and input data. The same provenance also (www.vistrails.org), an open source workflow- enables the result to be published on a website, based data exploration and visualization tool,2 where users and reviewers can execute it and ex- and it’s already being used by different groups of amine the results using a Web browser.4 scientists. Videos that illustrate the process to create reproducible publications using VisTrails VisTrails and Provenance are available at www.vistrails.org/index.php/ for Computational Experiments RepeatabilityCentral. In this article, we give an Provenance is a critical ingredient for reproduc- overview of this infrastructure and its components, ible experiments.5,6 If we know how a figure or how it can be used, and its benefits and limitations. table was generated (the computational processes and data used), we can incorporate them in the Creating Reproducible Papers paper so that the result can be reproduced. How- Before discussing how to create a reproducible pa- ever, because computational experiments can be per, let’s first examine a real reproducible paper. complex and their design involves many trial-and- Figure 1 illustrates the anatomy of a reproducible error steps, it’s easy to get lost. For example, it’s paper created using our infrastructure. This paper easy to forget the exact parameter values or the investigates Galois conjugates of quantum double version of an input file that was used to derive a models.3 Figures in the paper are accompanied specific result. Therefore, systematic mechanisms by their provenance, consisting of the workflow are needed to capture the provenance of these used to derive the plot, the underlying libraries experiments. invoked by the workflow, and links to the input In our infrastructure, we’ve adopted the Vis- data—simulation results stored in an archival site. Trails system as a means to capture provenance.

July/August 2012 3

CISE-14-4-Freire.indd 3 6/8/12 10:41 AM VisTrails and (www.avs.com) and ParaView (www.paraview.org), Vis- Trails makes advanced scientific and information visualiza- Related Systems tion techniques available to users, letting them explore and compare different visual representations of their data. As a isTrails (see www.vistrails.org/usersguide) is an open result, users can create complex workflows that encompass V source system designed to support exploratory com- important steps of scientific discovery, from data gathering putational experiments. VisTrails is written in Python and and manipulation to complex analyses and visualizations, uses Qt as its GUI toolkit (through PyQt Python bind- all integrated in one system. ings). It is multiplatform and runs on Windows, Mac, and There are two key aspects that distinguish VisTrails from Linux. Since its beta release in 2007, the system has been these systems. First, it provides comprehensive provenance downloaded more than 35,000 times. The VisTrails wiki has support: in addition to capturing data provenance (for had more than 1.2 million page views, and Google Analyt- example, the steps followed to create a given data prod- ics reports that visitors to the site come from 75 different uct), VisTrails also captures provenance of the exploration countries. process, including the trial-and-error refinements applied VisTrails includes and substantially extends useful to workflows. Second, VisTrails has been a pioneer in the features of scientific workflow and visualization systems. support of reproducible publications. It has introduced Similar to scientific workflow systems such as Kepler functionality for sharing and publishing computational (https://kepler-project.org) and Taverna (www.taverna.org. experiments, including the ability to link results reported in uk), VisTrails allows the specification of computational pro- a document to their provenance, run workflow in multiple cesses that integrate existing applications, loosely coupled environments, manage files manipulated by workflows, resources, and libraries according to a set of rules. As with and automatically upgrade workflows when the underlying visualization systems such as Advanced Visual Systems libraries change.

(For a basic overview of VisTrails and a discussion and visualization systems. To cater to a broader about other possible tools, see the sidebar “Vis- set of users, including many who don’t have pro- Trails and Related Systems.”) gramming expertise, it leverages provenance in- Compared to both scientific workflows and -vi formation to provide a series of operations and sualization systems, a distinguishing feature of user interfaces that simplify workflow design and VisTrails is its provenance infrastructure: Vis- use—including the ability to create and refine Trails was designed from the start to both capture workflows by analogy, to query workflows by -ex and leverage provenance information. VisTrails ample, and to suggest workflow completions as us- captures a detailed history of the steps followed ers interactively construct their workflows using a and data derived in the course of an exploratory recommendation system.5 We’ve also developed a task. Workflow systems have traditionally been framework that lets users create custom applica- used to automate repetitive tasks, but in applications (mashups) that can be more easily deployed tions that are exploratory in nature, such as simu- to end users.7 lations, data analysis, and visualization, not much is repeated—change is the norm. As a user gener- Sharing and Publishing Results ates and evaluates hypotheses about their data, a Although capturing provenance is a necessary series of different, but related, workflows are cre- step for creating reproducible results, other is- ated as they’re adjusted iteratively. VisTrails was sues should be considered when sharing these designed to manage these rapidly evolving work- results—notably, how they should be packaged so flows and maintains provenance of that they can be understood by people other than their authors as well as executed in environments • data products (such as visualizations and plots), different from the one where they were created. • the workflows that derive these products, • the workflow execution log, Specifying computations as workflows. In our in- • information about the underlying tools and li- frastructure, computational processes are speci- braries invoked by the workflows, and fied as pipelines (or workflows). Workflow systems • user-defined annotations that enrich the auto- support the creation of pipelines that combine matically captured provenance. multiple tools. As such, they enable the automa- tion of repetitive tasks and result reproducibility. VisTrails addresses important usability issues Workflows are rapidly replacing primitive shell that have hampered a wider adoption of workflow scripts in a wide range of tasks, as evidenced by

4 Computing in Science & Engineering

CISE-14-4-Freire.indd 4 6/8/12 10:41 AM several workflow-based applications, both com- and sometimes impossible. To address this issue, mercial (Apple’s Mac OS X Automator and Yahoo! we developed mechanisms (which we detail later) Pipes) and academic (NiPype, Kepler, and Taverna). that provide reliable links between data and their Workflows have several advantages compared provenance9 and extended VisTrails to support to scripts and programs written in high-level the evolution of software libraries used in the languages. They provide a simple programming workflows.10 model whereby a sequence of tasks is composed by connecting the outputs of one task to the inputs of Connecting data to provenance. The common another. Figure 1 shows an example workflow that practice of connecting workflows and data prod- reads a zip archive from the Web that contains a ucts through file names has important limitations. set of result files, uncompresses it, performs an Consider, for example, a workflow that runs a sim- analysis, and uses matplotlib to output a plot. ulation and outputs a file with a visualization of This simpler programming model lets workflow the simulation results. If the workflow outputs an systems provide intuitive visual programming image file to the file system, any future run will interfaces, which makes them more appealing overwrite that image file. Also, if we use differ- for users who don’t have substantial program- ent parameters or improve the simulation code ming expertise. It also makes it possible to cre- and run the updated workflow, the original image ate abstractions to represent a given workflow at is lost. different levels of granularity thereby simplifying If a version-control system managed that im- its presentation and making it easier for users to age file, the user could retrieve the old version understand its functionality.7 from the repository. However, if users revert the Another benefit of using workflows is that they output image to the original version, how would have an explicit structure: they consist of graphs, they know how it was created? Because there’s where nodes represent processes (or modules) no explicit link between the workflow instance along with their parameters, and edges capture (the workflow specification, parameters, and the flow of data between the processes. We can input files) and the different versions of its out- exploit this structure to allow rich queries (and put, determining provenance is challenging. This operations) over workflow collections, enabling problem is compounded when computations take knowledge reuse.8 place in multiple systems, and recording the com- To specify computations as a workflow, we must plete provenance requires tying together multiple integrate with the workflow system the user- workflows through their outputs and inputs. As defined functions and libraries (such as simula- files are overwritten, renamed, or moved, prov- tion codes and specialized visualizations) required enance information could be lost or become by the computations. VisTrails provides a pack- invalid. age API for this purpose. A VisTrails package is VisTrails implements a persistence framework simply a collection of Python classes—stored in that, by coupling workflow provenance with the one or more files—that respect specific conven- versioning of data produced and consumed by tions and that invoke the user-defined functions. workflows, captures the actual changes to data For example, the workflow in Figure 1 makes use as well as detailed information about how those of a package that wraps the ALPS library (http:// changes came about.9 The persistence package alps.comp-phys.org), which provides high-end contains modules that identify data by their con- simulation codes for strongly correlated quantum tents (via hashing), its use (via the workflow that mechanical systems. The system also provides a generated it), and its history (via a version-control mechanism, CLTools, that simplifies the process system). Each version of a result is connected, of wrapping command-line tools. For more infor- through a strong link, to the provenance that mation about these, see the VisTrails User’s Guide details how the result was generated. This ensures (www.vistrails.org/usersguide). that an author can always retrieve data used in Even when the provenance associated with a previous work, even if the original file has been result is represented as a workflow, shipping the changed or removed. workflow to be run in an environment different Instead of relying on users or ad hoc approaches from the one in which it has been designed still to automatically derive file names, strong links raises many challenges. From hard-coded loca- are identifiers derived from the file content, the tions for input data to dependencies on specific li- workflow specification, and any parameters used. brary and hardware versions, adapting workflows As a result, they accurately and reliably tie a given to run in a new environment can be challenging workflow instance to its input and derived data.

July/August 2012 5

CISE-14-4-Freire.indd 5 6/8/12 10:41 AM Besides simplifying the process of maintaining local machine can be challenging, so we’ve devel- data provenance, this approach provides a general oped workflow components to automate the mon- mechanism for the persistent caching of both in- itoring process. Another concern is a result that termediate and final results, which can be useful, uses a very large dataset or data that’s proprietary; for example, to package workflows that include although we might be able to run the workflow long-running computations. Furthermore, the locally, we might need to remotely query or ag- use of a managed data repository allows the cre- gregate the data. We developed modules that work ation of workflows that are location agnostic— with relational databases to support such remote that is, unlike workflows that point to files in the queries in local workflows. It’s also possible for au- file system, these workflows can be shared and run thors to create modules that are specific to their in multiple environments unchanged. data and provide remote access to their own computational infrastructure (hardware and software). Software and computational environment. When shipping experiments to be run in a new environ- Publishing ment, configuration can be challenging. Although To support provenance-rich results in papers and VisTrails is cross-platform, the code, libraries, the ability to link back to the workflows that de- and other dependencies underlying a workflow rived them, we developed code and plug-ins for often aren’t. A good practice, when possible, is to LaTeX, wiki, Microsoft Word, and PowerPoint. package the environment (or a subset of it) together This lets authors easily embed and reproduce re- with the experiments. This can be achieved sults, and lets readers follow links to and explore through the use of virtual machines or packaging the actual computations. Figure 2a shows the systems such as CDE that capture system-level LaTeX publishing interface provided by VisTrails. dependencies (www.pgbovine.net/cde.html). A user can configure how the results should be Another issue that must be addressed is soft- included—for example, to include a PDF fig- ware and hardware evolution. As systems evolve ure and ship the workflow Include( .vtl) with and hardware changes, simply archiving an ex- the paper. This interface, in turn, generates a ecutable won’t suffice. Even archiving code can stub that can be included in a LaTeX file. Here’s be problematic as language specifications change. an example of how to define the inclusion of a Archiving environments as virtual machines usu- provenance-rich figure in a paper: ally allows reproduction but presents issues with scale. Furthermore, while exact reproduction is \begin{figure} important to verify results, utilizing published \vistrail[filename=ladder_dyl_ gap_theta-2.xml,version=5,pdf, work to extend solutions is more efficient when buildalways,getvtl,embedworkflow, modern tools and algorithms can be substituted. execute]{width=8cm} Such upgrades can accelerate progress by allow- \caption{(color online) Ground-state ing readers to take advantage of the new hardware degeneracy splitting of the and infrastructure. To address this, VisTrails non-Hermitian doubled Yang-Lee model stores—as part of a workflow’s provenance—the when perturbed by a string exact version of each module used and provides tension ( = 0).} an automatic workflow upgrade mechanism.10 If \label{fig:figure} a reader downloads an old experiment, VisTrails \end{center} \end{figure} can automatically upgrade the computations to match the current environment. This is accom- plished in part by allowing package developers to By including the VisTrails LaTeX package specify upgrade paths when specifications change. (\usepackage{vistrails}) in the document, when the document is compiled, VisTrails is in- Local, remote, and mixed execution. Because special voked to execute the \vistrails macro and hardware used in experiments isn’t always readily insert the result in the PDF. The user can also available and input data are sometimes too large, choose to serve the workflow from a database, in it’s important to provide a flexible mechanism that which case, the \vistrail command specifies allows different kinds of execution. We’ve been the host and database name: exploring different strategies to support workflow execution on clusters using native code and via \vistrail[host=alps.ethz.ch,db= Kitware’s ParaView (www.paraview.org) for paral- vistrails,vtid=10,version=169,pdf] lel execution. Controlling these executions from a {width=8cm}

6 Computing in Science & Engineering

CISE-14-4-Freire.indd 6 6/8/12 10:41 AM (a) (b)

Figure 2. Embedding reproducible results in publications. VisTrails provides an interface that lets users include in papers provenance-rich results that can be executed on (a) the users’ client machine or (b) on a server (using the CrowdLabs site through a Web browser).

Additionally, it’s possible to have papers point discussed, by appropriately packaging the experi- to results that can be executed on a remote server. ments, it’s possible for a reviewer to reproduce and For this, we use CrowdLabs (www.crowdlabs. test the existing computations. In the absence of org), a social website where users can share their an infrastructure such as the one we developed, workflows with the associated provenance and re- the reviewing task can be frustrating and time- sults. A user can upload workflows to the Crowd- consuming. Labs server, where they can be executed from a A reviewer also needs to test different param- Web browser. As Figure 2b illustrates, CrowdLabs eter configurations and access and interact with also generates stubs that can be included in docu- the computations. VisTrails provides a param- ments, including LaTeX, a wiki, or an HTML eter exploration interface for quickly selecting page, so that these documents point to the execut- and setting ranges that’s coupled with an intui- able workflow on the server. This makes it easier tive spreadsheet interface for visually comparing for readers to reproduce the results, because they results.2 This lets reviewers assess how general a don’t need to install any special software on their solution is, how much tuning it requires, and de- machines. termine in which cases it fails. We should mention that VisTrails includes a For journals and conferences that evaluate com- mashup editor that lets authors create Web mash- putational results, the availability of a common, ups based on their workflows to make them more unified infrastructure (such as the one we devel- accessible to readers.7 For example, they can select oped) guarantees the uniformity of representation a subset of the workflow’s parameters and sug- across experiments so that reviewers need not re- gest values for reviewers to explore computations. learn the experimental setup for each submission. Similar to workflows, mashups can be made avail- This was an important motivation for suggest- able through CrowdLabs, where readers and re- ing the use of VisTrails in the guidelines for the viewers can interactively modify parameters and ACM Sigmod Repeatability initiative (see www. analyze the results. CrowdLabs uses Flash to sup- sigmod2011.org/calls_papers_sigmod_research_ port interactive visualizations, and we’re currently repeatability.shtml). exploring newer technology, such as HTML5 and Web Graphics Library (WebGL) to provide bet- Looking Ahead: ter interaction with graphical results. Challenges and Opportunities We’ve described our efforts to simplify the cre- Reviewing: Reproducibility and Workability ation, review, and reuse of reproducible experi- A reproducible paper has the potential to improve ments. Our infrastructure is in production; it has the quality of reviews, because reviewers have the been used to create a number of papers and to ability to explore and validate conclusions. As we support the ACM Sigmod Repeatability initiative.

July/August 2012 7

CISE-14-4-Freire.indd 7 6/8/12 10:41 AM Our design has been guided by close collabora- (for example, “find experiments similar to mine”) tions with scientists who are authoring reproduc- and better understand existing tools and how they’re ible papers. We offer a more detailed description used. Furthermore, such collections let the com- of real use cases for our reproducibility infrastruc- munity evaluate a contribution’s impact not only ture elsewhere.11 through the citations to a paper, but also through End-to-end and long-term reproducibility of a the use of the proposed software and data com- scientific result is hard to achieve due to factors ponents. Although some repositories—such such as the use of specialized hardware, propri- as nanoHub, myExperiment, and CrowdLabs— etary data, and inevitable changes in hardware already cater to different aspects of reproducibil- and software environments. Nonetheless, with ity, they’re still in their infancy, and questions re- the infrastructure we’ve built, it’s possible to ac- main about what their architecture should be and curately document the processes through prov- even how they will be used.8 enance capture, as well as to attain reproducibility for a result’s important subcomponents, such as Acknowledgments the analysis and visualization of data derived from We thank the VisTrails team for making this work simulations run on special hardware. possible, especially David Koop, Emanuele Santos, As it stands, the infrastructure consists of a set and Huy Vo. We also thank many collaborators and of core functionality required for reproducibility; users who have provided insightful feedback on the it’s by no means comprehensive. Our long-term design and implementation of our reproducibility goal is to use it as the basis for a general system infrastructure—in particular, Philippe Bonnet, Den- in which different components can be mixed and nis Shasha, Joel Tohline, and Matthias Troyer. The matched to cater to the range of requirements in Department of Energy (DOE) Office of Science, Bio- different domains and for different types of scien- logical and Environmental Research (BER) and the tific results. National Science Foundation partially supported Our infrastructure’s current version is based on this work under awards IIS-1139832, IIS- 1142013, and requires the use of the VisTrails system. We IIS-0905385, IIS-1153728, CNS-1153503, and recognize, however, that such an approach might AGS-0835821. not always be desirable—in particular, for research that requires the use of interactive tools that can’t References be wrapped within a workflow system. We’re cur- 1. D. Donoho et al., “Reproducible Research in Compu- rently extending our infrastructure to support tational Harmonic Analysis,” Computing in Science & other systems that capture provenance. We’ve Eng., vol. 11, no. 1, 2009, pp. 8–18. also developed a plug-in mechanism that lever- 2. J. Freire et al., “VisTrails,” The Architecture of Open ages the VisTrails provenance subsystem to add Source Applications, ch. 23, Lulu, 2011; www.aosabook. provenance support to other tools12—including, for org/en/vistrails.html. example, ParaView, VisIt, and Autodesk’s Maya. 3. M.H. Freedman et al., “Galois Conjugates of Topo- logical Phases,” Physical Rev. B, vol. 85, no. 4, 2012; http://link.aps.org/doi/10.1103/PhysRevB.85.045414. s collections of reproducible compu- 4. P. Mates et al., “Crowdlabs: Social Analysis and Visu- tational experiments (along with their alization for the Sciences,” Proc. 23rd Int’l Conf. Sci- source code, raw data, workflows, entific and Statistical Database Management (SSDBM), and provenance) become available in Spring-Verlag, 2011, pp. 555–564. Acommunity-accessible repositories, new soft- 5. J. Freire et al., “Provenance for Computational Tasks: ware can be built upon this verified base. In data- A Survey,” Computing in Science & Eng., vol. 10, no. 3, intensive areas of science, significant amounts of 2008, pp. 11–21. knowledge accumulated by practicing scientists as 6. C. Silva, J. Freire, and S.P. Callahan, “Provenance for best practices aren’t necessarily formalized. By pub- Visualizations: Reproducibility and Beyond,” Comput- lishing the provenance of exploratory processes, ing in Science & Eng., vol. 9, no. 5, 2007, pp. 82–89. such as that captured by VisTrails, it is possible to 7. E. Santos et al., “VisMashup: Streamlining the make this knowledge more precise, bring it to the Creation of Custom Visualization Applications,” IEEE light of science, and make it verifiable by others. Trans. Visualization and Computer Graphics, vol. 15, The availability of fully documented experiments no. 6, 2009, pp. 1539–1546. enables scientific advances that combine previous 8. J. Freire, P. Bonnet, and D. Shasha, “Exploring tools as well as ideas. For example, members of the Coming Repositories of Reproducible Experi- the community can search for related experiments ments: Challenges and Opportunities,” Proc. Very

8 Computing in Science & Engineering

CISE-14-4-Freire.indd 8 6/8/12 10:41 AM Large Databases (PVLDB), vol. 4, no. 12, 2011, University. Her research interests include data man- pp. 1494–1497. agement, large-scale information integration, Web 9. D. Koop et al., “Bridging Workflow and Data mining, visualization, and provenance. Freire has a Provenance Using Strong Links,” Proc. Scientific and PhD in computer science from the State University Statistical Database Management Conf. (SSDBM), of New York at Stony Brook. She’s a member of the Springer-Verlag, 2010, pp. 397–415. ACM and IEEE. Contact her at [email protected]. 10. D. Koop et al., “The Provenance of Workflow Up- grades,” Provenance and Annotation of Data and Pro- Claudio T. Silva is a professor of computer science and cesses, LNCS 5272, Springer-Verlag, 2010, pp. 2–16. engineering at the Polytechnic Institute of New York 11. D. Koop et al., “A Provenance-Based Infrastructure University. His research interests include visualiza- to Support the Life Cycle of Executable Papers,” tion, geometry processing, graphics, and high- Procedia Computer Science, vol. 4, 2011, pp. 648–657; performance computing. Silva has a PhD in comput- http://dx.doi.org/10.1016/j.procs.2011.04.068. er science from the State University of New York at 12. S.P. Callahan et al., “Towards Provenance-Enabling Stony Brook. He’s a member of IEEE, the ACM, Euro ParaView,” Provenance and Annotation of Data and graphics, and Sociedade Brasileira de Matematica. Processes, LNCS 5272, Springer-Verlag, 2008, Contact him at [email protected]. pp. 120–127. Selected articles and columns from IEEE Computer Juliana Freire is a professor of computer science and Society publications are also available for free at engineering at the Polytechnic Institute of New York http://ComputingNow.computer.org.

NEW from COST-BENEFIT ANALYSIS OF QUALITY PRACTICES Robert T. McCann A general approach to monitoring and analyz- ing rework costs that provides explicit formulae to support an understanding of cost-bene t analysis results. ISBN 978-0-7695-4659-9 • 7” x 10” • 75 pp.

Order .PDF ($15 / $12 members): http://bit.ly/AnO1JT

Order Paperback ($19): http://bit.ly/wHBAk4

July/August 2012 9

CISE-14-4-Freire.indd 9 6/8/12 10:41 AM