Flexible Generation of E-Learning Exams in R: Moodle Quizzes, OLAT Assessments, and Beyond

JSS Journal of Statistical Software June 2014, Volume 58, Issue 1. http://www.jstatsoft.org/

Achim Zeileis Nikolaus Umlauf Friedrich Leisch Universität Innsbruck Universität Innsbruck Universität fur¨ Bodenkultur Wien

Abstract The capabilities of the package exams for automatic generation of (statistical) exams in R are extended by adding support for learning management systems: As in earlier versions of the package exam generation is still based on separate Sweave files for each exercise – but rather than just producing different types of PDF output files, the package can now render the same exercises into a wide variety of output formats. These include HTML (with various options for displaying mathematical content) and XML specifications for online exams in learning management systems such as Moodle or OLAT. This flexibility is accomplished by a new modular and extensible design of the package that allows for reading all weaved exercises into R and managing associated supplementary files (such as graphics or data files). The manuscript discusses the readily available user interfaces, the design of the underlying infrastructure, and how new functionality can be built on top of the existing tools.

Keywords: exams, e-learning, multiple choice, arithmetic problems, Sweave, R,LATEX, HTML, XML, IMS QTI, Moodle, OLAT.

1. Introduction

The design for version 1 of the exams package was conceived eight years ago (in 2006) when the original authors (Grun¨ and Zeileis 2009) were involved in a redesign of the introductory statistics lecture at WU Wirtschaftsuniversität Wien. Back then the main goal was to be able to produce exams along with associated self-study materials as PDF (portable document format) files. Thus, the main focus was still on printable materials for classic classroom exams. Although e-learning systems started to become available more easily back at that time, they were still not very widely used and, more importantly, rather few easy-to-use standards for specifying e-learning exams were available (e.g., WU Wien used a partially self-developed 2 Flexible Generation of E-Learning Exams in R e-learning system based on .LRN, see Blesius et al. 2007). However, since 2006 the situation has clearly changed: E-learning systems are now abundant with many universities offering one (ore more) e-learning system(s) in which all students are readily registered. Consequently, many lecturers routinely offer online exams (or tests, quizzes, assessments) for large-lecture courses – either as self-study materials or as (part of) the main assessment of the course. Among the more popular choices of learning management systems are the open-source systems Moodle – developed by Dougiamas et al. (2014) and supported by a large world-wide user community – or OLAT (for online learning and training) – originally developed by Universität Zurich¨ (2012) and with the recent fork OpenOLAT being developed by frentix GmbH(2014) and a support community. A popular proprietary learning management system is Blackboard developed by Blackboard Inc.(2010). Standards for specifying and exchanging e-learning exams/assessements have also emerged (see Agea et al. 2009, for an overview). While Moodle specifies its own Moodle XML format (but can import and export several other formats), OLAT/OpenOLAT and Blackboard employ certain subsets of the international QTI (question & test interoperability) standard, version 1.2, maintained by the IMS Global Learning Consortium, Inc.(2002). The successor formats are QTI 2.0 and the current QTI 2.1 which is for example employed in the ONYX testsuite (BPS Bildungsportal Sachsen GmbH 2014) that also offers interfaces to OLAT and Blackboard. Therefore, although the PDF exams produced by version 1 of the exams package as introduced by Grun¨ and Zeileis(2009) are still useful for many types of courses, it would also be highly desirable to have support for generating e-learning exams from the same pool of exercises. In fact, this became an apparent need when the authors of the present manuscript took over new large-lecture statistics and mathematics courses at their respective institutions (Univer- sität Innsbruck and Universität fur¨ Bodenkultur Wien, BOKU, respectively). For example, the new “Mathematics 101” lecture at Universität Innsbruck is currently attended by about 1,600 students (mostly first-year business and economics students) and accompanied by bi- weekly online exams conducted in the university’s OLAT learning management system. This was a strong incentive to start developing version 2 of the exams package that is presented here and offers an extensible toolbox for generating e-learning exams, including easy-to-use functions for Moodle quizzes and OLAT assessments1. The new version of the exams package for the R system for statistical computing (R Core Team 2014) is available from the Comprehensive R Archive Network at http://CRAN.R-project. org/package=exams. Like prior versions it employs ideas and technologies from literate programming and reproducible research (see e.g., Knuth 1992; de Leeuw 2001; Leisch and Rossini 2003; Kuhn 2014) by using Sweave() (Leisch 2002) to combine data-generating processes in R with corresponding questions/solutions in LATEX(Knuth 1984; Lamport 1994). But in addition to producing exams in PDF format, the new version of exams includes extensible tools for generating other output formats without having to modify the pool of exercises. Thus, the design principles of the exams package are only somewhat extended compared to version 1:

Each exercise template (also called “exercise” for short) is a single Sweave file (.Rnw) interweaving R code for data generation and LATEX code for describing question and solution (possibly containing mathematical notation in LATEX). 1Currently, OLAT and OpenOLAT do not differ with respect to their specification of exams. Hence, essentially all discussion of OLAT in this manuscript applies to both OLAT and OpenOLAT. Journal of Statistical Software 3

Exams can be generated by randomly drawing diﬀerent versions of exercises from a pool of such Sweave exercise templates. The resulting exams can be rendered into various formats including PDF, HTML, Moodle XML, or QTI 1.22 (for OLAT/OpenOLAT).

Maintenance is simple as exercises are separate standalone ﬁles. Thus, large teams can work jointly on the pool of exercises in a multi-author and cross-platform setting because each team member can independently develop and edit a single exercise.

The remainder of this paper consists of two major parts: First, we illustrate in Section2 how to use both the old and new exam-generating functions that are readily available in the package. This serves as a first introduction and is sufficient for getting a good overview of the available features and how to get started. Second, Section3 provides details about the design underlying the toolbox for the new infrastructure. This section – as well as the subsequent Section4 showing how to extend the toolbox – may be skipped upon first reading but it contains many details that are likely to be important when actually starting to create course materials with the package. Finally, a discussion in Section5 concludes the paper.

2. Using the exams package

In this section we provide an overview of the most important user interfaces provided by the exams package. This serves as a first introduction, assuming only (basic) knowledge of Sweave (Leisch 2012a,b). First, the format of the exercise Sweave files is reviewed along with the old (version 1) exams() function. Subsequently, the new (version 2) functions of type exams2xyz() are introduced: exams2pdf() and exams2html() produce one PDF or HTML file for each exam, respectively. In case of just a single exam, this is shown interactively in a viewer/browser. exams2moodle() and exams2qti12() generate Moodle and QTI 1.2 exams, i.e., just a single XML or ZIP file, respectively, which can be easily uploaded into Moodle and OLAT.

2.1. Version 1: PDF exams() from Sweave exercises Exercise templates (or just “exercises” for short) are essentially separate standard Sweave ﬁles (Leisch 2012a,b). They are composed of the following elements:

R code chunks (as usual within <<>>= and @) for random data generation.

Question and solution descriptions contained in LATEX environments of corresponding names. Both can contain R code chunks again or include data via \Sexpr{}.

Metainformation about the exercise type (numeric, multiple choice, . . . ), its correct solution etc. All metainformation commands are in LATEX style but are actually commented out and hidden in the ﬁnal output ﬁle.

The underlying ideas are eplained in more detail by Grun¨ and Zeileis(2009) and Section3 provides more technical details. Here, we focus on illustrating how diﬀerent output formats can be generated from such exercises.

2An implementation of the newer QTI 2.1 standard is also available in the package, see ?exams2qti21. However, at the time of writing the function still needs more thorough testing and is hence not discussed here. 4 Flexible Generation of E-Learning Exams in R

<>= ## DATA GENERATION n <- sample(120:250, 1) mu <- sample(c(125, 200, 250, 500, 1000), 1) y <- rnorm(n, mean = mu * runif(1, min = 0.9, max = 1.1), sd = mu * runif(1, min = 0.02, max = 0.06))

## QUESTION/ANSWER GENERATION Mean <- round(mean(y), digits = 1) Var <- round(var(y), digits = 2) tstat <- round((Mean - mu)/sqrt(Var/n), digits = 3) @

\begin{question} A machine fills milk into $\Sexpr{mu}$ml packages. It is suspected that the machine is not working correctly and that the amount of milk filled differs from the setpoint $\mu_0 = \Sexpr{mu}$. A sample of $\Sexpr{n}$ packages filled by the machine are collected. The sample mean $\bar{y}$ is equal to $\Sexpr{Mean}$ and the sample variance $s^2_{n-1}$ is equal to $\Sexpr{Var}$.

Test the hypothesis that the amount filled corresponds on average to the setpoint. What is the absolute value of the $t$~test statistic? \end{question}

\begin{solution} The $t$~test statistic is calculated by: \begin{eqnarray*} t & = & \frac{\bar y - \mu_0}{\sqrt{\frac{s^2_{n-1}}{n}}} = \frac{\Sexpr{Mean} - \Sexpr{mu}}{\sqrt{\frac{\Sexpr{Var}}{\Sexpr{n}}}} = \Sexpr{tstat}. \end{eqnarray*} The absolute value of the $t$~test statistic is thus equal to $\Sexpr{format(abs(tstat), nsmall = 3)}$. \end{solution}

%% META-INFORMATION %% \extype{num} %% \exsolution{\Sexpr{format(abs(tstat), nsmall = 3)}} %% \exname{t statistic} %% \extol{0.01}

Figure 1: A simple Sweave exercise: tstat.Rnw. Journal of Statistical Software 5

\begin{question} A machine fills milk into $500$ml packages. It is suspected that the machine is not working correctly and that the amount of milk filled differs from the setpoint $\mu_0 = 500$. A sample of $226$ packages filled by the machine are collected. The sample mean $\bar{y}$ is equal to $517.2$ and the sample variance $s^2_{n-1}$ is equal to $262.56$.

Test the hypothesis that the amount filled corresponds on average to the setpoint. What is the absolute value of the $t$~test statistic? \end{question}

\begin{solution} The $t$~test statistic is calculated by: \begin{eqnarray*} t & = & \frac{\bar y - \mu_0}{\sqrt{\frac{s^2_{n-1}}{n}}} = \frac{517.2 - 500}{\sqrt{\frac{262.56}{226}}} = 15.958. \end{eqnarray*} The absolute value of the $t$~test statistic is thus equal to $15.958$. \end{solution}

%% META-INFORMATION %% \extype{num} %% \exsolution{15.958} %% \exname{t statistic} %% \extol{0.01}

Figure 2: LATEX output of Sweave("tstat.Rnw").

In Figure1, the Sweave file for a simple exercise asking students to compute a one-sample t test statistic is shown for illustration (as already used by Grun¨ and Zeileis 2009). The R chunk for the data-generating process (DGP), the question and solution environments, and the metainformation can be easily distinguished. The LATEX file resulting from an Sweave() call is shown in Figure2, and Figure3 shows the final compiled PDF output generated by

R> library("exams") R> set.seed(1090) R> exams("tstat.Rnw")

Here, the exams() function looks for the exercise template tstat.Rnw ﬁrst in the local working directory and then within the installed exams package where this ﬁle is provided. Then it 6 Flexible Generation of E-Learning Exams in R

1. Problem A machine fills milk into 500ml packages. It is suspected that the machine is not working correctly and that the amount of milk filled differs from the setpoint µ0 = 500. A sample of 226 packages filled by the machine are collected. The sample meany ¯ is equal to 517.2 2 and the sample variance sn−1 is equal to 262.56. Test the hypothesis that the amount filled corresponds on average to the setpoint. What is the absolute value of the t test statistic? Solution The t test statistic is calculated by:

y¯ − µ0 517.2 − 500 t = r = q = 15.958. s2 262.56 n−1 226 n

The absolute value of the t test statistic is thus equal to 15.958.

Figure 3: Display of a tstat exercise as PDF via exams() or exams2pdf().

Figure 4: Display of a tstat exercise as HTML via exams2html(). MathML is employed for mathematic equations, as rendered by a Firefox browser. Journal of Statistical Software 7

Argument Description file A (list of) character vector(s) specifying the (base) names of the Sweave exercise files. n The number of exams to be generated from the list of exercises. Default: 1. nsamp The number of exercise files sampled from each list element of file. Default: One for each list element. dir Path to output directory. Default: Single PDF or HTML files are shown directly in a viewer/browser (i.e., exams/exams2pdf/exams2html with n = 1). In all other cases the current working directory is used by default. edir Path to the directory in which the exercises in file are stored. Default: Working directory (or within the exams installation). tdir Path to a temporary directory in which Sweave() is carried out. Default: New tempdir(). sdir Path to the directory in which supplementary files (e.g., graphics or data files) are stored (except for exams()). Default: New tempdir(). name Name prefix for the resulting exam files. template Character specifying the (base) names of a LATEX, HTML, or XML file template for the exam (except for exams2moodle()). Default: A function-specific template provided within the exams installation. encoding Character specifying the encoding to be used (in version 2 interfaces only). verbose Should progress information be displayed (in version 2 interfaces only)?

Table 1: Common arguments of the main user interfaces for generating exams: exams(), exams2pdf(), exams2html(), exams2moodle(), exams2qti12(). The first group of arguments pertains to the specification of the exam(s), the second group to the handling of in- put/temporary/output directories, and the last group to name and setup for the resulting files. For further function-specific arguments and more details/examples, see the corresponding manual pages. copies the exercise .Rnw to a temporary directory, calls Sweave() to generate the .tex, and includes this in the default LATEX template for exams before producing the .pdf. As, by default, just a single .pdf exam is produced and no output directory is specified, a PDF viewer pops up and displays the resulting exam (as in Figure3). While applying exams() to just a single exercise is very useful while writing/programming an exercise, a full exam will typically encompass several different exercises. Also, it may require suppressing the solutions, including a title page with a questionnaire form, etc. The former can be achieved by supplying a (list of) vector(s) of exercises while the latter can be accomodated by using different templates:

R> myexam <- list( + "boxplots", + c("confint", "ttest", "tstat"), + c("anova", "regression"), + "scatterplot", + "relfreq") R> odir <- tempfile() R> set.seed(1090) 8 Flexible Generation of E-Learning Exams in R

R> exams(myexam, n = 3, dir = odir, template = c("exam", "solution"))

The myexam list contains five exercises: the first one is always boxplots.Rnw while the second exercise is randomly drawn from confint.Rnw, ttest.Rnw, tstat.Rnw, and so on for the remaining exercises. Then, exams() is used to draw n = 3 random exams and produce one exam and one solution PDF for each. The template argument takes names of LATEX files which provide the LATEX headers and footers. These templates can be used to create a title page with a questionnaire form (for student name, id, signature, etc.), show or suppress solutions, and set further formatting details. All involved .Rnw files (with exercises) and .tex templates employed in the example above are provided in the exams source package and its installed versions. The resulting output files are stored along with the extracted metainformation in the output directory:

R> dir(odir)

[1] "exam1.pdf" "exam2.pdf" "exam3.pdf" "metainfo.rda" [5] "solution1.pdf" "solution2.pdf" "solution3.pdf"

More details on basic usage and more advanced customization of this function are provided by the vignette vignette("exams", package = "exams") which is an updated version of Grun¨ and Zeileis(2009). The vignette actually uses the new and somewhat more ﬂexible exams2pdf() function rather than exams() and discusses the enhancements made for the former (see also below).

2.2. Version 2: Producing PDF, HTML, or XML for Moodle or OLAT The new infrastructure added to the exams package on the road to version 2 is providing more flexibility and enables a much broader variety of output formats while keeping the specification of the exercise templates fully backward compatible and only slightly extended. While the design of the underlying workhorse functions is rather different (see Section3), the new user interfaces are very similar to the old one, sharing most of its arguments (see Table1). Hence, for users of the previous version of the package, it is easy and straightforward to adapt to the new facilities.

Producing PDF documents: exams2pdf() As mentioned above, the function exams2pdf() is a more ﬂexible reimplementation of exams() using the new extensible infrastructure of the exams package. For the user virtually nothing changes:

R> set.seed(1090) R> exams2pdf("tstat.Rnw") pops up the same PDF as shown in Figure3. We refrain from further discussion of customization of the PDF output because this is discussed in vignette("exams", package = "exams") with details about LATEX master templates, additional auxiliary files, showing/hiding solutions etc. Here we only point out the main difference between the old exams() function and the new exams2pdf(): The latter not only returns the metainformation from Journal of Statistical Software 9 the exercise but additionally also the LATEX code for the question and solution environments as well as paths to supplementary materials (such as graphics or data files). Section3 explains the structure of the return values in more detail and illustrates how this can be used.3

Producing HTML documents: exams2html() As a first step towards including exams generated from Sweave files into e-learning exams, it is typically necessary to be able to generate an HTML version of the exams. Hence, the function exams2html() is designed analogously to exams()/exams2pdf() but produces HTML files. In case of just a single generated exam, this is displayed in a browser using base R’s browseURL() function4. Again, this is particularly useful while writing/programming a new exercise template. For example,

R> set.seed(1090) R> exams2html("tstat.Rnw") generates the HTML file shown in Figure4 which corresponds directly to the PDF file from Figure3. Note that for properly viewing the formulas in this HTML file, a browser with MathML support is required. This is discussed in more detail in Section 3.4. Here, the Firefox browser is used (in Debian Linux’s rebranded Iceweasel version) which has native MathML support. To transform the LATEX questions/solutions to something that a web browser can render, three options are available: translation of the LATEX to (1) plain HTML, (2) HTML plus MathML for mathematical formulas (default), or conversion of the corresponding PDF to (3) HTML with one embedded raster images for the whole question and solution, respectively. The former two options are considerably faster and more elegant – they just require the R package tth (Hutchinson, Leisch, and Zeileis 2013) that makes the ‘TEX-to-HTML’ converter TtH (Hutchinson 2012) easily available in R. Also, by default, the base64enc package (Urbanek 2012) is employed for embedding graphics in Base64 encoding. More details on this approach are provided in Section 3.4. The HTML files produced with approaches (1) and (2) can also easily contain hyperlinks to supplementary files. For example, if the R code in the Sweave file generates a file mydata.rda, say, then simply including \url{mydata.rda} in the question/solution will result in a suitable hyperlink. The supplementary data files for each random replication of the exercise is managed fully automatically and a copy of the data is created in an (exam-specific) sub-directory of the output directory. Run exams2html("boxhist.Rnw") for such an example. Just like exams()/exams2pdf(), exams2html() can also generate multiple replications of randomly drawn exams via exams2html(myexam, n = 3, dir = odir). Also multiple versions of the same replications can be generated by providing several templates, e.g., for showing/suppressing solutions.

3To obtain the same type of return value as from the exams() function, exams_metainfo(exams2xyz(...)) can be used. 4In RStudio (RStudio Team 2014), versions prior to 0.97.133, the "browser" option is set to a function that cannot browse local HTML ﬁles on some platforms. Recent versions of RStudio have resolved this problem and ?exams2html also provides workarounds for older RStudio versions. 10 Flexible Generation of E-Learning Exams in R

Producing Moodle XML: exams2moodle() To incorporate exams generated from Sweave exercises into learning management systems, such as Moodle, two building blocks are typically required: (1) questions/solutions are available in plain text or HTML format, and (2) questions/solutions can be embedded along with the metainformation about the possible and correct solutions into some exam description format. (1) can be accomplished as outlined in the previous subsection for exams2html() and for Moodle (2) requires embedding everything into Moodle XML format. Both steps can be easily carried out using the exams2moodle() function:

R> set.seed(1090) R> exams2moodle(myexam, n = 3, dir = odir)

This draws the same three random exams from the myexam list that were already generated in PDF format above. The output ﬁle, stored again in odir, is a single XML ﬁle.

R> dir(odir)

[1] "exam1.pdf" "exam2.pdf" "exam3.pdf" "metainfo.rda" [5] "moodlequiz.xml" "solution1.pdf" "solution2.pdf" "solution3.pdf"

This XML file moodlequiz.xml can be easily imported into a Moodle quiz and then further customized: First, the XML file is imported into the question bank in Moodle. Then, all replications of each exercise can be added as “random” questions into a quiz (and potentially further customized). Figure5 shows the first random draw of the boxplots exercise in the resulting Moodle quiz (again rendered by a Firefox browser). More details on how exams- generated questions can be integrated in Moodle are provided in Section5. The corresponding solutions are displayed upon completion of the exam in Moodle. As before, selected supplementary files are automatically managed and can easily be included using \url{} in the underlying LATEX code. To be able to include all these supplements in a single XML file, Base64 encoding is employed for all supplements. See the manual page for the list of all supported supplement file formats.

Producing QTI 1.2 XML (for OLAT): exams2qti12() The generation of QTI 1.2 assessments (for OLAT) proceeds essentially in the same way as for the Moodle quizzes, by default using ttm for transformation of the text to HTML5. The same three random draws of exams from myexam can be prepared in QTI 1.2 format via:

R> set.seed(1090) R> exams2qti12(myexam, n = 3, dir = odir)

This produces a single ZIP ﬁle qti.zip, again written to odir.

R> dir(odir) 5It may be of interest to OLAT users that we experienced problems with the display of MathML matrices in OLAT. The columns were not separated by spaces and we have not been able to adapt our OLAT installation to avoide this problem. Hence, if we want to display matrices in OLAT, we generate them with extra empty columns. The cholesky.Rnw exercise template has code that can automatically do this, if enabled. Journal of Statistical Software 11

Figure 5: Display of exercise 1 (boxplots) from myexam in Moodle (as rendered by a Firefox browser).

[1] "exam1.pdf" "exam2.pdf" "exam3.pdf" "metainfo.rda" [5] "moodlequiz.xml" "qti12.zip" "solution1.pdf" "solution2.pdf" [9] "solution3.pdf"

The ZIP file can again be easily imported into an OLAT test configuration where further customization can be performed6. The first boxplots exercise from the exam generated above is shown in OLAT in Figure6 (again as rendered by a Firefox browser). The corresponding solutions are displayed in OLAT immediately after incorrectly completing an individual exercise. The display of solutions can also be suppressed completely by setting solutionswitch = FALSE in exams2qti12().7 The main difference of the generated ZIP file for QTI 1.2, compared to the Moodle XML

6While customization of the features of the overall assessment was always possible for us, OLAT typically did not allow for modification of the individual exercise items. We were not able to track down which part of the QTI 1.2 XML specification causes this. 7In our e-learning exams, we typically employ these default settings (i.e., maxattemps = 1 and solutionswitch = TRUE). Alternatively, we give the students an unlimited number of attempts to solve an exercise (maxattempts = Inf) but then suppress solutions completely (solutionswitch = FALSE) because oth- erwise the correct solution would be displayed after the first incorrect attempt. 12 Flexible Generation of E-Learning Exams in R

Figure 6: Display of exercise 1 (boxplots) from myexam in OLAT (as rendered by a Firefox browser).

output, is that in addition to the qti.xml file further supplementary files can be included. Hence, supplements in all potential formats can be easily included and uploaded in one go into OLAT. Therefore, by default, Base64 is employed only for graphics but not for other files (such as data sets etc.) but either approach can be used optionally for all types of supplements. The QTI 1.2 standard allows for rather fine control of the properties of the exercises (also known as items in QTI 1.2) and the exams (also known as assessments). Hence, exams2qti12() provides a variety of options for controlling the appearance of exam/exercises, see the manual page ?exams2qti12 for details. Also, the underlying XML template can be adapted.

2.3. Creating the first exam When creating the first “real” exam with exams, i.e., when starting to prepare course mate- Journal of Statistical Software 13 rials with the help of the package, it is our experience that it works best to start (almost) from scratch with some simple examples. The package provides a wide range of examples for typical exercises (see Table3 in the appendix for an overview) which can serve as a starting point and it is often useful to copy parts of these exercises to create new ones. In particular, we recommend to keep the formatting as simple as possible for two reasons: (1) The resulting exercises are typically more robust and work well with different exams2xyz() interfaces (especially both in LATEX and in HTML). (2) Some formatting issues require attention to technical details, e.g., as discussed in Section3. Thus, we recommend to start with exercises taking inspiration from the available examples and only using basic LATEX markup for mathematical notation and formatting. To aid this process, exams provides the function exams_skeleton() (or equivalently exams.skeleton()) which creates a directory with a copy of the exercises from Table3 (in the appendix) and the required templates (e.g., for LATEX, HTML, or XML) along with a ‘demo.R’ script illustrating the use of the various exams2xyz() interfaces. As an illustration, assume that we are interested in using exams2moodle() and exams2pdf() for generating both Moodle and PDF exams (for printout). To test that the LATEX-to-HTML conversion for Moodle actually works for all exercises, we additionally want to inspect the results of exams2html(). Hence, the code below calls exams_skeleton() specifying these three writers. Here, we employ a temporary directory but users may set the dir argument to something like "C:/myexam/" or "~/myexam/" etc.

R> mydir <- file.path(tempdir(), "myexam") R> exams_skeleton(dir = mydir, absolute = TRUE, + writer = c("exams2html", "exams2pdf", "exams2moodle")) R> dir(mydir)

[1] "demo.R" "exercises" "templates"

The directory then contains the file ‘demo.R’ which can be opened in any editor for R scripts. The script illustrates how to create various kinds of output using the exams2html(), exams2pdf(), and exams2moodle() functions based on the exercises and templates in the subdirectories of the same name. Absolute paths are employed in the script to refer to these subdirectories (while the default absolute = FALSE would result in relative paths being used). The function exams_skeleton() always copies all exercise files to the directory but the ‘demo.R’ script only employs one example for each exercise type, i.e., num, schoice, mchoice, cloze, and string. To restrict this set of exercises, the type argument of exams_skeleton() can be used (e.g., type = c("num", "schoice")). In any case, it should be easy to modify the ‘demo.R’ script, omitting or adding exercises that are readily available in the subdirectory. Finally, to illustrate how different encodings can be used, exams_skeleton() can also be called with an encoding argument, e.g., setting encoding = "UTF-8". This modifies the demo.R script as well as the HTML and LATEX templates accordingly. The encodings "latin1" (or "ISO-8859-1") and "latin9" (or "ISO-8859-15") have also been tested. As usual, employing Sweave files in a particular encoding can be very convenient for special characters (such as accents or umlauts) but might also lead to problems if they are used in different locales (e.g., on different operating systems). An alternative route (employed by the authors of the exams package) is to employ Sweave ASCII files only, using LATEX commands for special characters. 14 Flexible Generation of E-Learning Exams in R

3. Design

All the new exams2xyz() interfaces for generating exams in different formats (with currently xyz ∈ {pdf, html, moodle, qti12}) are built by combining the modular building blocks provided by version 2 of exams. All functions have the same goal, i.e., to write exam files in a certain format to the hard disk. The approach is that the Sweave exercises are first weaved to LATEX, read into R, potentially transformed (e.g., to HTML), and then written to suitable output file formats on the disk. Different customizable driver functions (or even driver- generating functions) for performing the weave/read/transform/write steps are available in exams. Internally, all the exams2xyz() interfaces choose certain drivers and then call the new function xexams() (for extensible exams) that handles all temporary files/directories and suitably executes the drivers. In the following subsections, all these building blocks are introduced in detail.

3.1. Extended specification of exercises As discussed in Section2 and illustrated in Figure1, each exercise is simply an Sweave file containing R code for data generation, question/solution environments with LATEX text, and metainformation about the type of exercise and the correct solution etc. This design was introduced by Grun¨ and Zeileis(2009) but is slightly extended in the new version to provide some more options for the generation of e-learning exams. See Table2 for an overview for a list of exercise types and corresponding metainformation commands. Each exercise must specify at least an \extype{} and an \exsolution{} and should typically also have a short \exname{}. There are now five different extypes. Two types that have a single question and answer:

num for questions with a numeric answer, e.g., \exsolution{1.23}.

string for questions with a (short) text answer, e.g., \exsolution{glm}.

Three types have a list of questions (or statements):

mchoice for multiple-choice questions where each element of the question/statement can either be true or false, e.g., \exsolution{01011}.

schoice for single-choice questions where exactly one of the questions/statements is true and all others are false, e.g., \exsolution{01000}.

cloze for a combination of questions/statements with num, string, or mchoice answers. Thus, each element of the question has either a numeric, short text, or single/multiple- choice answer, e.g., \exsolution{1.23|001|glm}. To specify the individual cloze types, a clozetype has to be given, e.g., \exclozetype{num|schoice|string}.

The types schoice and cloze have been newly introduced. The purpose of the former is mainly to allow for diﬀerent processing of options (e.g., for assigning points to correct/wrong results) between mchoice and schoice. The cloze type was introduced because both Moodle and QTI 1.2 have support for it (albeit in slightly diﬀerent ways, for details see below). Possible evaluation strategies (with/without partial credits and/or with/without negative Journal of Statistical Software 15

Command Description \extype{} Specification of the type of exercise (required): num for questions with a numeric answer, mchoice for questions with multiple-choice answers, schoice for questions with single-choice answers (i.e., multiple-choice with exactly one correct solution), string for questions with a (short) text answer, or cloze for cloze solutions (i.e., combinations of the above). \exname{} Short name/description (to be used for printing within R). \extitle{} Pretty longer title (for Moodle). \exsection{} Section of the exercise (for Moodle, with slashes for subsections as in a URL). \exversion{} Version of the exercise. \exsolution{} Correct solution (required). It must contain a numeric solution for num, a string of zeros/ones for mchoice/schoice, or a character string of string. For cloze a combination of these can be specified, e.g., \exsolution{1.23|001|glm}. \extolerance{} Tolerance for num solutions or a vector of tolerances (expanded if necessary) for cloze solutions. If unspecified the tolerance is 0. \exclozetype{} List of types for the elements of a cloze exercise, e.g., \exclozetype{num|schoice|string} for the example above. \expoints{} Points for (fully) correct solution. Default is 1. \exextra[]{} Additional metainformation to be read and stored, e.g., for new custom interfaces. The default storage type is character, e.g., \exextra[myinfo]{1.23} yields a metainformation element myinfo of "1.23". The type can also be numeric or logical, e.g., \exextra[myinfo,logical]{FALSE|FALSE|TRUE}.

Table 2: Overview of metainformation commands in exercises. The commands in the ﬁrst section allow for a general description, those in the second section for question/answer spec- iﬁcation. Only extype and exsolution are always required (but exname is recommended additionally for nice printing in R). points) are discussed below for exams2moodle() and exams2qti12() and in AppendixB for further functionality within R. For the three types with lists of questions (mchoice, schoice, cloze), the question and solution environments should each contain at the end an answerlist environment. In the question this should list an \item for each question/statement and in the solution the corresponding answers/explanations can be provided (if any). The answerlist environment can either be written as usual “by hand” or by using the answerlist() function provided by the exams package. For illustration, we set up a multiple-choice question with three statements about Switzerland. First, we generate an answerlist with statements for the question.

R> qu <- c("Zurich is the capital of Switzerland.", + "Italian is an official language in Switzerland.", + "Switzerland is part of the European Union (EU).") R> answerlist(qu) 16 Flexible Generation of E-Learning Exams in R

\begin{answerlist} \item Zurich is the capital of Switzerland. \item Italian is an official language in Switzerland. \item Switzerland is part of the European Union (EU). \end{answerlist}

Then the corresponding answerlist for the solution is set up.

R> sol <- c(FALSE, TRUE, FALSE) R> ex <- c("The capital of Switzerland is Bern.", + "The official languages are: German, French, Italian, Romansh.", + "Switzerland is part of the Schengen Area but not the EU.") R> answerlist(ifelse(sol, "True", "False"), ex)

\begin{answerlist} \item False. The capital of Switzerland is Bern. \item True. The official languages are: German, French, Italian, Romansh. \item False. Switzerland is part of the Schengen Area but not the EU. \end{answerlist}

For more examples see the exercise files in the inst/exercises directoy of the exams source package. There are various multiple-choice questions with and without figures and/or ver- batim R output (e.g., anova, boxplots, cholesky, among others). The files tstat and tstat2 illustrate how the same type of exercise can be coded as a num or schoice question, respectively. The cloze type is employed in, e.g., boxhist or fourfold (with more flexible formatting specifically for Moodle in boxhist2 and fourfold2, respectively). See also Table3 in the appendix for an overview and AppendixC for more details on cloze exercises.

3.2. The xexams() wrapper function To avoid recoding certain tedious tasks – such as copying/reading ﬁles and handling temporary directories – for each of the user interfaces introduced in Section2, the new exams package provides a modular and extensible framework for building new exam-generating functions. This framework is tied together by the xexams() function which is typically not called by users directly but forms the basis for all new exams2xyz interfaces. To accomplish this, xexams() also takes the arguments listed in Table1 (except name and template), draws exams from the exercise file list, and does all the necessary ﬁle/directory handling. Furthermore, it takes a driver argument that needs to be a list of four functions driver = list(sweave, read, transform, write). These are utilized as follows:

1. Weave: For each of the selected exercise files (within all n exams) driver$sweave(file) is run to weave the .Rnw file into a .tex file. If sweave = NULL (the default), the standard Sweave() function is used. If sweave = list(...) is a list, e.g., list(pdf = FALSE, png = TRUE), this is passed as arguments to Sweave().

2. Read: Each resulting .tex ﬁle is read into R using driver$read(file). By default (read = NULL), the function read_exercise() is used (see below), resulting in a list of character vectors with the LATEX code for question/solution plus metainformation. Journal of Statistical Software 17

list( ## of 'exams', length: n list( ## of 'exercises', length: k list( ## of exercise content, length: 6 question, questionlist, solution, solutionlist, metainfo, supplements ) ) )

Figure 7: Structure of the return value of xexams(), when used with the default read driver read_exercises().

3. Transform: Each of these exercise-wise list objects can subsequently be transformed by driver$transform(object) which can be leveraged for transformations from LATEX to HTML etc. By default (transform = NULL), no transformation is applied.

4. Write: The (possibly transformed) lists of exercises, read into R for each exam object, can be written out to one ore more ﬁles per exam in an output directory via driver$write(object, dir, info = list(id, n)). By default (write = NULL), no ﬁles are written.

After performing each of the driver functions, xexams() returns invisibly a nested list object (currently unclassed) as illustrated in Figure7. It is a list of exams (of length n), each of which is a list of exercises (whose length depends on the length of file and nsamp), each of which is a list (whose length/contents depends on driver$read). When used with the default read_exercise(), each exercise is a list of length 6 containing the question/solution texts, metainformation, and paths to supplementary files. These will be introduced in more detail in the next section. All of the interfaces introduced in Section2 employ the standard Sweave() function for the weaving step (possibly with custom arguments) and the read_exercise() function for the reading step. They mainly differ in the transformation and writing step. exams2pdf() needs no transformation and the writer first sets up a .tex file for each exam, calls texi2dvi(pdf = TRUE), and then copies the resulting .pdf file to the output dir. exams2html() on the other hand uses a TEX-to-HTML transformation and the writer then sets up a .html file for each exam and copies it to the output dir. Finally, exams2moodle() and exams2qti12() both also use a transformation to HTML but have no writer. The reason for this is that they do not write one file per exam (i.e., with only one replication per exercise) but rather need to produce XML files that include all different replications of each exercise. Hence, they take the list returned by xexams() and process it subsequently in different ways. The details for all these steps are explained in the subsequent subsections. 18 Flexible Generation of E-Learning Exams in R

3.3. The read driver: read_exercise() and read_metainfo() The function read_exercise() reads the weaved exercises, i.e., ﬁles like that shown in Fig- ure2. It simply extracts the text lines from the question and solution environments and stores them in vectors of the same name. If these environments contain answerlist environments, these are extracted and stored separately in questionlist and solutionlist vectors, respectively. Finally, the metainformation is extracted using read_metainfo() which not only stores character vectors but also transforms them to suitable types (depending on the extype) and performs some sanity checks. The resulting metainformation is a list with elements essentially corresponding to the commands from Table2. For illustration, we run xexams() to select the same three exams as used in the Moodle and OLAT examples above. However, using the default driver speciﬁcation, xexams() just performs the weaving and reading steps (and has no transformation or writing step):

R> set.seed(1090) R> x <- xexams(myexam, n = 3)

The resulting object is a nested list as shown in Figure7 with 3 exams of 5 exercises each (drawn from the myexam list). Using x[[i]][[j]], the j-th exercise of the i-th exam can be accessed. Here, we explore the first exercise (boxplots, a multiple-choice question) from the first exam that is also shown in Figures5 and6. Its general question text (in L ATEX) is printed below – it requires a graphic which is stored in a supplementary file in a temporary directory.

R> writeLines(x[[1]][[1]]$question)

In Figure~\ref{fig:ch06-boxplots} the distributions of a variable given by two samples (A und B) are represented by parallel boxplots. Which of the following statements are correct? \emph{(Comment: The statements are either about correct or clearly wrong.)}

\setkeys{Gin}{width=0.7\textwidth} \begin{figure}[htb!] \begin{center} \includegraphics{boxplots-002} \caption{\label{fig:ch06-boxplots} Parallel boxplots.} \end{center} \end{figure}

R> x[[1]][[1]]$supplements

boxplots-002.pdf "/tmp/RtmpSoC12J/file7ff6d466d19/exam1/exercise1/boxplots-002.pdf" attr(,"dir") [1] "/tmp/RtmpSoC12J/file7ff6d466d19/exam1/exercise1"

The corresponding list of statements about the graphic is stored separately. It is shown below along with the most important metainformation elements. Journal of Statistical Software 19

R> x[[1]][[1]]$questionlist

[1] "The location of both distributions is about the same." [2] "Both distributions contain no outliers." [3] "The spread in sample A is clearly bigger than in B." [4] "The skewness of both samples is similar." [5] "Distribution A is about symmetric."

R> x[[1]][[1]]$metainfo[c("file", "type", "solution")]

$file [1] "boxplots"

$type [1] "mchoice"

$solution [1] TRUE TRUE FALSE TRUE TRUE

In summary, xexams() combined with the default readers is relatively straightforward to use in other progams (such as the exams2xyz functions). The return value is somewhat “raw” as it is not classed and has no dedicated methods for subsetting etc. However, we refrained from using a more elaborate structure as this function is not meant to be called by end-users while we expected other developers to ﬁnd the current structure suﬃciently simple to use in their programs.

3.4. LATEX-to-HTML transform driver generator When embedding statistical/mathematical exercises into web pages or learning management systems, the exercises’ LATEX text – typically containing mathematical notation – has to be transformed in some way so that it can be rendered by a browser. Until relatively recently, this posed the notorious problem of how to display the mathematical formulas and often the only good answer was to embed raster images of the formulas. However, this situation has clearly changed (see e.g., Vismor 2012) and there are now various convenient options: e.g., using the mathematical markup language MathML (W3C 2014; Wikipedia 2014) or keeping LATEX formulas in the web page and embedding some JavaScript for rendering them. Especially the display of MathML in web pages has become very easy: Firefox long had native support for it and for the Microsoft Internet Explorer the MathPlayer plugin (Design Science 2013b) has long been available. More recently, other major browsers like Opera or Safari also added support for MathML (see Vismor 2012, Section 1.2). Google Chrome briefly enabled MathML support but disabled it again due to instabilities. Furthermore, MathJax (Design Science 2013a), an open-source JavaScript engine, can be used to render MathML (or LATEX) formulas on a server rather than in the local browser. Therefore, the new exams package offers functionality for automatically transforming the LATEX exercises to HTML within R and by default employs MathML for all mathematical notation (e.g., as demonstrated in Figure4). More specifically, the package provides the driver generator make_exercise_transform_html(). It returns a function suitable for plug-in into the 20 Flexible Generation of E-Learning Exams in R transform driver in xexams() which then replaces the LATEX code in question/questionlist and solution/solutionlist with HTML code. For illustration, we set up a particular function trafo() below and apply it to the first exercise in the first exam within the object x that we had considered before:

R> trafo <- make_exercise_transform_html(converter = "ttm", base64 = FALSE) R> writeLines(trafo(x[[1]][[1]])$question)

In Figure the distributions of a variable given by two samples (A und B) are represented by parallel boxplots. Which of the following statements are correct? (Comment: The statements are either about correct or clearly wrong.)

Figure 1: Parallel boxplots.

It can be seen that the resulting exercise employs HTML text, e.g., uses instead of \emph or instead of \includegraphics.8 Internally, make_exercise_transform_html() can leverage three different converters: ttm (default), tth, or tex2image. The former two come from the R package tth (Hutchinson et al. 2013) and internally call the two C functions tth (TEX to HTML) and ttm (TEX to HTML/MathML) taken from the TtH suite of Hutchinson(2012). The last option, tex2image, is a function provided by the exams package itself. It proceeds by first run- ning texi2dvi(pdf = TRUE) from the base R package tools and subsequently converting the resulting PDF to a raster image in a system() call to ImageMagick’s convert function (Im- ageMagick Studio LLC 2014). Thus, for this function ImageMagick is assumed to be installed and in the search path. All three converters have their benefits and drawbacks:

 tth is typically preferable if there is no or only very simple mathematical notation. The resulting HTML can then be rendered in any modern browser.

ttm is preferable if there is some standard mathematical notation (e.g., fractions or equation arrays etc.). As argued above this can still be easily displayed in suitable browsers or by employing MathJax in the web page.

8It may be noteworthy to that the conversion (a) assumes the graphics to be in .png format and (b) does not resolve the ﬁgure reference at the beginning of the text correctly. For (a), we just need to make sure that the sweave driver in xexams() has png = TRUE (and pdf = FALSE) which is accounted for in exams2html() etc. Issue (b), however, needs to be avoided by formulating the underlying .Rnw diﬀerently (or by tolerating the missing number). Journal of Statistical Software 21

tex2image is the “last resort” if neither of the two previous approaches work, e.g., if more complex LATEX commands/packages need to be used which are not supported by tth/ttm. It is fairly slow while tth/ttm are typically even faster than calling LATEX.

To explore the diﬀerences of the results, the converters can also be called directly on character strings containing LATEX. Below we use two simple code lines for which tth() would probably be suﬃcient:

R> tex <- c("This is \\textbf{bold} and this \\textit{italic}.", + "Points on the unit circle: $x^2 + y^2 = 1$.") R> ttm(tex)

[1] "This is bold and this italic." [2] "Points on the unit circle: " [3] " $" [4] " " [5] " x^{2 " [6] "} " [7] " + " [8] " y^{2 " [9] "} " [10] " = 1$ ."

R> tth(tex)

[1] "This is bold and this italic." [2] "Points on the unit circle: x² + y² = 1."

R> (tex2image(tex, dir = odir, show = FALSE))

[1] "/tmp/RtmpSoC12J/file7ff679fd6b9e/tex2image_1.png"

Note that tex2image(tex) returns the path to a raster image file which by default is also shown directly in the browser. Finally, our illustration of make_exercise_transform_html() also employed a second option, base64 = FALSE, which still deserves more detailed explanation. After converting an exercise from LATEX to HTML code (using either of the three converters above), the HTML code may contain references to supplementary files (e.g., in tags). Optionally, by using the default base64 = TRUE, these images can be embedded directly into the HTML code in Base64 encoding (via the base64enc package in R, Urbanek 2012) and thus waiving the need for having them as supplementary files.

3.5. PDF and HTML write driver generators In the ﬁrst three steps of xexams(), exams are randomly drawn and weaved, read into R, and potentially transformed from LATEX to HTML (or some other format). However, so far, no output ﬁles have been generated. The original idea of Grun¨ and Zeileis(2009) was to produce 22 Flexible Generation of E-Learning Exams in R

Exam ##ID##

Exam ##ID##

##\exinput{exercises}##

Figure 8: Default HTML template ﬁle plain.html employed in make_exams_write_html(). Elements marked by ##...## are being replaced in each replication of the exam.

one or more output files for each of the n generated exams. To do so in xexams() a write driver can be specified. The package provides several generating functions for suitable drivers, especially for generating PDF and HTML files. As before, the idea is to pass customization options to the driver generator which can then be plugged into xexams(). For PDF output files, the following driver generator is available:

make_exams_write_pdf(template = "plain", name = NULL, inputs = NULL, header = list(Date = Sys.Date()), quiet = TRUE, control = NULL)

This is employed in exams2pdf() and proceeds in the same way as described by Grun¨ and Zeileis(2009) for the exams() function. It includes the question/questionlist and solution/solutionlist in a LATEX template, then runs texi2dvi(pdf = TRUE) from the base tools package, and finally copies the resulting PDF files to a desired output directory. The default plain.tex template is provided within the exams package and also more than one template can be employed as illustrated in Section2. Details about the remaining customization arguments are provided on the manual page and in Grun¨ and Zeileis(2009). For HTML output files, a similar driver generator is available:

make_exams_write_html(template = "plain", name = NULL, question = "

Question

", solution = "
Solution
", mathjax = FALSE) Journal of Statistical Software 23
This is employed in exams2html() and is also based on a template. By default the plain.html ﬁle is used that is provided within exams and shown in Figure8. This contains placeholders marked with ##...## that are to be replaced in each randomly drawn exam. The ##ID## is simply replaced with a numeric ID (1,..., n) and ##\exinput{exercises}## is replaced by an ordered list (
) containing the question/solution. If the question and solution arguments to make_exams_write_html() are character strings, these are added as titles in the list. Alternatively, either argument can also be set to FALSE which avoids inclusion of the corresponding element of the exercise in the resulting HTML ﬁle. As an additional convenience setting mathjax = TRUE includes the