1 BabeLO - An Extensible Converter of Programming Exercises Formats
Ricardo Queiros´ and Jose´ Paulo Leal
Abstract—In the last two decades there was a proliferation of programming exercise formats that hinders interoperability in automatic assessment. In the lack of a widely accepted standard a pragmatic solution is to convert content among the existing formats. BabeLO is a programming exercise converter providing services to a network of heterogeneous e-Learning systems such as contest management systems, programming exercise authoring tools, evaluation engines and repositories of learning objects. Its main feature is the use of a pivotal format to achieve greater extensibility. This approach simplifies the extension to other formats, just requiring the conversion to and from the pivotal format. This paper starts with an analysis of programming exercise formats representative of the existing diversity. This analysis sets the context for the proposed approach to exercise conversion and to the description of the pivotal data format. The abstract service definition is the basis for the design of BabeLO, its components and web service interface. The paper includes a report on the use of BabeLO in two concrete scenarios: to relocate exercises to a different repository; and to use an evaluation engine in a network of heterogeneous systems.
Index Terms—Interoperability, Web Services, REST, programming exercise formats, e-learning. !
1 INTRODUCTION that cannot be shared among automatic assessment systems. These systems whose interoperability is hin- SSESSMENT plays a vital role in learning. How- dered by the lack of a common format include contest A ever, automatic assessment of exercises other management systems, evaluation engines, repositories than multiple choice can be a rather a complex task. of learning objects and authoring tools. A pragmatical This is certainly the case with assessment of computer approach to remedy this problem is to create a service programs in two distinct learning contexts: curricular to convert among existing formats. A kind of trans- and competitive learning. lation service specialized in programming problems Introductory programming courses are part of the formats. curricula of many engineering and sciences programs. To convert programming exercises on-the-fly These courses rely on programming exercises, as- among the most used formats is the purpose of signments and practical examinations, to consolidate the BabeLO – a service to cope with the existing knowledge and evaluate students. The enrolment in Babel of Learning Object formats for programming these courses is usually very high, resulting in a great exercises. BabeLO was designed as a service to act as workload for the faculty and teaching assistants. In a middleware in a network of systems typically used this context the availability of many programming in automatic assessment of programs. It provides exercises from different sources is of great importance. support for multiple exercise formats and can be Competitive learning relies on the competitiveness used by: 1) evaluation engines to assess exercises of students to increase their programming skills. This regardless of its format; 2) repositories to import is the common goal of several programming contests exercises from various sources; 3) authoring systems where students at different levels compete such as: to create exercises in multiple formats or based on the International Olympiad in Informatics (IOI), for exercises from other sources. secondary school students; the ACM International The remainder of this paper is organized as fol- Collegiate Programming Contests (ICPC), for univer- lows. Section 2 analyses several of existing formats sity students; and the IEEExtreme, for IEEE student to highlight both their differences and their similar members. Contest management systems and on-line features. Based on this analysis Section 3 presents an judges where students train rely also on automatic as- approach to extensible format conversion and details sessment. In this context there is also the need for new the features of PExIL – the pivotal format in which the programming problems that may come from different conversion is based – and the service functions. The systems. Also, competitive learning is usually a source following section provides details on the design and or programming exercises for curricular learning that implementation of BabeLO, including the service API are recast as learning objects. and the interfaces required to extend the conversion In both contexts the lack of a standard – or at to a new format. To evaluate the effectiveness and least a widely used format – creates a modern Babel efficiency of this approach this paper reports on two tower made of Learning Objects, of assessment items actual uses of BabeLO: to relocate exercises to a dif- 2 ferent repository; and to use an evaluation engine in a 2.3 Mooshak Exchange Format network of heterogeneous systems. The final section Mooshak6 is a web based competitive learning system summarizes the main contributions of this research originally developed for managing programming con- and plans futures developments of this service. tests over the Internet [2]. Recently, it was upgraded to expose the evaluation functions as services. These services are expected to integrate in heterogeneous 2 PROGRAMMING EXERCISES FORMATS networks of e-Learning types of systems, including the Learning Management Systems (LMS), Integrated The increasing popularity of programming contests Development Environments (IDE) and Learning Ob- worldwide resulted in the creation of several contest jects Repositories (LOR) [3]. Despite the context where management systems. At the same time Computer it is used, Mooshak has its own internal format to Science courses use programming exercises to encour- describe problems called Mooshak Exchange Format age the practice on programming. The interoperability (MEF). MEF includes a XML manifest file referring between these type of systems is becoming a topic several types of resources such as problem state- of interest in the scientific community. In order to ments (e.g. PDF, HTML), image files, input/output address these interoperability issues several problem test files, correctors (static and dynamic) and solution formats were developed. The next subsections details programs. The manifest also allows the inclusion of four formats: CATS, FreeProblemSet (FPS), Mooshak feedback and points associated to each test. Exchange Format (MEF) and Peach Exchange Format Currently, Mooshak is being used in several Uni- (PEF). Then, their features are synthesize based on a versities worldwide to support learning activity. In specific exercises format expressiveness model. the competitive context, it was used as the official evaluation system for the IEEE programming contests for some years. 2.1 CATS 2.4 Peach Exchange Format CATS1 is a format for programming assignments [1]. Peach7 is a system for the presentation, collec- The format encoded in XML describes the conditions tion, storage, management and evaluation (automated of the problem and a set of files with additional tests, and/or manual) of assignments. The Peach Exchange test programs, etc. All these files are wrapped up in Format (PEF) [4] is a specific format for programming a ZIP file to facilitate deployment. A typical XML file task packages used in Peach. Peach task packages are contains the description of 1) the problem statement stored in a directory tree with a predefined structure. codified in the Simple Text Markup Language (STML) Currently, Peach is being used by the Eindhoven format - a simplified subset of HTML; 2) the format University of Technology8. of input and output files and samples; 3) the restric- tions on the input format; 4) references for external resources such as tests generators, special checkers, 2.5 Synthesis plug-ins and solution programs. Several approaches can be found in literature [4], [1], [5] to evaluate the expressiveness of programming assignments formats. This sub-section synthesizes the 2.2 FreeProblemSet formats described previously according to the model proposed by Verhoeff. This model describes concep- Freeproblemset (FPS)2 is a transport file format for tually the notion of a task package as an unit for the storage of all information about problems. It collecting, storing, archiving, and exchanging all in- aims to provide free problems sets for managers formation concerning with a programming task. The of ACM/ICPC Online Jugdes by transporting data choice of the Verhoeff model over the alternatives from one judge to another. The format uses XML to is due to its more comprehensive coverage of the formalize the description of a programming problem. required features. This model organizes the program- It includes information on problem, test data, special ming exercise data in five facets: judger(optional) and answer(optional). Currently, the 1) Textual information - programming task human- FPS format is supported by several online judge sys- readable texts; 3 4 tems including HUSTOJ , ACM-Server and Woj-land 2) Data files - source files and test data; 5 (OJ from Wuhan University) . 3) Configuration and recommendation parameters - resource limits; 1. http://imcs.dvgu.ru/cats/docs/format.html 4) Tools - generic and task-specific tools; 2. http://code.google.com/p/freeproblemset/ 3. http://code.google.com/p/hustoj/ 6. http://mooshak.dcc.fc.up.pt/ 4. http://code.google.com/p/acm-server/ 7. http://peach3.nl 5. http://code.google.com/p/woj-land/ 8. http://peach.win.tue.nl/ 3
5) Metadata - data to foster the exercises discovery memory limits. These recommendations depend on among systems. the actual platform used for evaluation runs. Thus, The textual information facet (Table 1) accommo- platform information should be associated for a more dates all the information (or pointers to such infor- accuracy control. mation) that the exercise author wants to offer to stu- dents or contestants about the exercise. It includes, for TABLE 3 instance, the exercise challenge, background informa- Configuration and recommendation parameters facet. tion, grading information and input/output samples. Feature CATS FPS MEF PEF These texts may be available in several languages and Compiler - - - - formatted in plain text or other open format standards Executer - - - - such as HTML or LAT X. Since this data are encoded Memory limit X X - X E Size limit - - - - in flexible text data formats they can be transformed Time limit X X - - into various (open) presentation formats such as PDF. Code lines - - - - Other texts (e.g. grading information and samples) should ideally be generated from the evaluation data. When creating, solving or evaluating an exercise several software tools are needed such as editors, TABLE 1 compilers, libraries, linkers, test and feedback gener- Textual information facet. ators, input/output format checkers, evaluators, etc. Feature CATS FPS MEF PEF The tools facet (Table 4) covers the support of the Multilingual - - X X formats either by referencing these tools or by formal- HTML format X X X X izing data that can be used as input for these tools. LATEXformat - - X - Image X X X X Attach files X - - X TABLE 4 Description X X X X Tools facet. Grading - - - - Samples - - - - Feature CATS FPS MEF PEF Compiler - - - X Besides human-readable texts, a programming ex- Test gen. X - - - Feedback gen. - - X - ercise can also include several other files, in both text Skeleton gen. - - - - or binary format. The data files facet (Table 2) covers Checker X - - X the following files: program and skeleton source files Corrector - - X - Library X - X X (ideally with support for multiple programming lan- guages), input/output test data (ideally with support The metadata facet (Table 5) comprises all the for grouping and multiple visibility mode), feedback data that provide useful information on the exercise files associated with a specific test case and others for classification and discovery purposes. There are files. several types of metadata that can be included in a TABLE 2 programming exercise such as: exercise metadata (ex- Data files facet. ercise type, keywords, difficulty level), authors meta- data (name, contact), event metadata (name, type, Feature CATS FPS MEF PEF local, date, number of participants, etc.), solver meta- Solution X X X X Skeleton - - - - data (platform, operating system, etc.) and manage- Multi-language X X - X ment metadata (status of development, version infor- Tests X X X X mation, etc.). Test groups X - - X Sample tests - X - - Grading X - X X TABLE 5 Feedback - - X - Metadata facet.
Feature CATS FPS MEF PEF The description of a programming exercise can exercise X X X X also include parameters related with the submission, author X - - X compilation and execution of the student attempts event - X - X keywords - - X X to solve the exercise. These parameters can be orga- license - - - - nized in terms of configuration and recommendation platform - - - X (Table 3). The former deals with the configuration management - - - X of compiler and linkers such as the compilation line of source files and associated parameters. The latter This study confirms the disparity of programming includes recommendations usually expressed in terms exercise formats highlighting both their differences of resource limits such as the size and number of lines and their similar features. This heterogeneity hinders of a submission, compilation and execution time and the interoperability among the typical systems found 4 on the automatic evaluation of exercises. Rather than 3.2 PExIL - a pivot exercise format attempting to harmonize the various specifications, a The Programming Exercises Interoperability Lan- pragmatic solution is to provide a service for exer- guage (PExIL) [7] is a XML dialect that aims to cises formats conversion. The next section presents an consolidate all the data required in the programming approach to extensible format conversion and details exercise life-cycle. the features of PExIL – the pivotal format in which The life cycle comprises several phases: in the cre- the conversion is based – and of an abstract view of ation phase the content author should have the means the conversion service functions. to automatically create some of the resources (assets) related with the programming exercise such as the 3 EXERCISE FORMAT CONVERSION exercise description and test cases and the possibility Data conversion is the conversion of computer data to package and distribute them in a standard format from one format to another. Data conversion can across all the compatible systems (e.g. learning man- typically occur based on two approaches: direct or agement systems, learning objects repositories); in the pivotal [6]. selection phase the teacher must be able to search In the direct conversion the converter receives the for a programming exercise based on its metadata input format and apply transformations according to from a repository of learning objects and store a the output format. One apt example is the transcoder9 reference to it in a learning management system; in developed by JISC Centre for Educational Technology the presentation phase the student must be able to and Interoperability Standards (JISC CETIS) that choose the exercise description in its native language enables the conversion between e-Learning content and a proper format (e.g. HTML, PDF); in the solving packages. It addresses conversions between the most phase the learner should have the possibility to use common e-Learning content formats in use such as test cases to test his attempt to solve the exercise and IMS Content Packaging (IMS CP), Sharable Content the possibility to automatically generate new ones; in Object Reference Model (SCORM) and IMS Common the evaluation phase the evaluation engine should Cartridge (IMS CC). receive specialized metadata to properly evaluate the The pivotal conversion is based on an intermediate learner’s attempt and return enlightening feedback. format allowing any source format to be converted The PExIL definition is formalized through the to its target. Compared with the previous approach, creation of a XML Schema depicted in Figure 1. The this pivotal encoding approach provides several schema comprises three groups of elements: advantages such as manageability. A data format • Textual - elements with general information converter would have to support a huge number about the exercise to be presented to the learner. of mappings for all the permutations of the data (e.g. title, date, challenge); formats supported. Using a intermediate format • Specification - elements with a set of restrictions scales down this number since only one mapping is that can be used for generating specialized re- needed for each format supported. Pivotal conversion sources (e.g. test cases, feedback); is often used in several areas. For instance, Office • Programs - elements with references to programs applications use the OpenDocument file format as a as external resources (e.g. solution program, cor- pivot for the conversion between office file formats. rectors) and metadata about those resources (e.g. Despite its use, the pivotal approach is also subject of compilation, execution line, hints). criticism [6] such as the augment of noise due to the The expressiveness of PExIL was validated accord- propagation of the translation errors and inaccuracy ing to the multi-facet model proposed by Verhoeff. or lost of data due to the conversion between formats Table 6 shows the PExIL elements coverage. that are conceptually different. Textual elements group (G1) in PExIL can be used in several phases of the programming exercise life- cycle: in the selection phase as exercise metadata to 3.1 Approach aid discoverability and to facilitate the interoperability Based on the current conversion approaches it was among systems (e.g. LMS, IDE); in the presentation decided to use the pivotal approach regarding the phase as content to be present to the learner (e.g. exercise formats conversion. exercise description); in the solving phase as skeleton Using a pivot format reduces drastically the number code to be included in the student’s project solution. of permutations needed from n×(n−1) to 2×n, where Specification elements group (G2) in PExIL can be n is the number of formats supported. used in several phases of the programming exercise Since one of the design requirements of the Con- life-cycle: by 1) the content author to automatically verter is its extensibility it is important to simplify generate an input and output test example to be the support for new formats. included on the exercise description for presentation purposes; 2) the learner to automatically generate new 9. http://purl.oclc.org/NET/transcoder test cases to validate his attempt; 3) the Evaluation 5
Exercise
1..* 0 ..1 0..* 1..* 0..* 1..1 1..1 0 ..1 0..* 0..* 0..* Title Creation Context Challenge Keywords Input Output FeedbackLevels Solution Corrector Skeleton
+ Levels + Depends 1..* 1..1 0..* 0..* + Incremental + Sho wAllLevels Author Date Event Institution InputOutput
0..* 1..1 0..* Program Description Specification Example + Id + LineTerminato r + Lang uag e + ValueSeparato r + Co mp iler + Executer + Version 0 ..1 0 ..1 0 ..1 + So urce When Repeat Line + Ob ject 0 ..1 0 ..1 1..* + Co mpilatio n + Co nd itio n + Co unt + Visible + Executio n
1..* 0 ..1 1..* 0 ..1 0..* Hints Feedback TestCaseVisibility Data
+ Pub lic + Id 0 ..1 0 ..1 0 ..1 + Typ e Submission Compilation Execution + Value + Min + Time_So lve + Line + Line + Max + Time_Sub mit + Time + Time + Sp ec + Attempts + Size + Visible + Co de_Lines + Leng th
Fig. 1. PExIL data model.
Engine to evaluate a submission using the test cases. Convert, ConvertSets, ConvertFromSet and Program elements group (G3) contain references to ConvertToSet. In the following sub-subsection de- program source files as external resources (e.g. solu- tail the five service functions. tion program, correctors) and metadata about those GetFormats resources (e.g. compilation, execution line, hints). 3.3.1 The function These resources are used mostly in the evaluation The GetFormats function provides the requester phase of the programming exercise life-cycle to allow with a list of all the formats supported by the service. the Evaluation Engine to produce an evaluation report In order to support a format the service must imple- of a students’ attempt to solve an exercise. ment the format conversion from and to the pivotal format. In this function, the request may omit the TABLE 6 parameter or have one representing the input format. PExIL coverage on the Verhoeff model. In the former the response returns a list of all formats of the converter. In the latter the response includes Verhoeff / PExIL G1 G2 G3 only the formats that can be converted from the input Text X X - Data files X X X format given as a parameter. In the response, each Parameters - - X format is described by its name and a list of formats Tools - - X that can be used as outputs and its corresponding Metadata X - X URL paths. This will allow client systems to automate the conversion request based on the available formats This analysis asserts the total coverage of PExIL returned by the GetFormats function. elements based on the Verhoeff model and guarantees PExIL as a good candidate to act as a pivot format 3.3.2 The Convert function for a conversion service of programming exercises The Convert function performs the conversion of a formats. More details about PExIL can be found in given programming exercise from an input format literature [7]. to an output format. The function includes three parameters: 1) the format of the exercise to convert; 2) the conversion output format; and 3) a reference 3.3 Abstract Functions (URL based) of the exercise to convert. The function This section describes the generic capabilities of a returns an archive with its contents complying with Converter service expressed in terms of their be- the output format. haviours, without prescribing how to make them operational. A service of this genre is responsi- 3.3.3 The ConvertSets function ble for the conversion of programming exercises The ConvertSets function converts a set of pro- formats. It supports five functions: GetFormats, gramming exercises from an input format to an output 6 format. This function is useful with formats that deal REST implementation uses Jersey10 – the reference only with exercise sets, or when they are convenient implementation of JAX-RS (The Java API for RESTful for a particular task. In a competitive setting a contest Web Services). In Table 7 each function is associated manager may want to feed a new contest based on with the corresponding operation in the REST flavour. a problem set of a existing programming contest. In a more pedagogical setting a teacher may want TABLE 7 to use in the classroom a set of problems from an BabeLO REST API. external source. In both cases the request parameters Functions REST API are similar to the previous function. The function GetFormats convert[/in] returns an archive with a set of exercises complying Convert convert/in/out/ex the given output format. ConvertSets convertsets/in/out/ex ConvertToSet converttoset/in/out/ex ConvertFromSet convertfromset/in/out/pos/ex 3.3.4 The ConvertFromSet function The ConvertFromSet function converts a single pro- The GetFormats function returns a list of the for- gramming exercise from a set of exercises described mats supported by the service. The syntax of the GET with the input format to a single exercise in the HTTP request is: output format. The exercise to convert is given by GET /convert[/inputFormat] > BRL its position within the set as an additional parameter. The function returns an archive with a single exercise The response is formalized using the BabeLO Re- complying the given output format. sponse Language (BRL). The Convert function con- verts a given programming exercise from an input 3.3.5 The ConvertToSet function format to an output format. The syntax of the GET HTTP request is: The ConvertToSet function converts a single pro- gramming exercise complying with the input format GET /convert/in/out/ex > Archive to a single collection in the output format. The func- The function includes three parameters: in – the tion returns an archive containing a collection with format of the exercise to convert; out – the conversion only one exercise in the given output format. output format; ex – reference (URL based) of the exercise to convert. The function returns an archive 4 BABELO with its contents complying the output format. In this section details the design and implementation Figure 3 shows a typical conversion between of BabeLO on the Tomcat servlet container. Figure 2 two formats: Mooshak and FPS. BabeLO uses the shows the architecture of the BabeLO service de- convertFrom() method of MooshakConverter to scribed by an UML component diagram. transform the programming exercise from mooshak format to the PExIL format. Then, it is uses the convertTo() method of the FPSConverter to transform the exercise from the PExiL format to the Mooshak FPS ... Converter Converter FPS format.
<
co nvertFro m(data) co nvertTo(data) <
+ convertFrom(Data : File) : File MooshakConverter FPSConverter + convertTo (Data : File) : File + convertSetsFro m(Data : File, [po sition: Integ er]) : File + convertSetsTo(Data : File) : File + g etFormats([Format: String]) : XML Fig. 3. The Convert function.
REST API
<
Automatic Learning Java interface defining the convertFrom() and Authoring Assessment Objects System System Repository convertTo() methods mentioned above, as well as other similar methods to handle collections.
Fig. 2. BabeLO architecture. 5 EVALUATION RESULTS This section focus the evaluation the effectiveness and In order to allow client systems to use the conver- efficiency of the proposed approach. It is based on sion features of BabeLO its core functions are exposed as services using a REST web service interface. The 10. http://jersey.java.net 7 two actual uses of BabeLO: to relocate exercises to a although the time for downloading a collection in- different repository; and to use an evaluation engine creases with the size of the collection, the average in a network of heterogeneous systems. time per exercise reduces significantly. This is due to the fact that a collection download avoids the time of establishing a connection to the repository for each 5.1 Case study 1: repositories exchange exercise, which is a significant part of the download. In the first case study the BabeLO service is used Analysing the columns for the BabeLO download one to create a repository collection based on a problem notices something similar; the time per exercise is set of an existing programming contest. Figure 4 reduced but it is not as expressive as in the direct depicts the interconnection of BabeLO with two other download. This is due to the fact that the conversion components to achieve this purpose. time is a larger share of the overall time and saving the time of establishing connection has less impact. At first sight an overhead factor of 10 may seam crimsonHex ACM/ICPC repository (Waterlo o local contest) impracticable for on-the-fly format conversion. How- ever, it should be noted that exercises only need to be downloaded from a remote site for the first time they BabeLO are used. As in any HTTP based system a cache can REST API <
TABLE 8 LTI (IMS) <
# Direct BabeLO Over ex. Coll. P.Ex. Coll. P.Ex. head Fig. 5. Case study 2 - automatic assessment. 1 7 7 78 78 10.1 10 22 2.2 693 69.3 30.5 100 145 1.45 5,970 59.7 40.2 To start using this network the teacher sets a num- 1000 1,395 1.395 56,348 56.3 39.4 ber of activities (exercises) in the LMS by selecting a set of relevant programming exercises from the The main conclusion is that BabeLO introduces LOR. Then, the learner tries to solve the exercises an overhead of a factor of 10 when converting an assigned by the teacher using the pivot component exercise. This overhead increases with the size of col- (launched by the LMS). The pivot component recovers lections when taking as benchmark the direct down- the exercise description from the LOR and shows it load of a collection. This fact is understandable since to the student. After coding the program in the IDE BabeLO has to process each exercise, one-by-one, the student uses the pivot component to send an and takes little advantage of collections. Analysing attempt to the EE. The first time that the user send the columns for direct download one notices that, an attempt to the EE, the engine uses the BabeLO 8 service to convert the exercise from its original format fear of unauthorized copy of learning objects, and to to the Mooshak internal format and then it caches encourage users of other formats to contribute with the BabeLO response for further use. After that the code to convert to and from the pivotal format. EE evaluates the student’s attempt and returns an evaluation report. The student may submit repeatedly, REFERENCES integrating the feedback received from the EE. In the [1] A. Klenin, “Common problem description format: Require- end, the EE sends a grade to the pivot component and ments.” ACM-ICPC World Final CLIS (Competitive Learning reports the LO usage data back to the LOR. Institute Symposium), 2011. In order to validate the efficiency of this network an [2] J. P. Leal and F. M. A. Silva, “Mooshak: a web-based multi- site programming contest system,” Softw., Pract. Exper., vol. 33, experiment was conducted at ESEIG - a school of the no. 6, pp. 567–581, 2003. Polytechnic Institute of Porto. Students from the first- [3] J. P. Leal and R. Queiros,´ “A programming exercise evaluation year of the course Algorithmic and Programming par- service for mooshak,” ACM-ICPC World Final CLIS (Compet- itive Learning Institute Symposium). ACM-ICPC World Final ticipated in a two-month experiment (6 classes) with CLIS (Competitive Learning Institute Symposium), 2011. several assignments comprising 18 exercises. From [4] T. Verhoeff, “Programming task packages: Peach exchange a extensive list of results it should be stressed the format,” International Journal Olympiads In Informatics, vol. 2, pp. 192–207, 2008. number of times that the EE accessed to BabeLO (18) [5] S. H. Edwards, J. Borstler,¨ L. N. Cassel, M. S. Hall, comparing it with the number of attempts submitted and J. Hollingsworth, “Developing a common format by students to solve the exercises (819). These results for sharing programming assignments,” SIGCSE Bull., vol. 40, no. 4, pp. 167–182, 2008. [Online]. Available: show two things: the EE cache mechanism is working http://dx.doi.org/10.1145/1473195.1473240 as expected and the BabeLO service made successfully [6] T. Tsunakawa, “Pivotal approach for lexical translation,” Ph.D. conversions since no more requests were made. dissertation, University of Tokyo, 2010. [7] R. Queiros´ and J. P. Leal, “Pexil: Programming exercises interoperability language.” XML: Aplicacoes e Tecnologias Associadas (XATA), 2011. 6 CONCLUSIONS [8] J. P. Leal and R. Queiros,´ “Crimsonhex: a service oriented The goal of the research presented in this paper is repository of specialised learning objects,” in ICEIS 09 - 11th International Conference on Enterprise Information Systems, Milan, to promote the interoperability among systems that Italy, ser. Lecture Notes in Business Information Processing, participate in the automatic assessment of programs. vol. 24, Springer-Verlag, LNBIP. Springer-Verlag, LNBIP, May The heterogeneity of theses systems is the least of the 2009, pp. 102–113. [9] E. Verdu,´ L. M. Regueras, M. J. Verdu,´ J. P. Leal, J. P. de Castro, obstacles to interoperability. The major problem is the and R. Queiros,´ “A distributed system for learning program- lack of a standard for assessment items, i.e. learning ming on-line,” Computers & Education, Elsevier, vol. 58, pp. 1– objects, understood by all the systems involved in 10, January 2012. [10] R. Fielding and R. Taylor, “Principled design of the modern assessment. The proposed approach is a service to web architecture,” in Software Engineering, 2000. Proceedings of convert programming exercises formats on-the-fly. the 2000 International Conference on, 2000, pp. 407 –416. The contribution of this research is twofold, the service abstract definition and its actual implemen- tation. The abstract definition comprehends the mod- ular design of the conversion service based on a pivot Ricardo Queiros´ Assistant professor at the School of Industrial Studies and Manage- format, the data model of this pivotal format and the ment (ESEIG) in Vila do Conde. He is a Ph.D. definition of the service functions. The actual imple- student of the Doctoral Program in Computer mentation of BabeLO is a web service implemented Sciences in the Faculty of Sciences of the University of Porto (FCUP). His scientific ac- in a Java servlet container supporting a few formats tivity is related with e-Learning standards and and with the ability to be extended to new formats. Interoperability. He is an associated mem- The effectiveness and efficiency of the proposed ber of the Center for Research in Advanced Computing Systems (CRACS). approach were evaluated with two case studies pre- sented in this paper. The BabeLO service is an open source project and was already deployed in networks of e-Learning sys- tems that require on-the-fly conversion among learn- Jose´ Paulo Leal Assistant professor at the ing object formats. department of Computer Science of the Fac- ulty of Sciences of the University of Porto Currently BabeLO supports only the conversion of (FCUP) and associate researcher of the programming exercises from and to Mooshak and FPS Center for Research in Advanced Comput- formats and this is its main drawback. As future work ing Systems (CRACS). His main research interests are e-learning system implemen- the authors will extend the BabeLO compliance to tation, structured document processing and other exercise formats. Also rather than maintaining software engineering. He has a special in- a single installation of the conversion service, the au- terest on automatic exercise evaluation, in particular on the evaluation of programming thors intend to distribute the source code of BabeLO exercises, on the semantic web and on web adaptability. and promote installations of this service at each site. With this approach they expect both to reduce the