Software testing in introductory programming courses: A systematic mapping

The following sections list all selected papers in the systematic mapping. Each section con- tains the papers mapped to the corresponding topic: curriculum (Section1), teaching methods (Section2), course materials (Section3), programming assignments (Section4), programming process (Section5), tools (Section6), program/test quality (Section7), perceptions/behaviors (Section8) and concept understanding (Section9). For each selected paper, we indicate the study ID (in the format S#), the publication year, the publication venue and the evaluation method used in the study.

1 Curriculum

The topic curriculum includes the papers listed in Table1, which discuss the integration of testing in the computing curriculum as a whole or in individual programming courses.

Table 1: Selected papers mapped to topic “curriculum” study ID reference year venue name evaluation method S1 [73] 2003 OOPSLA descriptive S2 [2] 2009 SIGCSE descriptive S3 [59] 2012 ICSE descriptive S4 [91] 2002 FIE experience report S5 [53] 2003 ITiCSE experience report S6 [161] 2004 Journal of Computing in Small Colleges experience report S7 [285] 2005 SIGCSE experience report S8 [68] 2007 SIGCSE Bulletin experience report S9 [99] 2018 SIGCSE experience report S10 [233] 2017 FIE literature review S11 [137] 2000 Australasian Conference on Computing Edu- not applicable cation S12 [139] 2001 SIGCSE not applicable S13 [135] 2001 FIE not applicable S14 [109] 2011 Journal of Computing in Small Colleges survey

2 Teaching methods

The topic teaching methods includes the papers listed in Table2, which investigate methods to teach programming with the integration of software testing.

1 Table 2: Selected papers mapped to topic “teaching methods” study ID reference year venue name evaluation method S15 [189] 2003 Journal of Computing in Small Colleges descriptive S16 [102] 2002 SIGCSE experience report S17 [74] 2003 OOPSLA experience report S18 [148] 2003 Computer Science Education experience report S19 [177] 2004 Journal of Computing in Small Colleges experience report S20 [162] 2005 Journal of Computing in Small Colleges experience report S21 [80] 2007 Journal of Computing in Small Colleges experience report S22 [282] 2007 Agile experience report S23 [47] 2008 Agile experience report S24 [106] 2008 SIGCSE Bulletin experience report S25 [219] 2008 Journal of Computing in Small Colleges experience report S26 [231] 2008 SIGCSE experience report S27 [209] 2009 SIGCSE experience report S28 [145] 2013 Journal of Computing in Small Colleges experience report S29 [104] 2014 ICCSE experience report S30 [265] 2015 FIE experience report S31 [141] 2016 T4E experience report S32 [20] 2002 SIGCSE Bulletin experimental S33 [72] 2003 Journal on Educational Resources in Comput- experimental ing S34 [76] 2004 SIGCSE experimental S35 [129] 2006 SIGCSE experimental S36 [126] 2008 SIGCSE experimental S37 [115] 2012 ITiCSE experimental S38 [122] 2012 ACM Inroads experimental S39 [226] 2013 SIGCSE experimental S40 [204] 2014 ICER experimental S41 [190] 2015 FIE experimental S42 [163] 2016 Journal of Computing in Small Colleges experimental S43 [203] 2016 SIGCSE experimental S44 [158] 2017 Journal of Computing in Small Colleges experimental S45 [172] 2017 FIE experimental S46 [138] 2000 ADMI not applicable S47 [224] 2003 Journal of Computing in Small Colleges not applicable S48 [247] 2004 Journal of Computing in Small Colleges not applicable S49 [101] 2006 FIE not applicable S50 [13] 2007 Journal of Computing in Small Colleges not applicable S51 [34] 2007 Journal of Computing in Small Colleges not applicable S52 [97] 2007 SIGITE not applicable (continued in the next page)

2 Selected papers mapped to topic “teaching methods”(continued) study ID reference year venue name evaluation method S53 [96] 2007 EISTA not applicable S54 [111] 2008 ICALT not applicable S55 [235] 2009 SIGCSE Bulletin not applicable S56 [208] 2011 book not applicable S57 [23] 2012 Journal of Computing in Small Colleges not applicable S58 [31] 2012 Conference on Information Technology Edu- not applicable cation S59 [116] 2012 ICETA not applicable S60 [6] 2002 IEEE Aerospace Conference survey S61 [18] 2003 CSEE&T survey S62 [284] 2011 Australasian Computing Education Confer- survey ence S63 [243] 2012 ICER survey S64 [51] 2013 ITiCSE survey S65 [98] 2013 SIGITE survey S66 [21] 2015 Learning Scale Conference survey S67 [221] 2017 SBES survey

3 Course materials

The topic course materials includes the papers listed in Table3, which investigate how to incor- porate testing concepts into course materials of introductory courses.

Table 3: Selected papers mapped to topic “course materials” study ID reference year venue name evaluation method S68 [3] 2006 SIGCSE experimental S69 [66] 2009 SIGCSE experimental S70 [19] 2008 FIE not applicable

4 Programming assignments

The topic programming assignments includes the papers listed in Table4, which discuss guide- lines to conduct programming assignments that include testing practices.

Table 4: Selected papers mapped to topic “programming assignments”

study ID reference year venue name evaluation method S71 [171] 2005 SIGCSE experience report S72 [201] 2017 Koli Calling experimental (continued in the next page)

3 Selected papers mapped to topic “programming assignments”(continued) study ID reference year venue name evaluation method S73 [262] 2017 FIE experimental S74 [136] 2000 ADMI not applicable S75 [100] 2001 Journal of Computing Sciences in Colleges not applicable S76 [123] 2001 Journal of Computing in Small Colleges not applicable S77 [78] 2008 SIGCSE Bulletin not applicable S78 [152] 2008 OOPSLA not applicable S79 [175] 2013 Journal of Computing in Small Colleges not applicable S80 [176] 2015 Journal of Computing in Small Colleges not applicable S81 [45] 2000 Australasian Conference on Computing Edu- qualitative cation S82 [154] 2015 Koli Calling qualitative S83 [37] 2011 Journal of Computing in Small Colleges survey S84 [223] 2011 Communications in Computer and Informa- survey tion Science

5 Programming process

The topic programming process includes the papers listed in Table5, which discuss program- ming processes for novices, binding the activities of programming and testing.

Table 5: Selected papers mapped to topic “programming process”

study ID reference year venue name evaluation method S85 [245] 2001 Journal of Computing in Small Colleges descriptive S86 [7] 2003 SIGCSE experience report S87 [214] 2006 CSEE&T experience report S88 [213] 2007 FIE experience report S89 [49] 2009 ACM TOCE experience report S90 [184] 2009 Journal of Computing in Small Colleges experience report S91 [196] 2016 Journal of Computing in Small Colleges experience report S92 [75] 2003 EISTA experimental S93 [87] 2005 IEEE Transactions on Software Engineering experimental S94 [128] 2006 CSEE&T experimental S95 [173] 2009 FIE experimental S96 [39] 2012 International Journal of Information and experimental Computer Science S97 [183] 2013 ICSE experimental S98 [191] 2016 SIGCSE experimental S99 [193] 2016 CLEI experimental S100 [178] 2017 COMPSAC experimental (continued in the next page)

4 Selected papers mapped to topic “programming process”(continued) study ID reference year venue name evaluation method S101 [234] 2017 CSEE&T experimental S102 [134] 2004 Journal of Computing in Small Colleges literature review S103 [65] 2008 SIGCSE Bulletin literature review S104 [194] 2000 Southeast Regional Conference not applicable S105 [48] 2006 OOPSLA not applicable S106 [118] 2010 Southeast Regional Conference not applicable S107 [26] 2005 SIGCSE qualitative S108 [147] 2006 Australasian Conference on Computing Edu- qualitative cation S109 [181] 2008 SIGCSE qualitative S110 [277] 2011 SIGCSE qualitative S111 [197] 2015 Journal of Computing in Small Colleges qualitative

6 Tools

The topic tools includes the papers listed in Table6, which investigate supporting tools for the integration of testing into programming courses.

Table 6: Selected papers mapped to topic “tools”

study ID reference year venue name evaluation method S112 [290] 2000 ITiCSE experience report S113 [50] 2003 Computers&Education experience report S114 [156] 2003 Software: Practice and Experience experience report S115 [220] 2003 Australasian Conference on Computing Edu- experience report cation S116 [278] 2003 Australasian Conference on Computing Edu- experience report cation S117 [52] 2005 Advances in Web-Based Learning experience report S118 [57] 2005 FIE experience report S119 [114] 2005 Journal of Computing in Small Colleges experience report S120 [17] 2006 SIGCSE Bulletin experience report S121 [90] 2006 PPPJ experience report S122 [269] 2006 OOPSLA experience report S123 [85] 2007 ICSE experience report S124 [105] 2007 PPPJ experience report S125 [15] 2008 ITiCSE experience report S126 [270] 2008 Software: Practice and Experience experience report S127 [210] 2009 PPPJ experience report S128 [230] 2009 SIGCSE experience report (continued in the next page)

5 Selected papers mapped to topic “tools”(continued) study ID reference year venue name evaluation method S129 [266] 2009 ETCS experience report S130 [155] 2010 ITiCSE experience report S131 [258] 2011 ITiCSE experience report S132 [272] 2011 ITiCSE experience report S133 [198] 2012 SIGCSE experience report S134 [215] 2012 ICAESM experience report S135 [289] 2012 Journal of Computing in Small Colleges experience report S136 [200] 2013 CSERC experience report S137 [242] 2013 IISA experience report S138 [279] 2013 ITiCSE experience report S139 [77] 2014 Learning Scale Conference experience report S140 [170] 2014 Western Canadian Conference on Computing experience report Education S141 [267] 2014 ISSTA experience report S142 [28] 2015 ICSE experience report S143 [30] 2015 SIGCSE experience report S144 [121] 2015 IIAI experience report S145 [274] 2015 FIE experience report S146 [95] 2016 ITiCSE experience report S147 [112] 2016 CSEE&T experience report S148 [153] 2016 Journal of Computing in Small Colleges experience report S149 [255] 2005 International Workshop on Mining Software experimental Repositories S150 [188] 2001 SIGCSE experimental S151 [60] 2003 IEEE Transactions on Education experimental S152 [232] 2006 International Workshop on Automation of experimental Software Test S153 [264] 2008 SIGCSE experimental S154 [11] 2009 SIGCSE experimental S155 [46] 2010 Australian Software Engineering Conference experimental S156 [54] 2010 OOPSLA experimental S157 [64] 2011 SIGCSE experimental S158 [70] 2011 CSEE&T experimental S159 [86] 2011 FIE experimental S160 [185] 2011 ITHET experimental S161 [248] 2011 CSEE&T experimental S162 [281] 2011 Computers & Education experimental S163 [10] 2012 SIGCSE experimental S164 [83] 2012 SIGCSE experimental S165 [146] 2012 AICERA experimental S166 [41] 2013 SIGCSE experimental (continued in the next page)

6 Selected papers mapped to topic “tools”(continued) study ID reference year venue name evaluation method S167 [127] 2013 ICSE experimental S168 [131] 2013 CSEE&T experimental S169 [237] 2013 SPLASH experimental S170 [239] 2013 ICER experimental S171 [280] 2013 Information and Software Technology experimental S172 [9] 2014 Software - Practice and Experience experimental S173 [43] 2014 ITiCSE experimental S174 [84] 2014 SIGCSE experimental S175 [205] 2014 ITiCSE experimental S176 [227] 2014 Computers in Human Behavior experimental S177 [249] 2014 FIE experimental S178 [29] 2015 SIGCSE experimental S179 [44] 2015 SIGCSE experimental S180 [217] 2015 Journal of Computing in Small Colleges experimental S181 [62] 2015 International Journal of Knowledge and experimental Learning S182 [25] 2016 ITiCSE experimental S183 [27] 2016 ITiCSE experimental S184 [32] 2016 SIGCSE experimental S185 [216] 2016 Australasian Computer Science Week Multi- experimental conference S186 [256] 2016 Learning Scale Conference experimental S187 [261] 2016 ITiCSE experimental S188 [244] 2017 ITiCSE experimental S189 [168] 2017 International Scientific Conference on Infor- experimental matics S190 [288] 2017 Joint Meeting on Foundations of Software experimental Engineering S191 [151] 2018 SIGCSE experimental S192 [5] 2005 Computer Science Education literature review S193 [69] 2005 Journal of Computing in Small Colleges literature review S194 [120] 2010 Koli Calling literature review S195 [222] 2010 International Symposium on Information literature review Technology S196 [12] 2008 Educational Data Mining not applicable S197 [107] 2000 FIE not applicable S198 [125] 2000 ITiCSE not applicable S199 [8] 2002 SIGCSE not applicable S200 [113] 2002 ITiCSE not applicable S201 [16] 2003 OOPSLA not applicable S202 [140] 2003 ITiCSE not applicable (continued in the next page)

7 Selected papers mapped to topic “tools”(continued) study ID reference year venue name evaluation method S203 [180] 2003 FIE not applicable S204 [195] 2003 ITiCSE not applicable S205 [212] 2004 SIGCSE Bulletin not applicable S206 [251] 2004 OOPSLA not applicable S207 [260] 2004 Southeast Regional Conference not applicable S208 [14] 2005 OOPSLA not applicable S209 [88] 2006 FIE not applicable S210 [254] 2006 OOPSLA not applicable S211 [110] 2007 SIGCSE not applicable S212 [119] 2007 Informatics in Education not applicable S213 [182] 2007 FIE not applicable S214 [271] 2007 SIGCSE not applicable S215 [79] 2008 ITiCSE not applicable S216 [92] 2008 SIGCSE not applicable S217 [225] 2008 ITiCSE not applicable S218 [211] 2009 SIGCSE not applicable S219 [56] 2010 ICST not applicable S220 [144] 2010 ITiCSE not applicable S221 [157] 2010 book not applicable S222 [218] 2010 SIGCSE not applicable S223 [24] 2011 Int. Workshop on Social Software Engineer- not applicable ing S224 [117] 2011 ITiCSE not applicable S225 [241] 2011 Int. Workshop on Games and Software Engi- not applicable neering S226 [246] 2011 SIGCSE not applicable S227 [292] 2011 CSEE&T not applicable S228 [165] 2012 Koli Calling not applicable S229 [61] 2013 Procedia Technology not applicable S230 [268] 2013 ASE not applicable S231 [291] 2013 Journal of Software not applicable S232 [293] 2013 SIGCSE not applicable S233 [4] 2014 CSCI not applicable S234 [169] 2014 ITiCSE not applicable S235 [187] 2014 Learning Scale Conference not applicable S236 [207] 2014 CEUR not applicable S237 [33] 2015 ITiCSE not applicable S238 [58] 2015 Workshop on Educational Software Engineer- not applicable ing S239 [202] 2015 ICETA not applicable S240 [206] 2015 MIPRO not applicable (continued in the next page)

8 Selected papers mapped to topic “tools”(continued) study ID reference year venue name evaluation method S241 [273] 2015 SIGCSE not applicable S242 [287] 2015 Int. Workshop on CrowdSourcing in Software not applicable Engineering S243 [133] 2016 SIGCSE not applicable S244 [164] 2016 SIGPLAN Symposium on Scala not applicable S245 [286] 2016 SIGCSE not applicable S246 [93] 2017 AINA not applicable S247 [55] 2017 ICSE not applicable S248 [67] 2017 ITiCSE not applicable S249 [142] 2005 Journal of Computing in Small Colleges qualitative S250 [228] 2001 ITiCSE survey S251 [108] 2004 Journal of Computing in Small Colleges survey S252 [143] 2005 FIE survey S253 [236] 2005 SIGCSE Bulletin survey S254 [252] 2006 SIGCSE survey S255 [186] 2007 Journal of Computing in Small Colleges survey S256 [259] 2008 SAICSIT survey S257 [36] 2012 ITiCSE survey S258 [132] 2012 ITiCSE survey S259 [94] 2014 GCCE survey S260 [192] 2016 ICSE survey S261 [276] 2017 FIE survey

7 Program/test quality

The topic program/test quality includes the papers listed in Table7, which investigate assess- ment of students’ submitted code (program or tests).

Table 7: Selected papers mapped to topic “program/test quality”

study ID reference year venue name evaluation method S262 [257] 2001 XP Universe Conference descriptive S263 [179] 2004 International Symposium on Software Met- experimental rics S264 [1] 2010 OOPSLA experimental S265 [35] 2012 CLEI Electronic Journal experimental S266 [275] 2013 ITiCSE experimental S267 [81] 2014 ICSE experimental S268 [283] 2014 Australasian Computing Education Confer- experimental ence (continued in the next page)

9 Selected papers mapped to topic “program/test quality”(continued) study ID reference year venue name evaluation method S269 [159] 2015 Int. Symposium on Software Reliability En- experimental gineering S270 [240] 2015 SIGCSE experimental S271 [71] 2016 Koli Calling experimental S272 [103] 2016 ICSE experimental S273 [160] 2017 Journal of Systems and Software experimental S274 [63] 2017 FIE experimental S275 [238] 2013 ICER not applicable S276 [167] 2013 ITiCSE qualitative

8 Perceptions/behaviors

The topic perceptions/behaviors includes the papers listed in Table8, which investigate stu- dents’ attitudes towards software testing.

Table 8: Selected papers mapped to topic “perceptions and behaviors” study ID reference year venue name evaluation method S279 [253] 2006 OOPSLA experimental S280 [130] 2007 ICSE experimental S281 [38] 2012 ITiCSE experimental S282 [40] 2013 ICER experimental S283 [89] 2013 Australasian Computing Education Confer- experimental ence S284 [250] 2013 ITiCSE experimental S285 [42] 2014 SIGCSE experimental S286 [82] 2014 ITiCSE experimental S287 [22] 2016 Journal of Computing in Small Colleges experimental S288 [149] 2005 ICER qualitative S289 [263] 2006 Australasian Conference on Computing Edu- qualitative cation S290 [150] 2008 Computer Science Education qualitative S291 [174] 2009 FIE qualitative S292 [199] 2014 Symposium on the Foundations of Software qualitative Engineering S293 [124] 2017 ITiCSE qualitative

9 Concept understanding

The topic concept understanding includes the papers listed in Table9, which investigate the assessment of students’ knowledge of programming and testing concepts.

10 Table 9: Selected papers mapped to topic “concept understanding” study ID reference year venue name evaluation method S277 [229] 2013 ITiCSE experimental S278 [166] 2017 ITiCSE exploratory

References

[1] K. Aaltonen, P. Ihantola, and O. Seppala. Mutation Analysis vs. Code Coverage in Auto- mated Assessment of Students’ Testing Skills. In Proceedings of the ACM International Conference Companion on Object Oriented Programming Systems Languages and Applica- tions Companion, OOPSLA ’10, pages 153–160, New York, NY, USA, 2010. ACM.

[2] J. Adams. Test-driven Data Structures: Revitalizing CS2. In Proceedings of the 40th ACM Technical Symposium on Computer Science Education, SIGCSE ’09, pages 143–147, New York, NY, USA, 2009. ACM.

[3] R. Agarwal, S. H. Edwards, and M. A. Perez-Qui´ nones.˜ Designing an adaptive learning module to teach software testing. In Proceedings of the 37th SIGCSE technical symposium on Computer science education, SIGCSE ’06, pages 259–263, New York, NY, USA, 2006. ACM.

[4] M. Akour. Towards Harnessing Testing Tools into Programming Courses Curricula: Case Study of Jordan. In 2014 International Conference on Computational Science and Compu- tational Intelligence, volume 2, pages 197–200, Mar. 2014.

[5] K. M. Ala-Mutka. A survey of automated assessment approaches for programming assign- ments. Computer Science Education, 15(2):83–102, 2005.

[6] I. Alkadi and G. Alkadi. To C++ or to Java, that is the question! Featuring a test plan and an automated testing assistant for object oriented testing. In Proceedings, IEEE Aerospace Conference, volume 7, pages 7–3679–7–3688 vol.7, 2002.

[7] E. Allen, R. Cartwright, and C. Reis. Production Programming in the Classroom. In Proceed- ings of the 34th SIGCSE Technical Symposium on Computer Science Education, SIGCSE ’03, pages 89–93, New York, NY, USA, 2003. ACM.

[8] E. Allen, R. Cartwright, and B. Stoler. DrJava: A Lightweight Pedagogic Environment for Java. In Proceedings of the 33rd SIGCSE Technical Symposium on Computer Science Edu- cation, SIGCSE ’02, pages 137–141, New York, NY, USA, 2002. ACM.

[9] A. Allevato and S. Edwards. Dereferee: Instrumenting C++ pointers with meaningful runtime diagnostics. Software - Practice and Experience, 44(8):973–997, 2014.

[10] A. Allevato and S. H. Edwards. RoboLIFT: Engaging CS2 Students with Testable, Automati- cally Evaluated Android Applications. In Proceedings of the 43rd ACM Technical Symposium on Computer Science Education, SIGCSE ’12, pages 547–552, New York, NY, USA, 2012. ACM.

11 [11] A. Allevato, S. H. Edwards, and M. A. Perez-Qui´ nones.˜ Dereferee: Exploring Pointer Mis- management in Student Code. In Proceedings of the 40th ACM Technical Symposium on Computer Science Education, SIGCSE ’09, pages 173–177, New York, NY, USA, 2009. ACM.

[12] A. Allevato, M. Thornton, S. H. Edwards, and M. A. Perez-Qui´ nones.˜ Mining data from an automated grading and testing system by adding rich reporting capabilities. In Educational Data Mining (EDM 2008), 2008.

[13] C. D. Allison. The Simplest Unit Test Tool That Could Possibly Work. J. Comput. Sci. Coll., 23(1):183–189, Oct. 2007.

[14] A. Allowatt and S. H. Edwards. Ide support for test-driven development and automated grading in both java and c++. In Proceedings of the 2005 OOPSLA Workshop on Eclipse Technology eXchange, eclipse ’05, pages 100–104, New York, NY, USA, 2005. ACM.

[15] M. Amelung, P. Forbrig, and D. Rosner.¨ Towards generic and flexible web services for e- assessment. In Proceedings of the 13th Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE ’08, pages 219–224, New York, NY, USA, 2008. ACM.

[16] S. K. Andrianoff, D. B. Levine, S. D. Gewand, and G. A. Heissenberger. A Testing-based Framework for Programming Contests. In Proceedings of the 2003 OOPSLA Workshop on Eclipse Technology eXchange, eclipse ’03, pages 94–98, New York, NY, USA, 2003. ACM.

[17] J. Baldwin, E. Crupi, and T. Estrellado. Webwork for programming fundamentals. SIGCSE Bull., 38(3):361–361, June 2006.

[18] E. F. Barbosa, J. C. Maldonado, R. LeBlanc, and M. Guzdial. Introducing testing practices into objects and design course. In Proceedings 16th Conference on Software Engineering Education and Training, 2003. (CSEE T 2003)., pages 279–286, March 2003.

[19] E. F. Barbosa, M. A. G. Silva, C. K. D. Corte, and J. C. Maldonado. Integrated teaching of programming foundations and software testing. In 2008 38th Annual Frontiers in Education Conference, pages S1H–5–S1H–10, Oct. 2008.

[20] E. G. Barriocanal, M.-n. S. Urban,´ I. A. Cuevas, and P. D. Perez.´ An Experience in Integrating Automated Unit Testing Practices in an Introductory Programming Course. SIGCSE Bull., 34(4):125–128, Dec. 2002.

[21] S. Basu, A. Wu, B. Hou, and J. DeNero. Problems before solutions: Automated problem clarification at scale. In Proceedings of the Second (2015) ACM Conference on Learning @ Scale, L@S ’15, pages 205–213, New York, NY, USA, 2015. ACM.

[22] L. Baumstark, Jr. and M. Orsega. Quantifying Introductory CS Students’ Iterative Software Process by Mining Version Control System Repositories. J. Comput. Sci. Coll., 31(6):97– 104, June 2016.

[23] T. Beaubouef and W. Zhang. Learning to Program Through Use of Code Verification. J. Comput. Sci. Coll., 27(5):78–84, May 2012.

12 [24] J. Bell, S. Sheth, and G. Kaiser. Secret Ninja Testing with HALO Software Engineering. In Proceedings of the 4th International Workshop on Social Software Engineering, SSE ’11, pages 43–47, New York, NY, USA, 2011. ACM.

[25] C. Benac Earle, L.-\. Fredlund, and J. Hughes. Automatic Grading of Programming Exer- cises Using Property-Based Testing. In Proceedings of the 2016 ACM Conference on Innova- tion and Technology in Computer Science Education, ITiCSE ’16, pages 47–52, New York, NY, USA, 2016. ACM.

[26] J. Bennedsen and M. E. Caspersen. Revealing the Programming Process. In Proceedings of the 36th SIGCSE Technical Symposium on Computer Science Education, SIGCSE ’05, pages 186–190, New York, NY, USA, 2005. ACM.

[27] G. Birch, B. Fischer, and M. Poppleton. Using Fast Model-Based Fault Localisation to Aid Students in Self-Guided Program Repair and to Improve Assessment. In Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education, ITiCSE ’16, pages 168–173, New York, NY, USA, 2016. ACM.

[28] J. Bishop, R. N. Horspool, T. Xie, N. Tillmann, and J. de Halleux. Code Hunt: Experience with Coding Contests at Scale. In Proceedings of the 37th International Conference on Software Engineering - Volume 2, ICSE ’15, pages 398–407, Piscataway, NJ, USA, 2015. IEEE Press.

[29] D. Blaheta. Unci: A C++-based Unit-testing Framework for Intro Students. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education, SIGCSE ’15, pages 475–480, New York, NY, USA, 2015. ACM.

[30] M. K. Bradshaw. Ante Up: A Framework to Strengthen Student-Based Testing of Assign- ments. In Proceedings of the 46th ACM Technical Symposium on Computer Science Educa- tion, SIGCSE ’15, pages 488–493, New York, NY, USA, 2015. ACM.

[31] E. Brannock and N. Napier. Real-world Testing: Using FOSS for Software Development Courses. In Proceedings of the 13th Annual Conference on Information Technology Educa- tion, SIGITE ’12, pages 87–88, New York, NY, USA, 2012. ACM.

[32] G. Braught and J. Midkiff. Tool Design and Student Testing Behavior in an Introductory Java Course. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education, SIGCSE ’16, pages 449–454, New York, NY, USA, 2016. ACM.

[33] S. A. Brian, R. N. Thomas, J. M. Hogan, and C. Fidge. Planting Bugs: A System for Test- ing Students’ Unit Tests. In Proceedings of the 2015 ACM Conference on Innovation and Technology in Computer Science Education, ITiCSE ’15, pages 45–50, New York, NY, USA, 2015. ACM.

[34] T. Briggs and C. D. Girard. Tools and Techniques for Test-driven Learning in CS1. J. Comput. Sci. Coll., 22(3):37–43, Jan. 2007.

[35] M. A. S. Brito, J. L. Rossi, S. R. S. de Souza, and R. T. V. Braga. An Experience on Applying Software Testing for Teaching Introductory Programming Courses. CLEI Electronic Journal, 15:5 – 5, 04 2012.

13 [36] C. Brown, R. Pastel, B. Siever, and J. Earnest. JUG: A JUnit Generation, Time Complexity Analysis and Reporting Tool to Streamline Grading. In Proceedings of the 17th ACM An- nual Conference on Innovation and Technology in Computer Science Education, ITiCSE ’12, pages 99–104, New York, NY, USA, 2012. ACM. [37] R. Bryce. Bug Wars: A Competitive Exercise to Find Bugs in Code. J. Comput. Sci. Coll., 27(2):43–50, Dec. 2011. [38] K. Buffardi and S. H. Edwards. Exploring Influences on Student Adherence to Test-driven Development. In Proceedings of the 17th ACM Annual Conference on Innovation and Tech- nology in Computer Science Education, ITiCSE ’12, pages 105–110, New York, NY, USA, 2012. ACM. [39] K. Buffardi and S. H. Edwards. Impacts of teaching test-driven development to novice pro- grammers. International Journal of Information and Computer Science, 1(6):135–143, 2012. [40] K. Buffardi and S. H. Edwards. Effective and Ineffective Software Testing Behaviors by Novice Programmers. In Proceedings of the Ninth Annual International ACM Conference on International Computing Education Research, ICER ’13, pages 83–90, New York, NY, USA, 2013. ACM. [41] K. Buffardi and S. H. Edwards. Impacts of Adaptive Feedback on Teaching Test-driven Development. In Proceeding of the 44th ACM Technical Symposium on Computer Science Education, SIGCSE ’13, pages 293–298, New York, NY, USA, 2013. ACM. [42] K. Buffardi and S. H. Edwards. A Formative Study of Influences on Student Testing Behav- iors. In Proceedings of the 45th ACM Technical Symposium on Computer Science Education, SIGCSE ’14, pages 597–602, New York, NY, USA, 2014. ACM. [43] K. Buffardi and S. H. Edwards. Responses to Adaptive Feedback for Software Testing. In Proceedings of the 2014 Conference on Innovation & Technology in Computer Science Education, ITiCSE ’14, pages 165–170, New York, NY, USA, 2014. ACM. [44] K. Buffardi and S. H. Edwards. Reconsidering Automated Feedback: A Test-Driven Ap- proach. In Proceedings of the 46th ACM Technical Symposium on Computer Science Educa- tion, SIGCSE ’15, pages 416–420, New York, NY, USA, 2015. ACM. [45] A. Carbone, J. Hurst, I. Mitchell, and D. Gunstone. Principles for designing programming ex- ercises to minimise poor learning behaviours in students. In Proceedings of the Australasian Conference on Computing Education, ACSE ’00, pages 26–33, New York, NY, USA, 2000. ACM. [46] R. Cardell-Oliver, L. Zhang, R. Barady, Y. H. Lim, A. Naveed, and T. Woodings. Automated Feedback for Quality Assurance in Software Engineering Education. In 2010 21st Australian Software Engineering Conference, pages 157–164, Apr. 2010. [47] B. Carlson. An Agile classroom experience: Teaching TDD and refactoring. In Proceedings - Agile 2008 Conference, pages 465–469, 2008. [48] M. E. Caspersen and M. Kolling. A Novice’s Process of Object-oriented Programming. In Companion to the 21st ACM SIGPLAN Symposium on Object-oriented Programming Sys- tems, Languages, and Applications, OOPSLA ’06, pages 892–900, New York, NY, USA, 2006. ACM.

14 [49] M. E. Caspersen and M. Kolling. STREAM: A First Programming Process. Trans. Comput. Educ., 9(1):4:1–4:29, Mar. 2009.

[50] B. Cheang, A. Kurnia, A. Lim, and W.-C. Oon. On automated grading of programming assignments in an academic institution. Comput. Educ., 41(2):121–131, Sept. 2003.

[51] W. K. Chen and B. R. Hall. Applying Software Engineering in CS1. In Proceedings of the 18th ACM Conference on Innovation and Technology in Computer Science Education, ITiCSE ’13, pages 297–302, New York, NY, USA, 2013. ACM.

[52] M. Choy, U. Nazir, C. K. Poon, and Y. T. Yu. Experiences in using an automated system for improving students’ learning of computer programming. In R. W. H. Lau, Q. Li, R. Cheung, and W. Liu, editors, Advances in Web-Based Learning – ICWL 2005, pages 267–272, Berlin, Heidelberg, 2005. Springer Berlin Heidelberg.

[53] H. B. e. Christensen. Systematic Testing Should Not Be a Topic in the Computer Science Curriculum! In Proceedings of the 8th Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE ’03, pages 7–10, New York, NY, USA, 2003. ACM.

[54] P. J. Clarke, A. A. Allen, T. M. King, E. L. Jones, and P. Natesan. Using a Web-based Repository to Integrate Testing Tools into Programming Courses. In Proceedings of the ACM International Conference Companion on Object Oriented Programming Systems Languages and Applications Companion, OOPSLA ’10, pages 193–200, New York, NY, USA, 2010. ACM.

[55] B. S. Clegg, J. M. Rojas, and G. Fraser. Teaching software testing concepts using a mutation testing game. In Proceedings of the 39th International Conference on Software Engineering: Software Engineering and Education Track, ICSE-SEET ’17, pages 33–36, Piscataway, NJ, USA, 2017. IEEE Press.

[56] J. Clements and D. Janzen. Overcoming Obstacles to Test-Driven Learning on Day One. In 2010 Third International Conference on Software Testing, Verification, and Validation Workshops, pages 448–453, Apr. 2010.

[57] J. Collofello and K. Vehathiri. An environment for training computer science students on software testing. In Proceedings Frontiers in Education 35th Annual Conference, pages T3E–6, Oct 2005.

[58] S. Combefis´ and A. Paques. Pythia Reloaded: An Intelligent Unit Testing-based Code Grader for Education. In Proceedings of the 1st International Workshop on Code Hunt Workshop on Educational Software Engineering, CHESE 2015, pages 5–8, New York, NY, USA, 2015. ACM.

[59] T. Cowling. Stages in teaching software testing. In 2012 34th International Conference on Software Engineering (ICSE), pages 1185–1194, June 2012.

[60] C. Daly and J. M. Horgan. An automated learning system for java programming. IEEE Transactions on Education, 47(1):10–17, Feb 2004.

[61] K. Danutama and I. Liem. Scalable Autograder and LMS Integration. Procedia Technology, 11:388 – 395, 2013.

15 [62] D. M. de Souza, S. Isotani, and E. F. Barbosa. Teaching novice programmers using progtest. International Journal of Knowledge and Learning, 2015.

[63] D. M. de Souza, M. Kolling, and E. F. Barbosa. Most common fixes students use to improve the correctness of their programs. In 2017 IEEE Frontiers in Education Conference (FIE), pages 1–9, Oct 2017.

[64] P. Denny, A. Luxton-Reilly, E. Tempero, and J. Hendrickx. CodeWrite: Supporting Student- driven Practice of Java. In Proceedings of the 42Nd ACM Technical Symposium on Computer Science Education, SIGCSE ’11, pages 471–476, New York, NY, USA, 2011. ACM.

[65] C. Desai, D. Janzen, and K. Savage. A survey of evidence for test-driven development in academia. SIGCSE Bull., 40(2):97–101, June 2008.

[66] C. Desai, D. S. Janzen, and J. Clements. Implications of Integrating Test-driven Development into CS1/CS2 Curricula. In Proceedings of the 40th ACM Technical Symposium on Computer Science Education, SIGCSE ’09, pages 148–152, New York, NY, USA, 2009. ACM.

[67] K. Dewey, P. Conrad, M. Craig, and E. Morozova. Evaluating test suite effectiveness and as- sessing student code via constraint logic programming. In Proceedings of the 2017 ACM Con- ference on Innovation and Technology in Computer Science Education, ITiCSE ’17, pages 317–322, New York, NY, USA, 2017. ACM.

[68] P. M. Dorin. Laboratory Redux. SIGCSE Bull., 39(2):84–87, June 2007.

[69] C. Douce, D. Livingstone, and J. Orwell. Automatic Test-based Assessment of Programming: A Review. J. Educ. Resour. Comput., 5(3), Sept. 2005.

[70] T. Dvornik, D. S. Janzen, J. Clements, and O. Dekhtyar. Supporting introductory test-driven labs with WebIDE. In 2011 24th IEEE-CS Conference on Software Engineering Education and Training (CSEE T), pages 51–60, May 2011.

[71] S. Edwards and Z. Li. Towards Progress Indicators for Measuring Student Programming Effort During Solution Development. In Proceedings of the 16th Koli Calling International Conference on Computing Education Research, Koli Calling ’16, pages 31–40, New York, NY, USA, 2016. ACM.

[72] S. H. Edwards. Improving Student Performance by Evaluating How Well Students Test Their Own Programs. J. Educ. Resour. Comput., 3(3), Sept. 2003.

[73] S. H. Edwards. Rethinking Computer Science Education from a Test-first Perspective. In Companion of the 18th Annual ACM SIGPLAN Conference on Object-oriented Program- ming, Systems, Languages, and Applications, OOPSLA ’03, pages 148–155, New York, NY, USA, 2003. ACM.

[74] S. H. Edwards. Teaching Software Testing: Automatic Grading Meets Test-first Coding. In Companion of the 18th Annual ACM SIGPLAN Conference on Object-oriented Program- ming, Systems, Languages, and Applications, OOPSLA ’03, pages 318–319, New York, NY, USA, 2003. ACM.

[75] S. H. Edwards. Using test-driven development in the classroom: Providing students with automatic, concrete feedback on performance. In International Conference on Education and Information Systems: Technologies and Applications (EISTA?03), 2003.

16 [76] S. H. Edwards. Using Software Testing to Move Students from Trial-and-error to Reflection- in-action. In Proceedings of the 35th SIGCSE Technical Symposium on Computer Science Education, SIGCSE ’04, pages 26–30, New York, NY, USA, 2004. ACM.

[77] S. H. Edwards. Work-in-progress: Program Grading and Feedback Generation with Web- CAT. In Proceedings of the First ACM Conference on Learning @ Scale Conference, L@S ’14, pages 215–216, New York, NY, USA, 2014. ACM.

[78] S. H. Edwards, J. Borstler,¨ L. N. Cassel, M. S. Hall, and J. Hollingsworth. Developing a common format for sharing programming assignments. SIGCSE Bull., 40(4):167–182, Nov. 2008.

[79] S. H. Edwards and M. A. Perez-Quinones. Web-CAT: Automatically Grading Programming Assignments. In Proceedings of the 13th Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE ’08, pages 328–328, New York, NY, USA, 2008. ACM.

[80] S. H. Edwards and M. A. Perez-Qui´ nones.˜ Experiences Using Test-driven Development with an Automated Grader. J. Comput. Sci. Coll., 22(3):44–50, Jan. 2007.

[81] S. H. Edwards and Z. Shams. Comparing Test Quality Measures for Assessing Student- written Tests. In Companion Proceedings of the 36th International Conference on Software Engineering, ICSE Companion 2014, pages 354–363, New York, NY, USA, 2014. ACM.

[82] S. H. Edwards and Z. Shams. Do Student Programmers All Tend to Write the Same Software Tests? In Proceedings of the 2014 Conference on Innovation & Technology in Computer Science Education, ITiCSE ’14, pages 171–176, New York, NY, USA, 2014. ACM.

[83] S. H. Edwards, Z. Shams, M. Cogswell, and R. C. Senkbeil. Running Students’ Software Tests Against Each Others’ Code: New Life for an Old ”Gimmick”. In Proceedings of the 43rd ACM Technical Symposium on Computer Science Education, SIGCSE ’12, pages 221– 226, New York, NY, USA, 2012. ACM.

[84] S. H. Edwards, Z. Shams, and C. Estep. Adaptively Identifying Non-terminating Code when Testing Student Programs. In Proceedings of the 45th ACM Technical Symposium on Com- puter Science Education, SIGCSE ’14, pages 15–20, New York, NY, USA, 2014. ACM.

[85] S. Elbaum, S. Person, J. Dokulil, and M. Jorde. Bug hunt: Making early software testing lessons engaging and affordable. In Proceedings - International Conference on Software Engineering, pages 687–697, 2007.

[86] E. Enstrom,¨ G. Kreitz, F. Niemela,¨ P. Soderman,¨ and V. Kann. Five years with kattis #x2014; Using an automated assessment system in teaching. In 2011 Frontiers in Education Confer- ence (FIE), pages T3J–1–T3J–6, Oct. 2011.

[87] H. Erdogmus, M. Morisio, and M. Torchiano. On the effectiveness of the test-first approach to programming. IEEE Transactions on Software Engineering, 31(3):226–237, March 2005.

[88] M. Y. Feng and A. McAllister. A tool for automated gui program grading. In Proceedings. Frontiers in Education. 36th Annual Conference, pages 7–12, Oct 2006.

17 [89] C. Fidge, J. Hogan, and R. Lister. What vs. How: Comparing Students’ Testing and Cod- ing Skills. In Proceedings of the Fifteenth Australasian Computing Education Conference - Volume 136, ACE ’13, pages 97–106, Darlinghurst, Australia, Australia, 2013. Australian Computer Society, Inc.

[90] G. Fischer and J. W. von Gudenberg. Improving the Quality of Programming Education by Online Assessment. In Proceedings of the 4th International Symposium on Principles and Practice of Programming in Java, PPPJ ’06, pages 208–211, New York, NY, USA, 2006. ACM.

[91] S. Frezza. Integrating testing and design methods for undergraduates: teaching software testing in the context of software design. In 32nd Annual Frontiers in Education, volume 3, pages S1G–1–S1G–4 vol.3, Nov 2002.

[92] X. Fu, B. Peltsverger, K. Qian, L. Tao, and J. Liu. APOGEE: Automated Project Grad- ing and Instant Feedback System for Web Based Computing. In Proceedings of the 39th SIGCSE Technical Symposium on Computer Science Education, SIGCSE ’08, pages 77–81, New York, NY, USA, 2008. ACM.

[93] N. Funabiki, R. Kusaka, N. Ishihara, and W. C. Kao. A Proposal of Test Code Generation Tool for Java Programming Learning Assistant System. In 2017 IEEE 31st International Conference on Advanced Information Networking and Applications (AINA), pages 51–56, Mar. 2017.

[94] N. Funabiki, T. Nakamura, and W.-C. Kao. A proposal of Javadoc hint function for Java Programming Learning Assistant System. In 2014 IEEE 3rd Global Conference on Consumer Electronics (GCCE), pages 304–308, Oct. 2014.

[95] J. Gao, B. Pang, and S. S. Lumetta. Automated Feedback Framework for Introductory Pro- gramming Courses. In Proceedings of the 2016 ACM Conference on Innovation and Technol- ogy in Computer Science Education, ITiCSE ’16, pages 53–58, New York, NY, USA, 2016. ACM.

[96] A. Gaspar and S. Langevin. Active learning in introductory programming courses through student-led ?live coding? and test-driven pair programming. In EISTA 2007, Education and Information Systems, Technologies and Applications, 2007.

[97] A. Gaspar and S. Langevin. Restoring ”Coding with Intention” in Introductory Programming Courses. In Proceedings of the 8th ACM SIGITE Conference on Information Technology Education, SIGITE ’07, pages 91–98, New York, NY, USA, 2007. ACM.

[98] A. Gaspar, S. Langevin, N. Boyer, and R. Tindell. A Preliminary Review of Undergraduate Programming Students’ Perspectives on Writing Tests, Working with Others, & Using Peer Testing. In Proceedings of the 14th Annual ACM SIGITE Conference on Information Technology Education, SIGITE ’13, pages 109–114, New York, NY, USA, 2013. ACM.

[99] P. Gestwicki. Design and evaluation of an undergraduate course on software development practices. In Proceedings of the 49th ACM Technical Symposium on Computer Science Edu- cation, SIGCSE ’18, pages 221–226, New York, NY, USA, 2018. ACM.

[100] A. Ghafarian. Incorporating a Semester-long Project into the CS 2 Course. J. Comput. Sci. Coll., 17(2):183–190, Dec. 2001.

18 [101] C. D. Girard and C. Wellington. Work in progress: A test-first approach to teaching cs1. In Proceedings. Frontiers in Education. 36th Annual Conference, pages 19–20, Oct 2006.

[102] M. H. Goldwasser. A Gimmick to Integrate Software Testing Throughout the Curriculum. In Proceedings of the 33rd SIGCSE Technical Symposium on Computer Science Education, SIGCSE ’02, pages 271–275, New York, NY, USA, 2002. ACM.

[103] O. S. Gomez,´ S. Vegas, and N. Juristo. Impact of cs programs on the quality of test cases generation: An empirical study. In Proceedings of the 38th International Conference on Software Engineering Companion, ICSE ’16, pages 374–383, New York, NY, USA, 2016. ACM.

[104] L. H. Gonzalez-Guerra and A. J. Leal-Flores. Tutoring model to guide students in program- ming courses to create complete and correct solutions. In 2014 9th International Conference on Computer Science Education, pages 75–80, Aug. 2014.

[105] O. Gotel, C. Scharff, and A. Wildenberg. Extending and Contributing to an Open Source Web-based System for the Assessment of Programming Problems. In Proceedings of the 5th International Symposium on Principles and Practice of Programming in Java, PPPJ ’07, pages 3–12, New York, NY, USA, 2007. ACM.

[106] O. Gotel, C. Scharff, and A. Wildenberg. Teaching Software Quality Assurance by Encour- aging Student Contributions to an Open Source Web-based System for the Assessment of Programming Assignments. SIGCSE Bull., 40(3):214–218, June 2008.

[107] D. Gustafson and M. Dwyer. Work in progress : the adversarial testing system. In 30th Annual Frontiers in Education Conference. Building on A Century of Progress in Engineering Education. Conference Proceedings (IEEE Cat. No.00CH37135), volume 1, pages T1C/19– T1C/20 vol.1, 2000.

[108] J. A. Harris, E. S. Adams, and N. L. Harris. Making program grading easier: But not totally automatic. J. Comput. Sci. Coll., 20(1):248–261, Oct. 2004.

[109] J. Heliotis and R. Zanibbi. Moving Away from Programming and Towards Computer Science in the CS First Year. J. Comput. Sci. Coll., 26(3):115–125, Jan. 2011.

[110] M. T. Helmick. Interface-based Programming Assignments and Automatic Grading of Java Programs. In Proceedings of the 12th Annual SIGCSE Conference on Innovation and Tech- nology in Computer Science Education, ITiCSE ’07, pages 63–67, New York, NY, USA, 2007. ACM.

[111] I. Hernan-Losada,´ C. Pareja-Flores, and J. A. Velazquez-Iturbide.´ Testing-Based Automatic Grading: A Proposal from Bloom’s Taxonomy. In 2008 Eighth IEEE International Confer- ence on Advanced Learning Technologies, pages 847–849, July 2008.

[112] P. Herout and P. Brada. Uml-test application for automated validation of students’ uml class diagram. In 2016 IEEE 29th International Conference on Software Engineering Education and Training (CSEET), pages 222–226, April 2016.

[113] C. Higgins, P. Symeonidis, and A. Tsintsifas. The marking system for coursemaster. In Pro- ceedings of the 7th Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE ’02, pages 46–50, New York, NY, USA, 2002. ACM.

19 [114] C. A. Higgins, G. Gray, P. Symeonidis, and A. Tsintsifas. Automated assessment and expe- riences of teaching programming. J. Educ. Resour. Comput., 5(3), Sept. 2005.

[115] M. Hilton and D. S. Janzen. On Teaching Arrays with Test-driven Learning in WebIDE. In Proceedings of the 17th ACM Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE ’12, pages 93–98, New York, NY, USA, 2012. ACM.

[116] R. Horvath.´ Software testing in introductory programming courses. In 2012 IEEE 10th International Conference on Emerging eLearning Technologies and Applications (ICETA), pages 133–134, Nov. 2012.

[117] M. J. Hull, D. Powell, and E. Klein. Infandango: Automated Grading for Student Program- ming. In Proceedings of the 16th Annual Joint Conference on Innovation and Technology in Computer Science Education, ITiCSE ’11, pages 330–330, New York, NY, USA, 2011. ACM.

[118] J. Hundley. Imprinting Community College Computer Science Education with Software Engineering Principles: Work in Progress. In Proceedings of the 48th Annual Southeast Regional Conference, ACM SE ’10, pages 54:1–54:4, New York, NY, USA, 2010. ACM.

[119] P. Ihantola. Creating and visualizing test data from programming exercises. Informatics in Education, 2007.

[120] P. Ihantola, T. Ahoniemi, V. Karavirta, and O. Seppal¨ a.¨ Review of recent systems for au- tomatic assessment of programming assignments. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research, Koli Calling ’10, pages 86–93, New York, NY, USA, 2010. ACM.

[121] N. Ishihara and N. Funabiki. A Proposal of Statement Fill-in-Blank Problem in Java Pro- gramming Learning Assistant System. In 2015 IIAI 4th International Congress on Advanced Applied Informatics, pages 247–252, July 2015.

[122] V. Isomott¨ onen¨ and V. Lappalainen. Csi with games and an emphasis on tdd and unit testing: Piling a trend upon a trend. ACM Inroads, 3(3):62–68, Sept. 2012.

[123] J. Isong. Developing an automated program checkers. In Proceedings of the Twelfth Annual CCSC South Central Conference on The Journal of Computing in Small Colleges, pages 218–224, USA, 2001. Consortium for Computing Sciences in Colleges.

[124] C. Izu, C. Pope, and A. Weerasinghe. On the ability to reason about program behaviour: A think-aloud study. In Proceedings of the 2017 ACM Conference on Innovation and Tech- nology in Computer Science Education, ITiCSE ’17, pages 305–310, New York, NY, USA, 2017. ACM.

[125] D. Jackson. A semi-automated approach to online assessment. In Proceedings of the 5th Annual SIGCSE/SIGCUE ITiCSEconference on Innovation and Technology in Computer Sci- ence Education, ITiCSE ’00, pages 164–167, New York, NY, USA, 2000. ACM.

[126] D. Janzen and H. Saiedian. Test-driven Learning in Early Programming Courses. In Proceed- ings of the 39th SIGCSE Technical Symposium on Computer Science Education, SIGCSE ’08, pages 532–536, New York, NY, USA, 2008. ACM.

20 [127] D. S. Janzen, J. Clements, and M. Hilton. An Evaluation of Interactive Test-driven Labs with WebIDE in CS0. In Proceedings of the 2013 International Conference on Software Engineering, ICSE ’13, pages 1090–1098, Piscataway, NJ, USA, 2013. IEEE Press.

[128] D. S. Janzen and H. Saiedian. On the Influence of Test-Driven Development on Software Design. In 19th Conference on Software Engineering Education Training (CSEET’06), pages 141–148, Apr. 2006.

[129] D. S. Janzen and H. Saiedian. Test-driven Learning: Intrinsic Integration of Testing into the CS/SE Curriculum. In Proceedings of the 37th SIGCSE Technical Symposium on Computer Science Education, SIGCSE ’06, pages 254–258, New York, NY, USA, 2006. ACM.

[130] D. S. Janzen and H. Saiedian. A Leveled Examination of Test-Driven Development Accep- tance. In Proceedings of the 29th International Conference on Software Engineering, ICSE ’07, pages 719–722, Washington, DC, USA, 2007. IEEE Computer Society.

[131] P. Je?ek, M. Malohlava, and T. Pop. Automated evaluation of regular lab assignments: A bittersweet experience? In 2013 26th International Conference on Software Engineering Education and Training (CSEE T), pages 249–258, May 2013.

[132] C. Johnson. SpecCheck: Automated Generation of Tests for Interface Conformance. In Proceedings of the 17th ACM Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE ’12, pages 186–191, New York, NY, USA, 2012. ACM.

[133] D. E. Johnson. Itch: Individual testing of computer homework for scratch assignments. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education, SIGCSE ’16, pages 223–227, New York, NY, USA, 2016. ACM.

[134] C. G. Jones. Test-driven Development Goes to School. J. Comput. Sci. Coll., 20(1):220–231, Oct. 2004.

[135] E. Jones. An experiential approach to incorporating software testing into the computer sci- ence curriculum. In 31st Annual Frontiers in Education Conference, volume 2, pages F3D– 7–F3D–11 vol.2, 2001.

[136] E. L. Jones. Grading Student Programs - a Software Testing Approach. In Proceedings of the Fourteenth Annual Consortium on Small Colleges Southeastern Conference, CCSC ’00, pages 185–192, USA, 2000. Consortium for Computing Sciences in Colleges.

[137] E. L. Jones. Software testing in the computer science curriculum – a holistic approach. In Proceedings of the Australasian Conference on Computing Education, ACSE ’00, pages 153–157, New York, NY, USA, 2000. ACM.

[138] E. L. Jones. The sprae framework for teaching software testing in the undergraduate curricu- lum. In Proceedings ADMI 2000, 2000.

[139] E. L. Jones. Integrating Testing into the Curriculum &Mdash; Arsenic in Small Doses. In Proceedings of the Thirty-second SIGCSE Technical Symposium on Computer Science Edu- cation, SIGCSE ’01, pages 337–341, New York, NY, USA, 2001. ACM.

[140] E. L. Jones and C. S. Allen. Repositories for CS Courses: An Evolutionary Tale. In Pro- ceedings of the 8th Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE ’03, pages 119–123, New York, NY, USA, 2003. ACM.

21 [141] G. Joshi and P. Desai. Building Software Testing Skills in Undergraduate Students Using Spiral Model Approach. In 2016 IEEE Eighth International Conference on Technology for Education (T4E), pages 244–245, Dec. 2016.

[142] M. Joy, N. Griffiths, and R. Boyatt. The boss online submission and assessment system. J. Educ. Resour. Comput., 5(3), Sept. 2005.

[143] D. W. Juedes. Web-based grading: further experiences and student attitudes. In Proceedings Frontiers in Education 35th Annual Conference, pages F4E–18, Oct. 2005.

[144] V. Karavirta and P. Ihantola. Serverless Automatic Assessment of Javascript Exercises. In Proceedings of the Fifteenth Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE ’10, pages 303–303, New York, NY, USA, 2010. ACM.

[145] M. Kart. Test First Programming, Design by Contract, and Intriguing Coursework: Ingredi- ents for Increasing Student Engagement. J. Comput. Sci. Coll., 28(4):35–41, Apr. 2013.

[146] R. Kaushal and A. Singh. Automated evaluation of programming assignments. In 2012 IEEE International Conference on Engineering Education: Innovative Practices and Future Trends (AICERA), pages 1–5, July 2012.

[147] K. Keefe, J. Sheard, and M. Dick. Adopting xp practices for teaching object oriented pro- gramming. In Proceedings of the 8th Australasian Conference on Computing Education - Volume 52, ACE ’06, pages 91–100, Darlinghurst, Australia, Australia, 2006. Australian Computer Society, Inc.

[148]M.K olling,¨ B. Quig, A. Patterson, and J. Rosenberg. The bluej system and its pedagogy. Computer Science Education, 13(4):249–268, 2003.

[149] Y. B.-D. Kolikant. Students’ Alternative Standards for Correctness. In Proceedings of the First International Workshop on Computing Education Research, ICER ’05, pages 37–43, New York, NY, USA, 2005. ACM.

[150] Y. B.-D. Kolikant and M. Mussai. ?so my program doesn?t run!? definition, origins, and practical expressions of students? (mis)conceptions of correctness. Computer Science Edu- cation, 18(2):135–151, 2008.

[151] S. Krusche and A. Seitz. Artemis: An automatic assessment management system for inter- active learning. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education, SIGCSE ’18, pages 284–289, New York, NY, USA, 2018. ACM.

[152] C. L. Kussmaul. Scaffolding for Multiple Assignment Projects in CS1 and CS2. In Com- panion to the 23rd ACM SIGPLAN Conference on Object-oriented Programming Systems Languages and Applications, OOPSLA Companion ’08, pages 873–876, New York, NY, USA, 2008. ACM.

[153] A. Kyrilov and D. C. Noelle. Do Students Need Detailed Feedback on Programming Exer- cises and Can Automated Assessment Systems Provide It? J. Comput. Sci. Coll., 31(4):115– 121, Apr. 2016.

[154] A.-J. Lakanen, V. Lappalainen, and V. Isomott¨ onen.¨ Revisiting rainfall to explore exam questions and performance on cs1. In Proceedings of the 15th Koli Calling Conference on

22 Computing Education Research, Koli Calling ’15, pages 40–49, New York, NY, USA, 2015. ACM.

[155] V. Lappalainen, J. Itkonen, V. Isomottonen, and S. Kollanus. ComTest: A Tool to Impart TDD and Unit Testing to Introductory Level Programming. In Proceedings of the Fifteenth Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE ’10, pages 63–67, New York, NY, USA, 2010. ACM.

[156] J. P. Leal and F. Silva. Mooshak: a web-based multi-site programming contest system. Soft- ware: Practice and Experience, 33(6):567–581, 5 2003.

[157] J. P. Leal and F. Silva. A new learning paradigm: Competition supported by technology, chapter Using Mooshak as a competitive learning tool, pages 91–106. Sello Editorial, 2010.

[158] Y. Lee, D. B. Marepalli, and J. Yang. Teaching test-drive development using dojo. J. Comput. Sci. Coll., 32(4):106–112, Apr. 2017.

[159] O. A. L. Lemos, F. C. Ferrari, F. F. Silveira, and A. Garcia. Experience report: Can software testing education lead to more reliable code? In 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE), pages 359–369, Nov 2015.

[160] O. A. L. Lemos, F. F. Silveira, F. C. Ferrari, and A. Garcia. The impact of Software Testing education on code reliability: An empirical assessment. Journal of Systems and Software, pages –, 2017.

[161] C. Leska. Testing across the curriculum: Square one! J. Comput. Sci. Coll., 19(5):163–169, May 2004.

[162] C. Leska and J. Rabung. Refactoring the CS1 Course. J. Comput. Sci. Coll., 20(3):6–18, Feb. 2005.

[163] J. J. Li and P. Morreale. Enhancing CS1 Curriculum with Testing Concepts: A Case Study. J. Comput. Sci. Coll., 31(3):36–43, Jan. 2016.

[164] T. v. d. Lippe, T. Smith, D. Pelsmaeker, and E. Visser. A scalable infrastructure for teaching concepts of programming languages in scala with weblab: An experience report. In Proceed- ings of the 2016 7th ACM SIGPLAN Symposium on Scala, SCALA 2016, pages 65–74, New York, NY, USA, 2016. ACM.

[165] L. Llana, E. Martin-Martin, and C. Pareja-Flores. FLOP, a Free Laboratory of Programming. In Proceedings of the 12th Koli Calling International Conference on Computing Education Research, Koli Calling ’12, pages 93–99, New York, NY, USA, 2012. ACM.

[166] A. Luxton-Reilly, B. A. Becker, Y. Cao, R. McDermott, C. Mirolo, A. Muhling,¨ A. Petersen, K. Sanders, Simon, and J. Whalley. Developing assessments to determine mastery of pro- gramming fundamentals. In Proceedings of the 2017 ITiCSE Conference on Working Group Reports, ITiCSE-WGR ’17, pages 47–69, New York, NY, USA, 2017. ACM.

[167] A. Luxton-Reilly, P. Denny, D. Kirk, E. Tempero, and S.-Y. Yu. On the differences between correct student solutions. In Proceedings of the 18th ACM Conference on Innovation and Technology in Computer Science Education, ITiCSE ’13, pages 177–182, New York, NY, USA, 2013. ACM.

23 [168] M. Madeja and J. Poruban. Automatic assessment of assignments for android application programming courses. In 2017 IEEE 14th International Scientific Conference on Informatics, pages 232–237, Nov 2017.

[169] J. Marcos-Abed. Learning Computer Programming: A Study of the Effectiveness of a COAC#. In Proceedings of the 2014 Conference on Innovation & Technology in Computer Science Education, ITiCSE ’14, pages 333–333, New York, NY, USA, 2014. ACM.

[170] J. Marcos-Abed. Using a COAC# for CS1. In Proceedings of the Western Canadian Confer- ence on Computing Education, WCCCE ’14, pages 10:1–10:3, New York, NY, USA, 2014. ACM.

[171] W. Marrero and A. Settle. Testing First: Emphasizing Testing in Early Programming Courses. In Proceedings of the 10th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education, ITiCSE ’05, pages 4–8, New York, NY, USA, 2005. ACM.

[172] C. Matthies, A. Treffer, and M. Uflacker. Prof. ci: Employing continuous integration services and github workflows to teach test-driven development. In 2017 IEEE Frontiers in Education Conference (FIE), pages 1–8, Oct 2017.

[173] A. Mendonc¸a, D. Guerrero, and E. Costa. An approach for problem specification and its ap- plication in an Introductory Programming Course. In 2009 39th IEEE Frontiers in Education Conference, pages 1–6, Oct. 2009.

[174] A. Mendonc¸a, C. d. Oliveira, D. Guerrero, and E. Costa. Difficulties in solving ill-defined problems: A case study with introductory computer programming students. In 2009 39th IEEE Frontiers in Education Conference, pages 1–6, Oct. 2009.

[175] D. Middleton. Developing Students’ Testing Skills: Distinguishing Functions. J. Comput. Sci. Coll., 28(5):73–74, May 2013.

[176] D. Middleton. Developing Students’ Testing Skills: Covering Space: Nifty Assignment. J. Comput. Sci. Coll., 30(5):29–31, May 2015.

[177] K. W. Miller. Test Driven Development on the Cheap: Text Files and Explicit Scaffolding. J. Comput. Sci. Coll., 20(2):181–189, Dec. 2004.

[178] M. Missiroli, D. Russo, and P. Ciancarini. Teaching test-first programming: Assessment and solutions. In 2017 IEEE 41st Annual Computer Software and Applications Conference (COMPSAC), volume 1, pages 420–425, July 2017.

[179] M. Morisio, M. Torchiano, and G. Argentieri. Assessing quantitatively a programming course. In 10th International Symposium on Software Metrics, 2004. Proceedings., pages 326–336, Sept. 2004.

[180] D. S. Morris. Automatic grading of student’s programming assignments: an interactive pro- cess and suite of programs. In 33rd Annual Frontiers in Education, 2003. FIE 2003., vol- ume 3, pages S3F–1–6 vol.3, Nov 2003.

[181] L. Murphy, G. Lewandowski, R. McCauley, B. Simon, L. Thomas, and C. Zander. De- bugging: The good, the bad, and the quirky – a qualitative analysis of novices’ strategies. In Proceedings of the 39th SIGCSE Technical Symposium on Computer Science Education, SIGCSE ’08, pages 163–167, New York, NY, USA, 2008. ACM.

24 [182] M. C. Murphy and B. Yildirim. Work in progress - testing right from the start. In 2007 37th Annual Frontiers In Education Conference - Global Engineering: Knowledge Without Borders, Opportunities Without Passports, pages F1H–25–F1H–26, Oct. 2007.

[183] V. L. Neto, R. Coelho, L. Leite, D. S. Guerrero, and A. P. Mendonc¸a. POPT: A Problem- oriented Programming and Testing Approach for Novice Students. In Proceedings of the 2013 International Conference on Software Engineering, ICSE ’13, pages 1099–1108, Pis- cataway, NJ, USA, 2013. IEEE Press.

[184] J. Nino.˜ Introducing API Design Principles in CS2. J. Comput. Sci. Coll., 24(4):109–116, Apr. 2009.

[185] T. Nishimura, S. Kawasaki, and H. Tominaga. Monitoring system of student situation in introductory C programming exercise with a contest style. In 2011 International Conference on Information Technology Based Higher Education and Training, pages 1–6, Aug. 2011.

[186] P. Nordquist. Providing Accurate and Timely Feedback by Automatically Grading Student Programming Labs. J. Comput. Sci. Coll., 23(2):16–23, Dec. 2007.

[187] C. O’Brien, M. Goldman, and R. C. Miller. Java Tutor: Bootstrapping with Python to Learn Java. In Proceedings of the First ACM Conference on Learning @ Scale Conference, L@S ’14, pages 185–186, New York, NY, USA, 2014. ACM.

[188] E. Odekirk-Hash and J. L. Zachary. Automated feedback on programs means students need less help from teachers. In Proceedings of the Thirty-second SIGCSE Technical Symposium on Computer Science Education, SIGCSE ’01, pages 55–59, New York, NY, USA, 2001. ACM.

[189] M. Olan. Unit Testing: Test Early, Test Often. J. Comput. Sci. Coll., 19(2):319–328, Dec. 2003.

[190] R. A. P. Oliveira, L. B. R. Oliveira, B. B. P. Cafeo, and V. H. S. Durelli. Evaluation and assessment of effects on exploring mutation testing in programming courses. In 2015 IEEE Frontiers in Education Conference (FIE), pages 1–9, Oct. 2015.

[191] B. H. Pachulski Camara and M. A. Graciotto Silva. A Strategy to Combine Test-Driven Development and Test Criteria to Improve Learning of Programming Skills. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education, SIGCSE ’16, pages 443–448, New York, NY, USA, 2016. ACM.

[192] S. Pape, J. Flake, A. Beckmann, and J. Jurjens. STAGE: A Software Tool for Automatic Grading of Testing Exercises: Case Study Paper. In Proceedings of the 38th International Conference on Software Engineering Companion, ICSE ’16, pages 491–500, New York, NY, USA, 2016. ACM.

[193] E. Parodi, S. Matalonga, D. Macchi, and M. Solari. Comparing technical debt in student exercises using test driven development, test last and ad hoc programming. In 2016 XLII Latin American Computing Conference (CLEI), pages 1–10, Oct 2016.

[194] A. Parrish, D. Cordes, B. Dixon, and J. McGregor. Class Development and Testing in the Small. In Proceedings of the 38th Annual on Southeast Regional Conference, ACM-SE 38, pages 139–145, New York, NY, USA, 2000. ACM.

25 [195] A. Patterson, M. Kolling,¨ and J. Rosenberg. Introducing unit testing with bluej. In Pro- ceedings of the 8th Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE ’03, pages 11–15, New York, NY, USA, 2003. ACM.

[196] J. Paul. Test-driven Approach in Programming Pedagogy. J. Comput. Sci. Coll., 32(2):53–60, Dec. 2016.

[197] J. L. Pearce, M. Nakazawa, and S. Heggen. Improving Problem Decomposition Ability in CS1 Through Explicit Guided Inquiry-based Instruction. J. Comput. Sci. Coll., 31(2):135– 144, Dec. 2015.

[198] J. Petit, O. Gimenez,´ and S. Roura. Jutge.Org: An Educational Programming Judge. In Pro- ceedings of the 43rd ACM Technical Symposium on Computer Science Education, SIGCSE ’12, pages 445–450, New York, NY, USA, 2012. ACM.

[199] R. Pham, S. Kiesling, O. Liskin, L. Singer, and K. Schneider. Enablers, inhibitors, and perceptions of testing in novice software teams. In Proceedings of the ACM SIGSOFT Sym- posium on the Foundations of Software Engineering, volume 16-21-November-2014, pages 30–40. Association for Computing Machinery, 2014.

[200] V. Pieterse. Automated assessment of programming assignments. In Proceedings of the 3rd Computer Science Education Research Conference on Computer Science Education Re- search, CSERC ’13, pages 4:45–4:56, Open Univ., Heerlen, The Netherlands, The Nether- lands, 2013. Open Universiteit, Heerlen.

[201] V. Pieterse and J. Liebenberg. Automatic vs manual assessment of programming tasks. In Proceedings of the 17th Koli Calling International Conference on Computing Education Re- search, Koli Calling ’17, pages 193–194, New York, NY, USA, 2017. ACM.

[202] E. Pietrikova, J. Juhar, and J. Stastna. Towards automated assessment in game-creative pro- gramming courses. In 2015 13th International Conference on Emerging eLearning Technolo- gies and Applications (ICETA), pages 1–6, Nov. 2015.

[203] J. G. Politz, J. M. Collard, A. Guha, K. Fisler, and S. Krishnamurthi. The Sweep: Essential Examples for In-Flow Peer Review. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education, SIGCSE ’16, pages 243–248, New York, NY, USA, 2016. ACM.

[204] J. G. Politz, S. Krishnamurthi, and K. Fisler. In-flow peer-review of tests in test-first program- ming. In Proceedings of the Tenth Annual Conference on International Computing Education Research, ICER ’14, pages 11–18, New York, NY, USA, 2014. ACM.

[205] J. G. Politz, D. Patterson, S. Krishnamurthi, and K. Fisler. Captainteach: Multi-stage, in- flow peer review for programming assignments. In Proceedings of the 2014 Conference on Innovation & Technology in Computer Science Education, ITiCSE ’14, pages 267–272, New York, NY, USA, 2014. ACM.

[206] M. Pozenel, L. Furst, and V. Mahnic. Introduction of the automated assessment of home- work assignments in a university-level programming course. In 2015 38th International Convention on Information and Communication Technology, Electronics and Microelectron- ics (MIPRO), pages 761–766, May 2015.

26 [207] I. Pribela, D. Pracner, Z. Budimac, and T. Grbac. Tool for testing bad student programs. In G. T. Budimac Z., editor, CEUR Workshop Proceedings, volume 1266, pages 67–74. CEUR- WS, 2014.

[208] V. Proulx. Introductory Computing: The Design Discipline. In I. Kala? and R. Mittermeir, editors, Informatics in Schools. Contributing to 21st Century Education, volume 7013 of Lecture Notes in Computer Science, pages 177–188. Springer Berlin Heidelberg, 2011. DOI: 10.1007/978-3-642-24722-4 16.

[209] V. K. Proulx. Test-driven Design for Introductory OO Programming. In Proceedings of the 40th ACM Technical Symposium on Computer Science Education, SIGCSE ’09, pages 138–142, New York, NY, USA, 2009. ACM.

[210] V. K. Proulx and W. Jossey. Unit Test Support for Java via Reflection and Annotations. In Proceedings of the 7th International Conference on Principles and Practice of Programming in Java, PPPJ ’09, pages 49–56, New York, NY, USA, 2009. ACM.

[211] V. K. Proulx and W. Jossey. Unit Testing in Java. In Proceedings of the 14th Annual ACM SIGCSE Conference on Innovation and Technology in Computer Science Education, ITiCSE ’09, pages 349–349, New York, NY, USA, 2009. ACM.

[212] V. K. Proulx and R. Rasala. Java io and testing made simple. SIGCSE Bull., 36(1):161–165, Mar. 2004.

[213] S. M. Rahman. Applying the TBC method in introductory programming courses. In 2007 37th Annual Frontiers In Education Conference - Global Engineering: Knowledge Without Borders, Opportunities Without Passports, pages T1E–20–T1E–21, Oct. 2007.

[214] S. M. Rahman and P. L. Juell. Applying Software Development Lifecycles in Teaching Introductory Programming Courses. In 19th Conference on Software Engineering Education Training (CSEET’06), pages 17–24, Apr. 2006.

[215] D. Rajaguru, A. Rajeswari, V. Bhuvaneshwari, and K. Vagheesan. E-assessment of pro- gramming assignments in web service. In IEEE-International Conference On Advances In Engineering, Science And Management (ICAESM -2012), pages 484–489, Mar. 2012.

[216] T. Rajala, E. Kaila, R. Linden,´ E. Kurvinen, E. Lokkila, M.-J. Laakso, and T. Salakoski. Automatically Assessed Electronic Exams in Programming Courses. In Proceedings of the Australasian Computer Science Week Multiconference, ACSW ’16, pages 11:1–11:8, New York, NY, USA, 2016. ACM.

[217] L. Reynolds, Q. Mayo, D. Adamo, and R. Bryce. Improving Conceptual Understanding of Code with Bug Fixer. J. Comput. Sci. Coll., 31(2):87–94, Dec. 2015.

[218] M. Ricken and R. Cartwright. Test-first Java Concurrency for the Classroom. In Proceedings of the 41st ACM Technical Symposium on Computer Science Education, SIGCSE ’10, pages 219–223, New York, NY, USA, 2010. ACM.

[219] B. A. Ring, J. Giordan, and J. S. Ransbottom. Problem Solving Through Programming: Motivating the Non-programmer. J. Comput. Sci. Coll., 23(3):61–67, Jan. 2008.

27 [220] G. H. B. Roberts and J. L. M. Verbyla. An Online Programming Assessment Tool. In Proceedings of the Fifth Australasian Conference on Computing Education - Volume 20, ACE ’03, pages 69–75, Darlinghurst, Australia, Australia, 2003. Australian Computer Society, Inc.

[221] P. L. R. Rodrigues, L. P. Franz, J. F. P. Cheiran, J. a. P. S. da Silva, and A. S. Bordin. Coding dojo as a transforming practice in collaborative learning of programming: An experience report. In Proceedings of the 31st Brazilian Symposium on Software Engineering, SBES’17, pages 348–357, New York, NY, USA, 2017. ACM.

[222] R. Romli, S. Sulaiman, and K. Z. Zamli. Automatic programming assessment and test data generation a review on its approaches. In 2010 International Symposium on Information Technology, volume 3, pages 1186–1192, June 2010.

[223] R. Romli, S. Sulaiman, and K. Zuhairi Zamli. Current practices of programming assessment at higher learning institutions. Communications in Computer and Information Science, 179 CCIS(PART 1):471–485, 2011.

[224] J. A. Rosiene and C. Pe Rosiene. Testing in the ’Small’. J. Comput. Sci. Coll., 19(2):314–318, Dec. 2003.

[225] G. Rossling and S. Hartte. WebTasks: Online Programming Exercises Made Easy. In Pro- ceedings of the 13th Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE ’08, pages 363–363, New York, NY, USA, 2008. ACM.

[226] M. J. Rubin. The Effectiveness of Live-coding to Teach Introductory Programming. In Proceeding of the 44th ACM Technical Symposium on Computer Science Education, SIGCSE ’13, pages 651–656, New York, NY, USA, 2013. ACM.

[227] M. Rubio-Sanchez, P. Kinnunen, C. Pareja-Flores, and A. Velazquez-Iturbide. Student per- ception and usage of an automated programming assessment tool. Computers in Human Behavior, 31:453 – 460, 2014.

[228] R. Saikkonen, L. Malmi, and A. Korhonen. Fully automatic assessment of programming ex- ercises. In Proceedings of the 6th Annual Conference on Innovation and Technology in Com- puter Science Education, ITiCSE ’01, pages 133–136, New York, NY, USA, 2001. ACM.

[229] K. Sanders, M. Ahmadzadeh, T. Clear, S. H. Edwards, M. Goldweber, C. Johnson, R. Lister, R. McCartney, E. Patitsas, and J. Spacco. The canterbury questionbank: Building a repository of multiple-choice cs1 and cs2 questions. In Proceedings of the ITiCSE Working Group Reports Conference on Innovation and Technology in Computer Science Education-working Group Reports, ITiCSE -WGR ’13, pages 33–52, New York, NY, USA, 2013. ACM.

[230] J. A. Sant. ”mailing it in”: Email-centric automated assessment. In Proceedings of the 14th Annual ACM SIGCSE Conference on Innovation and Technology in Computer Science Education, ITiCSE ’09, pages 308–312, New York, NY, USA, 2009. ACM.

[231] J. P. Sauve´ and O. L. Abath Neto. Teaching Software Development with ATDD and Easy- accept. In Proceedings of the 39th SIGCSE Technical Symposium on Computer Science Education, SIGCSE ’08, pages 542–546, New York, NY, USA, 2008. ACM.

[232] J. P. Sauve,´ O. L. Abath Neto, and W. Cirne. Easyaccept: A tool to easily create, run and drive development with automated acceptance tests. In Proceedings of the 2006 International

28 Workshop on Automation of Software Test, AST ’06, pages 111–117, New York, NY, USA, 2006. ACM.

[233] L. P. Scatalon, E. F. Barbosa, and R. E. Garcia. Challenges to integrate software testing into introductory programming courses. In 2017 IEEE Frontiers in Education Conference (FIE), pages 1–9, Oct 2017.

[234] L. P. Scatalon, J. M. Prates, D. M. de Souza, E. F. Barbosa, and R. E. Garcia. Towards the role of test design in programming assignments. In 2017 IEEE 30th Conference on Software Engineering Education and Training (CSEE&T), pages 170–179, Nov 2017.

[235] S. Schaub. Teaching CS1 with Web Applications and Test-driven Development. SIGCSE Bull., 41(2):113–117, June 2009.

[236] S. C. Shaffer. Ludwig: An online programming tutoring and assessment system. SIGCSE Bull., 37(2):56–60, June 2005.

[237] Z. Shams. Automated Assessment of Students’ Testing Skills for Improving Correctness of Their Code. In Proceedings of the 2013 Companion Publication for Conference on Systems, Programming, & Applications: Software for Humanity, SPLASH ’13, pages 37–40, New York, NY, USA, 2013. ACM.

[238] Z. Shams. Automatically Assessing the Quality of Student-written Tests. In Proceedings of the Ninth Annual International ACM Conference on International Computing Education Research, ICER ’13, pages 189–190, New York, NY, USA, 2013. ACM.

[239] Z. Shams and S. H. Edwards. Toward Practical Mutation Analysis for Evaluating the Quality of Student-written Software Tests. In Proceedings of the Ninth Annual International ACM Conference on International Computing Education Research, ICER ’13, pages 53–58, New York, NY, USA, 2013. ACM.

[240] Z. Shams and S. H. Edwards. Checked Coverage and Object Branch Coverage: New Al- ternatives for Assessing Student-Written Tests. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education, SIGCSE ’15, pages 534–539, New York, NY, USA, 2015. ACM.

[241] S. Sheth, J. Bell, and G. Kaiser. Halo (highly addictive, socially optimized) software en- gineering. In 1st International Workshop on Games and Software Engineering (GAS’11), 2011.

[242] A. Sioson. Experiences on the use of an automatic C++ solution grader system. In IISA 2013 - 4th International Conference on Information, Intelligence, Systems and Applications, pages 18–21, 2013.

[243] J. Smith, J. Tessler, E. Kramer, and C. Lin. Using Peer Review to Teach Software Testing. In Proceedings of the Ninth Annual International Conference on International Computing Education Research, ICER ’12, pages 93–98, New York, NY, USA, 2012. ACM.

[244] R. Smith, T. Tang, J. Warren, and S. Rixner. An automated system for interactively learning software testing. In Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education, ITiCSE ’17, pages 98–103, New York, NY, USA, 2017. ACM.

29 [245] S. Smith and S. Stoecklin. What We Can Learn from Extreme Programming. J. Comput. Sci. Coll., 17(2):144–151, Dec. 2001.

[246] J. Snyder, S. H. Edwards, and M. A. Perez-Qui´ nones.˜ LIFT: Taking GUI Unit Testing to New Heights. In Proceedings of the 42Nd ACM Technical Symposium on Computer Science Education, SIGCSE ’11, pages 643–648, New York, NY, USA, 2011. ACM.

[247] R. M. Snyder. Teacher Specification and Student Implementation of a Unit Testing Method- ology in an Introductory Programming Course. J. Comput. Sci. Coll., 19(3):22–32, Jan. 2004.

[248] D. M. d. Souza, J. C. Maldonado, and E. F. Barbosa. ProgTest: An environment for the sub- mission and evaluation of programming assignments based on testing activities. In 2011 24th IEEE-CS Conference on Software Engineering Education and Training (CSEE&T), pages 1– 10, May 2011.

[249] D. M. d. Souza, B. H. Oliveira, J. C. Maldonado, S. R. S. Souza, and E. F. Barbosa. Towards the use of an automatic assessment system in the teaching of software testing. In 2014 IEEE Frontiers in Education Conference (FIE) Proceedings, pages 1–8, Oct. 2014.

[250] J. Spacco, D. Fossati, J. Stamper, and K. Rivers. Towards Improving Programming Habits to Create Better Computer Science Course Outcomes. In Proceedings of the 18th ACM Con- ference on Innovation and Technology in Computer Science Education, ITiCSE ’13, pages 243–248, New York, NY, USA, 2013. ACM.

[251] J. Spacco, D. Hovemeyer, and W. Pugh. An Eclipse-based Course Project Snapshot and Submission System. In Proceedings of the 2004 OOPSLA Workshop on Eclipse Technology eXchange, eclipse ’04, pages 52–56, New York, NY, USA, 2004. ACM.

[252] J. Spacco, D. Hovemeyer, W. Pugh, F. Emad, J. K. Hollingsworth, and N. Padua-Perez. Experiences with Marmoset: Designing and Using an Advanced Submission and Testing System for Programming Courses. In Proceedings of the 11th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education, ITICSE ’06, pages 13–17, New York, NY, USA, 2006. ACM.

[253] J. Spacco and W. Pugh. Helping Students Appreciate Test-driven Development (TDD). In Companion to the 21st ACM SIGPLAN Symposium on Object-oriented Programming Sys- tems, Languages, and Applications, OOPSLA ’06, pages 907–913, New York, NY, USA, 2006. ACM.

[254] J. Spacco, W. Pugh, N. Ayewah, and D. Hovemeyer. The Marmoset Project: An Automated Snapshot, Submission, and Testing System. In Companion to the 21st ACM SIGPLAN Sym- posium on Object-oriented Programming Systems, Languages, and Applications, OOPSLA ’06, pages 669–670, New York, NY, USA, 2006. ACM.

[255] J. Spacco, J. Strecker, D. Hovemeyer, and W. Pugh. Software Repository Mining with Mar- moset: An Automated Programming Project Snapshot and Testing System. In Proceedings of the 2005 International Workshop on Mining Software Repositories, MSR ’05, pages 1–5, New York, NY, USA, 2005. ACM.

[256] S. Sridhara, B. Hou, J. Lu, and J. DeNero. Fuzz Testing Projects in Massive Courses. In Proceedings of the Third (2016) ACM Conference on Learning @ Scale, L@S ’16, pages 361–367, New York, NY, USA, 2016. ACM.

30 [257] D. H. Steinberg. The effect of unit tests on entry points, coupling and cohesion in an intro- ductory java programming course. In Proceedings of XP Universe Conference, 2001.

[258] M. Striewe and M. Goedicke. Using run time traces in automated programming tutoring. In Proceedings of the 16th Annual Joint Conference on Innovation and Technology in Computer Science Education, ITiCSE ’11, pages 303–307, New York, NY, USA, 2011. ACM.

[259] H. Suleman. Automatic Marking with Sakai. In Proceedings of the 2008 Annual Research Conference of the South African Institute of Computer Scientists and Information Technolo- gists on IT Research in Developing Countries: Riding the Wave of Technology, SAICSIT ’08, pages 229–236, New York, NY, USA, 2008. ACM.

[260] Y. Sun and E. L. Jones. Specification-driven Automated Testing of GUI-based Java Programs. In Proceedings of the 42Nd Annual Southeast Regional Conference, ACM-SE 42, pages 140– 145, New York, NY, USA, 2004. ACM.

[261] T. Tang, R. Smith, S. Rixner, and J. Warren. Data-Driven Test Case Generation for Automated Programming Assessment. In Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education, ITiCSE ’16, pages 260–265, New York, NY, USA, 2016. ACM.

[262] R. Teusner, T. Hille, and C. Hagedorn. Aspects on finding the optimal practical programming exercise for moocs. In 2017 IEEE Frontiers in Education Conference (FIE), pages 1–8, Oct 2017.

[263] E. Thompson, L. Hunt, and K. Kinshuk. Exploring Learner Conceptions of Programming. In Proceedings of the 8th Australasian Conference on Computing Education - Volume 52, ACE ’06, pages 205–211, Darlinghurst, Australia, Australia, 2006. Australian Computer Society, Inc.

[264] M. Thornton, S. H. Edwards, R. P. Tan, and M. A. Perez-Qui´ nones.˜ Supporting Student- written Tests of Gui Programs. In Proceedings of the 39th SIGCSE Technical Symposium on Computer Science Education, SIGCSE ’08, pages 537–541, New York, NY, USA, 2008. ACM.

[265] V. Thurner and A. Bottcher. An objects first, tests second approach for software engineering education. In 2015 IEEE Frontiers in Education Conference (FIE), pages 1–5, Oct 2015.

[266] W. Tiantian, S. Xiaohong, M. Peijun, W. Yuying, and W. Kuanquan. AutoLEP: An Auto- mated Learning and Examination System for Programming and its Application in Program- ming Course. In 2009 First International Workshop on Education Technology and Computer Science, volume 1, pages 43–46, Mar. 2009.

[267] N. Tillmann, J. de Halleux, T. Xie, and J. Bishop. Constructing Coding Duels in Pex4fun and Code Hunt. In Proceedings of the 2014 International Symposium on Software Testing and Analysis, ISSTA 2014, pages 445–448, New York, NY, USA, 2014. ACM.

[268] N. Tillmann, J. d. Halleux, T. Xie, and J. Bishop. Pex4fun: A web-based environment for educational gaming via automated test generation. In 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 730–733, Nov. 2013.

31 [269] H. Tr\a etteberg and T. Aalberg. JExercise: A Specification-based and Test-driven Exer- cise Support Plugin for Eclipse. In Proceedings of the 2006 OOPSLA Workshop on Eclipse Technology eXchange, eclipse ’06, pages 70–74, New York, NY, USA, 2006. ACM.

[270] G. Tremblay, F. Guerin,´ A. Pons, and A. Salah. Oto, a generic and extensible tool for marking programming assignments. Software: Practice and Experience, 38(3):307–333, 2008.

[271] G. Tremblay, L. Laforest, and A. Salah. Extending a Marking Tool with Simple Support for Testing. In Proceedings of the 12th Annual SIGCSE Conference on Innovation and Tech- nology in Computer Science Education, ITiCSE ’07, pages 313–313, New York, NY, USA, 2007. ACM.

[272] G. Tremblay and P. Lessard. A marking language for the oto assignment marking tool. In Proceedings of the 16th Annual Joint Conference on Innovation and Technology in Computer Science Education, ITiCSE ’11, pages 148–152, New York, NY, USA, 2011. ACM.

[273] S. A. Turner. Looking Glass: A C++ Library for Testing Student Programs Through Reflec- tion. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education, SIGCSE ’15, pages 528–533, New York, NY, USA, 2015. ACM.

[274] L. C. Ureel and C. Wallace. WebTA: Automated iterative critique of student programming assignments. In 2015 IEEE Frontiers in Education Conference (FIE), pages 1–9, Oct. 2015.

[275] I. Utting, A. E. Tew, M. McCracken, L. Thomas, D. Bouvier, R. Frye, J. Paterson, M. Caspersen, Y. B.-D. Kolikant, J. Sorva, and T. Wilusz. A Fresh Look at Novice Program- mers’ Performance and Their Teachers’ Expectations. In Proceedings of the ITiCSE Working Group Reports Conference on Innovation and Technology in Computer Science Education- working Group Reports, ITiCSE -WGR ’13, pages 15–32, New York, NY, USA, 2013. ACM.

[276] P. H. D. Valle, A. M. Toda, E. F. Barbosa, and J. C. Maldonado. Educational games: A contribution to software testing education. In 2017 IEEE Frontiers in Education Conference (FIE), pages 1–8, Oct 2017.

[277] T. VanDeGrift, T. Caruso, N. Hill, and B. Simon. Experience Report: Getting Novice Pro- grammers to THINK About Improving Their Software Development Process. In Proceedings of the 42Nd ACM Technical Symposium on Computer Science Education, SIGCSE ’11, pages 493–498, New York, NY, USA, 2011. ACM.

[278] A. Venables and L. Haywood. Programming students need instant feedback! In Proceedings of the Fifth Australasian Conference on Computing Education - Volume 20, ACE ’03, pages 267–272, Darlinghurst, Australia, Australia, 2003. Australian Computer Society, Inc.

[279] A. Vihavainen, T. Vikberg, M. Luukkainen, and M. Partel.¨ Scaffolding Students’ Learning Using Test My Code. In Proceedings of the 18th ACM Conference on Innovation and Tech- nology in Computer Science Education, ITiCSE ’13, pages 117–122, New York, NY, USA, 2013. ACM.

[280] M. Vujosevic-Janicic, M. Nikolic, D. Tosic, and V. Kuncak. Software verification and graph similarity for automated evaluation of students’ assignments. Information and Software Tech- nology, 55(6):1004 – 1016, 2013.

32 [281] T. Wang, X. Su, P. Ma, Y. Wang, and K. Wang. Ability-training-oriented automated assess- ment in introductory programming course. Computers & Education, 56(1):220 – 226, 2011. Serious Games. [282] C. A. Wellington, T. H. Briggs, and C. D. Girard. Experiences Using Automated 4ests and 4est Driven Development in Computer 9cience I. In Agile 2007 (AGILE 2007), pages 106– 112, Aug. 2007. [283] J. Whalley and N. Kasto. How difficult are novice code writing tasks?: A software metrics approach. In Proceedings of the Sixteenth Australasian Computing Education Conference - Volume 148, ACE ’14, pages 105–112, Darlinghurst, Australia, Australia, 2014. Australian Computer Society, Inc. [284] J. L. Whalley and A. Philpott. A Unit Testing Approach to Building Novice Programmers’ Skills and Confidence. In Proceedings of the Thirteenth Australasian Computing Education Conference - Volume 114, ACE ’11, pages 113–118, Darlinghurst, Australia, Australia, 2011. Australian Computer Society, Inc. [285] M. Wick, D. Stevenson, and P. Wagner. Using Testing and JUnit Across the Curriculum. In Proceedings of the 36th SIGCSE Technical Symposium on Computer Science Education, SIGCSE ’05, pages 236–240, New York, NY, USA, 2005. ACM. [286] C. Wilcox. Testing Strategies for the Automated Grading of Student Programs. In Proceed- ings of the 47th ACM Technical Symposium on Computing Science Education, SIGCSE ’16, pages 437–442, New York, NY, USA, 2016. ACM. [287] T. Xie, J. Bishop, R. N. Horspool, N. Tillmann, and J. d. Halleux. Crowdsourcing Code and Process via Code Hunt. In 2015 IEEE/ACM 2nd International Workshop on CrowdSourcing in Software Engineering, pages 15–16, May 2015. [288] J. Yi, U. Z. Ahmed, A. Karkare, S. H. Tan, and A. Roychoudhury. A feasibility study of using automated program repair for introductory programming assignments. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, pages 740–751, New York, NY, USA, 2017. ACM. [289] B. V. Zanden, D. Anderson, C. Taylor, W. Davis, and M. W. Berry. Codeassessor: An Interactive, Web-based Tool for Introductory Programming. J. Comput. Sci. Coll., 28(2):73– 80, Dec. 2012. [290] A. Zeller. Making Students Read and Review Code. In Proceedings of the 5th Annual SIGCSE/SIGCUE ITiCSE Conference on Innovation and Technology in Computer Science Education, ITiCSE ’00, pages 89–92, New York, NY, USA, 2000. ACM. [291] G. Zhu and Y. Chen. Knowledge-based links for automatic interaction with programming online judges. Journal of Software, 8(5):1209–1218, 2013. [292] D. M. Zimmerman, J. R. Kiniry, and F. Fairmichael. Toward instant gradeification. In 2011 24th IEEE-CS Conference on Software Engineering Education and Training (CSEE&T), pages 406–410, May 2011. [293] D. Zingaro, Y. Cherenkova, O. Karpova, and A. Petersen. Facilitating Code-writing in PI Classes. In Proceeding of the 44th ACM Technical Symposium on Computer Science Educa- tion, SIGCSE ’13, pages 585–590, New York, NY, USA, 2013. ACM.

33