US006761631B2 (12) United States Patent (10) Patent N0.: US 6,761,631 B2 Lydon et al. (45) Date of Patent: Jul. 13, 2004
(54) APPARATUS AND SYSTEM FOR 6,397,197 B1 5/2002 Gindlesperger FACILITATING ONLINE CODING 6,408,283 B1 6/2002 Alaia et 91 COMPETITIONS 6,513,042 B1 1/2003 Anderson et a1. 6,569,012 B2 * 5/2003 Lydon et a1...... 463/9 . - . 2001/0032189 A1 10/2001 Powell (75) Inventors‘ lg'l‘llollllésHl‘gfgggncglglsgs’) 2001/0034631 A1 10/2001 Kiselik ' g ’ ’ 2001/0037281 A1 11/2001 French et a1. (73) Assignee: Topcoder, Inc., Glastonbury, CT (US) 2001/0039529 A1 11/2001 Hoffman FOREIGN PATENT DOCUMENTS (ak) Notice: Subject/O any disclaimer’. the term Ofthis W0 WO 97/39811 10/1997 ...... 9/24 patent is extended or ad]usted under 35 U.s.c. 154(1)) by 0 days. OTHER PUBLICATIONS Fink, William & Joyce Dick. Computer Programming Tour (21) Appl. N0.: 10/377,344 nament as a learning Experience. The Computer Education _ Quarterly. Summer 1989, vol. 11, Issue 2 pp. 46—52.* (22) Flled: Feb‘ 28’ 2003 O’Sullivan, Kate. Testing recruits. Net Style. Inc. Dec. 2000, (65) Prior Publication Data PP~ 149_152'*_ _ _ Flipcode. Coding contest [online]. May 2000 [retrieved on Us 2003/0153386 A1 Aug- 14, 2003 Jun. 26, 2002]. Retrieved from the Internet:
1405 US 6,761,631 B2 Page 2
OTHER PUBLICATIONS Fink, William & Joyce Dick. Computer Programming Tour nament as a learning Experience. The Computer Education The Eiffel Struggle 2000, Call for participation [online]. Quarterly. Summer 1989, vol. 11, Issue 2 pp. 46—52.* Aug. 2000, [retrieved on Jun. 26, 2002]. Retrieved from O’Sullivan, Kate. Testing recruits. Net Style. Inc. Dec. 2000, Internet:
“Hack the Web for Fun!” [online], [Retrieved on Aug. 23, The Eiffel Struggle 2000, Call for participation [online]. 2000]. Retrieved from the Internet: http://java.sun.com/con Aug. 2000, retrieved on Jun. 26, 2002. Retrieved from the test/eXternal/html. Internet: URL: http://WWW.elj.com/eiffel/struggle2000/cfp/, “Brainbench the measure of achievement” [online], [Re entire document. trieved on Nov. 9, 2000], Retrieved from the Internet: ICFP Functional Programming Contest, [online] 1998, [re http://WWW.brainbench.com/Xml/bb/homepage.Xml. trieved on Jun. 26, 2002]. Retrieved from the internet: URL: “Problem Set Archive With Online Judge” [online], [Re http://WWW.ai.mit.edu/eXtra/icfp—contest. trieved on Nov. 8, 2001]. Retrieved from the Internet: 1999 ICFP Programming Contest, [online]1999, [retrieved http://acm.uva.es/problemset/. on Jun. 26, 2002]. Retrieved from the internet: URL: http:// “An OvervieW of PC2” [online], [Retrieved on Aug. 23, WWW.es.virginia.edu/~jks6b/ocfp/. 2000]. Retrieved from the Internet: http://hera.ecs.csus.edu/ US. patent application Ser. No. 10/378,539, Lydon et al., pc2/pc2desc.html, Revised Jun. 22, 2000. ?led Feb. 28, 2003. “IOI Software Team” [online], [Retrieved on Aug. 23, 2001], Retrieved from the Internet: http://olympiads.Win.tu US. patent application Ser. No. 10/377,343, Lydon et al., e.nl/ioi/st/. ?led Feb. 28, 2003. “Ed’s Programming Contest Problem Archive” [online], US. patent application Ser. No. 10/408,402, Hughes, Apr. 7, [Retrieved on Aug. 23, 2000]. Retrieved from the Inter 2003. net:http://WWW.karrels.org/Ed/ACM/indeX.html. “About CBDi” [online], [Retrieved on Jul. 29, 2003]. “ACM/IBM Quest for Java 2000” [online]. [Retrieved on Retrieved from the Internet: http://WWW.cbdiforum.com/ Aug. 23, 2000]. Retrieved from the Internet: http://WWW.ac public/about.php3. m.org/jquest. “ComponentSource: About Us” [online], [Retrieved on Jul. “JavaWorld Code Masters” [online], [Retrieved on Aug. 23, 29, 2003]. Retrieved from the Internet: http://WWW.compo 2000]. Retrieved from the Internet: http://WWW.javaWorld nentsource.com/Marketplace/Home.asp?Type=AboutUs. .com/javaWorld/codemasters/fiindex.html. “Flashline 4 Provides SoftWare Asset Management for “Hack the Web for Fun!” [online], [Retrieved on Aug. 23, Enterprise Programs” [online], [Retrieved on Jul. 30, 2003]. 2000]. Retrieved from the Internet: http://java.sun.com/con Retrieved from the Internet: http://WWW.?ashline.com/fcm/ test/eXternal/html. fcm.jsp. “Brainbench the measure of achievement” [online], [Re “About ComponentOne” [online], [Retrieved on Jul. 29, trieved on Nov. 9, 2000]. Retrieved from the Internet: 2003]. Retrieved from the Internet: http://WWW.componen http://WWW.brainbench.com/Xml/bb/homepage.Xml. tone.com/AboutComponentOne.aspX. “Problem Set Archive With Online Judge” [online], [Re trieved on Nov. 8, 2001], Retrieved from the Internet: “Rogue Wave Corporate Information” [online], [Retrieved http://acm.uva.es/problemset/. on Jul. 29, 2003]. Retrieved from the Internet: http://WW W.rogueWave.com/corp/. LeWis, “Tournaments become latest high—tech recruiting tool,” The Boston Globe at C1, May 6, 2002. “IJogicLibrary Company OvervieW”[online], [Retrieved on Annavajjala, “JAVA Challenge SoftWare Project” Master’s Jul. 29, 2003]. Retrieved from the Internet: http://WWW.logi Thesis submitted to the Department of Computer Science clibrary.com/companyovervieW.html. and Electrical Engineering at West Virginia University. “/n softWare inc.—General Corporate Information” [online], Astrachan, “The Internet Programming Contest: A Report [Retrieved on Jul 29, 2003]. Retrieved from the Internet: and Philosophy” Association for Computing Machinery, http://WWW.nsoftWare.com/company/corporateinfo.aspX. Feb. 1993. “ILOG Corporate Pro?le” [online], [Retrieved on Jul. 29, “URCSC Computer Programming Contest” [online], [Re 2003]. Retrieved from the Internet: http://WWW.ilog.com/ trieved on Jun. 26, 2002], Retrieved from the Internet: corporate/pro?le/. http://WWW.mathcs.richmond.edu/~urcsc/contest/ “What is SourceForge.net” [online], [Retrieved on Jul. 29, rules2.html. 2003]. Retrieved from the Internet: http://sourceforge.net/ Blecourt, “The Legacy of Arpad Elo: The Development of a docman/displayidoc.php?docid=6025 &groupiid= 1. Chess Rating System” University of Amsterdam, Dec. 1, “CollabNet Corporate Background” [online], [Retrieved on 1998. Jul. 30, 2003]. Retrieved from the Internet: http://WWW.col US. Chess Federation [online], [Retrieved on Dec. 3, 2002]. lab.net/media/pdfs/collabnetibackground.pdf. Retrieved from the Internet: http://WWW.uschess.org/about/ Of?ce Action for US Patent No. 6,569,012 dated Jun. 27, about/html. 2002. Carlton, Jim. Strutting their data, geeks go to battle for bucks and glory. Wall Street Journal, Mar. 11, 1994, col. 4, pA1(W) Of?ce Action for US Patent No. 6,569,012 dated Aug. 21, & pA1(e), ISSN 0193—241. 2002. Fink, William & Joyce Dick. Computer Programming Tour Final Of?ce Action for US Patent No. 6,569,012 dated Dec. nament as a learning Experience. The Computer Education 6, 2002. Quarterly. Summer 1989, vol. 11, Issue 2 pp. 46—52. Of?ce Action for Patent Application Ser. No. 10/377,343 O’Sullivan, Kate. Testing recruits. Net Style. Inc. Dec. 2000, dated Jul. 25, 2003. pp. 149—152. International Search Report for International Patent Appli Flipcode, Coding contest [online], May 2000 [retrieved on cation Ser. No. PCT/US03/10537 date Jul. 23, 2003. Jun. 26, 2002]. Retrieved from the Internet: URL:http:// WWW.?ipcode.com/contest.shtml. * cited by examiner U.S. Patent Jul. 13, 2004 Sheet 1 0f 16 US 6,761,631 B2
N:
§\@Nr\IQ526<9m2Ewzo352.6 .n:52.6mmm;__nriwaiw?wiu:Ewjumm>>__mw>> m: mwm<>>Eow mmw>>ommmm<>>Eowm:“H"wm<>>tow mmw>>ommmww>>omm IN...... L_| ...... l. .1...... 1 MMMmW:aw1m."anmm.33@Q: ...... 3355mm U.S. Patent Jul. 13, 2004 Sheet 2 0f 16 US 6,761,631 B2
TO CLIENT 108
CLIENT COMMUNICATION MODULE g0_6 : A I 1
APPLICATION SERVER Q2
1
I, r
I CONTEST SERVER m DATABASE SERVER Z_2_0
SERVER 104
FIG. 2 U.S. Patent Jul. 13, 2004 Sheet 3 0f 16 US 6,761,631 B2
CLlENTlC? 124
______. _ _ _ _ ...\ WEB
SERVER 104
vvEB SERVER ggg
308 HTTP /- 32s INTERFACE INTERFACE L‘STENER SERVER 4 SERVER R MANAGER PROCESS \ L / II\ \ APPLICATION SE ER _21_2 * “>4 5 : BUSINESS /- 316 332 \ ; I 324 OBJECT SERVl-ET I I \ ENGINE I I BUS‘NESS /- 316 A I i MESSAGE OBJECT 320 336 E a PRgggggoR h BUSINESS ,316 / \ ‘ I I OBJECT \ DATA XSL I I ACCESS PROCESSOR I .. BUSINESS LAYER -I I OBJECT 316 .
II CONTEST SERVEE m A- 348 DATABASE SER ER m I I RATING 1 ‘II 344\ ENGINE f 352 340 ;a
I5 COMPILATIONCODE TESTINGCODE DATABASE ; I ENGINE ENGINE I U.S. Patent Jul. 13, 2004 Sheet 4 0f 16 US 6,761,631 B2
STEP 404 —4 REGISTRATION E
4-“ STEP 406 — COMMUNICATE PROBLEM(S) '
STEP 40a — CODING ON-LINE I COMPETITION
‘ “ROUND"
STEP 412 — CHALLENGE
STEP 416 —— AUTOMATED TESTING
______-J
FIG. 4
U.S. Patent Jul. 13, 2004 Sheet 6 6f 16 US 6,761,631 B2
REFERENCE CODE RESPONSE 638
REFERENCE CODE
636 TEST '* COMPARSON ---> DATAQQQ e40
CONTESTANTS CODE RESPONSE 644 CONTESTANT'S CODE
SERVER 104
FIG. 6 U.S. Patent Jul. 13, 2004 Sheet 7 0f 16 US 6,761,631 B2
TRANSMIT QUALIFYING STEP 704 _— OUESTION(S) TO 4 CONTESTANT
‘
__ RECEIVE ANSWER TO STEP 708 QUALIFYING QUESTION
STEP 712 —% EvALUATE ANSWER
i NOTIFY CONTESTANT i STEP 716 ; OF RESULTS 5
N STEP 720 QUALIFYING TEST COMPLETE?
STEP 724 ——1j EvALUATE
I NOTIFY CONTESTANT OF I STEP 728 I EVALUATION RESULTS 5
BEGIN COMPETITION FIG. 7 U.S. Patent Jul. 13, 2004 Sheet 8 0f 16 US 6,761,631 B2
QUALIFYING TEST SCORE STEP 804 GREATER THAN CANNOT PARTICIPATE PREDETERMINED TEST SCORE? / STEP 808
STEP 812— INCREMENT CONTESTANT COUNTER
WAIT FOR ADDITIONAL COMPETITION COUNTER < = CONTESTANTS STEP 816 PREDETERMINED COUNT? / STEP 820
STEP 824 -— COMPETE
FIG. 8 U.S. Patent Jul. 13, 2004 Sheet 9 0f 16 US 6,761,631 B2
COMMUNICATE CODING ‘ ______STEP 904—— -E PROBLEM i
RECEIvECODE STEP 90s ——
STEPSTEP 912_916__1: EVALUATEDETERMINE RECEIVED POINTS CODE ...... __J
STEP 920__ DETERMINE RATING
FIG. 9 U.S. Patent Jul. 13, 2004 Sheet 10 0f 16 US 6,761,631 B2
STEP 1004 __ RECEIVE CONTESTANT INFORMATION I
STEP 1008 _ PROVIDE ON-LINE CODING COMPETITION I ASSIGN A RATING TO EACH STEP 1012 —-— CONTESTANT I PROVIDE CONTESTANT STEP 1016 —— INFORMATION TO SPONSOR 4 ------1
STEP 1020 —— RECEIVE CRITERIA FROM SPONSOR I" """ "J
FIG. 10 U.S. Patent Jul. 13, 2004 Sheet 11 0f 16 US 6,761,631 B2
STEP 1104 -— PROVIDE ON-LINE COMPETITION
‘
STEP 1108 —— AWARD POINTS
STEP 1116 / N STEP 1112 NO pR|ZE
Y
STEP 1120 ——-— AWARD PRIZE
FIG. 11 U.S. Patent Jul. 13, 2004 Sheet 12 0f 16 US 6,761,631 B2
/ 1200
Tournament Initial Participants Rounds
1204 1208 1212 1216 1220
FIG. 12 U.S. Patent Jul. 13, 2004 Sheet 13 0f 16 US 6,761,631 B2
1304 1310 1312 1316 f 1300
1308
1322 1324
1328
1332 FIG. 13 U.S. Patent Jul. 13, 2004 Sheet 14 0f 16 US 6,761,631 B2
1400 1412 1404 f’; “'05 U.S. Patent Jul. 13, 2004 Sheet 15 0f 16 US 6,761,631 B2
{4" 1500
mmle'duum 66:: 1515 1
FIG. 15 U.S. Patent Jul. 13, 2004 Sheet 16 0f 16 US 6,761,631 B2
1804
FIG. 16 US 6,761,631 B2 1 2 APPARATUS AND SYSTEM FOR from other coders, Which bene?ts both the coders and their FACILITATING ONLINE CODING employers. RecogniZably better coders can command COMPETITIONS greater salaries, and employers have an objective means to compare candidates and ?nd good coders. In some CROSS-REFERENCE TO RELATED embodiments, good coders have incentives to participate as APPLICATIONS competitions provide one or more of entertainment, aWards, and future employment opportunities. This application is a division of co-pending US. patent application Ser. No. 10/041,393, entitled “Systems and In some embodiments, entities that Wish to gain access to Methods for Coding Competitions”, ?led on Jan. 8, 2002, skilled coders sponsor the coding competitions. In some Which claims priority to US. provisional patent application 10 embodiments, a rating system is used to provide a metric Ser. No. 60/260,610, entitled “Systems and Methods for that can be used to evaluate a programmer’s proven skills, Coding Competitions”, ?led Jan. 9, 2001. knoWledge, and ability. In various embodiments, coding competitions are pro TECHNICAL FIELD vided “on-line,” in the sense that they are provided using 15 computers that communicate over a netWork. Coding com This invention relates to computer-based methods and petition participants, also referred to as contestants, use a systems for competing and, more particularly, to methods client computer that communicates With a competition and systems providing coding competitions. server. The contestants may be located anyWhere in the BACKGROUND INFORMATION World if (for example) the netWork is a global netWork such 20 as the Internet, or the coding competition participants may In the United States and elseWhere, computers have be located nearby, even in the same room, if (for example) become part of people’s everyday lives, both in the Work the netWork is a local area netWork. place and in personal endeavors. This is because a general In some embodiments, as part of a coding competition, a purpose computer can be programmed to run a variety of server computer provides to one or more contestants a softWare programs each providing different processing and 25 computer coding problem (eg a test question) that requires networking functions. Computer programmers develop each contestant to develop computer code. The computer computer code. Some companies hire large numbers of code may be, for example, an entire computer application computer programmers to develop code on the company’s program or a large or small portion of a computer applica behalf. tion program (eg a subroutine or a subsystem, one or more 30 Like many other activities, coding requires a collection of objects, etc.). The code may be Written in a speci?c pro skills. For example, some coders are extremely skilled, and, gramming language (e.g. C++, Java, C#, Fortran, per, etc.), for example, are remarkably accurate and efficient. Profes or may be coded in one of a choice of programming sional certi?cations provide information about the base languages or coding formats. skills possessed by an individual, but generally there is no 35 The coding problem may require design and or analysis of objective Way to easily distinguish highly-skilled coders the problem in addition to coding. The computer code may from average coders. This lack of objective measure means also be for something other than a computer application that highly-skilled coders may not be compensated for their program; for example, it may be a design for a computer true Worth. In addition, in the United States and elseWhere, program, or it may be an electronic circuit design or Workers considered to be “professionals,” such as laWyers, 40 implementation, or for another engineering design and/or accountants, doctors, and managers, command higher sala implementation. The computer code may be a model that ries than the general population. MeanWhile, highly skilled Will be forWard engineered into a machine readable format programmers, Who typically have good education, signi? for testing purposes. The code developed by a contestant is cant responsibility, and relatively unique abilities, are treated generally referred to as computer code, regardless of like production and administrative staff, rather than profes 45 Whether it is all or a portion of an application program, or sionals. This is also true for computer coders in other something else. disciplines (e.g. hardWare design, and other engineering In some embodiments, portions of the coding competi disciplines). tions take place in virtual “rooms” in Which tWo or more In the hiring process, ?rms typically subject coders to an contestants compete. Each contestant in a virtual room intervieW process that may even include some actual testing 50 receives a coding problem or problems, Which problems of candidates’ coding skills. But it is very dif?cult for a may be of varying dif?culty. Contestants Write computer company, even With some skills testing, to de?nitively code in response to one or more of the presented coding identify those programmers Who have the greatest skills, are problem(s). When a contestant’s code is ready, the respec the most accurate and ef?cient, and so on. This makes it tive client computer transmits the contestant’s computer dif?cult for a company that Wants to hire premier coders to 55 code to a server for evaluation. The time the contestant took identify and hire them, and it makes it more dif?cult for to complete the computer code may be recorded. premier coders to be hired at the compensation levels that In some embodiments, the server evaluates the computer they may deserve. code, and assigns a number of points to the contestants’ submission. In some embodiments, the competition includes SUMMARY OF THE INVENTION 60 multiple rounds in Which the server evaluates contestant Aneed therefore exists for mechanisms for skilled coders computer code for each round. The server may also deter to build marketability and raise their income to re?ect the mine a skill level rating or ratings for the contestant. The value their Work product creates. In general, in one aspect of determination of skill level of contestants may, for example, the invention, systems and methods for providing coding result in the aWarding of a priZe to a contestant, present competitions address these and other goals. Through the 65 employment opportunities to the contestant, attract sponsor provision of computer-based coding competitions, success ship for the competition, and present hiring opportunities to ful coding competition participants can be distinguished the sponsors of the competition.