A Metric Evaluation of Game Application

Alan C. Jamieson, Nicholas A. Kraft, Jason O. Hallstrom and Brian A. Malloy Computer Science Department Clemson University Clemson, SC 29634, USA {ajamies,nkraft,jasonoh,malloy}@cs.clemson.edu

Abstract art, music, and voice. To manage large teams of program- mers, most current game developers use the C++ language In this paper we evaluate the exploitation of object tech- and attempt to exploit the benefits of object technology. nology as it is used in a test suite of game application soft- In this paper we evaluate the exploitation of object tech- ware. We use several well-known metrics and apply them nology as it is used in a test suite of game application soft- to both the game application software and a test suite of ware. We use several well-known metrics and apply them language processing tools to form a basis of comparing the to both the game application software and a test suite of two groups. We have developed a metric computation sys- language processing tools to form a basis of comparing the tem that uses the g4re infrastructure to analyze the C++ two groups. We have developed a metric computation sys- programs. We present some results for the two groups of tem that uses the g4re infrastructure to analyze the C++ pro- applications and draw some conclusions about the modu- grams [10, 11]. We present some results for the two groups larity, use of inheritance, and the complexity of methods in of programs and draw some conclusions about the modu- these applications. larity, use of inheritance and the complexity of methods in the applications. The literature on object-oriented software metrics is ex- 1. Introduction tensive; see, for example, the detailed surveys presented in [3] and [5]. By contrast, there has been relatively little As software developers shift their priorities to the con- work focused on applying metrics to assess the relative de- struction of complex large scale systems that are easy to sign characteristics of systems in different application do- extend, modify, and maintain, the inadequacy of tradi- mains. A unique contribution of our work is the application tional approaches and methodologies becomes apparent. of object-oriented software metrics to the consideration of The object-oriented approach to software development ad- gaming applications. dresses some of these inadequacies and makes it possible to In the next section we provide background about the model systems that are close to their real world analogues. terminology that we use, about game APIs, and about the 4 The goal of object-oriented design is to accurately identify g re system. In Section 3 we describe the construction of the principle roles in a process, assign responsibilities to our metric computation system and define the metrics that these roles and encapsulate them in an object. The benefits we use to evaluate the exploitation of object orientation in of object technology are extensibility, ease of modification game application software. In Section 4 we provide results and ease of reuse. from our evaluation of the test suite of game software and The development of game application software has fol- language processing tools. In Section 5 we describe the lowed this pattern of migration from traditional approaches limited research that is similar to our work. Finally, in Sec- to the exploitation of object technology for its acknowl- tion 6 we draw conclusions and describe our future work. edged benefits. In fact, the object-oriented paradigm seems to have a natural application to the domain of graphics, 2. Background Graphical User Interfaces (GUIs) and game application programming. Early games such as and Doom were In this section we provide background about the termi- implemented in C because of its small learning curve and nology and tools that we use in this paper. In Section 2.1 its fast compilation and execution speed. However, cur- we review some of the Application Programmer Interfaces rent game development requires large teams of program- (APIs) used in game development. In Section 2.2 we re- mers and analysts as well as other kinds of talent such as view the g4re system that analyzes the applications in our

Submitted to The Future Play Conference October 13–15, 2005, Michigan State University MI test suite [10, 11]. Finally, in Section 2.3 we review the use port application development [4, 23, 25]. The lack of tool of metrics in the software life cycle. support for applications using the C++ language is espe- cially noteworthy. 2.1. Game APIs One explanation for the lack of software tools for C++ is the difficulty in constructing a front-end for the language, In early game development, DOS-based games were as described in references [2, 9, 12, 20, 21, 22]. This diffi- generally implemented with commands issued directly to culty results, in part, from the complexity and scale of the the computer’s hardware. These early DOS games used language. However, a more important problem is the ambi- calls to device drivers for input devices such as a mouse or guity inherent in many C++ language constructs[9, 12, 19, and calls to specific sound cards such as Creative 22]. Many C++ constructs cannot be recognized through Labs’ Sound Blaster. Video programming was the most syntactic considerations alone. For example, the difficulty difficult aspect of game development where the generation in distinguishing a declaration from an expression can only of fast and smooth graphics required significant program- be resolved by performing name lookup[1, 9]. ming skill. Graphics code frequently exploited the speed of 4 assembly language programming and depended on certain The g re tool chain exploits the gcc Abstract Semantic hardware-level features of the VGA graphics adapter. Graph (ASG), GENERIC, to provide an Application Pro- grammers Interface (API) to facilitate easy access to infor- Currently, few game developers write register-level mation about declarations, including classes, functions, and video code, instead relying on prewritten Application Pro- variables, as well as information about scopes, types, and grammer Interfaces (APIs) that form a layer of software be- control statements. The advantages of the g4re tool chain is tween the game and the hardware. The most popular API that it can analyze any program that can be compiled by the in current usage is DirectX using the C++ language vehi- gcc C++ compiler. We use the g4re tool chain to construct cle [15, 16]. The DirectX API provides low-level access our metric computation system and we describe this system to multimedia hardware in a device-independent manner. in Section 3. New versions of DirectX are released to permit game de- velopers to take advantage of hardware advances as they occur, even after games have shipped. However, the Di- 2.3. Object-Oriented Metrics rectX API is specific to the platform. With the popularity of the game Software metrics are quantitative measures that enable developers became interested in platform-independent software developers, testers, and maintainers to evaluate game programming and several APIs have been introduced, the static properties of a software system [5]. Software including SVGALib, ClanLib and SDL. The most popular metrics are computed and the resultant data are collected, of the platform-independent game APIs is the Simple Di- analyzed, and compared throughout the lifetime of a soft- rectmedia Layer (SDL) [18]. SDL is a cross-platform mul- ware system to evaluate improvement or deterioration of timedia library that has already been used to port a number the software system. Software metrics are also useful for of Windows-based games to Linux. identifying problem modules of a software system. The SDL API supports virtually all of the major op- Object-oriented metrics were introduced to measure erating systems including Linux, Windows, Solaris and software properties specific to object-oriented software BSD variants including FreeBSD and MacOS. In addi- systems, including properties pertaining to classes and their tion to fast graphics support, SDL provides interfaces for object instances [3, 5]. The primary focus of object- playing sound, accessing CD-ROM drives and achieving oriented metrics is measuring properties of classes and their portable multi-threaded applications. SDL is released un- instances. Properties of interest include scope of properties, der the GNU LGPL and has accumulated a collection of object complexity, coupling, and cohesion. user-contributed libraries that provide additional function- ality for game developers. 3. Methodology 2.2. The g4re Tool Chain In this section we present an overview of our metrics Software tools are fundamental to the comprehen- computation system that enables us to evaluate the ex- sion, analysis, testing and debugging of application sys- ploitation of object technology in game application soft- tems. Tools can automate repetitive tasks and, with large ware. In Section 3.1 we describe details of the system and scale systems, can enable computation that would be pro- its use of the g4re tool chain [10, 11]. In Section 3.2 we hibitively time consuming if performed manually. The Java describe the metrics that we compute to facilitate our eval- language is well supported with libraries and tools to sup- uation of the game software.

2 Figure 1. System overview. This figure illustrates the important components in our metrics computation system that we constructed to compute the metrics defined in Section 3.2 and to perform the case study presented in Section 4. The metrics computation system consists of the g4re tool chain, including the CppInfo API, and a Metrics Tool that interacts with the API to extract information about a C++ program.

3.1. Overview of the Metrics Computation System classes and functions needed to compute the metrics de- fined in Section 3.2. Output of the Metrics Tool, shown in Figure 1 provides an overview of our metrics compu- the far left of the bottom row of the figure, is available in a tation system that we constructed to compute the metrics variety of formats and consists of a set of statistics for each defined in Section 3.2 and to perform the case study pre- computed metric. sented in Section 4. Our metrics computation system con- sists of the g4re tool chain, including the CppInfo API, and 3.2. Overview of the Computed Metrics a Metrics Tool that interacts with the API to extract infor- mation about a C++ program. Output of our system is a set In this section we define the object-oriented metrics that of statistics for each computed metric. we compute for the case study presented in Section 4. We define one metric measuring complexity, Weighted Meth- Input to our system is the source code for a C++ pro- gram, shown in the far left of the top row of the figure, ods per Class, and three metrics measuring the use of inher- which is used as input to the gcc compiler. Using the itance: Depth of Inheritance Tree, Number of Ancestors, -fdump-translation-unit-all option, we obtain and Number of Children. We also use additional metrics that measure modularity and delegation. a plain text representation of the ASG for each C++ transla- tion unit in the program. We use these plain text ASG rep- 3.2.1. Metric 1: Weighted Methods per Class (WMC). resentations, known as tu files, as input to our TUxformer WMC measures the complexity of an object and is an indi- subsystem, shown in the middle of the top row of the fig- cator of the time and effort required to develop and main- ure. For each tu file, the TUxformer subsystem creates an tain a class. in-memory representation of the encoded ASG, prunes the Given a class C with methods M ,M ,...,M , weighted ASG, and serializes the ASG to GXL. 1 2 n with cyclomatic complexity c1,c2,...,cn, respectively, the We use the set of GXL files produced by TUxformer as metric is computed as input to the g4xformer subsystem, shown in the far right of the top row of the figure. The g4xformer subsystem n WMC(C) = ∑ ci parses each GXL file and creates an in-memory represen- i=1 tation of the encoded ASG. The subsystem then links the representations of each individual ASG to create a unified Given a method M with control flow graph G = (V,E), let representation of the entire C++ program. After linking is D equal the set of decision nodes in V, where a decision complete, the subsystem filters, from the unified represen- node represents one of { if, switch, for, while, do while, tation, language elements that are identified as defined in a catch }. The cyclomatic complexity, c, of M is the number filename contained in the filter lists, shown in the far right of linearly independent paths in G and is computed as of the top row of the figure. c(M) = |D| + 1 The CppInfo API, shown in the far right of the bottom row of the figure, provides access to information from the 3.2.2. Metric 2: Depth of Inheritance Tree (DIT). DIT unified representation of a whole C++ program created by is the length of the maximum path from a class to the root the g4xformer subsystem. Our Metrics Tool, shown in of its inheritance hierarchy, relates to scope of properties, the middle of the bottom row of the figure, instantiates and and is an indicator of the number of ancestor classes that queries the API to gain access to the information about can potentially affect a class.

3 SDL Game Applications Language Processing Applications ASC AvP Freespace2 Scorched3D Doxygen g4re Jikes Keystone Version 1.16.1.0 cvs cvs 38.1 1.3.9.1 1.0.4 1.22 0.2.3 Source Files 436 509 652 1069 260 128 75 123 Translation Units 199 222 220 513 122 60 38 52 C++ Translation Units 194 95 220 492 90 60 38 52 LOC (≈) 130 K 318 K 365 K 110 K 200 K 10 K 70 K 30 K

Table 1. Testsuite of SDL Game Application Software and Language Processing Tools.

Given a class C with a set of base classes BC, the metric use of inheritance in game applications. In Section 4.4 we is computed as provide results describing the complexity of member func- tions in classes. Finally, in Section 4.5 we provide results  0 if |BC| = 0 DIT(C) = for games implemented in DirectX. max({DIT(Bi) : Bi ∈ BC}) + 1 if |BC| > 0 3.2.3. Metric 3: Number of Ancestors (NOA). NOA is 4.1. The Test suite of Game Applications and Lan- the total number of ancestor classes of a class. In the ab- guage Processing Tools sence of multiple inheritance, NOA is equivalent to DIT. In the presence of multiple inheritance, care must be taken to Table 1 lists eight applications, or test cases, that form avoid counting an ancestor class more than once, due to the the test suite that we use in our study, together with size possibility of a diamond-shaped inheritance hierarchy. statistics about each test case. The top row of the ta- ble lists the names that we use to refer to each of the 3.2.4. Metric 4: Number of Children (NOC). NOC is test cases. The game applications are listed in the first the number of immediate successors of a class and mea- four columns and the language processing applications are sures the breadth of inheritance. listed in the last four columns. The four game applications Given a class C with a set of derived classes DC, the are: Allied Strategic Command (ASC), Alien vs Preda- metric is computed as tor (AvP), Freespace 2 (Freespace2), and Scorched 3d NOC(C) = |DC| (Scorched3D). The Application Programmer’s Interface (API) used for the four games is the Simple Directmedia 4. Case Study Layer (SDL), described in Section 2. The four language processing applications, listed in the last four columns of 4 In this section we describe the results that we obtained Table 1, are: Doxygen, g re, Jikes, and Keystone. Doxy- using our metrics tool to evaluate game application soft- gen is a documentation system for C++, C, and Java [26] 4 ware. We evaluate game software by comparing metrics and g re is part of the infrastructure for reverse engineer- computed for four games with four popular language pro- ing that we use to construct our metrics tool [10, 11]. Jikes cessing applications. The results that we report in this sec- is a Java compiler system [6] and Keystone is a parser and tion capture information about the sizes of the programs front-end for ISO C++ [8, 14]. and the exploitation of object technology as measured by The rows of Table 1 list some statistics and coarse- metrics described in Section 3. All experiments were ex- grained size metrics for the test cases: the first row lists ecuted on a workstation with an AMD Athlon64 3000+ the version number, Version; the second row lists the num- processor, 1024 MB of PC3200 DDR RAM, and a 7200 ber of source files, Source Files, for each test case; the RPM SATA hard drive, running the Slackware 10.1 operat- third row lists the number of translation units, Translation ing system. The programs were compiled using gcc version Units, which includes both C++ and C translation units; the 3.3.4. fourth row lists the number of C++ translation units (C++ In the next section we describe the eight (8) applica- Translation Units), which is only C++ code; and finally, tions that form our test suite: four game applications im- the last row of the table lists the (approximate) thousands plemented using the SDL API and four language process- of lines of code (KLOC) for each test case, not counting ing applications, and provide some results using coarse- blank or comment lines. For example, the largest game grained size metrics for the applications. In Section 4.2 we in our test suite is Freespace 2, a Version that we ob- provide results describing the modularity of game applica- tained from a cvs repository (on July 22, 2005), consist- tions and in Section 4.3 we provide results describing the ing of 652 source files, 220 Translation Units and 220

4 Classes Functions Count Abstract Root Leaf Count Member Virtual Pure ASC 1389 58 901 390 8693 7775 2170 208 AvP 1732 28 1369 327 11548 9350 1216 90 Freespace2 332 0 320 12 9468 1687 48 0 Scorched3D 799 50 405 364 8432 7210 1907 112 Doxygen 315 9 153 157 5422 4570 2159 249 g4re 78 17 27 37 849 798 303 106 Jikes 378 5 210 158 5717 5685 602 16 Keystone 160 14 49 87 2354 2306 1178 189

Table 2. Modularity. This table presents results about the modularity of game application software.

C++ Translation Units. Since the number of Transla- plication in our test suite; the second highest class total is tion Units is the same as the number of C++ Translation the ASC game containing 1389 classes. It is somewhat Units, the Freespace 2 test case contains no C code. The surprising that the AvP game contains a large number of Freespace 2 test case consists of 365 KLOC, as illustrated classes in view of the large number of C files in the pro- on the last row, third column of Table 1. gram. The results listed in Table 1 show that the AvP game The results in Table 1 suggest that, for the test cases contains 222 Translation Units, but only 95 of these are that we have chosen for our study, the game applications C++ Translation Units; which means that over half of the are larger than the language processing applications. For translation units consist of C code. However, the g4re tool example, the average number for the game applications is uses an Abstract Syntax Graph (ASG) representation of 231 KLOC, whereas the average number of KLOC for the the input application; at the level of the ASG, all template language processing applications is 78 KLOC; thus, the av- classes are instantiated so that the count of classes listed in erage game application in our test suite is three times as the first column of data in Table 2 includes classes and in- large as the average language processing application. stantiated template classes and all of these structures may participate in inheritance and appear anywhere in an inher- 4.2. Modularity in Game Application Software itance tree constructed for the Depth of Inheritance Tree, Number of Children and Number of Ancestor metrics. Table 2 presents results that capture information about The results in Table 2 suggest that, for our test suite of the modularity of game application software. The rows in four game applications and four language processing tools, Table 2 list the test cases, where the game applications are the game applications may be more modular than the lan- listed in the first four rows and the language processing guage processing tools. For example, the total number of tools are listed in the last four rows. The columns sum- classes in the four game applications is 4252 classes. The marize results describing the classes and functions in the results in Section 4.1 show that the total KLOC for the four respective test cases: the first four columns summarize in- game applications is 923 KLOC, for a classes to KLOC ra- formation about the classes and the last four columns sum- tio of 4.6. Similarly, the total number of classes in the four marize information about the functions. For example, the language processing tools is 931 classes. The results in Alien vs Predator (AvP) game, listed on the second row Section 4.1 show that the total KLOC for the four language of the table, contains 1732 classes: 28 of the classes are ab- processing tools is 310 KLOC, for a classes to KLOC ratio stract, Abstract, 1369 of the classes contain no ancestors of 3.0. in the inheritance tree, Root, and 327 of the classes contain no children and have at least one ancestor in the inheritance 4.3. Inheritance in Game Application Software tree, Leaf. The number of Root and Leaf classes for the AvP game together comprise 1696 classes so that the in- Tables 3 and 4 present results for the Depth of Inher- heritance tree contained only 36 interior classes. Moreover, itance Tree (DIT) and Number of Children (NOC) met- the (AvP) game contains 11,548 functions, Count, 9350 of rics respectively. Due to space limitations, we have elided these functions were members of a class, Member, 2170 the results for the Number of Ancestors (NOA) metric; the of these functions were virtual, Virtual, and 208 of these interested reader may consult reference [7] for results on functions were pure virtual, Pure. the NOA metric. The rows in Tables 3 and 4 list the test The AvP game contains more classes than any other ap- cases. The columns list results for the metrics, where the

5 h ags envle .91,adtelretstandard largest the and 0.09312, value, mean largest the the Moreover, of all programs. of depth test inheritance the longest the 9, of depth for maximum results cases. DIT lists test 3 eight Table the of the row for fourth the computed example, we For that DIT for (DIT). results Inheritance of Depth NOC 4.3.1. the tree. and inheritance the tree of inheritance breadth the the measures metric of metric DIT depth the Intuitively, the tree. measures inheritance the the in and class successors tree a immediate of inheritance of the number the of measures root metric the NOC to class a from path the value. below common most and the above is lie mode values the of median; number equal an which mode, standard the the and list mean, tables the the from in columns deviation three final The rics. minimum, the Max list columns three first Keystone Jikes g Doxygen Scorched3D Freespace2 AvP ASC h I ercmaue h egho h maximum the of length the measures metric DIT The g Doxygen Scorched3D Freespace2 AvP ASC Keystone Jikes Scorched3D 4 4 re n h mean, the and re al 3. Table al 4. Table Mode aewt iiu et f0ada and 0 of depth minimum a with game et fIhrtneTree. Inheritance Of Depth 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Mean Min Min h einvlei h au for value the is value median The . ubro Children. of Number 10 26 48 11 12 30 27 6 Max 5 3 5 4 9 1 5 4 Max t Dev Std ausfrtersetv met- respective the for values , 1.7938 0.7143 1.8974 0.9270 0.9312 0.0361 0.3043 0.6451 0.6063 0.4550 0.4872 0.3587 0.2003 0.0361 0.1207 0.3139 Mean Mean Scorched3D h median, the , al rsnsthe presents 3 Table Min 1.4498 0.9596 1.5921 1.1220 1.4957 0.1869 0.7068 0.9906 1.6563 2.4741 1.1705 2.7672 0.9122 0.6586 0.8497 1.5496

h maximum, the , Std Dev Std Dev aehad game 2 0 2 1 0 0 0 0 0 0 0 0 0 0 0 0 Median Median Median 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Mode Mode 6 r o l ftegm plcto etcases. test application game they the as of zero, all are for values these are and and row median fourth The the of columns tree. inheritance the its for of mode vari- depth largest the the had in case ance test this that so 1.4957, deviation, d.Tefia he oun ntetbe ittestandard mode, the the and list mean, tables the the in from columns deviation three final The ods. minimum, the list mum, columns where metric, three WMC first the for the results list columns The cases. Class 4.4. language the 212.6. for is and tools 41.37 is of tools ratio processing breadth language maximum the to and classes the classes However, 22.5 classes. is the of tools average processing the the language of average for The inheri- values the applications. game than for broader trees are tance tools processing game language the for of all for are row they second as cases. the test zero, application of are columns values two these last the and the for in mode and listed median are The game cases. test NOC the in of variance any test largest for the the 0.8497, of of deviation all standard of a chil- tree of inheritance The number broadest programs. maximum the a 30, and of 0 dren of children results of NOC number lists 4 Table cases. the of test row for eight second the the for example, computed For we that NOC for (NOC). results Children of Number 4.3.2. Keystone Jikes g Doxygen Scorched3D Freespace2 AvP ASC 4 al rsnsrslsfrthe for results presents 5 Table trees inheritance the that suggest 4 Table in results The re opeiyi aeApiainSoftware Application Game in Complexity Max WC erc h osi h al ittetest the list table the in rows The metric. (WMC) al 5. Table le sPredator vs Alien n h mean, the and Scorched3D Mode AvP egtdMtosprClass. per Methods Weighted 0 0 0 0 0 0 0 0 Max Min aehdama au f010,and 0.1207, of value mean a had game . 2016 557 206 430 240 123 107 561 ausfrgm plctosi 20 is applications game for values

Mean Max t Dev Std aeaelse ntels two last the in listed are game ae( game 24.3875 32.3968 17.7564 27.6762 17.3717 12.9770 6.6596 7.2898 ausfrwihe meth- weighted for values , AvP

egtdMtosper Methods Weighted Mean h median, the , al rsnsthe presents 4 Table ihaminimum a with ) 119.1240 52.4735 30.2694 57.4967 19.0581 15.7072 10.5944 30.3646 Min

h maxi- the , Std Dev Median Max AvP 15 13 13 12 7 3 3 4 Median 14 10 0 7 3 3 3 0 Mode h atrwo h al it h ie fcd LC not KLOC, code of lines lines. the comment lists or table blank including the fourth of the Finally, row (NOC). and last children the of (DIT) number tree maximum the inheritance lists max- row the the lists of 6 Table depth of imum not row and third computing classes The In template instantiations. counted their we classes. classes of of number number the the and lists files row source second of the number the about information contains and Interactive (F Empire from avereage the whereas 25.55, Mean is tools average is processing the applications language Similarly, game the 257.75. of only value maximum 802.25, the is tools whereas processing language average the the of value example, For maximum software. application methods game the the than in complex more are tools processing guage 2 are Research, Microsoft games from The API. rectX section. this of part first language the the in presented and tools software processing application comparison game for SDL basis hand the some by provide with computed to were them present section we this and compu- in metric results and that The analysis platform our tation. Windows automate the to us for enable system will analysis construc- an the of includes work tion ongoing Our with API. implemented DirectX games the commercial four for obtained we 4.5. Software. 6. Table LOC( NOC DIT Classes oreFiles Source (C REESPACE h eut nTbe5so httemtosi h lan- the in methods the that show 5 Table in results The al it orgmsipeetdwt h Di- the with implemented games four lists 6 Table that results preliminary some describe we section this In IV eut o h iet API DirectX the for Results au o h aeapiain sol 11.07. only is applications game the for value :C ≈ ) T 2 rmActivision, from P2) etut fDrcXGm Application Game DirectX of Testsuite )fo oiinIc h rtrwo h table the of row first The Inc. from 2) 331K 1938 1135 49 9 Allegiance 665K 1963 2081 312 iiiain alt Power to Call Civilization: 7 Allegiance Civ:CtP2 ecn:Fesae2 Freespace Descent: ateo Britain of Battle 451K 1132 123 715 11 Mean BoB (A 416K au o the for value 110 649 LLEGIANCE 39 7 Freespace2 (B O B) ) 7 h ieo h etut ae tdfcl oda definitive draw to difficult it that makes conclusions. note testsuite We the of system. size is open-source the it the better: than fares modular which more system proprietary the major of a redesign evaluate authors the however, Later, modular. more re- Paulson the one of contradict and figures sults system resulting the open-source system, one proprietary of consists which suite, a to allocating by to elements cost various higher the responsi- individuals implementing the for estimates between ble metric coordination The of cost the files. interdependent groups The of that algorithm collections clustering change. a system on based arbitrary is files metric an second of by consider number average, authors the on estimates The metric affected, first call. The a metrics. such two of in absence defined the function denotes a from call a file position of at existence one the of value denotes A files implementation. source given the a between in dependencies the captures matrix of use over its in defined novel metrics is approach Their software. open-source evaluating complex- for in metric similar interest a complexity. authors’ use and the modularity, share and ity we that note We ( complex, oterporeaycutrat,oe-orepoet ( ( faster, projects grow open-source not relative do counterparts, that proprietary conclude authors their the to three figures, of resulting Based the consisting projects. on testsuite proprietary a three and to projects metrics open-source of series a apply ( 5. iesfiedmnin fcmaio:( comparison: ( of con- dimensions study five Their per- siders common projects. of open-source validity regarding the evaluate ceptions to is soft- goal proprietary Their and ware. open-source between differences the this In own. studies. our relevant several to consider spirit briefly we in section, similar are that considered gaming however, comparisons of have, consideration researchers its Other in applications. unique is work design domains. our relative application Further, different the in assess systems to of characteristics metrics work applying little relatively on been has focused in there presented contrast, By survey [5]. detailed and the [3] example, for see, tensive; v ii ouaiy o ahdmnino oprsn they comparison, of dimension each For modularity. ) eincetvt,( creativity, design ) intra MacCormack Paulson ex- is metrics software object-oriented on literature The eae Work Related i oafnto endi file in defined function a to cutrdpnece.We ple otertest- their to applied When dependencies. -cluster iv tal. et r oerlal,ad( and reliable, more are ) inter tal. et tal. et 1]proma miia vlainof evaluation empirical an perform [17] cutrdpnece,adalwrcost lower a and dependencies, -cluster ii einsrcuematrixes structure design otrmr raiiy ( creativity, more foster ) iii h pnsuc ytmappears system open-source the : 1]fcso h ouaiyof modularity the on focus [13] opeiy ( complexity, ) j iial,azr value zero a Similarly, . v r esmodular. less are ) iv i ytmgrowth, system ) eiblt,and reliability, ) iii 2] Each [24]. r more are ) ( i , j i ) ) 6. Conclusions and Future Work International Workshop on Program Comprehension, Pitts- burgh, PA, USA, May 5-7 1999. 4 In this paper we have evaluated the exploitation of ob- [10] N. A. Kraft, B. A. Malloy, and J. F. Power. g re: Harness- ject technology in a test suite of game application software. ing gcc to reverse engineer C++ applications. In Seminar We used metrics that describe the size, modularity, delega- No. 05161: Transformation Techniques in Software Engi- neering, Schloss Dagstuhl, Germany, April 17-22 2005. tion, inheritance, and complexity of the applications. We 4 [11] N. A. Kraft, B. A. Malloy, and J. F. Power. Toward an infras- developed a metric computation system that uses the g re tructure to support interoperability in reverse engineering. infrastructure to analyze the C++ programs [10, 11], and we In Proceedings of the 12th Working Conference on Reverse use the system to apply the metrics to the game application Engineering, WCRE’005, Pittsbrugh, PA, November 2005. software and a test suite of language processing tools to [12] John Lilley. PCCTS-based LL(1) C++ parser: Design and form a basis of comparing the applications in the two do- theory of operation. Version 1.5, February 1997. mains. Our game applications are implemented using the [13] A. MacCormack, J. Rusnak, and C. Baldwin. Exploring the SDL [18] and DirectX APIs. structure of complex software designs: An empirical study We have shown that, for our test suite, the game applica- of open source and proprietary code. Working Paper 05-016, tions are more modular than the language processing tools Harvard Business School, Boston, MA, USA, 2004. [14] B. A. Malloy, T. H. Gibbs, and J. F. Power. Decorat- and that the methods in the game applications are less com- ing tokens to facilitate recognition of ambiguous language plex than the methods in the language processing tools. Our constructs. Software, Practice & Experience, 33(1):19–39, metrics computation system does not include library code 2003. in its analysis and it is likely that much of the complexity [15] I. Parberry. Learn Computer Game Programming with Di- of the methods is game applications lies in the SDL or Di- rectX 7.0. Woodward Publishing Co., Plano, TX, USA, rectX APIs. Our future work includes modification of the 2000. g4re infrastructure to extrapolate the number of template [16] Ian Parberry, Timothy Roden, and Max B. Kazemzadeh. Ex- classes rather than the number of instantiations of template perience with an industry-driven capstone course on game classes. We are also extending the g4re infrastructure to programming: extended abstract. SIGCSE Bull., 37(1):91– automate computation of the DirectX metrics. 95, 2005. [17] J.W. Paulson, G. Succi, and A. Eberlein. An empirical study of open-source and closed-source software products. References IEEE Transactions on Software Engineering, 30(4):246– 256, 2004. [1] American National Standards Institute. International [18] E. Pazera. Focus on SDL. Premier Press, Cincinnati, OH, Standard: Programming Languages - C++. Number 2003. 14882:1998(E) in ASC X3. ISO/IEC JTC 1, September [19] J. F. Power and B. A. Malloy. Metric-based analysis of 1998. context-free grammars. In Proceedings of the 8th Inter- [2] F. Bodin, P. Beckman, D. Gannon, J. Gotwals, S. Narayana, national Workshop on Program Comprehension, Limerick, S. Srinivas, and B. Winnicka. Sage++: An object-oriented Ireland, June 2000. toolkit and class library for building Fortran and C++ re- [20] J. F. Power and B. A. Malloy. Symbol table construction structuring tools. In The second annual object-oriented and name lookup in ISO C++. In 37th International Con- numerics conference (OON-SKI), pages 122–136, Sunriver, ference on Technology of Object-Oriented Languages and Oregon, USA, 1994. Systems, (TOOLS Pacific 2000), pages 57–68, Sydney, Aus- [3] Shyam R. Chidamber and Chris F. Kemerer. A metrics tralia, November 2000. suite for object oriented design. IEEE Trans. Software Eng., [21] S.P. Reiss and T. Davis. Experiences writing object-oriented 20(6):476–493, 1994. compiler front ends. Technical report, Brown University, [4] S. F. Cohen. Quest for Java. Communications of the ACM, January 1995. 41(1):81–83, January 1997. [22] J.A. Roskind. A YACC-able C++ 2.1 grammar, and the re- [5] Norman E. Fenton and Shari Lawrence Pfleeger. Software sulting ambiguities. Independent Consultant, Indialantic FL, Metrics: A Rigorous and Practical Approach. PWS Pub- 1989. lishing Co., Boston, MA, USA, 1998. [23] S. Singhal and B. Nguyen. The Java factor. Communications [6] IBM Jikes Project. Jikes version 1.22. Available at of the ACM, 41(6):34–37, June 1998. http://jikes.sourceforge.net. [24] D.V. Steward. The design structure system: A method for [7] A. C. Jamieson, N. A. Kraft, J. O. Hallstrom, and B. A. managing the design of complex systems. IEEE Transac- Malloy. A metric evaluation of game software. Technical tions on Engineering Management, 28(3):71–84, 1981. report, Clemson University, 2005. [25] P. Tyma. Why are we using Java. Communications of the [8] Keystone Project. Keystone version 0.2.3. Available at ACM, 41(6):38–42, June 1998. http://keystone.sourceforge.net. [26] D. van Heesch. Doxygen version 1.3.9.1. Available at [9] Gregory Knapen, Bruno Lague, Michel Dagenais, and Et- http://stack.nl/ dimitri/doxygen. tore Merlo. Parsing C++ despite missing declarations. In 7th

8