Profiler Tool Selection for Curricular Support Abstract Profiling software is the process of analyzing the structure and performance of software. Profiling can be used to detect patterns of use, to verify performance, to optimize code, to identify data corruption and to expose memory leaks or excessive resource demands. Components of a large system can be profiled individually or together. Profiling is accomplished through software tools – software that runs and/or instruments the application under study. Since profiling reinforces understanding of conceptual processes, it can be used effectively in computer science to strengthen programming skills and awareness of design issues. Essentially, such tool usage exposes hidden details. For example, analysis of memory usage can highlight different types of memory management. Tracing function calls can illustrate the impact of different portions of a software system (i.e. functions executed). Students thus learn to think critically in a practical setting, fostering the capacity to evaluate vendor claims and counter asumptions (such as ‘Java has no memory leaks’). In this paper, we summarize our attempts to select, evaluate, and compare profiling tools, where the computer science classroom is the targeted use of such tools. First, we derived desired characteristics for profiling tools. Then we determined availability, subject to the constraints of an academic environment. Finally, we evaluated implementation and usage details. Profiling Tool Criteria Profiling is not a standard topic in the curriculum. However, profiling can be employed as a supplementary tool to reinforce material in programming and software design courses. Hence, we reviewed the fundamental concepts that CS students need to understand. Ideally, profiling should be demonstrated at a level that students can easily understand, that can reinforce concepts that are being taught in the classroom, and that students find useful in their computer science education. We focused on selecting a tool that students could easily acquire, apply to their own code, and interpret the feedback so provided. Our criteria considered computer equipment accessible to students and programming languages familiar to most students. We also surveyed the computer science faculty, soliciting their feedback on appropriate tool usage and desirable features. Faculty were asked to rank preferred features. Presentation details such as the presence of a GUI and the display of profiling results (charts, graphs) were two features queried. Other features focused on utility. Specifically, whether the profiling tool would work with multiple languages, C++, as well as the Unix, Windows and MacOS environments. Faculty were also asked to briefly answer the following questions: • Which languages do you feel are important to consider? • Which platform or environment is most important? • What features or capabilities would you most like to see introduced to the classroom? • Is there a particular profiling focus or result you feel would benefit CS students most? We balanced results of the faculty survey with our own classroom experiences and perceived student expectations. In sum, we derive a set of features that described a tool that profiles C++, and is available for both Windows and UNIX systems. A graphical interface was considered to be less important, as long as the formatted results were easy to read and interpret. Since Java is another important language in the curriculum, profiling different languages, in particular C, C++, and Java, would be ideal. In terms of conceptual material to reinforce with a profiling tool, memory usage was emphasized. Other important concepts included: profiling disk accesses, procedure-level profiling, I/O performance tracking, identification of excessive resource use, and algorithm performance. To aid in teaching profiling, professors anticipated using a profiler to compare algorithms. This could be done by comparing different software programs, or by using one program that changes its control flow over time. Profiling Tool Availability Choosing a profiling tool was challenging. In fact, selection proved to be one of the toughest tasks encountered. Anticipated as a fairly straightforward step, we were surprised to spend (too much) time sorting through various profiling tools trying to find one that met all of our needs. The major challenges we faced were: finding free tools, encountering poor documentation, and determining whether tools met all of our criteria for language, platform, cost, features. Our project aims to integrate profiling tools into classrooms with students as the target. Thus, we sought an open-source tool. Purchasing a tool might make it inconvenient and infeasible for many students to acquire the tools for their own use and experimentation. Also, academic departments might want free distribution. Many of the most appealing tools were not free, so we did not test them. Generally, these tools had user friendly GUIs and well-developed graphical features. Most of these tools, however, were targeted to very specific audiences. The second, and perhaps the most frustrating, problem encountered was poor or completely missing documentation. With some of the most promising and free tools, we were unable to get past the installation stage because there was no documentation to support investigation and no tutorials. We suspected incomplete documentation would be much less of an issue with a purchased tool because well-documented user manuals are usually a selling point. After many hours, we abandoned two tools with good potential – most notably Eclipse and NetBeans. Eclipse is a development platform that is open source. Many Eclipse developers have created plug-in tools with various capabilities, including profiling. Although Eclipse offers a wide variety of features, the major obstacle was lack of appropriate introductory documentation. Even after hours of research, experimentation, and searching for beginner (step-by-step) instructions, we were unable to get the profiling plug-in tool installed and working. Without more detailed instruction, Eclipse did not appear to be a suitable tool for easy installation targeted toward small, low-level profiling. NetBeans presented a similar problem. The last challenge faced was finding a tool that met a variety of our desired features. Aside from cost, these included language, platform, and graphical features. We found tools that met some, but not all of our criteria, so we decided not to explore them further. For example, many tools profile only Java programs, but we desired multi-language support. In the end, we compromised and selected a tool that met all of the criteria but not necessarily in the optimal manner. For example, some tools had better graphical user interfaces than others but did not have multi-language support. Profiling Tools Examined To prepare for presentation of a profiling tool to a university classroom, we experimented with a wide range of profiling tools ourselves. Online searches and investigation led to many sources for profiling software. Experimenting with the available software provided a comprehension of the breadth of features, strengths, weaknesses, and capabilities of the accessible tools. From our initial exploration, we were able to gain a more thorough understanding of the profiling tools currently available. By understanding the limitations of the tools, we were able to determine a set of minimum functionality to demonstrate in our classroom instruction. We narrowed our selection from a couple dozen profiling tools down to three strong candidates, and then performed tests on each tool. We ran a small set of programs in each one, evaluating the ease of installation, ease of use, user interface, existence and clarity of documentation and instructions, and the usefulness and readability of the results. The programs we ran were small to medium sized C++ programs – approximately equivalent to the length and complexity of a typical program written in an introductory data structures class. Programs included different types of loops, pointers, arrays, lists, linked lists, etc. During each test, we noted any unexpected errors, confusing instructions or unclear results. These factors determined our evaluation of the tool’s ease of use. Finding answers to issues, explanations of results, and troubleshooting details during our examination of each tool helped us assess the tool’s documentation and instruction quality. The final result set for each run was considered for usefulness and readability. We compared the breadth of the tool’s analysis, the clarity of the output, and any other extra information that the tool gathered. Our three candidates were GProf, DevPartner and ValGrind. All three are available as freeware, work with C++ programs, and were downloaded and installed without problems. GProf runs on a Linux shell, which is readily available on university lab computers. It is conveniently built-in on all Unix/Linux platforms. The tool includes thorough help files. We were also able to find extensive documentation online to provide instructions and explanations of its features. GProf does not, however, include a graphical interface. Results are displayed as text in the Linux shell it is run in. The results are precise, but the impact of the tool’s output is not presented in an easily comparable format. DevPartner is a free download, providing limited features of the full cost version. The software is easy to download and install. It runs on