Bachelor’s thesis Information and Communications Technology 2019 Aleksandr Osipov SELECTING C# PROFILING TOOLS BACHELOR’S THESIS | ABSTRACT TURKU UNIVERSITY OF APPLIED SCIENCES Bachelor of Engineering, Information and Communications Technology Degree programme 2019 | 56 Aleksandr Osipov SELECTING C# PROFILING TOOLS The usage of profiling tools could certainly be beneficial for game development since it helps to identify performance bottlenecks. However, the number of available profilers is high and can cause frustration among developers. The objective of this thesis is to research the most recommended profilers and determine the best one based on following criteria: affordability, learnability, intuitive UI, optimized performance, deep analysis, and customizability. The investigation was carried out on Redgate ANTS memory and performance profilers, namely, the CodeTrack, JetBrains dotMemory and dotTrace profilers and the built-in Visual Studio profilers. As a target for testing, the game Barotrauma was used, which was developed in cooperation with FakeFish ltd and Undertow Games. In order to ensure that all profilers have similar conditions, the requirements were to play the game in single player for 10 to 30 minutes, encounter at least one monster attack and experience one breach of the submarine. This thesis investigates all profiler findings and assesses them according to the six aforementioned criteria, based on the author’s own user experience. The results show that built-in Visual Studio profilers have the highest grades. JetBrains and CodeTrack are tied for second place, considering JetBrains is more user-friendly and performant, while CodeTrack is a fully free software. ANTS profilers take the last place; nevertheless, they are helpful in optimization matters. This thesis offers advice on selecting the right profiling tool for game development companies and makes a small practical contribution on optimization of future games. KEYWORDS: Profiler, performance, C#, .NET framework, software development CONTENTS LIST OF ABBREVIATIONS 5 1 INTRODUCTION 6 2 CURRENT PROFILING APPROACHES AND TOOLS 8 2.1 Testing approaches 8 2.2 Testing levels and types 9 3 PROFILERS AND METHODS 12 3.1 The most common profilers include: 12 3.2 Methods 13 3.2.1 ANTS 13 3.2.2 CodeTrack 24 3.2.3 JetBrains 31 3.2.4 Visual Studio profiler 41 4 RESULTS AND DISCUSSION 50 5 CONCLUSION 55 REFERENCES 56 FIGURES Figure 1. Memory profiler main menu. 14 Figure 2. Memory profiler configuration. 15 Figure 3. Memory profiler results. 16 Figure 4. Memory profiler class list. 17 Figure 5. Memory profiler instance categorizer. 18 Figure 6. Performance profiler main menu. 19 Figure 7. Performance profiler configuration. 19 Figure 8. Performance profiler results. 20 Figure 9. Performance profiler results options. 21 Figure 10. Performance profiler low profiling overhead mode. 22 Figure 11. Performance profiler lowest profiling overhead mode. 23 Figure 12. CodeTrack main menu. 24 Figure 13. CodeTrack configuration. 25 Figure 14. CodeTrack sampling profiling option. 26 Figure 15. CodeTrack tracing profiling option. 27 Figure 16. CodeTrack deep trace profiling option. 27 Figure 17. CodeTrack results. 28 Figure 18. CodeTrack results in flame view. 29 Figure 19. CodeTrack results in list view. 29 Figure 20. CodeTrack results in timeline view. 30 Figure 21. dotMemory main menu. 31 Figure 22. dotMemory results. 32 Figure 23. dotMemory snapshot comparison. 33 Figure 24. dotMemory memory traffic. 33 Figure 25. dotMemory single snapshot. 34 Figure 26. dotTrace main menu. 35 Figure 27. dotTrace control panel. 36 Figure 28. dotTrace sampling snapshots. 37 Figure 29. dotTrace results. 37 Figure 30. dotTrace tracing snapshots. 38 Figure 31. dotTrace tracing results. 39 Figure 32. dotTrace tracing results in hot spots view. 39 Figure 33. dotTrace timeline results. 40 Figure 34. Visual Studio Diagnostic tools, CPU. 42 Figure 35. Visual Studio Diagnostic tools, Memory. 43 Figure 36. Visual Studio Diagnostic tools memory results. 44 Figure 37. Visual Studio Diagnostic tools CPU profiling results. 44 Figure 38. Visual Studio profiler target options. 45 Figure 39. Visual Studio profiler options. 46 Figure 40. Visual Studio profiler memory snapshots. 47 Figure 41. Visual Studio profiler memory snapshot data. 47 Figure 42. Visual Studio profiler CPU snapshot data. 48 Figure 43. Visual Studio profiler CPU call chain. 49 TABLES Table 1. Profiler comparison. 54 LIST OF ABBREVIATIONS CLR Common language runtime CPU Central processing unit GC Garbage collection GPU Graphics processing unit IDE Integrated development environment IIS Internet information services JIT Just-in-time RAM Random access memory UI User interface 6 1 INTRODUCTION Scheduled profiling in software development process increases the chance of enhancing slow sections of code (Bertolino & Faedo, 2007, p. 4). It is important to choose a reliable and convenient profiler for a profiling routine. The usage of a performance profiler will guide developers through resource-demanding sections of code, highlighting issues and potential improvements. To achieve an optimized product, performance analysis is advised (Ammann and Offutt 2008, pp. 3-4). Developers can tackle the problem by simply viewing the code, which is a very time- consuming and low efficiency method, or developers can use help of available profilers (Dolan & Moré, 2002, p. 1). “Performance profilers are software development tools designed to help you analyze the performance of your applications and improve poorly performing sections of code.” (SmartBear 2019) Performance analysis is part of software testing and forms part of the ANSI/IEEE 1059 software testing standard, which reads: “…a technique to validate application. The definition of testing is that testing is the process of analysing a software item to detect the differences between existing and required conditions (that is defects/errors/bugs) and to evaluate the features of the software item. The purpose of testing is verification, validation and error detection in order to find problems – and the purpose of finding those problems is to get them fixed.” There are two main types of software testing: functional and non-functional (Hailpern and Santhanam 2010, pp. 9-10). Each consists of subtypes whose purpose is to assess the system's compliance with its specified requirements in functional types and test a software for the requirements which are nonfunctional in nature but important such as performance, security, user interface etc. Method, used for most tests, includes test groups with isolated environment and limited knowledge about software, and test specialists with given requirements. For performance test, it is compulsory to have software to test performance impact. Nowadays there are so many options and this creates confusion among developers. The problem, discussed in this thesis, is how to choose the most relevant and trustful profiler. TURKU UNIVERSITY OF APPLIED SCIENCES THESIS | Aleksandr Osipov 7 The next chapter, Chapter 2 explains the current methods the software testing is performed. Each approach, level and type will be described in detail, without diverting from the main topic, which is a part of software performance testing. Chapter 3 describes the most recommended profilers, among different online resources. It also introduces existing problems and solutions alongside with methods, used to solve the problems and find optimal profiler. TURKU UNIVERSITY OF APPLIED SCIENCES THESIS | Aleksandr Osipov 8 2 CURRENT PROFILING APPROACHES AND TOOLS Profiling is a part of software testing. Many approaches exist in software testing, for example: static and dynamic approach, exploratory approach, white, grey and black box approach etc. Among the different approaches, there is a minimum of three distinctive levels of testing: unit testing, integration testing and system testing, and many software testing types or techniques such as compatibility, smoke and sanity, functional and non- functional testing. 2.1 Testing approaches Static testing is a manual or automated review of code to find errors at the early stage of development and it is carried out when the code is not executed, in contrast with dynamic testing where code is executed to determine resource usage, overall performance and confirm that software meets business requirements (Pan 1999). Exploratory testing is an important approach for agile projects (Jorgensen and Jorgensen 2006), because its aim to keep testers on the same page with developers of a fast evolving project. It emphasizes the engagement of testers, the lack of plan and freedom to choose the path for testing, only using one’s own sense of where the problems might lie. White, grey and black box approaches regulate the openness of software source code for testers. White box testing is completely open, also called glass box, transparent box, etc., so testers have access to source code and design documents, to help create test cases using exposed internal structure and systems. Black box testing implies a closed code base with only input and output data available for testers. The grey box approach is a combination of white and black box, which means that the testers have limited knowledge of the testing system such as working models and architecture diagram, but generally have detailed design documents available. The white box approach is commonly used at unit level although it can be applied at integration and system levels. Black box testing is applicable to any level of testing.