Distributed password cracking with BOINC and hashcat Radek Hranicky´a,∗, Luka´sˇ Zobala, Ondrejˇ Rysavˇ y´b, Dusanˇ Kola´rˇb aBrno University of Technology, Faculty of Information Technology, Department of Information Systems, Brno, Czech Republic bBrno University of Technology, Faculty of Information Technology, Department of Information Systems, IT4Innovations Centre of Excellence, Brno, Czech Republic Abstract Considering today’s challenges in digital forensics, for password cracking, distributed computing is a necessity. If we limit the selection of password-cracking tools strictly to open-source software, hashcat tool unambiguously wins in speed, repertory of supported hash formats, updates, and community support. Though hashcat itself is by design a single-machine solution, its interface makes it possible to use the tool as a base construction block of a larger distributed system. Creating a “distributed hashcat” which supports the maximum of hashcat’s original features requires a smart controller that employs different distribution strategies in different cases. In the paper, we show how to use BOINC framework to control a network of hashcat-equipped nodes and provide a working solution for performing different cracking attacks. We also provide experimental results of multiple cracking tasks to demonstrate the applicability of our approach. Last but not least, we compare our solution to an existing hashcat-based distributed tool - Hashtopolis. Keywords: hashcat, BOINC, cracking, distributed computing, GPGPU 1. Introduction b) the range of supported formats, c) supported attack modes, and d) portability to different platforms. From existing open- With the escalating use of computers and mobile devices, source software, we chose hashcat2, a self-proclaimed “World’s forensic investigators frequently face encrypted data which fastest password cracker” which is distributed under MIT li- could hide substantial evidence. Though General-purpose cense. Considering speed, team hashcat won 5 of 7 years of computing on graphics processing units (GPGPU) introduced Crack me if you can (CMIYC3) contest. Assessing features, a massive speedup to password cracking, the developers of hashcat supports over 200 different hash formats, and several software supporting data encryption tend to improve the pass- different attack modes: brute-force attack (also reffered to as word verification procedures, making the protection harder to mask attack), dictionary attack, combinator attack and hybrid be cracked [6]. For example, using hashcat and exhaustive attacks; moreover, it supports the use of password-mangling search (brute-force) with a single NVIDIA GTX 1080 Ti GPU, 4 1 rules including the ones used by popular John the Ripper tool. cracking MS Office 2013 documents is 37840x harder than Another reason for choosing hashcat was its support for dif- ≤ cracking older MS Office 2003 using SHA1 + RC4 algo- ferent operating systems, and hardware platforms. Hashcat de- rithms. Using 8 alphanumeric characters, one can create a to- velopers provide both sources, and pre-compiled binaries for ∗ 14 tal of 2:18 10 different passwords. Using a GPU denoted 32-bit, and 64-bit Windows, and Linux. Cracking with hash- above, a MS Office 2003 document can be cracked within 5 cat can be performed on various OpenCL-compatible CPUs, days, while cracking MS Office 2013 document may take up to GPUs, and even FPGAs, DSPs, and co-processors. 559 years. With the limited number of GPUs in a computer, As a framework for distributed computing, we used Berkeley cracking today’s password protection in a meaningful time may Open Infrastructure for Network Computing (BOINC5) which require employing a massive network of multi-GPU nodes. was initially designed [1] as a public-resource computing so- Our research aims at finding an open-source software solu- lution. However, in our previous research, we have shown its tion for fast and efficient distributed password cracking, which applicability in password cracking even in private networks [5]. could be easily deployed to any TCP/IP-based computer net- In our use-case, BOINC handles the authentication of comput- work made of commodity hardware. At first, we searched for an ing nodes, provides the distribution and automatic updates of application which could serve as a “cracking engine” of com- executable binaries, OpenCL kernels, and the input/output data putational nodes in a network. The key critieria were: a) speed, of each cracking task. ∗I am corresponding author Email addresses: [email protected] (Radek Hranicky),´ [email protected] (Luka´sˇ Zobal), [email protected] (Ondrejˇ 2https://hashcat.net/ Rysavˇ y),´ [email protected] (Dusanˇ Kola´r)ˇ 3https://contest.korelogic.com/ 1https://gist.github.com/epixoip/ 4http://www.openwall.com/john/ 973da7352f4cc005746c627527e4d073 5https://boinc.berkeley.edu/ Preprint submitted to Digital Investigation October 8, 2019 1.1. Contribution the hash only once, and compare the result with all the hashes By redesigning Fitcrack6 system [5], we created a solution we are trying to crack. for high-efficiency GPGPU password cracking using hashcat Bengtsson showed the practical use of MPI-based brute-force tool as a client-side computing engine. The offered solution em- and dictionary attack using Beowulf high-performance comput- ploys the BOINC framework to handle host management, net- ing (HPC) cluster for cracking MD5-based Unix shadow files work communication, and work assignment. For each hashcat’s and DES-based passwd files [3]. Both attacks were based on attack mode, we propose a convenient strategy for task distri- simple password-by-password keyspace division. bution to utilize the maximum of hardware resources. We show Apostal et al. brought another enhancement to HPC-based how dictionary segmentation, fine-grained adaptive scheduling, password cracking. The divided dictionary algorithm evenly and batch assignment of workunits help to reduce the overhead divided the dictionary words between MPI node equipped with of the entire cracking process. We experimentally proved that GPUs. Using CUDA, GPUs on each MPI node locally calcu- the new solution is capable of performing distributed attacks in lated the hashes and compared them with the ones which should a reliable and efficient way. Moreover, we compared our soft- be cracked [2]. ware with the Hashtopolis tool, underlying pros, and cons of Marks et al. designed a hybrid CPU/GPU cluster formed each solution. by devices from different vendors (Intel, AMD, NVIDIA) [12]. The design includes both hardware, and software solution. The 1.2. Structure of the paper distributed network consisted of Management/storage nodes which control the calculation and handle user inputs, and Com- Section 2 introduces the reader to the current findings in dis- putation nodes responsible for the work itself. For intercon- tributed password cracking. Section 3 describes the architecture nection, three different lines were used: 10 Gb/s Ethernet for of the proposed distributed cracking system. In section 4 we de- data transfer, 1 Gb/s Ethernet and Infiband for controlling the scribe how to use a distributed environment to perform the same computation process. Marks proposed a software framework attacks hashcat supports on a single machine. Section 5 shows called Hybrid GPU/CPU Cluster (HGPC) utilizing a master- experimental results supplemented by the comparison with our slave communication model using an XML-based protocol over previous solution, and Hashtopolis tool. Our research is con- TCP/IP network. A proof-of-concept implementation was able cluded by section 6 which brings together obtained experience to crack MD5, SHA-1, and four versions of SHA-2 hashes. and denotes possible future work. Experimental results of cracking on up to 24 nodes showed great power, and scalability. We, however, suppose that us- 2. Related work ing a time-proven tool of computation nodes would increase the performance even more. While cracking MD5 hashes on Much of related work is based on the popular John the Rip- NVIDIA Tesla M2050, Marks achieved the speed around 800 8 per (JtR) tool. Up to this day, the John’s wiki7 enlists 15 differ- Mh/s, while hashcat users report cracking over 1200 Mh/s us- ent approaches on parallel and distributed processing with the ing the same GPU. tool, some of them were later abandoned. The first published Previous solutions work well for a “classic HPC” system us- academic work on the case was performed by Lim who modi- ing a homogenous cluster with a static set of nodes which is, fied the sources by adding MPI support for incremental (brute- however, not our use-case. Using a simple text-based proto- force) cracking mode [11]. The solution used a master proces- col, loosely modeled on HTTP, Zonenberg created a distributed sor and a fixed number of slave processors. The master pro- solution for cracking MD5 hashes using a brute-force attack cessor divided the keyspace (a set of all password candidates) [19]. The architecture consisted of a master server and a set to a pre-defined number of chunks, while each slave processor of compute nodes, which were either CPU-based, or used GPU received an equal chunk to solve. The principle of keyspace acceleration based on CUDA. division was adopted in many subsequent solutions, however, Crumpacker came with the idea of using BOINC [1] to dis- with various alterations since Lim’s original technique is only tribute work, and implemented a proof-of-concept tool for dis- feasible for use in a stable, homogenous environment. tributed cracking with JtR [4]. Similarly as in our previous Pippin et al. then proposed the parallel dictionary attack for prototype [5], he decided to standardize the size of the worku- cracking multiple hashes [16]. Instead of dividing keyspace, nit (chunk in terms of BOINC) by counting out the number they assigned different hashes to each node while all nodes used of passwords that one computer could check in a reasonable the same password dictionary. We consider the approach to be amount of time. Crumpacker, however, reports he was unable efficient for large hashlists and simple hash algorithms only.