Know Your Opponent: My CCDC Toolbox

Alright guys! This has been several years in the works. Sorry for the delay (and the rambling that ensues in this post). Hope it helps!

A very common question @ CCDC events are “What tools did you use?” and “How did you hack us?”

I do use a number of off the shelf tools such as metasploit and nmap, but a good part of my work during the competition is done with custom tools. This is true for a number of us on the National Red Team, with each of us having particularly effective toolboxes. Every year, we take this further and further down the rabbit hole and the toolchain used this year across all red teamers was by far the most custom and effective. With the exception of a few collaborative efforts, I’m going to withhold speaking about other red teamer’s tools and attempt to focus on mine. But rest assured, what you see here is simply a glimpse into the arsenal you meet in San Antonio.

This has not been an easy post to write. The last few years I’ve been a big cagey about discussing these custom tools publicly. Unlike professional adversaries, we have to do much of this work in our spare time. CCDC is ultimately an educational experience, so I absolutely want to share as much as possible. My devil’s advocate position has been that the experience is part of the competition, and if I spill the beans on our tools, we won’t be able to provide the realistic threat you’ve come to expect.

It’s definitely a gray area, but I’m going to go ahead and talk about the tools I’ve spent the last few years working on. The toolbox gets divided into two categories: infrastructure and malware.

Keep in mind as we dive in, that I’ve only red teamed Western Regional and Nationals, with Pacific Rim last year being the exception. If you’ve never competed in any of those, then you likely have never seen these. I’m going to work this year on getting some parts of this tool chain distributed out to the regionals. Even if you have done those competitions, you likely haven’t seen everything in this list either. With CCDC environments vastly different from scenario to scenario, tool reliability is one of our biggest challenges. This makes making tools work a per competition mission. I can only prep so much, so they end up getting deployed in order of their importance. Also, I’ve intentionally withheld some details. I’ll leave those up to you guys to figure out 🙂

A Brief Note on Codenames

Anyone who’s worked with me knows that I love a good codename. We do a lot of R&D and codenames help keep projects in their lane. Sometimes the codename makes sense; less so for others. What you can expect is the following tools can be identified by CODENAME because they will be represented in ALLCAPS.

Also, I’ve attached random pictures from the inter webs to make this long, arduous post more entertaining.

So without further ado!

Infrastructure

Infrastructure tools are things that make us more efficient as a Red Team. These pieces of software run on our servers, never on blue team hosts. If you were at Nationals this year and saw the “Red Team Infrastructure” slide in Dave Cowen’s debrief, this is what he’s talking about.

This year, I migrated every infrastructure component to a service oriented architecture. With the exception of TRAPHOUSE, everything else follows SOA principles with HTTP based API’s for inter process communication. This allows these services to integrate with each other in ways that weren’t possible before, as well as deal with problems of scale more effectively.

TRAPHOUSE

Deathstar_negwt

This is where the fun begins. TRAPHOUSE is the main brain for the red team. TRAPHOUSE has three main components:

  • Web UI
  • Backend
  • API

TRAPHOUSE maintains source of truth state for most pieces of our infrastructure. We do a lot of things with TRAPHOUSE, but most importantly, we file our compromise reports there. Black Team has access and manages Red Team scoring based on the data in TRAPHOUSE. To give you a peek into its “brain”, here’s a screenshot of the entity-relation diagram for the PostgreSQL database behind it:

Screenshot 2017-05-08 15.58.08

Another key feature of TRAPHOUSE is its Pads. These are basically Etherpads within TRAPHOUSE that let us collaboratively share and store information between red teamers.

TRAPHOUSE has evolved over several years and is one of my oldest tools. It was the product of a marathon hacking session and until this year, was basically duct tape over duct tape. This year, it got a complete re-write with the help of some awesome contributors (Email me if you’re looking to hire Rails devs! They’re great!):

  • Reuben Brandt
  • Jonathon Nordquist
  • Jeremy Oltean

Together, we were able to take the old crappy code and make it far more modular and extensible. There will be more work to come on TRAPHOUSE, but it’s in a great place to continue to be a centerpiece of our red team activities.

The next big feature I’ll be shipping in TRAPHOUSE is a C2 API. The goal is that other red teamers can integrate their C2s into TRAPHOUSE for other people to interface with and use.

OG

OG

While a lot of stuff happens in TRAPHOUSE, the other interface into my tool chain is a Slack bot called OG. OG has extremely tight integration with TRAPHOUSE and is the main interface for most red teamers. You interact with OG with “!” commands, and OG is channel and user aware. You can invite him to rooms, private message him, etc. and he will always respond where appropriate.

OG also manages authentication and accounts for TRAPHOUSE. You register for TRAPHOUSE through OG, and OG won’t talk to you unless you’ve got a valid TRAPHOUSE account.

With inspiration from the “ChatOps” movement, I’ve attempted to make all my tools (especially the infrastructure) available to the rest of the red team via OG commands.

DOOBY

dooby

DOOBY might have been the most challenging piece of code I’ve worked on. ever. DOOBY is a custom DHCP Server that I wrote specifically for CCDC Red Teams. The solutions it solved over traditional DHCP servers:

  • It’s hard to manage the IPs you use as a Red Teamer in a CCDC environment.
  • Humans are not as good as computers at computing random numbers.
  • If two red teamers collide IPs, we lose time, and potentially shells.

As Blue Teams have gotten better with regard to network defenses, we’ve had to evolve too. It has taken a few years for DOOBY to be realized. DHCP is an incredibly complex protocol when you get into the nitty gritty, and implementing it from the ground up took countless hours of testing to get it stable.

DOOBY fixes some major red team pain points in several ways:

  • Records all IP leasing in TRAPHOUSE and ties it to the Red Teamer for correlation w/ their compromise reports
  • Distributes leases across several red team networks
  • Allows IPs to be rotated very rapidly without fear of stomping someone else’s IP

In addition to red teamer laptops and VMs, DOOBY manages addressing for my other infrastructure components as well. As you read on, keep that in mind. DOOBY is perhaps the least sexy, but one of the most integral pieces of our infrastructure besides TRAPHOUSE.

DOOBY got its first prime time action this year, but I’m looking forward to iterating on it. There’s still work to be done to make it more user friendly for other Red Teamers, but it supported much of this infrastructure during Nationals this year.

BORG

borg

BORG was born a few years ago in the wee hours of the morning before Western Regionals. I wanted some way to do MSF exploits from random IP addresses but have second stages be centrally managed, and since I’m lazy – let’s integrate it with Slack!

What we ended up with when the sun came up was a beast. BORG is a fleet of “Drone” VMs that do our bidding. We use BORG for all sorts of things – scans, exploit launching, and spraying. The Drone VM itself has several valuable tools preloaded on it:

  • metasploit
  • crackmapexec
  • spraywmi
  • nmap
  • sqlmap
  • database tools
  • snmp tools
  • phantomjs
  • + many more

TRAPHOUSE manages a work queue and the Drones continuously check in seeing if any work needs to be done. Now, when a red teamer wants something done, they simply issue a command in slack:

!borg run nmap -sS -T4 -Pn -oA team_6_quick 10.6.6.2-254

This queues the NMap scan job up. When a free worker retrieves it from the queue, it’ll hop IPs (using DOOBY), perform the task, and return all output (STDOUT and any new files) to TRAPHOUSE. OG then sends a notification to the Slack user who submitted the job with download links to their files.

GRID

thegrid

GRID is one of those tools when you explain it to people, they give you a funny look somewhere in between “u wot m8?” and “i cant even”. Together, BORG and GRID act as a yin and yang as our red team “bot net”. Where BORG “throws” things from a distributed block of IPs, GRID “catches”.

In one sentence, GRID is a piece of physical hardware which acts as tens of thousands of infected hosts on the network. These “hosts” can proxy call backs and serve malicious files.

GRID has been an incredible piece of technology for us. With GRID effectively acting as a multi-tier, distributed, C2, red teamers can host their implant C2s on static IPs provided by DOOBY. With GRID, they never have to change that IP, and you as the blue team never have direct contact. The C2 remains effectively hidden in the tens of thousands of IPs within GRID’s network.

This abomination of CCDC technology actually has several components that make this possible, so I’ll outline them below.

GRID-KERNEL

GRID distributes itself on the network by creating tens of thousands of virtualized network interfaces. This requires a custom Linux kernel that we’ve named the “GRID-KERNEL”. Getting this stable was one of the hardest parts of GRID due to extremely slow development loop needed to test this.

GRID-CONFIG

The next step of the stack is configuration utility. This tool is what GRID uses to bootstrap its network interfaces. It talks over the SOA backbone to DOOBY and TRAPHOUSE to get a block of permanent addresses for the competition (one for each of its roughly 50,000 interfaces).

These addresses are then blacklisted in DOOBY from being leased. TRAPHOUSE also holds a record of them for black team correlation and for other things we’ll talk about later.

GRID-PROXY

The next major component of GRID is its layer 4 proxy. From within Slack, a red teamer can “reserve” frontend ports in GRID and have them reverse proxy back to them:

!grid forward --frontend_port 1337 --backend_ip 10.0.0.123 --backend_port 4444

This command will now listen on port 1337 across GRID’s network interfaces and forward the traffic to 10.0.0.123:4444. Now when a red teamer needs to know a call back IP:

!grid ip

OG will grab a random IP from GRID’s IP table in TRAPHOUSE and return it to Slack. The !grid command in Slack is extensive with several more capabilities. It is completely self service for the red team and supports both TCP and UDP.

GRID-CDN

There are a few ports that are reserved in GRIDs proxy. Two of them are 80 and 443. These ports actually get reversed proxied to GRID’s CDN service. The CDN service acts as a layer 7 router sending traffic matching rules to pre-defined places. As of now, this is defined by me at game time, but next year I’ll integrate this into OG for layer 7 HTTP self service.

If none of the routes are matched though, GRID then attempts a lookup against a local DB for a matching URI. If the URI is known, the CDN will lookup the corresponding file in its CDN Origin and serve it.

This allows us to “push” files to GRID and retrieve them in a distributed fashion. The file object can be referenced across many of my tools for interpolation with its GRID file ID. An example of this might look like:

!grid host --url https://my.super/leet.malware --name dabomb

OG will push a CDN cache job to TRAPHOUSE, which will make an SOA call to GRID’s API. GRID will download the file at the specified –url, hash it, and keep a reference using the –name field. It will generate a shortcode for you and pass it back to TRAPHOUSE, which OG will pick up and respond to you with the following:

http://[RANDOM_GRID_IP]/[RANDOM_SHORTCODE]

That file can now be retrieved from anywhere in the CCDC network using that URL. If Dan wants a different URL, he can simply issue the command:

!grid link --name dabomb

and he will get a new URL with a different IP and different short code. A number of options exist too, such as DNS hostnames, ports, # of downloads you want the file to be available for, etc.

GRID’s CDN gives us totally ephemeral staging of binaries and payloads. When a stage 2 needs to be retrieved on an infected host, it can pull from an IP address that you’ve never seen, and likely won’t again.

Over the last few years, blue teams have become more and more effective at network defenses.

While the CCDC scenario is realistic in many ways, one of the ways blue teams have found ways to gain an advantage is in monitoring network traffic. BORG and GRID give us the capability to completely negate this tactic. I’ve yet to see a blue team effectively defend against this.

GRID-CRYPTO

The final component of GRID deals with how GRID manages cryptographic material. TCP443 is a reserved port on GRID and serves a wide variety of HTTP virtual hosts on it. GRID this year was resolvable by 67 different domains that I own, each with unique and interesting subdomains. It was too much work to manually get HTTPS certificates for all of them, so I wrote a piece of code to use AWS Route53 and and EC2 instance to automatically generate valid SSL certificates for all domains and their subdomains. This tool would then spit out a configuration file for GRID that it could then use to serve valid HTTPS on requests that matched the correct Host parameter.

THEFOOT

thefoot

In order for GRID to get DNS resolution, we needed a way to dynamically serve GRID IPs via DNS. We originally ran a complicated chain of BIND servers, but overnight at Western Regionals this year, I wrote THEFOOT. THEFOOT is a custom DNS server that very simply serves a random GRID IP for any valid GRID domain. It retrieves the DNS zones at startup from TRAPHOUSE and responds to any recursive query for those domains, so we set the NS records of all our domains at THEFOOT’s IP during the competition and any resolver would return competition IPs to you.

RTDNS

rtdns

RTDNS is a separate DNS server that red teamers can choose to point to when using DOOBY. In many CCDC environments, every team has the same FQDN “foo.ccdc” for example. This means that to resolve “http://shop.foo.ccdc” in the browser, we have to point to your DNS server.

RTDNS solves this by letting a red teamer “lock” their resolution for a domain to a specific upstream resolver. This allows them to stick to their team’s DNS for HTTP/S resolution.

In previous years, we’ve actually run individual RTDNS servers for each team in Docker containers, but by 2018, RTDNS will be its own stand alone singular application with Slack Integration.

MRWIZARD

mrwizard

MRWIZARD is a distributed scanner made by Lucas Morris. MRWIZARD continuously runs a variety of port and service discovery scans against all the teams from a giant block of IPs (similar to how GRID manages its IPs).

MRWIZARD then uploads its scans to TRAPHOUSE for red teamers to grab when they need it. It also watches services and when new services are seen, it alerts TRAPHOUSE to this wonderful new discovery.

CRACKLORD

foxconn

This is another Lucas Morris and Crow creation. CRACKLORD is an on demand hash cracking system. Maus used his DevOps magic and worked closely with Lucas to get a CRACKLOARD  autoscaling EC2 version deployed for this year. The more hashes you want to give us, the more we’ll crack!!!

Currently in progress is CRACKLORD integration with OG and TRAPHOUSE so that cracking jobs can be scheduled directly from Slack as well as hash lookups from previously cracked hashes.

FOXCONN

cracklord

This was something Maus and I dreamed up years ago that has fallen a bit stale as of late, but is something we’ve used successfully. FOXCONN basically acts as an MSFVENOM server that generates MSFVENOM payloads over an HTTP API. It even has the ability to generate a payload based on request user-agent, so that whenever you get on a box, you can just hit the FOXCONN API and you’ll get returned a payload for that platform.

Now that we’ve re-architected TRAPHOUSE and OG, FOXCONN is up next for its makeover with even more exciting features.

BANEX

BANEX

To test much of our infrastructure, we employed a huge VMWare infrastructure. BANEX is our vSphere cluster with several dozen VMs of varying operating systems and architectures. We use this as a firing range to test our payloads and tools.

Custom Malware

These are the custom pieces of software I’ve written over the years to stay on your boxes and give you a seriously hard challenge.

GOOBY

gooby

In order to seriously decrease our coding effort, Dan, Vyrus, and myself converted almost all of our malware tools and techniques into shared libraries which could be included on a per function basis and compiled into our implants.

GOOBY is the base library that all of these extend from. Most of the tools below, are actually library functions implemented in GOOBY. This allows implants to share modules depending on what purpose we want the implant to serve.

For individual binaries, they’ll have their own build scripts that will (usually) statically compile the included GOOBY modules into the final implant.

GENESIS

genesis

Genesis is the successor to the much alluded to [REDACTED] tool which acted as a all-in-one persistence script. One common technique we use is “bundling” of payloads so that when exploitation occurs, you can run a single payload and get multiple pieces of persistence deployed. [REDACTED] was the product of one of the darkest of wizards on our Red Team. It evolved over the years to support both *nix and Windows.

This year, Dan, Lucas, Vyrus, and myself decided to test some new, unique features and created GENESIS. GENESIS is a small program that can be dynamically built and obfuscated with an arbitrary number of payloads included in it. This allowed us to abstract much of our local persistence techniques that originally lived in the [REDACTED] code into GOOBY, for easy use by any of our payloads.

We spent a lot of time working out obfuscation and anti-RE techniques into GENESIS. This is extremely important because now if a GENESIS binary is intercepted, significant effort will be needed in order to determine the other payload it encompasses. We’re lagging behind on some platforms in this area, but by next year we expect GENESIS and its supported obfuscation to be broadly supported.

Huge huge props to original developer (he knows who he is) and all the contributors to this effort over the years.

AMBER

amber

AMBER is a custom shellcodeexec I wrote in San Antonio this year. I wrote my own for two reasons:

  1. Many shellcodeexec implementations are easily detected by AV (we don’t want that!)
  2. AMBER is implemented as a GOOBY module for easy use in other implants

AMBER also supports in memory process injection and lets us migrate payloads into other processes with only a few lines of code. Extremely handy.

EXODUS

exodus

EXODUS is a library that builds on top of AMBER to perform similar functionality as GENESIS, but with shellcode instead of persistence.

In essence, EXODUS allows the operator to “bundle” an arbitrary group of shellcode payloads and create a loader which will execute the shellcode, either spawning a new process or injecting into an existing one (operator configurable).

This gives us the same “spray” capability that GENESIS gives, but with raw shellcode.

QBERT

qbert

You can think of QBERT as somewhere between a rootkit and the void of space. It’s extremely difficult to hunt down, it leaves very little signatures, and QBERTs sole purpose is to keep other GOOBY based malware persistent on the infected host. It does this through a variety of techniques that span OS functions, but suffice to say, it’s one of the dirtiest tools in the box. As of this blog post, it still is almost entirely undetectable by any major IDS/AV/Audit vendor.

GEMINI

gemini

GEMINI is an implant I’ve slowly developed over the last few years to keep me in systems when most other persistence techniques have been discovered. It doesn’t have a ton of functionality, but it’s small and extremely portable – it cross compiles to basically any operating system.

GENESIS allows access to the infected hosts filesystem and command execution by the operator and is also the first Implant that used the GRID network for its C2 callbacks. GEMINI will assess network conditions on the infected host to determine what time of covert channel to use for a call back.

The callbacks are asynchronous, so as an operator, you have to consider lag time between when you issue a command and when GEMINI comes home, but it’s been a reliable implant. A huge thank you to Vyrus for putting in countless hours improving its capabilities this year.

Vyrus and I also wrote a SPEC for generic C2 communication. This new protocol will be the basis for many different implants moving forward and the foundation for extending C2 interaction to OG and TRAPHOUSE.

WONKA

wonka

Dan talks often about his troll payloads, but I have a few of my own. WONKA historically has been a text file of special *nix local persistence techniques. Linux systems at CCDC events have a tendency to be very specific in their build and configuration, so I’d just keep the text file open and pick and choose what little one liners to deploy at competition time.

This year, I started the process of converting WONKA into a GOOBY module and adding meta wrappers around it. Some of the functions you might have seen if you’ve encountered WONKA:

  • “BASHLOTTO” – A bash trick that will lottery a command entered into BASH for a potentially other command. (Usually 25% chance on the dice roll)
  • “ANTIEDIT” – A custom wrapper around common editors that reverts changes on files you change.

There’s several more, but that gives you a few basic ideas of how WONKA causes you headaches during the competition.

I’m a big believer in WONKA because I think troubleshooting and performing root cause analysis is actually incredibly difficult. WONKA’s goal is to challenge you to think critically about how your systems may or may not be behaving and to validate assumptions about your tools.

SSHELLY

sshelly

SSHELLY is a persistence technique that attacks the OpenSSH server. SSHELLY has several options for nefarious behavior, two of which are:

  • “GODKEY” – Allow access for any user with a specific SSH key
  • “LOUDPACK” – Saves password credentials for users in memory. When a specific backdoor user connects, it dumps the credentials and disconnects the client.

SSHELLY still has a few bugs here and there, but it’s maturing nicely and is an amazing complement to my toolbox. By deploying SSHELLY, many linux hosts which must maintain a running SSH service are now able to be persisted in a difficult to detect manner.

ALFRED

alfred

Every year, there are things I want to do in a recurring manner on your systems. While cron and Scheduled Tasks are reliable, they’re easily detected. ALFRED is my own lightweight “cron” runner. I can either compile tasks into the ALFRED binary, or leave a config file on disk for ALFRED to search and find.

While it’s not functional yet, I’m experimenting with memory cradles that will allow other implants to schedule tasks simply by writing to a shared memory segment.

HALITOSIS

halitosis

I’ve been playing with audio/video drivers at CCDC competitions for a few years now. The obscure operating systems and the lack of packages generally makes intereacting with peripherials challenging. HALITOSIS is my GOOBY module that for listening to microphones on your system. I’m still in the process of porting much of my old code into the new GOOBY module, but it’s a great tool for listening in on whats going on in the blue team rooms.

TATTLER

k668-VKJ

Network visibility is something we spend a lot of time working on during the competition. Teams often put up firewalls and NAT their internal networks. This makes scanning from our external position difficult. TATTLER is a GOOBY module designed to perform adhoc port scanning and service discovery of your networks FROM THE INSIDE. It’s not as feature rich as NMap, but it’s extremely fast, portable, and configurable based on the operator’s desire. Any GOOBY based implant now has access to theTATTLER library to perform recon on otherwise unknown network segments.

MOTHRA

mothra

To keep it short and sweet (and not reveal too much), MOTHRA is fly paper for system credentials. At less than 1k, it’s definitely my smallest payload, but it’s one of the most deadly. It was only in use for a few hours at the end of day 2 NCCDC, and it managed to gather over 500 UNIQUE credentials.

Tony, you know you da man. This is going to be a very fun tool for years to come.

 Conclusion

So there you have it. On top of the industry tools (Metasploit, Cobalt Strike, Empire, etc.), we have to bring custom tools. If we didn’t develop tools like these, you wouldn’t be facing a realistic simulation of a world class adversary. (Looking at you CIA *cough*VAULT7*cough*). As put best in The Art Of War:

So it is said that if you know your enemies and know yourself, you will not be put at risk even in a hundred battles.
If you only know yourself, but not your opponent, you may win or may lose.
If you know neither yourself nor your enemy, you will always endanger yourself.

I know it’s not a tell all, but hopefully this gives you some insights into the lengths we go to challenge you. This is exactly why CCDC is the best learning environment for information security.

I cannot, cannot, cannot stress enough how awesome it is to work with some of the world’s leading experts in information security. Between the pull requests and the mentorship, working on these systems with my fellow red teamers is some of the most fun I get to have all year. Shoutouts to several brethren who contributed to these tools:

  • ahhh
  • vyrus
  • deadmaus_
  • cmc
  • emperorcow
  • javuto
  • tony

With that, I’ll leave you guys to prep for next year! Best of luck >:)

Attacker Mindset 101 — Welcome to the Red Team

Security engineers who possess red team skillsets are the most powerful offensive weapon toward uncovering risk across an organization — but they must also know the technology basics in order to be most effective.

A dedicated group that assess security methodologies through adversarial simulation is commonly referred to as a “red team.” These exercises are a hallmark of the information security lifecycle due to their ability to identify systemic risk that had not been taken into account. This model provides incredibly valuable insights to security teams. From testing security controls to assessing the human element, the ethos of a red team embodies reflection, awareness, and a constant determination to disprove assertion through real world action.

In security, we use the Johari Window to analyze the delta between our systems defenses and from what we’re defending our systems. By bucketing threats in this way, we gain better understanding of our weaknesses, which ultimately make us safer and more secure.

The most difficult quadrant to defend against is the “blind spot.” In the last few years, we’ve witnessed some of the world’s largest organizations become compromised in overwhelming fashion. In 2010, a high-security French bank was breached by unknown attackers in ways the bank had previously considered impossible. This is just one of several examples where the cunning criminal drastically outsmarted well-funded, and frankly overly confident, defenses.

So, how can companies develop defenses for complex risk that, by definition, they have no awareness?

Enter the world of the red team.

A red team’s mission is to simulate threats to identify previously unknown and unmitigated risk. Through a unique blend of adversarial engineering and standard penetration testing methodologies, red team exercises are meant to challenge your organization’s belief in the strength of their security. By emulating a wide array of threat actors, red teams bring perspective uniquely suited to the iterative development models commonly practiced in information security programs.

As my career has progressed, I’ve developed a firm belief that security engineering candidates should possess red team skillsets. It’s often, and truly, said that the best defender is also the best attacker. We live in a world where nerds are attempting to outsmart criminals — being able to embody an attacker’s mindset is a top priority. Even if your organization doesn’t have an official red team, your security engineering team will be much more effective and strategic if required to hone red team skillsets. Following the practice of screening candidates for these skillsets will help you develop a stronger team that will uncover issues at a reliable pace.

One caveat: your work will include much more than red teaming. Eventually, you’ll have engineers scattered across your security organization with a fantastic ability to uncover risk. Some of them might be building logging pipelines, while others are doing forensics. Fortunately, there are amazing opportunities that enable your team to continue to sharpen their skills.

One of the pinnacle events for me is the annual Collegiate Cyber Defense Competition (CCDC). CCDC is a purely defensive security competition designed to measure college students’ abilities to protect and secure information systems. During the event, competing teams of eight students, known as a blue team, are placed in identical enterprise-grade networks that are configured to emulate a real company. The student teams must then race against the clock to secure the systems and maintain the integrity of the services they provide.

In order to make this work, the competition recruits a singular red team to emulate real world threats. The red team will measure the effectiveness of both the competitors’ security methodologies, as well as conduct gap analysis to identify vulnerabilities the students have missed. These are documented and factored heavily into the overall score and outcome of the competition.

CCDC is nationwide, with everything from state-level competitions to the national championship hosted by the Center for Information Assurance and Security. They employ several full-time staff to design and architect the complex infrastructure required for the event. Last weekend was my sixth time red teaming the National Championship. I’ve used this opportunity over the years to improve my capabilities, regardless of my day job. While the blue teams spend the months leading up to the competition practicing security hardening and vulnerability management, I take the opportunity to improve my skills by studying cutting edge offensive security techniques. These skills are invaluable, for both the competition and my day job.

One of the questions I am asked the most is, “How do I get into red teaming?” or “How do I become red team?” Red teaming has such broad appeal in our industry, and it’s a hotter topic than ever right now. It’s also a bit taboo because, after all, you are breaking rules to expose flaws, so people are even more drawn to it. Attacks are often mystifying because, as stated previously, the unknown unknown will leave the mind wandering, attempting to uncover the methods.

Thusly, I’ve started tweaking my response to that question. One of the most common mores I witness is how worshipped offensive security is by a large segment of our community. With hacker culture moving from underground to mainstream populism in society, the security community has evolved into a global industry. This has shifted the power of balance in a number of ways. The Pareto Principle is stretched incredibly thin, with the vast majority consuming tools and software produced by a small but growing group. This phenomena leads to misconceptions about what is really involved in participating in a red team.

As valuable as it is to able to channel the attacker’s mindset, it is incredibly beneficial to be able to develop your tools. No effective outcome has ever occurred from leveraging only offensive tactics before developing more basic, albeit seemingly mundane in comparison, skillsets. Writing effective software takes both strong muscle memory and lots of practice. Learn to embrace the less glamorous, often overlooked skills. Developing software engineering skillsets, including a deep knowledge of computer systems, libraries, and protocols will act as a force multiplier in your security career.

I’d also strongly encourage people with those questions to really understand the effort that goes into red teaming. Dave Cowen, our NCCDC red team captain, and I were chatting after the competition and talked about the traits that really make a strong red teamer. These are the four:

Knowledge – Possess a broad understanding of the systems and technology you’re attempting to circumvent. This is where I strongly encourage folks to study and read about emerging technology. The more you’re familiar with databases and web servers and routers, the better you’ll be able to understand their weaknesses.

Methodology – Adhere to a methodology that best sets you up for success. Red teaming can be frustrating and if you abandon your principles, you’re going to end up down the wrong rabbit hole with nothing to show for it.

Agility – Adapt to chaos. In the real world, things change for thousands of reasons. It’s not your job to figure out why, it’s your job to adapt and continue moving forward.

Grit – Persevere. I can say first-hand the difference between success and failure can be measured in grit. Though last on the list, this is most important.

For example, when I was at Lares, I learned so much from Chris and Eric. The most valuable lesson of all was learning to stand up to adversity and see the job through. I’ve never met a more determined group of individuals and the results speak for themselves. Again, at Western Regionals, while everyone was hunting vulnerabilities, I sat and made wordlists by hand. I scraped IMDB for potential usernames. I used awk and sed to generate the wordlists. I narrowed a password list down. Miraculously, one of them worked. It was the much-needed entry point to our first shell of the competition. It wasn’t pretty, it wasn’t glamorous — but it got the job done by the force of sheer will.

I want to see more companies embrace the practice of red teaming as one of learning and self assessment. I want to see our community stop glorifying red teaming as the end all, be all. We only cover one quadrant of that Johari Window. There are three times the amount of work needed to cover the other three. Don’t follow the crowd, be yourself. The required grit I wrote about above will show success in any career path you take. I promise.

I’m working on a more detailed, technical post about the infrastructure tech we built for red team this year. You probably saw in Dave’s NCCDC debrief about our infrastructure and were wondering “WTF?” I’ll try and answer your burning questions in that post set to come out in the next week or so.
Cheers!

Alex

CCDC 2015 Debrief: Red Team Identity

Welcome back folks! It’s been almost a year since I updated this, but given that the Collegiate Cyber Defense Competition (CCDC) has wrapped up for this year, a new post is needed.

This was an important year for the competition. It was the 10th anniversary and Dwayne, Kevin, and Jessica put forth the best competition I’ve experienced yet. They’ve refined their craft over the years and the final product has become the pinnacle information security competition event in the country in my opinion.

For me, it marks my 11th CCDC event and 8th time red teaming. The last four years, I’ve red teamed both the Western Regionals and National competitions. As those hours have added up, I’ve thought a lot about the dynamics of the red team. We have a rather complex relationship with the whole event that I think should be better understood. A few of the hats we wear while at the event are:

Simulated Adversaries

The competition couldn’t claim to be a test of defenses without an offense to face off against it. The red team must effectively hold a realistic adversarial position against the competitors. We are comprised mostly of experienced information security professionals who use their skills and knowledge to emulate a top skill attacker on the network.

The best red team members not only bring technical knowledge to the table, but a metaphorical mask by which they become this digital assassin, executing attacks with precision and ingenuity. When you sit down, you are the bad guy. You are the adversary. A speaker during the closing ceremony today described the red team at Nationals as a “Motley Crew” of mad hackers. It’s true. We have different backgrounds and mindsets, but when you put us in a room together we become the enemy in fierce pursuit of the blue teams.

For me, that passion is fueled by a desire to demonstrate a world class threat. It’s rare you get to experience an attacker first hand outside of a real breach. As a former competitor, I can tell you that’s a huge selling point of the event to blue team members.  No longer are you inside of a lab at school or messing around with virtual machines at home. A real person who’s job is to infiltrate your systems is sitting on the other end of a wire and if you don’t act, it will have consequences.

Which brings me to my next job of red team:

Human Scoring Engines

Our impact on competitors has direct consequences to the competition. A successful breach will cause a competitor to loose points. That means we have to take our job every bit as seriously as the white team. Our Motley Crew might appear to be a band of trolls to an outsider, but we fully understand our ethical responsibility in fairly and accurately scoring the competition. Underneath the fun, there is stone cold dedication to the success of the competition.

While often the term “RED VS BLUE” is used to describe the competition, I find it to be inaccurate. This competition is about BLUE VS BLUE. The red team is no more competing than the white or the black team is. While we wear the mask of the hackers, we are there as a learning simulation with scoring implications. Period.

But maybe the most important role we take on is as…

Mentors

A large part of the reason the concept of RED VS BLUE continues to persist is the camaraderie weaved throughout the event. Those of us who have previously been blue team, have come through the program and become friends and co-workers with the red team. I don’t believe that kinship would exist without the fun and humor of the back and forth between the blue teams and the red team. Every year we see pictures in debriefs of funny hacks and situations that we can all laugh about for years to come.

That builds just the sort of relationships that blue teams need. In the end, we’re there to help you learn and develop your skills that will ultimately define your career. It’s no secret that we exist in a small industry and building connections to the red team has incredible benefits, regardless of the outcome of the event.

When I blue team’d at Nationals, we didn’t even place, yet the connections I made there are now directly responsible for my employment with Lares. I encourage every competitor to engage with the red team.We are there to support you, your career, and share in the memories of the competition with you. Next time you see us hanging around a table in the hotel late at night after the competition ends, come hang out and lets talk about the entire experience.

Which at the end of the day is what this competition is all about.We might be rebels, loud and opinionated, but we’re there to fill the void that no one else can and volunteer an experience that competitors year after year will find no where else.

Conclusion

Thanks again to EVERYONE involved – white, black, red, blue, orange, gold teams – this competition wouldn’t be what it is without it. Congratulations to the University of Central Florida for their 2nd win in a row! Now my work begins planning and building new stuff for next year.

CCDC 2014 Year in Review

As I’ve been sitting on this plane heading back to San Francisco from the 2014 National Collegiate Cyber Defense Competition (CCDC), I’ve been trying to come up with one word to encompass my views on this event. This will conclude my fifth year involved in CCDC. The first two I spent as a blue teamer; the latter three as a member of the regional and national red teams.  What is it about CCDC that keeps bringing me back? I’m not getting paid nor am I hiring anyone. Yet, I keep returning year after year. I think I get it now.

passion |ˈpaSHən|
noun
• an intense desire or enthusiasm for something

Thats the word I’m looking for. Passion.

From my very first year competing, there was something indefinable that pushed me to pour every ounce I had into the competition. Even though at that time I was a junior member of our team, the desire to learn as much as possible was very real. I think this is what professional athletes must experience. There is something beyond your skill that keeps you up at night, fueling your train down the tracks.

And I think the reason CCDC and I have such a strong bond is because it’s one of the rare places where others bring that same energy to the table. All teams (white, red, black, orange, blue, gold) and sponsors are continuously adding to that pool, mixing and stirring a fervency you simply can’t find anywhere else.

This is why the competition is getting better from all angles. The tradecraft is getting refined by all parties. It’s an incredible effort to be a part of.

On that note, I’d like to now take some time and give some positive feedback to individual sets of those involved (both regionally and nationally).

To red teams:

Don’t get complacent in your duty. The blue teams are getting better and as they do, so will we. Our participation, while energetic, is only a single component to this competition. Others are expecting you to challenge blue teams in a way only you can do. Embrace that and play your part. Contribute and coordinate – the more you do, the better your red team will be.

To white/black/gold teams:

Continue to make the competition as real world as you can. It’s easy for those of us who have worked in the security industry to understand what a real corporate environment is like, but there is a very real possibility a blue teamer has had zero exposure, and therefore zero understanding. Even in the immense differences CCDC environments have versus a real enterprise, it is still a goal worth chasing because that is what makes CCDC participants valuable. They’ve had that exposure to something other than a class room or text book in a situation that will push them to think critically under pressure.

To blue teams:

Don’t worry about the results. The most valuable part of CCDC is not the trophy – its the experience. You spent months preparing for this – learning every moment along the way. As someone who’s been in your shoes, let me tell you – thats the true prize. The knowledge you’ll walk away with is priceless. Yes, the competition side is fun especially when you develop a rivalry with another team, but once you graduate and are out, you’ll be working side by side with the same people you competed against and who won or lost will seem trivial. Focus on getting better, and be proud of yourself and your team when you do.

To sponsors:

Thank you for your support. Without you, all CCDC events would not be possible. My only feed back to you is this: When you send volunteers to the competition, who you send will reflect on you far more than the amount on the check. And that image will been seen by more than just competitors – other sponsors, red teamers, and other volunteers will pick up on it.

One of the best examples of this I’ve seen was FireEye‘s choice for their red team seat at Nationals. They sent Dan Borges, a young, but talented engineer, to fill that seat. They picked someone who possessed passion just as I talked about above – and that passion drove Dan to produce tangible results in his position. That dedication can only come from someone who deeply understands what CCDC is about – something Dan can bring to the table given his previous security competition experience.

I encourage all sponsors to take seriously who they send – recruiters, management, and even technical folks. You want to connect with the participants. Having someone like Dan who can connect and relate to the competitors is the single best contribution you can make to the competition.

To those not involved:

Seriously ask yourself why.

If you’re faculty or an administrator at a college which doesn’t participate – what are you waiting for? The fact that schools such as Stanford and MIT who have long been attributed to the best and the brightest don’t participate is disappointing. Get on board. You will very quickly learn whether your curriculum is adequately preparing graduates for jobs in the information security space (likely not).

If you’re an organization, what is stopping you? I’m looking at all the companies out here in the Silicon Valley. I’m shocked that you don’t throw more support behind this. Your fast moving companies are on the forefront of technological innovations, yet you don’t tap into one of the deepest veins for young, fresh security talent. It’s not just about filling positions. This is the next generation of professionals in the security industry. Connect with them in a rich, meaningful way.

Lastly, to myself:

Keep giving into that passion. Every year I come through with more skills than previously, and it’s only making me better. I have no certifications worth mentioning nor do I frequent regularly frequent conferences. CCDC is my venue to give back to those who have made me better. I have to give a huge thanks to my employer Lares Consulting for all their contributions to me and CCDC as well. Chris and others have been strong supporters of CCDC events both regionally and nationally. Without question, they supported my desire to contribute as much as I could because they have recognized the theme of this post: passion. These are the most passionate group of people I’ve ever worked with and I couldn’t be happier.

Truth be told, I wouldn’t have the opportunity to work with them today if it wasn’t for a CCDC event four years ago.

 

And that concludes the 2014 CCDC season. Congratulations to University of Central Florida on your win, and huge kudos to everyone else involved. Time for me to go back to the drawing board and begin cooking up new ideas for next year.

Times to Let Pass

In the last six months, my two remaining grandparents, my dad’s dad, and my mom’s mom, have passed away. Both of them were fantastic individuals, dedicated to their families. They couldn’t have been more opposite if they tried. Grandpa “Buzz” was a scholastic enthusiast, constantly pushing everyone around him to learn more. “Mimi” was the epitome of the “Southern Belle”, a classic woman born and raised in Alabama who’s spirit was always vibrant. One would feed me books, the other, biscuits and gravy. Their presence in my life now forever seared into my memory. The good, the bad, and the ugly.

In a few months, I’ll hit 25 in age. According to Wikipedia, life expectancy in the US is a little over 77 years old. Take that down to 75 to account for my less than acceptable habits, and I’ve just marked the first third of my life complete. The three generations I grew up with has now reduced down to two.

The question I keep pondering is by what metric is our satisfaction derived from? Accomplishments? Experiences? Financial stability? I think it’s obvious that each of us has to figure that out for ourselves, but do we really contemplate it? Nevertheless act upon its guidance.

I don’t think I’ve figured out my answer yet. It could be something that evolves or that you constantly chase? It’s a question to at least try and get out in front of.

As I sat next to Mimi on her final days, it became clear to me that when you find yourself approaching deaths doorstep, finding salvation might be the most important thing. That salvation isn’t simply a religious experience, although I don’t deny it is the most critical part to some. I put myself in her shoes over and over again. “What would I tell myself? How would I come to terms with the situation?” She loved her life, just as I do. Putting the macho-ness of our masculine society in it’s place for a moment, I’m going to say that I believe the vast majority of us aren’t prepared for that reality. Witnessing someone else I deeply loved and cared for go through that was a wake up call for me.

I’ve made the choice to live a life of broad experiences. As I passively survey my Facebook friends I grew up with, it becomes clear I’ve taken the plunge out of the nest of normality. That’s not to say I’m doing better or worse than others – it just happened to be the decision I made. Look at the United States in the late 1700s. Much of the country continued to develop along the eastern seaboard while others, for whatever their reasoning was, moved west into the unknown. Regardless of the motivation, I deeply understand that thirst.

As the last 25 years has been one hell of a ride, I now look to the future. My salvation lies in that quest to charter the unknown, to brave the storm, and come out on the other side into the oasis untraveled. The wilderness is calling. Time to do what I do best.

Stay tuned for future updates.

CCDC 2013 Red Team Feedback

Thanks to the efforts of so many people, we’ve finished another great Collegiate Cyber Defense Competition season. I was fortunate to be invited to take part of the Western Regional and National red team. Being a former blue teamer, I thought it’d be worthwhile to share some feedback from my red team point of view. While related to CCDC, many of these points have been valuable for me in the work place.

Disclosure: My attacks targeted *nix systems, as well as web and databases, so my feedback will be scoped as such. The principles though should apply across the board.

Good Operations are a MUST

Any time I’m at a USCC or CCDC event, I ask the competitors about what they want to do. The most common answer I get is “I want to do penetration testing” or some other form of “offensive” security. While I don’t deny it’s a fun area within security, the parity of offensive and defensive skills cannot be underestimated. If you want to attack a network, show me you can defend one. Period. So how do you do that at CCDC?

When I was on blue team, I debated with others whether it was more valuable to be good at security or good at systems administration. Having now gone out into the field and worked as both an operations and a security person, I think I’ve settled on a conclusion.

They are symbiotic, and without one, the other WILL struggle.

You should practice being an operator of systems. Run linux at home. Use it. I can’t tell you how many times at CCDC I’ve watched a blue teamer Google search (while we watch through VNC) something like:

– How to reset MySQL password
– How to stop a service on Fedora
– How to install a package on FreeBSD

If you find yourself doing this, then you need to spend more time learning how common operating systems and network services work. To secure a system, you’ll have to know how that system works. You’ll never be effective until you do. Here is a short list you should know how to do without Googling on multiple OSes:

– OS User Administration: Users, Groups, Sudo, Permissions, Change Passwords
– Remote Access: OpenSSH Server, FTPd, VNC/RDP, Define ACLs
– Database Access Control: Change Passwords, Set Host Masks, Investigate possible PII

You should know how to use operating systems just as you know how to use Facebook or your cell phone. Which brings me to my next point:

Please, know how to use logs.

Since I started red teaming, I hear blue teams talking about rumors of kernel rootkits or “0day” that the red team has deployed. While complex attacks like that can happen, there is absolutely no need for me to use it if I can just log in via SSH as root with a default password. Carl, a fellow red teamer, put it best:

“Why would I waste valuable time packing malware in a way that won’t get detected if blue teams aren’t going to install anti-virus anyway?”

At both Regionals and Nationals, I start by leaving breadcrumbs of my access on a system. If you catch it, I know I need to cover all my tracks, but if you don’t, hell, I won’t waste my time. This is one of the key areas where good operations will create the foundation for good security.  Knowing where the important logs are on a system are and how to tail and search them is going to determine whether you retain ownership of your server or I claim it as my prize and eat popcorn while watching you Google.

You can do this in several ways. You can get fancy and use Splunk (free trial available) or simply use some tail -f log | grep. With these skills, you’ll be able to identify breaches more rapidly and begin the proper incident response process to fix the problem. Which is another great segway into my final point:

Think twice before killing my shell.

When you see (where root is not you):

alex@ccdc-test-1:~$ who
alex pts/0 2013-04-23 18:24 (192.168.1.1)
root pts/1 2013-04-23 17:14 (192.168.1.129)
alex@ccdc-test-1:~$

Figure out how root got access before killing his shell. This gets brought up every red team debrief, but I cannot stress this enough. I know the feeling. I was blue team once. When you know that red team is on the other end of a keyboard logged into your box, you want nothing more than to fix the situation, you panic and overreact. In that moment, you need to rely on your operations skill and do quality incident response. The basic incident response process should look like this:

  1. Gather Information: Figure out what is going on, gather evidence from multiple sources
  2. Create Hypothesis Based on Information: Now you have a global view of whats going on
  3. Determine Remediation Steps: Note Plural. If this comes out to be “Kill The Process” or “Kick Him Out!” then you’ve failed this step. Make this technically detailed. It should be supported by your evidence and hypothesis.
  4. Execute Plan: Now you act.
  5. Detail Report: If you’ve followed steps 1-4, you’ll have already done the work to submit quality incident reports to the white team. Every point counts and following this process will help ensure you get something for the inevitable breach.

Conclusion

Whether for CCDC or work, these points will make you a better security professional. I don’t care if you’re interested in offensive or defensive security, your operations kung fu will be a valuable asset to you and your team.

For all the competitors this year, great work. Regardless of winning, simply competing is experience that you will use for a lifetime and thats worth more than any medal or award. I do want to extend a special congratulations to my old school RIT for taking home the Cup. Feel free to join myself and others in the #ccdc channel inside Freenode. This is a great community and I’m proud to be a part of it.

$ ssh 2014.ccdc
$ /etc/init.d/planning start

The Biggest Need in Cyber Security

After seeing this come across my news feed and several people comment on it, I thought I’d dump some thoughts. Maybe someday I’ll collect this into a more coherent editorial, but for now this will have to do (lots of code to write, haha).

Basically, if you didn’t read the article, the White House Military Office had a piece of malware installed on it through a spear-phishing attack. Someone clicked on a link in an email that infected their system. The White House has confirmed it, but also confirmed what I suspected – it was on a declassified civilian network and not the ones with the “nuclear codes” or other top secret information. And while everyone freaks out over this, I’d like to present some thoughts on why the WHMO hack is less important than you think, and this other problem is far more urgent than anyone, IT professional or otherwise, realizes.

My biggest fear is not that the US Government is going to get hacked and a nuclear weapon is going to be launched. What keeps me up at night is the fact that hundreds of thousands of US businesses don’t maintain any sort of security in their computer systems. Estimates have put the figure as high as 85% of small to medium businesses are not doing enough to secure themselves and are therefore vulnerable. Their failures are not negligence either. Technology has progressed at such a high rate, the systems administrators have hardly been able to keep up. Information Security used to be a job that could be managed by the administrators, now it’s such a complex science(see: art) that you need highly specialized people just to meet requirements. This is a very hard subjects for Republicans and Democrats alike. Democrats cry out for regulation. Republicans cry out for defense spending.

Here’s the problem with both: there is no where near the resources needed to do either. Same with Republican defense spending. You can give the DoD all the money in the world or regulate all you’d like, we simply don’t have the human capital to protect the countless computer networks our world has grown to rely on. Half the reason US Government networks are insecure is the man power. Regulation auditors or Cyber “forces”. Take your pick, the skill sets required for each are very similar.

And the estimates right now for human capital say we need 10,000 top tier cyber experts immediately and another 30,000 over the next 5 years. Currently, it’s estimated that there are less than 1000 people in the United States with enough skill to be effective. The Chinese can hammer our government networks all they’d like. They do, but don’t get far. Trust me.  I’ve worked closely with several people on this problem. Alan Paller, Chair and Director on Board, at the SANS Institute as well as Karen Evans, Former CIO of the United States under President Bush, have both extensively researched this lack. Here’s a paper delivered to President Obama in November 2010 outlining this issue.

Hackers also hammer any business they can in the United States, with most small businesses completely oblivious to their penetration. They infiltration, they steal – and usually not money, mostly intellectual property – and they spread. They maintain a low profile and most anti-virus that even up to date won’t protect against it. Good luck having your contract IT worker cleaning them out. As the young generation grows up learning technology, it becomes taking candy from a baby to do these things. If Sony’s breach showed us anything, it was that even the biggest companies are extremely vulnerable. And the best hackers? You never hear about their work. That’s how good they are.

It’s these businesses that are being hurt, and in turn our economy, by this cyber “war”. Being part of that elite 1000 has motivated me to attempt to grow the size. Although I’m blessed with some of the best job security of any career field in the world right now, it pains me to see us losing this fight. I have dedicated a substantial amount of my time to volunteer high school and college programs geared towards training and identifying the best young cyber talent in the country and put it on a path to an effective career. So far in the US Cyber Challenge (USCC), we have brought through over 1000 competitors nationally who have shown potential in the cyber field and gotten them in better training programs.

How do I know these programs work? I was one of them. I came through the US Cyber Challenge in the Summer of 2010. I’ve since dedicated as much time to them outside of work as possible. I also have strongly supported the Collegiate Cyber Defense Competition (CCDC). I was a competitor throughout my time as a student at the Rochester Institute of Technology and also as a volunteer red teamer, both regionally and nationally. CCDC now has over 100 schools competing in a national bracket. It is truly the NCAA of cyber security. Dwayne Williams and his staff are the best in class for competitive cyber security.

The biggest gap in security we have is security education. And as much as I have spoken about above, there is still a huge need.

  • We need to find and groom the best young talent to increase the work force.
  • We need to re-tool our current IT work force with more security knowledge that they can apply to their jobs.
  • We need to bring security into the mainstream. Typing classes are nice, but with kids sitting behind screens from such young ages, it’s important that we train all youth in good cyber “hygiene”.

Wherever you fall in those three points, make a difference. The safety of our digital world depend on it.