The Evolution of CCDC

Me setting up the Red Team Room at NCCDC 2022.

It’s been a while (ok too long) since I took the time to talk about CCDC, but after getting back in person for the 2022 National Collegiate Cyber Defense Competition, it seemed timely. This past Thursday, Friday, and Saturday, NCCDC resumed in person competition, returning to its roots in San Antonio. In some ways, it was the same old NCCDC, but there were a number of surprises that left myself and the rest of the Red Team leadership reflecting.

Some Things Never Change

Effort & Determination

My working desk for the 6 days leading up to NCCDC 2022 (with Dan Borges and Joe DeMesy)

All teams – Blue, Red, Black, White, Orange, etc. – demonstrated that this constant is what truly sets everyone at CCDC apart. I know I’m not alone when I say the spirit of Winston Churchill’s quote “I have nothing to offer but blood, toil, tears and sweat.” is alive and well. Just as when I was a competitor, results be damned. The countless hours and relentless pursuit of something is what defines a champion’s spirit. To everyone who showed up, I congratulate you. 

The Thirst For Knowledge

MORIA code composition by language and line count.

It’s hard for me to look back at a NCCDC event and not think about a detailed list of something I learned. Whether researching ways to make the Red Team more effective, ways to infect and persist, or learning how some cool system the Black team has cooked into the environment, one of the nicest things I take home each year (besides a cool backpack!) is some sweet, sweet knowledge about something I didn’t know before. This year, for me, it was modern Linux persistence. (Shoutouts to MORIA – my new Linux kit). This is a huge reason why I put the effort I do into CCDC – I believe this informational growth is best achieved in a hands-on, pressure-fueled environment.

Team Purpose

2022 NCCDC Red Team hard at work.

Regardless of in person or virtual, the purpose of the CCDC Red Team remains. We are there to provide the competitors a taste of a highly sophisticated, determined, and “adversarial” attacker. You can read all you want on Unit42, Talos, or any one of the numerous Threat Research Group’s blogs about sophisticated actors and their illicit actions across the internet all day, but only at events such as NCCDC will you actually get a chance to go up against one. This is an incredibly unique opportunity for college students to learn through experience. Once you get out into the real world, you’re likely to face the real version of this threat. Thankfully, you’ll be prepared.

Some things Do Change

Blue Team Efficiency

NCCDC 2022 Bracket of Blue Teams

After doing this now at dozens of CCDC events, (this year being my 11th year Red Teaming NCCDC), I can say unequivocally that Blue Team’s, as a whole, are getting better. It used to be very clear and obvious the difference between a first year and a returning Blue Team in their tactics. Returning teams have that experience of knowing how fast we move, how silent we tend to be, and how ruthless we’ll be about exploiting small weaknesses. It’s simply hard to get that knowledge without going through it. This is especially true because every Blue Team has their strengths and weaknesses – you have to understand those, then layer that experience of NCCDC over it to formulate a strong strategy.

This is changing. I don’t know why, but it’s a good thing. This year we saw first year teams demonstrate a level of threat hunting and incident response that you typically don’t see. They were ready given the limited tooling at hand to weed out and remediate Red Team persistence beyond any previous year. When you combine that returning teams’ automation and blazingly fast gamework, it’s clear the bar has been raised – and in a way I don’t think there’s any going back either. I used to believe that part of what made NCCDC challenging was the “scale” – the competitor to host ratio (teams of 8, 24 hosts per team, means roughly 3 hosts per person) seemed to be a high water mark. I think we’re starting to find this ratio to be more digestible to today’s script happy Blue Teams.

System Stability

I think part of what makes this reasonable is as we advance forward in time, operating systems, services, and software just generally are:

1. More capable than previous iterations and versions
2. More stable than previous versions
3. The evolution of #1 and #2 are better documented, leading to better knowledge of how to utilize these recent version’s featureset. 

The radical difference in Linux kernel major releases just don’t have the effect they once did on compatibility. I think this helps many teams – Blue, Black, Red – be able to make broader assumptions about a system’s capabilities, allowing the operator to plan with expectations set. We used to have fun on Day #2 with things like Slowloris attacks – but sincerely gone are those days. Combine that with an ever decreasing amount of local privilege escalation, and the days of getting root on *nix systems are slowly becoming more and more elusive. Even technologies like AppArmor or SELinux were present on a majority of systems. Hell, one box had auditd on it by default, out of the gate. 

As Red Team, we’re used to planning for the worst – a crazy mix of 32bit and 64bit, strange operating systems, outdated kernels, no package manager, etc. With all of those requirements, our systems, processes, and kits expect a frankly hostile environment – one that we’re seeing less and less of as years go by.

Red Team Identity

Dave Cowen giving his signature Red Team Debrief at the Closing Ceremonies.

So after all that, where does that leave the NCCDC Red Team? It’s no secret this year was unique for us. Our roster, for various reasons, was significantly different compared to previous years’ relatively stable composition of Core and Volunteer members. That pushed us to improvise more, but at the same time, consider what it means to deliver on our mission. As I sat with Dave after the event, I thought about this. What if the same Red Team we’d been accustomed to for the better part of 10 years showed up? Would anything have changed? 

To be honest, I don’t think so. The definition of insanity is doing the same thing over and over again, expecting different results. Our previous rosters relied on tools and kits that were showing their age, even last year. I think it would be against our nature to ignore the reality I’ve outlined above. We’re a team that puts it all out there in order to make the game what it is – and to do that moving forward likely requires a bit of soul searching. I’m in no way saying we don’t have incredibly talented folks – we do – but our game has to evolve to continue to serve its purpose. We have to recognize that technologies like Windows Defender are really good – and just like the blackhats we’re emulating – we too have to be ready to overcome such defensive technologies. Just as Blue Teams, and the systems they’re securing have evolved, so too must the Red Team. Our previous standards and expectations are simply insufficient for the future of CCDC. This evolution will level up our tools, personnel, and strategy. A challenge for sure, but one that we must tackle to continue our legacy. 

Wrapping Up

In conclusion, 2022 was a masterful year all around. Huge congrats to the ever impressive University of Central Florida for another victory, but shoutouts to everyone who came. I know coming back after two pandemic years carried a lot of uncertainty, but the magic that makes CCDC was back in full swing. To Dwayne, Kevin, Keith, Brandy, and the rest of the CIAS staff – chefs kiss. None of us would be here without you and I’m so grateful you give us the opportunity to participate year in and year out.

A fun easter egg screenshot of some of the code for my MORIA Linux Persistence Kit.

It felt so good to get back to and do early prep with Joe DeMesy, my Red Team partner, and Dan Borges, who is now on the Core Red Team. We spent days leading up to the event getting ready and building some amazing stuff. When it was all said and done though, the lessons learned were profound. This year taught the Red Team as much about itself as any year I can remember and I have the amazing Blue Teams to thank for that. I took more notes this year than in the last 3-4 years combined. I made tools like GRID, BORG, and Traphouse to keep up with evolving Blue Team tactics. The need for those tools hasn’t stopped – in fact they’re showing their age and need a fresh coat of paint, along with new tools to meet the challenges of tomorrow. The “builder” culture within the Red Team is now fundamental to our ability to operate successfully.

I’m looking forward to a renewed, retooled, and reinvigorated Red Team that ultimately continues to provide that unique experience only NCCDC can.

Building A New Security Team!

For my friends who don’t know, I joined Scale almost 1 year ago as their Head of Security. This has been one of the coolest companies I’ve ever worked for and I’m stoked at the growth and opportunity that awaits Scale.

I’m excited to announce that we’ve started building out our Security Operations team at Scale AI. Blue Team is some of the most amazing work you can do in security, and the impact you can have is incredibly rewarding.

I’m looking for a passionate Security Engineer who’s ready to come in and hit the ground running. This is an opportunity to greenfield a Security Operations team – from tooling to process – at one of the world’s fastest growing start ups. This is a unique time to join Scale – while the company has established itself as a world leader in accelerating AI development, there’s so much impact you can make in this role.

Having been a Security engineer for a long time, I truly understand the hiring market. Companies are banging down your door to talk to hire you. Beyond compensation and perks and all the standard levers companies will offer you, what I’m selling is different:

1. A chance to truly green field a Security Operations team. Not “maintain the status quo” or “fix a mountain of tech debt”. Actual problem solving, planning, and execution.

2. A unique environment. Companies like Scale that are growing fast present unique challenges that you won’t see at any other company. The speed of growth and and ability to engineer solutions that stand up to the rigor of the business allow creative Security Engineers to thrive and never get bored.

3. A chance to work with an incredible team of smart people. AI and ML is a highly technical arena and the Engineers at Scale are some of the smartest people I’ve ever worked with. The problems they’re solving are unique and are as “cutting edge” in technology as it gets today. Our customers span multiple industries – from Autonomous Vehicles, to Big Tech and Start Ups, to the US Government. AI is changing the world and Scale might be ground zero to see all of this happening.

Here is the link to the job description and open req. Feel free to apply or message me if you want to talk more about the position.

And any friends out there who’d be willing to re-share this post to their networks, I’d be super grateful!

More to come!

CCDC Is The “Real World” And Here’s Why

Anyone who knows me, knows that I strongly believe Collegiate Cyber Defense Competition (CCDC) is literally the best extracurricular activities students interested in cybersecurity can participate in. This belief is rooted in own personal experience – from my time as a student at the Rochester Institute of Technology, I competed in CCDC multiple times, including at the national level in 2011. Every year since, I’ve returned as a volunteer red teamer, both regionally and nationally. As of this writing, that totals 21 CCDC events.

Many outsiders would frame CCDC as “just a game” and not realistic. By outsiders, I mean people who’ve never competed in CCDC, and therefore cannot understand how CCDC may or may not be applicable to the real world.

Let’s look into why this argument is fake news by analyzing the competition as both a blue teamer and a red teamer.

Blue Team Perspective

I started my blue team path by barely making my school’s team. I was assigned to manage an inconsequential Slackware Linux desktop, a distribution I had never touched prior to walking into the room. That position molded me into a team leader who helped shape the composition of our blue team for years to come. I wanted to hoist the Alamo Cup over my head.

And to that end, I did think of it as a game. How can I get more points than other teams? Given all the various ways CCDC is scored, answering this question in any meaningful way takes a strategy. You have to prioritize – are you going to focus on injects or uptime? What about filing compromise reports? I spent hundreds of hours with CCDC teammates John O’Connor and Alex Shagla-McKotch, iterating over these details.

We concluded that a really good blue team needs three things to be successful: experience, skill, and flexibility.

You can’t buy experience. It must be earned through sweat and tears and many sleepless nights. We even coined an expletive phrase to describe the moment when you sit in your blue team chair for the first time at CCDC – “s**ting your pants”.

And anyone who’s blue teamed before knows EXACTLY what I’m talking about. You can have a perfect GPA and know the kernel like it was your girlfriend, but when you sit down at “T0” and the competition begins, a wave of panic will hit you like a brick to the face. It happened to me and everyone I ever saw in that position. How long it lasts is irrelevant – everyone always comes to sooner or later. So if red team was starting at the same time, how do you ensure your team doesn’t freeze? Experience. I take a lot of inspiration from our veteran colleagues and their service. I’ll let the Marine’s say it best: “Improvise, Adapt and Overcome.”

We would prioritize seat assignments first, by experience, and second, by skill. When you figure out mitigations for the fact rookie players are going to need a moment to get their head in the game, you start to understand the first lesson of why blue teaming at CCDC is the real world. Zynga, who recruited me directly from CCDC, brought me onto their incident response team. Almost immediately, this experience of having to handle myself under pressure gave me an obvious leg up compared to my peers. Even if the technology was foreign, I had a much faster reaction time and generally thought with more clarity. Classrooms and certifications and Twitter can’t teach this.

So what about skill? Where does that play into it? Literally everywhere. People have often (wrongfully) asserted that CCDC is more of a game of systems administration than security. This fallacy shows how in the dark the individuals who makes these assertions may be. I’ve worked both incident response and red team positions for the better part of a decade, and I can tell you one thing with certainty:

Your capacity to operate and manage infrastructure (systems administration) is directly proportional to your effectiveness in your security engineering position. In July 2010, I worked with Karen Evans of the Center for Strategic & International Studies on a publication titled “A Human Capital Crisis in Cybersecurity“. In that paper, we showed that skills in areas like systems administration and IT were critically important to a cybersecurity professional operating at an expert level.

Six years later, I spoke at the White House on this same topic. Not only did I maintain the same belief, I now had the experience and wisdom to confirm that the skills you develop participating in CCDC are as real as it gets. Beyond the obvious soft skills you learn participating in CCDC (technical writing, crisis communication, project management, etc.), there’s vast areas of knowledge you get better in. (Shameless plug) Next month, I’m speaking at the Rocky Mountain Information Security Conference on my experience building out Uber’s Incident Response Team. In that presentation, I talk about what I’d learned to look for when evaluating candidates compared to how most people assessed skill. Ordinarily, most candidates were examined in the following areas (or some combination of):

• Security Engineering (Defense)
• Malware Knowledge (Offense)
• Reverse Engineering
• Forensics

I argue that while those are important, you should also vet candidates in these areas:

• Software Engineering
• Systems Administration
• Systems Architecture
• Scalability & Resilience (SRE)
• Automation
• Logging
• Deep knowledge of:
  • Operating Systems
  • Networking
  • Databases
• UI/UX Design
• Data Science
• Technical Writing
• Time Management
• Project Management
• Security Policy & Compliance

I’ve bolded the topics that directly apply to blue teaming at CCDC. So we’ve shown that experience and skill are fundamentally critical to both CCDC and the real world. So let’s wrap up why blue teaming is literally the real world with the final component: flexibility.

Remember that little old Slackware Desktop I said I got assigned to for my very first competition? There was no better teacher for flexibility than getting assigned to this insanely old, completely broken, Slackware linux machine and told to “figure it out.”Legitimate security personnel at some of the worlds biggest companies deal with this quandary on a daily basis, regardless of budget or technology. You need to be prepared for the unknown. This is applicable to everything from your team’s overall strategy, to what variants of BSD you might encounter, to the subtle but important differences between PostgreSQL and MySQL. Almost all of the knowledge I bring to work everyday is a direct result of being able to learn on the fly and use the power of associative memory to rapidly deconstruct complex systems, thereby translating needs into solutions.

Red Team Perspective

I’ve done red teaming for two major technology companies, as well as worked at Lares, one of the world’s most renowned red teams. Every year, I’ve been able to transpose my experience red teaming CCDC with my experience in the real world. Not only is CCDC absolutely real world from an attacker perspective, in fact, I’d argue that most professional red teamers are actually less realistic than the CCDC red team! I will explain.

Let’s compare and contrast the TTPs of the following groups: nation state actors, cyber criminals, CCDC red teams, professional red teamers.

	Nation States	Cyber Criminals	CCDC Red Teams	“Real” Red Teams
Compromise systems in ways that could impact a business.	YES	YES	YES	NO
Compromise collateral targets and use those positions against one another.	YES	YES	YES	NO
Steal large volumes of data – not just a record or two for “confirmation”.	YES	YES	YES	NO
Can decide arbitrarily to corrupt or destroy data in a material way.	YES	YES	YES	NO
Required to follow all laws, regulations, and policies.	NO	NO	NO	YES
Required to conform to a set amount of effort and time.	NO	NO	NO	YES

Notice a pattern? There are simply limits to what you can do as a professional red teamer that you aren’t beholden to in the other three groups. Malicious actors around the world are something most security professionals have never experienced in a controlled environment. This is why CCDC is actually more real world than basic penetration tests. The blue teamers get to experience a determined and sophisticated adversary that disregards collateral damage and adjacent victims.

And as a red teamer, that is incredibly valuable. After CCDC last year, I wrote at length about the philosophy that guides good red teams and the “Attacker’s Mindset”. In that article, I describe the mission of a professional red team:

“A red team’s mission is to simulate threats to identify previously unknown and unmitigated risk. Through a unique blend of adversarial engineering and standard penetration testing methodologies, red team exercises are meant to challenge your organization’s belief in the strength of their security. By emulating a wide array of threat actors, red teams bring perspective uniquely suited to the iterative development models commonly practiced in information security programs.”

This captures the essence of how a penetration tester is different from a red teamer. The most valuable skill you bring to the table during a red team is to act like an evil villain, without actually being one. Hollywood sets a phenomenal example in how this works in life. Consider Bane and Tom Hardy, the actor who played him. His performance will put chills in your blood. A tribute to the apex villain. But do you think that when the cameras stopped rolling, Tom continued to break spines and blow up Heinz Field? Absolutely not! He’s capable of this because he’s acting. He has developed a skill known as method acting to be able to be the bad guy on camera without carrying the monster off screen with him. This type of emulation takes thousands of hours of practice and a few lessoned learned to get right. But it’s immensely important to do so. Without being able to maliciously hack, you can never simulate a malicious hacker. If only there was a place red teamers could sharpen their skills in a safe and beneficial manner!

In all seriousness, what are the options? Capture the Flags (CTFs) too often are just computer science puzzles. Training courses and certifications? You mean a basic exercise in knowledge regurgitation, no sign of the skill, experience, or flexibility? Gross. Myself and many of my veteran red teamers use CCDC quite literally as a dojo for developing our professional skills in an ethical and community building way. Not only do we get to practice strategies, tactics, and techniques that will make us better at our day jobs, we get to give back to the community by providing blue teams the adversary they deserve to face.

So let’s get into some of the details behind what goes into red teaming CCDC.

Tooling and the software development done in preparation are the biggest differentiator between CCDC red teamers who succeed and fail. One of the reasons I’ve stayed away from big name conferences the last few years is that I got sick of sharing drinks with “lab hackers”. A lab hacker is someone who spends an exorbitant amount of time and effort in a controlled setting, then makes sweeping, hyperbolic comments about how that work is applicable everywhere in the world. Then, more times than not, when it’s looked at by others, it’s uncovered what little to no merrit the comment had in the first place. Where as the lab hacker puts their effort and focus on getting accepted to a conference, we focus on weaponization. This means that 90%+ of the lab hacker’s work is not ready for the field.

So CCDC is a great way to test yourself and your tools. You’re walking into an environment that you have no visibility into – will your tools work at gametime or will you make excuses about how you can’t apt install some dependency or the victim doesn’t have “.NET”. Let me play you sad little song on my little violin.

But what about stealth? Clearly we must trade stealth for speed and that’s unrealistic right? Wrong. Look at some of the major worm that’s hit in the last decade or so:

• Conficker
• Mirari
• RPCDCOM Worm
• WannaCry

Would you say any of those were “quiet” or “stealthy”? Not even a little bit. They slashed and burned around the world without resistance. Does the bubonic plague need to be stealthy to be effective? Hell no and neither do your adversaries. In the “real world”, competent red teams (and of course the attackers they model) use what works, whether that’s default creds, phishing, unpatched vulns, and yes, even 0day (something that has been produced during a CCDC event in fact). Also, if you don’t know this already, not all phishing involves email.

I’ve spent years developing my CCDC red team toolbox for nothing but the purpose of CCDC. So when Vault7 hits and describes an exploit that you independently stumbled upon and weaponized specifically for using in CCDC, you start to take notice how that toolbox stacks up to quite literal nation state actors. In all seriousness, the only open source or commercial security tools I used at all this year were nmap and Paragon VMDK Mounter. As the maintainer of the CCDC Red Team’s GitHub, I have the stats to prove it. In the last 2-3 years, we’ve had well over 1000 commits made 30 different volunteers. Much of that code is inspired by field work all of us have done. That code transcends over 12 different languages targeting virtually every major operating system, including historic versions. It’s gotten to the point where a group of us actually shared a house for the week prior to Nationals and wrote code the entire time. (Stay tuned for another blog post in the coming days detailing what the infrastructure behind all of this is and why you are unable to simply “block” us.)

This massive undertaking is mutually beneficial. Blue teams get the simulated experience they can expect from real world threats and we can sharpen and refine our skills in a morally, ethically, and legally sound way. In another post I wrote for work, I wrote:

“The majority of Uber’s security response team volunteers on the CCDC Red Team, providing us an opportunity to continuously improve our ability to defend Uber’s infrastructure.”

And don’t take it from me, other companies also do the same. In fact, it’s highly competitive. At the national level, outside of the core red team there is only 10 volunteer spots and 10 sponsor spots. To earn a volunteer slot, you must apply directly. This process will take only a very select few of the total applicants and happens new again every year. I encourage you to apply. We love to red team with people who put in the blood, sweat, and tears to better themselves and their tradecraft.

Conclusion

As you read above, there are many aspects of both blue teaming and red teaming in a competition like CCDC that prepare students and volunteers alike in ways nothing else can. If CCDC was so detached from the real world, that could not happen. I’ve seen it happen for many students and, in my particular case, this real-world exposure changed the course of my life and my career.

I’d like to tell share with you where my attacker’s mindset comes from. I cut my teeth in security as a pre-teen cracking, hacking, and botting video games like Runescape and Diablo II. By the time I was 14, I had dropped out of high school and spent the next four years of my life running the streets with gang bangers and dope dealers. After turning 18 with mounting legal issues, I decided it was time to grow up. I had found a small community college in San Francisco that offered an associates degree in network security. I had no idea the skills I learned growing up on the streets could be used for good. Most people would repress such painful experiences, but I’ve been able to take those memories and wisdom and apply it to being a white hat security professional. Over time, I’ve met other professionals with similar stories. Hands down, their some of the most qualified cybersecurity professionals I know. Experience is without question the best patch for human deficiencies.

This unique experience is a scar I cherish. It gave me a serious leg up in our field. People can spend their entire careers being nerds chasing the skills to emulate criminals, and here I was, the criminal turned college freshman who needed to learn to become a nerd. Having given up the mask in exchange for the cape, there has been no better conduit for personal growth and development than CCDC. Blue team or red team, it doesn’t matter. The skills derived through that experience is quite literally what makes me good at my job. There are people with more certifications, more degrees, more retweets, but when push comes to shove, when I sit down at that keyboard, my CCDC experiences give me a far greater advantage than any piece of paper you have.

It doesn’t get any more real than that.

Know Your Opponent: My CCDC Toolbox

Alright guys! This has been several years in the works. Sorry for the delay (and the rambling that ensues in this post). Hope it helps!

A very common question @ CCDC events are “What tools did you use?” and “How did you hack us?”

I do use a number of off the shelf tools such as metasploit and nmap, but a good part of my work during the competition is done with custom tools. This is true for a number of us on the National Red Team, with each of us having particularly effective toolboxes. Every year, we take this further and further down the rabbit hole and the toolchain used this year across all red teamers was by far the most custom and effective. With the exception of a few collaborative efforts, I’m going to withhold speaking about other red teamer’s tools and attempt to focus on mine. But rest assured, what you see here is simply a glimpse into the arsenal you meet in San Antonio.

This has not been an easy post to write. The last few years I’ve been a big cagey about discussing these custom tools publicly. Unlike professional adversaries, we have to do much of this work in our spare time. CCDC is ultimately an educational experience, so I absolutely want to share as much as possible. My devil’s advocate position has been that the experience is part of the competition, and if I spill the beans on our tools, we won’t be able to provide the realistic threat you’ve come to expect.

It’s definitely a gray area, but I’m going to go ahead and talk about the tools I’ve spent the last few years working on. The toolbox gets divided into two categories: infrastructure and malware.

Keep in mind as we dive in, that I’ve only red teamed Western Regional and Nationals, with Pacific Rim last year being the exception. If you’ve never competed in any of those, then you likely have never seen these. I’m going to work this year on getting some parts of this tool chain distributed out to the regionals. Even if you have done those competitions, you likely haven’t seen everything in this list either. With CCDC environments vastly different from scenario to scenario, tool reliability is one of our biggest challenges. This makes making tools work a per competition mission. I can only prep so much, so they end up getting deployed in order of their importance. Also, I’ve intentionally withheld some details. I’ll leave those up to you guys to figure out 🙂

A Brief Note on Codenames

Anyone who’s worked with me knows that I love a good codename. We do a lot of R&D and codenames help keep projects in their lane. Sometimes the codename makes sense; less so for others. What you can expect is the following tools can be identified by CODENAME because they will be represented in ALLCAPS.

Also, I’ve attached random pictures from the inter webs to make this long, arduous post more entertaining.

So without further ado!

Infrastructure

Infrastructure tools are things that make us more efficient as a Red Team. These pieces of software run on our servers, never on blue team hosts. If you were at Nationals this year and saw the “Red Team Infrastructure” slide in Dave Cowen’s debrief, this is what he’s talking about.

This year, I migrated every infrastructure component to a service oriented architecture. With the exception of TRAPHOUSE, everything else follows SOA principles with HTTP based API’s for inter process communication. This allows these services to integrate with each other in ways that weren’t possible before, as well as deal with problems of scale more effectively.

TRAPHOUSE

This is where the fun begins. TRAPHOUSE is the main brain for the red team. TRAPHOUSE has three main components:

• Web UI
• Backend
• API

TRAPHOUSE maintains source of truth state for most pieces of our infrastructure. We do a lot of things with TRAPHOUSE, but most importantly, we file our compromise reports there. Black Team has access and manages Red Team scoring based on the data in TRAPHOUSE. To give you a peek into its “brain”, here’s a screenshot of the entity-relation diagram for the PostgreSQL database behind it:

Another key feature of TRAPHOUSE is its Pads. These are basically Etherpads within TRAPHOUSE that let us collaboratively share and store information between red teamers.

TRAPHOUSE has evolved over several years and is one of my oldest tools. It was the product of a marathon hacking session and until this year, was basically duct tape over duct tape. This year, it got a complete re-write with the help of some awesome contributors (Email me if you’re looking to hire Rails devs! They’re great!):

• Reuben Brandt
• Jonathon Nordquist
• Jeremy Oltean

Together, we were able to take the old crappy code and make it far more modular and extensible. There will be more work to come on TRAPHOUSE, but it’s in a great place to continue to be a centerpiece of our red team activities.

The next big feature I’ll be shipping in TRAPHOUSE is a C2 API. The goal is that other red teamers can integrate their C2s into TRAPHOUSE for other people to interface with and use.

OG

While a lot of stuff happens in TRAPHOUSE, the other interface into my tool chain is a Slack bot called OG. OG has extremely tight integration with TRAPHOUSE and is the main interface for most red teamers. You interact with OG with “!” commands, and OG is channel and user aware. You can invite him to rooms, private message him, etc. and he will always respond where appropriate.

OG also manages authentication and accounts for TRAPHOUSE. You register for TRAPHOUSE through OG, and OG won’t talk to you unless you’ve got a valid TRAPHOUSE account.

With inspiration from the “ChatOps” movement, I’ve attempted to make all my tools (especially the infrastructure) available to the rest of the red team via OG commands.

DOOBY

DOOBY might have been the most challenging piece of code I’ve worked on. ever. DOOBY is a custom DHCP Server that I wrote specifically for CCDC Red Teams. The solutions it solved over traditional DHCP servers:

• It’s hard to manage the IPs you use as a Red Teamer in a CCDC environment.
• Humans are not as good as computers at computing random numbers.
• If two red teamers collide IPs, we lose time, and potentially shells.

As Blue Teams have gotten better with regard to network defenses, we’ve had to evolve too. It has taken a few years for DOOBY to be realized. DHCP is an incredibly complex protocol when you get into the nitty gritty, and implementing it from the ground up took countless hours of testing to get it stable.

DOOBY fixes some major red team pain points in several ways:

• Records all IP leasing in TRAPHOUSE and ties it to the Red Teamer for correlation w/ their compromise reports
• Distributes leases across several red team networks
• Allows IPs to be rotated very rapidly without fear of stomping someone else’s IP

In addition to red teamer laptops and VMs, DOOBY manages addressing for my other infrastructure components as well. As you read on, keep that in mind. DOOBY is perhaps the least sexy, but one of the most integral pieces of our infrastructure besides TRAPHOUSE.

DOOBY got its first prime time action this year, but I’m looking forward to iterating on it. There’s still work to be done to make it more user friendly for other Red Teamers, but it supported much of this infrastructure during Nationals this year.

BORG

BORG was born a few years ago in the wee hours of the morning before Western Regionals. I wanted some way to do MSF exploits from random IP addresses but have second stages be centrally managed, and since I’m lazy – let’s integrate it with Slack!

What we ended up with when the sun came up was a beast. BORG is a fleet of “Drone” VMs that do our bidding. We use BORG for all sorts of things – scans, exploit launching, and spraying. The Drone VM itself has several valuable tools preloaded on it:

• metasploit
• crackmapexec
• spraywmi
• nmap
• sqlmap
• database tools
• snmp tools
• phantomjs
• + many more

TRAPHOUSE manages a work queue and the Drones continuously check in seeing if any work needs to be done. Now, when a red teamer wants something done, they simply issue a command in slack:

!borg run nmap -sS -T4 -Pn -oA team_6_quick 10.6.6.2-254

This queues the NMap scan job up. When a free worker retrieves it from the queue, it’ll hop IPs (using DOOBY), perform the task, and return all output (STDOUT and any new files) to TRAPHOUSE. OG then sends a notification to the Slack user who submitted the job with download links to their files.

GRID

GRID is one of those tools when you explain it to people, they give you a funny look somewhere in between “u wot m8?” and “i cant even”. Together, BORG and GRID act as a yin and yang as our red team “bot net”. Where BORG “throws” things from a distributed block of IPs, GRID “catches”.

In one sentence, GRID is a piece of physical hardware which acts as tens of thousands of infected hosts on the network. These “hosts” can proxy call backs and serve malicious files.

GRID has been an incredible piece of technology for us. With GRID effectively acting as a multi-tier, distributed, C2, red teamers can host their implant C2s on static IPs provided by DOOBY. With GRID, they never have to change that IP, and you as the blue team never have direct contact. The C2 remains effectively hidden in the tens of thousands of IPs within GRID’s network.

This abomination of CCDC technology actually has several components that make this possible, so I’ll outline them below.

GRID-KERNEL

GRID distributes itself on the network by creating tens of thousands of virtualized network interfaces. This requires a custom Linux kernel that we’ve named the “GRID-KERNEL”. Getting this stable was one of the hardest parts of GRID due to extremely slow development loop needed to test this.

GRID-CONFIG

The next step of the stack is configuration utility. This tool is what GRID uses to bootstrap its network interfaces. It talks over the SOA backbone to DOOBY and TRAPHOUSE to get a block of permanent addresses for the competition (one for each of its roughly 50,000 interfaces).

These addresses are then blacklisted in DOOBY from being leased. TRAPHOUSE also holds a record of them for black team correlation and for other things we’ll talk about later.

GRID-PROXY

The next major component of GRID is its layer 4 proxy. From within Slack, a red teamer can “reserve” frontend ports in GRID and have them reverse proxy back to them:

!grid forward --frontend_port 1337 --backend_ip 10.0.0.123 --backend_port 4444

This command will now listen on port 1337 across GRID’s network interfaces and forward the traffic to 10.0.0.123:4444. Now when a red teamer needs to know a call back IP:

!grid ip

OG will grab a random IP from GRID’s IP table in TRAPHOUSE and return it to Slack. The !grid command in Slack is extensive with several more capabilities. It is completely self service for the red team and supports both TCP and UDP.

GRID-CDN

There are a few ports that are reserved in GRIDs proxy. Two of them are 80 and 443. These ports actually get reversed proxied to GRID’s CDN service. The CDN service acts as a layer 7 router sending traffic matching rules to pre-defined places. As of now, this is defined by me at game time, but next year I’ll integrate this into OG for layer 7 HTTP self service.

If none of the routes are matched though, GRID then attempts a lookup against a local DB for a matching URI. If the URI is known, the CDN will lookup the corresponding file in its CDN Origin and serve it.

This allows us to “push” files to GRID and retrieve them in a distributed fashion. The file object can be referenced across many of my tools for interpolation with its GRID file ID. An example of this might look like:

!grid host --url https://my.super/leet.malware --name dabomb

OG will push a CDN cache job to TRAPHOUSE, which will make an SOA call to GRID’s API. GRID will download the file at the specified –url, hash it, and keep a reference using the –name field. It will generate a shortcode for you and pass it back to TRAPHOUSE, which OG will pick up and respond to you with the following:

http://[RANDOM_GRID_IP]/[RANDOM_SHORTCODE]

That file can now be retrieved from anywhere in the CCDC network using that URL. If Dan wants a different URL, he can simply issue the command:

!grid link --name dabomb

and he will get a new URL with a different IP and different short code. A number of options exist too, such as DNS hostnames, ports, # of downloads you want the file to be available for, etc.

GRID’s CDN gives us totally ephemeral staging of binaries and payloads. When a stage 2 needs to be retrieved on an infected host, it can pull from an IP address that you’ve never seen, and likely won’t again.

Over the last few years, blue teams have become more and more effective at network defenses.

While the CCDC scenario is realistic in many ways, one of the ways blue teams have found ways to gain an advantage is in monitoring network traffic. BORG and GRID give us the capability to completely negate this tactic. I’ve yet to see a blue team effectively defend against this.

GRID-CRYPTO

The final component of GRID deals with how GRID manages cryptographic material. TCP443 is a reserved port on GRID and serves a wide variety of HTTP virtual hosts on it. GRID this year was resolvable by 67 different domains that I own, each with unique and interesting subdomains. It was too much work to manually get HTTPS certificates for all of them, so I wrote a piece of code to use AWS Route53 and and EC2 instance to automatically generate valid SSL certificates for all domains and their subdomains. This tool would then spit out a configuration file for GRID that it could then use to serve valid HTTPS on requests that matched the correct Host parameter.

THEFOOT

In order for GRID to get DNS resolution, we needed a way to dynamically serve GRID IPs via DNS. We originally ran a complicated chain of BIND servers, but overnight at Western Regionals this year, I wrote THEFOOT. THEFOOT is a custom DNS server that very simply serves a random GRID IP for any valid GRID domain. It retrieves the DNS zones at startup from TRAPHOUSE and responds to any recursive query for those domains, so we set the NS records of all our domains at THEFOOT’s IP during the competition and any resolver would return competition IPs to you.

RTDNS

RTDNS is a separate DNS server that red teamers can choose to point to when using DOOBY. In many CCDC environments, every team has the same FQDN “foo.ccdc” for example. This means that to resolve “http://shop.foo.ccdc” in the browser, we have to point to your DNS server.

RTDNS solves this by letting a red teamer “lock” their resolution for a domain to a specific upstream resolver. This allows them to stick to their team’s DNS for HTTP/S resolution.

In previous years, we’ve actually run individual RTDNS servers for each team in Docker containers, but by 2018, RTDNS will be its own stand alone singular application with Slack Integration.

MRWIZARD

MRWIZARD is a distributed scanner made by Lucas Morris. MRWIZARD continuously runs a variety of port and service discovery scans against all the teams from a giant block of IPs (similar to how GRID manages its IPs).

MRWIZARD then uploads its scans to TRAPHOUSE for red teamers to grab when they need it. It also watches services and when new services are seen, it alerts TRAPHOUSE to this wonderful new discovery.

CRACKLORD

This is another Lucas Morris and Crow creation. CRACKLORD is an on demand hash cracking system. Maus used his DevOps magic and worked closely with Lucas to get a CRACKLOARD  autoscaling EC2 version deployed for this year. The more hashes you want to give us, the more we’ll crack!!!

Currently in progress is CRACKLORD integration with OG and TRAPHOUSE so that cracking jobs can be scheduled directly from Slack as well as hash lookups from previously cracked hashes.

FOXCONN

This was something Maus and I dreamed up years ago that has fallen a bit stale as of late, but is something we’ve used successfully. FOXCONN basically acts as an MSFVENOM server that generates MSFVENOM payloads over an HTTP API. It even has the ability to generate a payload based on request user-agent, so that whenever you get on a box, you can just hit the FOXCONN API and you’ll get returned a payload for that platform.

Now that we’ve re-architected TRAPHOUSE and OG, FOXCONN is up next for its makeover with even more exciting features.

BANEX

To test much of our infrastructure, we employed a huge VMWare infrastructure. BANEX is our vSphere cluster with several dozen VMs of varying operating systems and architectures. We use this as a firing range to test our payloads and tools.

Custom Malware

These are the custom pieces of software I’ve written over the years to stay on your boxes and give you a seriously hard challenge.

GOOBY

In order to seriously decrease our coding effort, Dan, Vyrus, and myself converted almost all of our malware tools and techniques into shared libraries which could be included on a per function basis and compiled into our implants.

GOOBY is the base library that all of these extend from. Most of the tools below, are actually library functions implemented in GOOBY. This allows implants to share modules depending on what purpose we want the implant to serve.

For individual binaries, they’ll have their own build scripts that will (usually) statically compile the included GOOBY modules into the final implant.

GENESIS

Genesis is the successor to the much alluded to [REDACTED] tool which acted as a all-in-one persistence script. One common technique we use is “bundling” of payloads so that when exploitation occurs, you can run a single payload and get multiple pieces of persistence deployed. [REDACTED] was the product of one of the darkest of wizards on our Red Team. It evolved over the years to support both *nix and Windows.

This year, Dan, Lucas, Vyrus, and myself decided to test some new, unique features and created GENESIS. GENESIS is a small program that can be dynamically built and obfuscated with an arbitrary number of payloads included in it. This allowed us to abstract much of our local persistence techniques that originally lived in the [REDACTED] code into GOOBY, for easy use by any of our payloads.

We spent a lot of time working out obfuscation and anti-RE techniques into GENESIS. This is extremely important because now if a GENESIS binary is intercepted, significant effort will be needed in order to determine the other payload it encompasses. We’re lagging behind on some platforms in this area, but by next year we expect GENESIS and its supported obfuscation to be broadly supported.

Huge huge props to original developer (he knows who he is) and all the contributors to this effort over the years.

AMBER

AMBER is a custom shellcodeexec I wrote in San Antonio this year. I wrote my own for two reasons:

1. Many shellcodeexec implementations are easily detected by AV (we don’t want that!)
2. AMBER is implemented as a GOOBY module for easy use in other implants

AMBER also supports in memory process injection and lets us migrate payloads into other processes with only a few lines of code. Extremely handy.

EXODUS

EXODUS is a library that builds on top of AMBER to perform similar functionality as GENESIS, but with shellcode instead of persistence.

In essence, EXODUS allows the operator to “bundle” an arbitrary group of shellcode payloads and create a loader which will execute the shellcode, either spawning a new process or injecting into an existing one (operator configurable).

This gives us the same “spray” capability that GENESIS gives, but with raw shellcode.

QBERT

You can think of QBERT as somewhere between a rootkit and the void of space. It’s extremely difficult to hunt down, it leaves very little signatures, and QBERTs sole purpose is to keep other GOOBY based malware persistent on the infected host. It does this through a variety of techniques that span OS functions, but suffice to say, it’s one of the dirtiest tools in the box. As of this blog post, it still is almost entirely undetectable by any major IDS/AV/Audit vendor.

GEMINI

GEMINI is an implant I’ve slowly developed over the last few years to keep me in systems when most other persistence techniques have been discovered. It doesn’t have a ton of functionality, but it’s small and extremely portable – it cross compiles to basically any operating system.

GENESIS allows access to the infected hosts filesystem and command execution by the operator and is also the first Implant that used the GRID network for its C2 callbacks. GEMINI will assess network conditions on the infected host to determine what time of covert channel to use for a call back.

The callbacks are asynchronous, so as an operator, you have to consider lag time between when you issue a command and when GEMINI comes home, but it’s been a reliable implant. A huge thank you to Vyrus for putting in countless hours improving its capabilities this year.

Vyrus and I also wrote a SPEC for generic C2 communication. This new protocol will be the basis for many different implants moving forward and the foundation for extending C2 interaction to OG and TRAPHOUSE.

WONKA

Dan talks often about his troll payloads, but I have a few of my own. WONKA historically has been a text file of special *nix local persistence techniques. Linux systems at CCDC events have a tendency to be very specific in their build and configuration, so I’d just keep the text file open and pick and choose what little one liners to deploy at competition time.

This year, I started the process of converting WONKA into a GOOBY module and adding meta wrappers around it. Some of the functions you might have seen if you’ve encountered WONKA:

• “BASHLOTTO” – A bash trick that will lottery a command entered into BASH for a potentially other command. (Usually 25% chance on the dice roll)
• “ANTIEDIT” – A custom wrapper around common editors that reverts changes on files you change.

There’s several more, but that gives you a few basic ideas of how WONKA causes you headaches during the competition.

I’m a big believer in WONKA because I think troubleshooting and performing root cause analysis is actually incredibly difficult. WONKA’s goal is to challenge you to think critically about how your systems may or may not be behaving and to validate assumptions about your tools.

SSHELLY

SSHELLY is a persistence technique that attacks the OpenSSH server. SSHELLY has several options for nefarious behavior, two of which are:

• “GODKEY” – Allow access for any user with a specific SSH key
• “LOUDPACK” – Saves password credentials for users in memory. When a specific backdoor user connects, it dumps the credentials and disconnects the client.

SSHELLY still has a few bugs here and there, but it’s maturing nicely and is an amazing complement to my toolbox. By deploying SSHELLY, many linux hosts which must maintain a running SSH service are now able to be persisted in a difficult to detect manner.

ALFRED

Every year, there are things I want to do in a recurring manner on your systems. While cron and Scheduled Tasks are reliable, they’re easily detected. ALFRED is my own lightweight “cron” runner. I can either compile tasks into the ALFRED binary, or leave a config file on disk for ALFRED to search and find.

While it’s not functional yet, I’m experimenting with memory cradles that will allow other implants to schedule tasks simply by writing to a shared memory segment.

HALITOSIS

I’ve been playing with audio/video drivers at CCDC competitions for a few years now. The obscure operating systems and the lack of packages generally makes intereacting with peripherials challenging. HALITOSIS is my GOOBY module that for listening to microphones on your system. I’m still in the process of porting much of my old code into the new GOOBY module, but it’s a great tool for listening in on whats going on in the blue team rooms.

TATTLER

Network visibility is something we spend a lot of time working on during the competition. Teams often put up firewalls and NAT their internal networks. This makes scanning from our external position difficult. TATTLER is a GOOBY module designed to perform adhoc port scanning and service discovery of your networks FROM THE INSIDE. It’s not as feature rich as NMap, but it’s extremely fast, portable, and configurable based on the operator’s desire. Any GOOBY based implant now has access to theTATTLER library to perform recon on otherwise unknown network segments.

MOTHRA

To keep it short and sweet (and not reveal too much), MOTHRA is fly paper for system credentials. At less than 1k, it’s definitely my smallest payload, but it’s one of the most deadly. It was only in use for a few hours at the end of day 2 NCCDC, and it managed to gather over 500 UNIQUE credentials.

Tony, you know you da man. This is going to be a very fun tool for years to come.

 Conclusion

So there you have it. On top of the industry tools (Metasploit, Cobalt Strike, Empire, etc.), we have to bring custom tools. If we didn’t develop tools like these, you wouldn’t be facing a realistic simulation of a world class adversary. (Looking at you CIA *cough*VAULT7*cough*). As put best in The Art Of War:

So it is said that if you know your enemies and know yourself, you will not be put at risk even in a hundred battles.
If you only know yourself, but not your opponent, you may win or may lose.
If you know neither yourself nor your enemy, you will always endanger yourself.

I know it’s not a tell all, but hopefully this gives you some insights into the lengths we go to challenge you. This is exactly why CCDC is the best learning environment for information security.

I cannot, cannot, cannot stress enough how awesome it is to work with some of the world’s leading experts in information security. Between the pull requests and the mentorship, working on these systems with my fellow red teamers is some of the most fun I get to have all year. Shoutouts to several brethren who contributed to these tools:

• ahhh
• vyrus
• deadmaus_
• cmc
• emperorcow
• javuto
• tony

With that, I’ll leave you guys to prep for next year! Best of luck >:)

Attacker Mindset 101 — Welcome to the Red Team

Security engineers who possess red team skillsets are the most powerful offensive weapon toward uncovering risk across an organization — but they must also know the technology basics in order to be most effective.

A dedicated group that assess security methodologies through adversarial simulation is commonly referred to as a “red team.” These exercises are a hallmark of the information security lifecycle due to their ability to identify systemic risk that had not been taken into account. This model provides incredibly valuable insights to security teams. From testing security controls to assessing the human element, the ethos of a red team embodies reflection, awareness, and a constant determination to disprove assertion through real world action.

In security, we use the Johari Window to analyze the delta between our systems defenses and from what we’re defending our systems. By bucketing threats in this way, we gain better understanding of our weaknesses, which ultimately make us safer and more secure.

The most difficult quadrant to defend against is the “blind spot.” In the last few years, we’ve witnessed some of the world’s largest organizations become compromised in overwhelming fashion. In 2010, a high-security French bank was breached by unknown attackers in ways the bank had previously considered impossible. This is just one of several examples where the cunning criminal drastically outsmarted well-funded, and frankly overly confident, defenses.

So, how can companies develop defenses for complex risk that, by definition, they have no awareness?

Enter the world of the red team.

A red team’s mission is to simulate threats to identify previously unknown and unmitigated risk. Through a unique blend of adversarial engineering and standard penetration testing methodologies, red team exercises are meant to challenge your organization’s belief in the strength of their security. By emulating a wide array of threat actors, red teams bring perspective uniquely suited to the iterative development models commonly practiced in information security programs.

As my career has progressed, I’ve developed a firm belief that security engineering candidates should possess red team skillsets. It’s often, and truly, said that the best defender is also the best attacker. We live in a world where nerds are attempting to outsmart criminals — being able to embody an attacker’s mindset is a top priority. Even if your organization doesn’t have an official red team, your security engineering team will be much more effective and strategic if required to hone red team skillsets. Following the practice of screening candidates for these skillsets will help you develop a stronger team that will uncover issues at a reliable pace.

One caveat: your work will include much more than red teaming. Eventually, you’ll have engineers scattered across your security organization with a fantastic ability to uncover risk. Some of them might be building logging pipelines, while others are doing forensics. Fortunately, there are amazing opportunities that enable your team to continue to sharpen their skills.

One of the pinnacle events for me is the annual Collegiate Cyber Defense Competition (CCDC). CCDC is a purely defensive security competition designed to measure college students’ abilities to protect and secure information systems. During the event, competing teams of eight students, known as a blue team, are placed in identical enterprise-grade networks that are configured to emulate a real company. The student teams must then race against the clock to secure the systems and maintain the integrity of the services they provide.

In order to make this work, the competition recruits a singular red team to emulate real world threats. The red team will measure the effectiveness of both the competitors’ security methodologies, as well as conduct gap analysis to identify vulnerabilities the students have missed. These are documented and factored heavily into the overall score and outcome of the competition.

CCDC is nationwide, with everything from state-level competitions to the national championship hosted by the Center for Information Assurance and Security. They employ several full-time staff to design and architect the complex infrastructure required for the event. Last weekend was my sixth time red teaming the National Championship. I’ve used this opportunity over the years to improve my capabilities, regardless of my day job. While the blue teams spend the months leading up to the competition practicing security hardening and vulnerability management, I take the opportunity to improve my skills by studying cutting edge offensive security techniques. These skills are invaluable, for both the competition and my day job.

One of the questions I am asked the most is, “How do I get into red teaming?” or “How do I become red team?” Red teaming has such broad appeal in our industry, and it’s a hotter topic than ever right now. It’s also a bit taboo because, after all, you are breaking rules to expose flaws, so people are even more drawn to it. Attacks are often mystifying because, as stated previously, the unknown unknown will leave the mind wandering, attempting to uncover the methods.

Thusly, I’ve started tweaking my response to that question. One of the most common mores I witness is how worshipped offensive security is by a large segment of our community. With hacker culture moving from underground to mainstream populism in society, the security community has evolved into a global industry. This has shifted the power of balance in a number of ways. The Pareto Principle is stretched incredibly thin, with the vast majority consuming tools and software produced by a small but growing group. This phenomena leads to misconceptions about what is really involved in participating in a red team.

As valuable as it is to able to channel the attacker’s mindset, it is incredibly beneficial to be able to develop your tools. No effective outcome has ever occurred from leveraging only offensive tactics before developing more basic, albeit seemingly mundane in comparison, skillsets. Writing effective software takes both strong muscle memory and lots of practice. Learn to embrace the less glamorous, often overlooked skills. Developing software engineering skillsets, including a deep knowledge of computer systems, libraries, and protocols will act as a force multiplier in your security career.

I’d also strongly encourage people with those questions to really understand the effort that goes into red teaming. Dave Cowen, our NCCDC red team captain, and I were chatting after the competition and talked about the traits that really make a strong red teamer. These are the four:

Knowledge – Possess a broad understanding of the systems and technology you’re attempting to circumvent. This is where I strongly encourage folks to study and read about emerging technology. The more you’re familiar with databases and web servers and routers, the better you’ll be able to understand their weaknesses.

Methodology – Adhere to a methodology that best sets you up for success. Red teaming can be frustrating and if you abandon your principles, you’re going to end up down the wrong rabbit hole with nothing to show for it.

Agility – Adapt to chaos. In the real world, things change for thousands of reasons. It’s not your job to figure out why, it’s your job to adapt and continue moving forward.

Grit – Persevere. I can say first-hand the difference between success and failure can be measured in grit. Though last on the list, this is most important.

For example, when I was at Lares, I learned so much from Chris and Eric. The most valuable lesson of all was learning to stand up to adversity and see the job through. I’ve never met a more determined group of individuals and the results speak for themselves. Again, at Western Regionals, while everyone was hunting vulnerabilities, I sat and made wordlists by hand. I scraped IMDB for potential usernames. I used awk and sed to generate the wordlists. I narrowed a password list down. Miraculously, one of them worked. It was the much-needed entry point to our first shell of the competition. It wasn’t pretty, it wasn’t glamorous — but it got the job done by the force of sheer will.

I want to see more companies embrace the practice of red teaming as one of learning and self assessment. I want to see our community stop glorifying red teaming as the end all, be all. We only cover one quadrant of that Johari Window. There are three times the amount of work needed to cover the other three. Don’t follow the crowd, be yourself. The required grit I wrote about above will show success in any career path you take. I promise.

I’m working on a more detailed, technical post about the infrastructure tech we built for red team this year. You probably saw in Dave’s NCCDC debrief about our infrastructure and were wondering “WTF?” I’ll try and answer your burning questions in that post set to come out in the next week or so.
Cheers!

Alex

CCDC 2015 Debrief: Red Team Identity

Welcome back folks! It’s been almost a year since I updated this, but given that the Collegiate Cyber Defense Competition (CCDC) has wrapped up for this year, a new post is needed.

This was an important year for the competition. It was the 10th anniversary and Dwayne, Kevin, and Jessica put forth the best competition I’ve experienced yet. They’ve refined their craft over the years and the final product has become the pinnacle information security competition event in the country in my opinion.

For me, it marks my 11th CCDC event and 8th time red teaming. The last four years, I’ve red teamed both the Western Regionals and National competitions. As those hours have added up, I’ve thought a lot about the dynamics of the red team. We have a rather complex relationship with the whole event that I think should be better understood. A few of the hats we wear while at the event are:

Simulated Adversaries

The competition couldn’t claim to be a test of defenses without an offense to face off against it. The red team must effectively hold a realistic adversarial position against the competitors. We are comprised mostly of experienced information security professionals who use their skills and knowledge to emulate a top skill attacker on the network.

The best red team members not only bring technical knowledge to the table, but a metaphorical mask by which they become this digital assassin, executing attacks with precision and ingenuity. When you sit down, you are the bad guy. You are the adversary. A speaker during the closing ceremony today described the red team at Nationals as a “Motley Crew” of mad hackers. It’s true. We have different backgrounds and mindsets, but when you put us in a room together we become the enemy in fierce pursuit of the blue teams.

For me, that passion is fueled by a desire to demonstrate a world class threat. It’s rare you get to experience an attacker first hand outside of a real breach. As a former competitor, I can tell you that’s a huge selling point of the event to blue team members.  No longer are you inside of a lab at school or messing around with virtual machines at home. A real person who’s job is to infiltrate your systems is sitting on the other end of a wire and if you don’t act, it will have consequences.

Which brings me to my next job of red team:

Human Scoring Engines

Our impact on competitors has direct consequences to the competition. A successful breach will cause a competitor to loose points. That means we have to take our job every bit as seriously as the white team. Our Motley Crew might appear to be a band of trolls to an outsider, but we fully understand our ethical responsibility in fairly and accurately scoring the competition. Underneath the fun, there is stone cold dedication to the success of the competition.

While often the term “RED VS BLUE” is used to describe the competition, I find it to be inaccurate. This competition is about BLUE VS BLUE. The red team is no more competing than the white or the black team is. While we wear the mask of the hackers, we are there as a learning simulation with scoring implications. Period.

But maybe the most important role we take on is as…

Mentors

A large part of the reason the concept of RED VS BLUE continues to persist is the camaraderie weaved throughout the event. Those of us who have previously been blue team, have come through the program and become friends and co-workers with the red team. I don’t believe that kinship would exist without the fun and humor of the back and forth between the blue teams and the red team. Every year we see pictures in debriefs of funny hacks and situations that we can all laugh about for years to come.

That builds just the sort of relationships that blue teams need. In the end, we’re there to help you learn and develop your skills that will ultimately define your career. It’s no secret that we exist in a small industry and building connections to the red team has incredible benefits, regardless of the outcome of the event.

When I blue team’d at Nationals, we didn’t even place, yet the connections I made there are now directly responsible for my employment with Lares. I encourage every competitor to engage with the red team.We are there to support you, your career, and share in the memories of the competition with you. Next time you see us hanging around a table in the hotel late at night after the competition ends, come hang out and lets talk about the entire experience.

Which at the end of the day is what this competition is all about.We might be rebels, loud and opinionated, but we’re there to fill the void that no one else can and volunteer an experience that competitors year after year will find no where else.

Conclusion

Thanks again to EVERYONE involved – white, black, red, blue, orange, gold teams – this competition wouldn’t be what it is without it. Congratulations to the University of Central Florida for their 2nd win in a row! Now my work begins planning and building new stuff for next year.

CCDC 2014 Year in Review

As I’ve been sitting on this plane heading back to San Francisco from the 2014 National Collegiate Cyber Defense Competition (CCDC), I’ve been trying to come up with one word to encompass my views on this event. This will conclude my fifth year involved in CCDC. The first two I spent as a blue teamer; the latter three as a member of the regional and national red teams.  What is it about CCDC that keeps bringing me back? I’m not getting paid nor am I hiring anyone. Yet, I keep returning year after year. I think I get it now.

passion |ˈpaSHən|
noun
• an intense desire or enthusiasm for something

Thats the word I’m looking for. Passion.

From my very first year competing, there was something indefinable that pushed me to pour every ounce I had into the competition. Even though at that time I was a junior member of our team, the desire to learn as much as possible was very real. I think this is what professional athletes must experience. There is something beyond your skill that keeps you up at night, fueling your train down the tracks.

And I think the reason CCDC and I have such a strong bond is because it’s one of the rare places where others bring that same energy to the table. All teams (white, red, black, orange, blue, gold) and sponsors are continuously adding to that pool, mixing and stirring a fervency you simply can’t find anywhere else.

This is why the competition is getting better from all angles. The tradecraft is getting refined by all parties. It’s an incredible effort to be a part of.

On that note, I’d like to now take some time and give some positive feedback to individual sets of those involved (both regionally and nationally).

To red teams:

Don’t get complacent in your duty. The blue teams are getting better and as they do, so will we. Our participation, while energetic, is only a single component to this competition. Others are expecting you to challenge blue teams in a way only you can do. Embrace that and play your part. Contribute and coordinate – the more you do, the better your red team will be.

To white/black/gold teams:

Continue to make the competition as real world as you can. It’s easy for those of us who have worked in the security industry to understand what a real corporate environment is like, but there is a very real possibility a blue teamer has had zero exposure, and therefore zero understanding. Even in the immense differences CCDC environments have versus a real enterprise, it is still a goal worth chasing because that is what makes CCDC participants valuable. They’ve had that exposure to something other than a class room or text book in a situation that will push them to think critically under pressure.

To blue teams:

Don’t worry about the results. The most valuable part of CCDC is not the trophy – its the experience. You spent months preparing for this – learning every moment along the way. As someone who’s been in your shoes, let me tell you – thats the true prize. The knowledge you’ll walk away with is priceless. Yes, the competition side is fun especially when you develop a rivalry with another team, but once you graduate and are out, you’ll be working side by side with the same people you competed against and who won or lost will seem trivial. Focus on getting better, and be proud of yourself and your team when you do.

To sponsors:

Thank you for your support. Without you, all CCDC events would not be possible. My only feed back to you is this: When you send volunteers to the competition, who you send will reflect on you far more than the amount on the check. And that image will been seen by more than just competitors – other sponsors, red teamers, and other volunteers will pick up on it.

One of the best examples of this I’ve seen was FireEye‘s choice for their red team seat at Nationals. They sent Dan Borges, a young, but talented engineer, to fill that seat. They picked someone who possessed passion just as I talked about above – and that passion drove Dan to produce tangible results in his position. That dedication can only come from someone who deeply understands what CCDC is about – something Dan can bring to the table given his previous security competition experience.

I encourage all sponsors to take seriously who they send – recruiters, management, and even technical folks. You want to connect with the participants. Having someone like Dan who can connect and relate to the competitors is the single best contribution you can make to the competition.

To those not involved:

Seriously ask yourself why.

If you’re faculty or an administrator at a college which doesn’t participate – what are you waiting for? The fact that schools such as Stanford and MIT who have long been attributed to the best and the brightest don’t participate is disappointing. Get on board. You will very quickly learn whether your curriculum is adequately preparing graduates for jobs in the information security space (likely not).

If you’re an organization, what is stopping you? I’m looking at all the companies out here in the Silicon Valley. I’m shocked that you don’t throw more support behind this. Your fast moving companies are on the forefront of technological innovations, yet you don’t tap into one of the deepest veins for young, fresh security talent. It’s not just about filling positions. This is the next generation of professionals in the security industry. Connect with them in a rich, meaningful way.

Lastly, to myself:

Keep giving into that passion. Every year I come through with more skills than previously, and it’s only making me better. I have no certifications worth mentioning nor do I frequent regularly frequent conferences. CCDC is my venue to give back to those who have made me better. I have to give a huge thanks to my employer Lares Consulting for all their contributions to me and CCDC as well. Chris and others have been strong supporters of CCDC events both regionally and nationally. Without question, they supported my desire to contribute as much as I could because they have recognized the theme of this post: passion. These are the most passionate group of people I’ve ever worked with and I couldn’t be happier.

Truth be told, I wouldn’t have the opportunity to work with them today if it wasn’t for a CCDC event four years ago.

 

And that concludes the 2014 CCDC season. Congratulations to University of Central Florida on your win, and huge kudos to everyone else involved. Time for me to go back to the drawing board and begin cooking up new ideas for next year.

Times to Let Pass

In the last six months, my two remaining grandparents, my dad’s dad, and my mom’s mom, have passed away. Both of them were fantastic individuals, dedicated to their families. They couldn’t have been more opposite if they tried. Grandpa “Buzz” was a scholastic enthusiast, constantly pushing everyone around him to learn more. “Mimi” was the epitome of the “Southern Belle”, a classic woman born and raised in Alabama who’s spirit was always vibrant. One would feed me books, the other, biscuits and gravy. Their presence in my life now forever seared into my memory. The good, the bad, and the ugly.

In a few months, I’ll hit 25 in age. According to Wikipedia, life expectancy in the US is a little over 77 years old. Take that down to 75 to account for my less than acceptable habits, and I’ve just marked the first third of my life complete. The three generations I grew up with has now reduced down to two.

The question I keep pondering is by what metric is our satisfaction derived from? Accomplishments? Experiences? Financial stability? I think it’s obvious that each of us has to figure that out for ourselves, but do we really contemplate it? Nevertheless act upon its guidance.

I don’t think I’ve figured out my answer yet. It could be something that evolves or that you constantly chase? It’s a question to at least try and get out in front of.

As I sat next to Mimi on her final days, it became clear to me that when you find yourself approaching deaths doorstep, finding salvation might be the most important thing. That salvation isn’t simply a religious experience, although I don’t deny it is the most critical part to some. I put myself in her shoes over and over again. “What would I tell myself? How would I come to terms with the situation?” She loved her life, just as I do. Putting the macho-ness of our masculine society in it’s place for a moment, I’m going to say that I believe the vast majority of us aren’t prepared for that reality. Witnessing someone else I deeply loved and cared for go through that was a wake up call for me.

I’ve made the choice to live a life of broad experiences. As I passively survey my Facebook friends I grew up with, it becomes clear I’ve taken the plunge out of the nest of normality. That’s not to say I’m doing better or worse than others – it just happened to be the decision I made. Look at the United States in the late 1700s. Much of the country continued to develop along the eastern seaboard while others, for whatever their reasoning was, moved west into the unknown. Regardless of the motivation, I deeply understand that thirst.

As the last 25 years has been one hell of a ride, I now look to the future. My salvation lies in that quest to charter the unknown, to brave the storm, and come out on the other side into the oasis untraveled. The wilderness is calling. Time to do what I do best.

Stay tuned for future updates.

CCDC 2013 Red Team Feedback

Thanks to the efforts of so many people, we’ve finished another great Collegiate Cyber Defense Competition season. I was fortunate to be invited to take part of the Western Regional and National red team. Being a former blue teamer, I thought it’d be worthwhile to share some feedback from my red team point of view. While related to CCDC, many of these points have been valuable for me in the work place.

Disclosure: My attacks targeted *nix systems, as well as web and databases, so my feedback will be scoped as such. The principles though should apply across the board.

Good Operations are a MUST

Any time I’m at a USCC or CCDC event, I ask the competitors about what they want to do. The most common answer I get is “I want to do penetration testing” or some other form of “offensive” security. While I don’t deny it’s a fun area within security, the parity of offensive and defensive skills cannot be underestimated. If you want to attack a network, show me you can defend one. Period. So how do you do that at CCDC?

When I was on blue team, I debated with others whether it was more valuable to be good at security or good at systems administration. Having now gone out into the field and worked as both an operations and a security person, I think I’ve settled on a conclusion.

They are symbiotic, and without one, the other WILL struggle.

You should practice being an operator of systems. Run linux at home. Use it. I can’t tell you how many times at CCDC I’ve watched a blue teamer Google search (while we watch through VNC) something like:

– How to reset MySQL password
– How to stop a service on Fedora
– How to install a package on FreeBSD

If you find yourself doing this, then you need to spend more time learning how common operating systems and network services work. To secure a system, you’ll have to know how that system works. You’ll never be effective until you do. Here is a short list you should know how to do without Googling on multiple OSes:

– OS User Administration: Users, Groups, Sudo, Permissions, Change Passwords
– Remote Access: OpenSSH Server, FTPd, VNC/RDP, Define ACLs
– Database Access Control: Change Passwords, Set Host Masks, Investigate possible PII

You should know how to use operating systems just as you know how to use Facebook or your cell phone. Which brings me to my next point:

Please, know how to use logs.

Since I started red teaming, I hear blue teams talking about rumors of kernel rootkits or “0day” that the red team has deployed. While complex attacks like that can happen, there is absolutely no need for me to use it if I can just log in via SSH as root with a default password. Carl, a fellow red teamer, put it best:

“Why would I waste valuable time packing malware in a way that won’t get detected if blue teams aren’t going to install anti-virus anyway?”

At both Regionals and Nationals, I start by leaving breadcrumbs of my access on a system. If you catch it, I know I need to cover all my tracks, but if you don’t, hell, I won’t waste my time. This is one of the key areas where good operations will create the foundation for good security.  Knowing where the important logs are on a system are and how to tail and search them is going to determine whether you retain ownership of your server or I claim it as my prize and eat popcorn while watching you Google.

You can do this in several ways. You can get fancy and use Splunk (free trial available) or simply use some tail -f log | grep. With these skills, you’ll be able to identify breaches more rapidly and begin the proper incident response process to fix the problem. Which is another great segway into my final point:

Think twice before killing my shell.

When you see (where root is not you):

alex@ccdc-test-1:~$ who
alex pts/0 2013-04-23 18:24 (192.168.1.1)
root pts/1 2013-04-23 17:14 (192.168.1.129)
alex@ccdc-test-1:~$


Figure out how root got access before killing his shell. This gets brought up every red team debrief, but I cannot stress this enough. I know the feeling. I was blue team once. When you know that red team is on the other end of a keyboard logged into your box, you want nothing more than to fix the situation, you panic and overreact. In that moment, you need to rely on your operations skill and do quality incident response. The basic incident response process should look like this:

1. Gather Information: Figure out what is going on, gather evidence from multiple sources
2. Create Hypothesis Based on Information: Now you have a global view of whats going on
3. Determine Remediation Steps: Note Plural. If this comes out to be “Kill The Process” or “Kick Him Out!” then you’ve failed this step. Make this technically detailed. It should be supported by your evidence and hypothesis.
4. Execute Plan: Now you act.
5. Detail Report: If you’ve followed steps 1-4, you’ll have already done the work to submit quality incident reports to the white team. Every point counts and following this process will help ensure you get something for the inevitable breach.

Conclusion

Whether for CCDC or work, these points will make you a better security professional. I don’t care if you’re interested in offensive or defensive security, your operations kung fu will be a valuable asset to you and your team.

For all the competitors this year, great work. Regardless of winning, simply competing is experience that you will use for a lifetime and thats worth more than any medal or award. I do want to extend a special congratulations to my old school RIT for taking home the Cup. Feel free to join myself and others in the #ccdc channel inside Freenode. This is a great community and I’m proud to be a part of it.

$ ssh 2014.ccdc
$ /etc/init.d/planning start


The Biggest Need in Cyber Security

After seeing this come across my news feed and several people comment on it, I thought I’d dump some thoughts. Maybe someday I’ll collect this into a more coherent editorial, but for now this will have to do (lots of code to write, haha).

Basically, if you didn’t read the article, the White House Military Office had a piece of malware installed on it through a spear-phishing attack. Someone clicked on a link in an email that infected their system. The White House has confirmed it, but also confirmed what I suspected – it was on a declassified civilian network and not the ones with the “nuclear codes” or other top secret information. And while everyone freaks out over this, I’d like to present some thoughts on why the WHMO hack is less important than you think, and this other problem is far more urgent than anyone, IT professional or otherwise, realizes.

My biggest fear is not that the US Government is going to get hacked and a nuclear weapon is going to be launched. What keeps me up at night is the fact that hundreds of thousands of US businesses don’t maintain any sort of security in their computer systems. Estimates have put the figure as high as 85% of small to medium businesses are not doing enough to secure themselves and are therefore vulnerable. Their failures are not negligence either. Technology has progressed at such a high rate, the systems administrators have hardly been able to keep up. Information Security used to be a job that could be managed by the administrators, now it’s such a complex science(see: art) that you need highly specialized people just to meet requirements. This is a very hard subjects for Republicans and Democrats alike. Democrats cry out for regulation. Republicans cry out for defense spending.

Here’s the problem with both: there is no where near the resources needed to do either. Same with Republican defense spending. You can give the DoD all the money in the world or regulate all you’d like, we simply don’t have the human capital to protect the countless computer networks our world has grown to rely on. Half the reason US Government networks are insecure is the man power. Regulation auditors or Cyber “forces”. Take your pick, the skill sets required for each are very similar.

And the estimates right now for human capital say we need 10,000 top tier cyber experts immediately and another 30,000 over the next 5 years. Currently, it’s estimated that there are less than 1000 people in the United States with enough skill to be effective. The Chinese can hammer our government networks all they’d like. They do, but don’t get far. Trust me.  I’ve worked closely with several people on this problem. Alan Paller, Chair and Director on Board, at the SANS Institute as well as Karen Evans, Former CIO of the United States under President Bush, have both extensively researched this lack. Here’s a paper delivered to President Obama in November 2010 outlining this issue.

Hackers also hammer any business they can in the United States, with most small businesses completely oblivious to their penetration. They infiltration, they steal – and usually not money, mostly intellectual property – and they spread. They maintain a low profile and most anti-virus that even up to date won’t protect against it. Good luck having your contract IT worker cleaning them out. As the young generation grows up learning technology, it becomes taking candy from a baby to do these things. If Sony’s breach showed us anything, it was that even the biggest companies are extremely vulnerable. And the best hackers? You never hear about their work. That’s how good they are.

It’s these businesses that are being hurt, and in turn our economy, by this cyber “war”. Being part of that elite 1000 has motivated me to attempt to grow the size. Although I’m blessed with some of the best job security of any career field in the world right now, it pains me to see us losing this fight. I have dedicated a substantial amount of my time to volunteer high school and college programs geared towards training and identifying the best young cyber talent in the country and put it on a path to an effective career. So far in the US Cyber Challenge (USCC), we have brought through over 1000 competitors nationally who have shown potential in the cyber field and gotten them in better training programs.

How do I know these programs work? I was one of them. I came through the US Cyber Challenge in the Summer of 2010. I’ve since dedicated as much time to them outside of work as possible. I also have strongly supported the Collegiate Cyber Defense Competition (CCDC). I was a competitor throughout my time as a student at the Rochester Institute of Technology and also as a volunteer red teamer, both regionally and nationally. CCDC now has over 100 schools competing in a national bracket. It is truly the NCAA of cyber security. Dwayne Williams and his staff are the best in class for competitive cyber security.

The biggest gap in security we have is security education. And as much as I have spoken about above, there is still a huge need.

• We need to find and groom the best young talent to increase the work force.
• We need to re-tool our current IT work force with more security knowledge that they can apply to their jobs.
• We need to bring security into the mainstream. Typing classes are nice, but with kids sitting behind screens from such young ages, it’s important that we train all youth in good cyber “hygiene”.

Wherever you fall in those three points, make a difference. The safety of our digital world depend on it.