CCDC 2013 Red Team Feedback

Thanks to the efforts of so many people, we’ve finished another great Collegiate Cyber Defense Competition season. I was fortunate to be invited to take part of the Western Regional and National red team. Being a former blue teamer, I thought it’d be worthwhile to share some feedback from my red team point of view. While related to CCDC, many of these points have been valuable for me in the work place.

Disclosure: My attacks targeted *nix systems, as well as web and databases, so my feedback will be scoped as such. The principles though should apply across the board.

Good Operations are a MUST

Any time I’m at a USCC or CCDC event, I ask the competitors about what they want to do. The most common answer I get is “I want to do penetration testing” or some other form of “offensive” security. While I don’t deny it’s a fun area within security, the parity of offensive and defensive skills cannot be underestimated. If you want to attack a network, show me you can defend one. Period. So how do you do that at CCDC?

When I was on blue team, I debated with others whether it was more valuable to be good at security or good at systems administration. Having now gone out into the field and worked as both an operations and a security person, I think I’ve settled on a conclusion.

They are symbiotic, and without one, the other WILL struggle.

You should practice being an operator of systems. Run linux at home. Use it. I can’t tell you how many times at CCDC I’ve watched a blue teamer Google search (while we watch through VNC) something like:

– How to reset MySQL password
– How to stop a service on Fedora
– How to install a package on FreeBSD

If you find yourself doing this, then you need to spend more time learning how common operating systems and network services work. To secure a system, you’ll have to know how that system works. You’ll never be effective until you do. Here is a short list you should know how to do without Googling on multiple OSes:

– OS User Administration: Users, Groups, Sudo, Permissions, Change Passwords
– Remote Access: OpenSSH Server, FTPd, VNC/RDP, Define ACLs
– Database Access Control: Change Passwords, Set Host Masks, Investigate possible PII

You should know how to use operating systems just as you know how to use Facebook or your cell phone. Which brings me to my next point:

Please, know how to use logs.

Since I started red teaming, I hear blue teams talking about rumors of kernel rootkits or “0day” that the red team has deployed. While complex attacks like that can happen, there is absolutely no need for me to use it if I can just log in via SSH as root with a default password. Carl, a fellow red teamer, put it best:

“Why would I waste valuable time packing malware in a way that won’t get detected if blue teams aren’t going to install anti-virus anyway?”

At both Regionals and Nationals, I start by leaving breadcrumbs of my access on a system. If you catch it, I know I need to cover all my tracks, but if you don’t, hell, I won’t waste my time. This is one of the key areas where good operations will create the foundation for good security.  Knowing where the important logs are on a system are and how to tail and search them is going to determine whether you retain ownership of your server or I claim it as my prize and eat popcorn while watching you Google.

You can do this in several ways. You can get fancy and use Splunk (free trial available) or simply use some tail -f log | grep. With these skills, you’ll be able to identify breaches more rapidly and begin the proper incident response process to fix the problem. Which is another great segway into my final point:

Think twice before killing my shell.

When you see (where root is not you):

alex@ccdc-test-1:~$ who
alex pts/0 2013-04-23 18:24 (192.168.1.1)
root pts/1 2013-04-23 17:14 (192.168.1.129)
alex@ccdc-test-1:~$


Figure out how root got access before killing his shell. This gets brought up every red team debrief, but I cannot stress this enough. I know the feeling. I was blue team once. When you know that red team is on the other end of a keyboard logged into your box, you want nothing more than to fix the situation, you panic and overreact. In that moment, you need to rely on your operations skill and do quality incident response. The basic incident response process should look like this:

1. Gather Information: Figure out what is going on, gather evidence from multiple sources
2. Create Hypothesis Based on Information: Now you have a global view of whats going on
3. Determine Remediation Steps: Note Plural. If this comes out to be “Kill The Process” or “Kick Him Out!” then you’ve failed this step. Make this technically detailed. It should be supported by your evidence and hypothesis.
4. Execute Plan: Now you act.
5. Detail Report: If you’ve followed steps 1-4, you’ll have already done the work to submit quality incident reports to the white team. Every point counts and following this process will help ensure you get something for the inevitable breach.

Conclusion

Whether for CCDC or work, these points will make you a better security professional. I don’t care if you’re interested in offensive or defensive security, your operations kung fu will be a valuable asset to you and your team.

For all the competitors this year, great work. Regardless of winning, simply competing is experience that you will use for a lifetime and thats worth more than any medal or award. I do want to extend a special congratulations to my old school RIT for taking home the Cup. Feel free to join myself and others in the #ccdc channel inside Freenode. This is a great community and I’m proud to be a part of it.

$ ssh 2014.ccdc
$ /etc/init.d/planning start