Concerning Blizzard and Diablo 3’s Disappointing Launch

Well it’s finally here. May 15th, 2012. Launch day for Diablo 3.

and….fail. Cannot connect to the game. One of several error screens. This is true for most people across the twittersphere and multitude of Skype and IRC channels I’m in.

Here’s what I want to know: How can a company with Blizzard’s resources, Diablo 3’s time, and Activision’s budgets flop a launch day this bad? I mean, they’ve had a beta test for over six months now. I’ve got a hypothesis. It’s in their IT Management’s “business” philosophy. Let’s break it down.

Typical IT management mindset is we should minimize cost by making the most out of the least amount of resources as possible. Too many unused resources and thats a waste, too few and you’re over utilized. This serves most IT budgets quite well. But you’re the IT manager at Blizzard trying to plan resource allocation for launch day. You’ve already benchmarked through the beta how much resources each user’s connection consumes (hopefully). Now you need to estimate how many people are going to try to connect at launch. So you look at historical data – Diablo II, WoW, Starcraft II, etc. and from there estimate a number. Then plan for that. There’s your IT Capacity Planning for a Diablo 3 launch.

I think it was done poorly, and here’s why.

Put that formula aside. Say you sold 1 million pre-release copies. History shows games like Diablo 3 make their money over time – the hardcore fans upfront, then popularity from those watching the hardcore fan generates post revenue buzz. (aka I play D3 for six months then introduce it to an unknowing friend). And now with the Auction House and other new features, I’m sure there will be additional revenue generated post launch. The point is, you don’t sell most of your licenses in a pre-release. But those pre-release licenses are your most important players (monetarily). Those are the most likely to purchase other things – expansion packs, DLC, etc. Making these customers happy should be your bottom line. Without them, you do not have the religious following for Diablo.

This is why capacity planning for a launch like Diablo 3 should have had much more importance put on it than it appears Blizzard did. Where is your cloud computing technology Blizzard? Capacity planning is not just in throwing more servers at it. You must build your server architectures to scale quickly and efficiently. Zynga’s Hybrid Cloud does this.

Here are some steps you can take to make sure your company doesn’t repeat something like this (and your fans aren’t blogging about you instead of using your product):

  • Build a platform that can automatically scale with demand
  • Write good horizontal scaling support into your application code
  • Make sure your IT department has the data center space (public / private cloud?) ready to meet the demand of the growth.

This is a great example of where a combination of good engineering and smart management can easily tackle a large and complex problem.

Thoughts or comments? Follow me on Twitter.

Why #OccupyWallStreet Matters

Up until now, I haven’t been very involved or excited about the #OccupyWallStreet protests. While the protests are circling around economic reform, a new subject of debate, and ultimately protest, has surfaced.

The big three media companies – Fox, NBC, & CNN have knowingly shelved the story as much as possible. They’ve executed the common practice of not completely turning it off, but giving it as minimal coverage as possible. Even the liberal slanted MSNBC only has it as a secondary story under their U.S. section.

And guess what – these companies are traded publicly on where else? Wall Street.

This is not right. I believe these companies to be denying coverage to such an important event in an attempt to stem the growth of this movement. Regardless of whether I agree with the protest methods or not, I think this blatant media shun is exactly why it’s time to take this protest big.

The largest problem in the world today is the systemic corruption that is plaguing every corner of mankind. Politics, Finance, Business. You name it. Corruption has made it easy for the top 1% to grow off the shoulders of the bottom 99%.

All of the issues the world faces are solvable except when it goes against the interests of the people running the show.

I don’t care if you’re rich or poor, educated or not. I think at this point in our history, we need to start taking a stand against this great pandemic of greed that has swept our culture.

I’m an engineer with a good job and even though I’m not the majority, I’ll stand with them on this one. I encourage you to show you support and spread the word. Attend the protests if you can. Show that this executive mentality will not be tolerated in our society anymore.

#OccupyWallStreet #OccupySF

Applegate: Beyond the Hype

Today, MacRumors came public with an e-mail exchange with Steve Jobs in regards to the Applegate controversy:

Q: Steve,

Could you please explain the necessity of the passive location-tracking tool embedded in my iPhone? It’s kind of unnerving knowing that my exact location is being recorded at all times. Maybe you could shed some light on this for me before I switch to a Droid. They don’t track me.

A: Oh yes they do. We don’t track anyone. The info circulating around is false.

Sent from my iPhone

Considering my original post and it’s follow up, I think it’s safe to say that this issue has been over-hyped into a whirlwind of speculation. The brass tax of the situation is this:

  • Apple iOS Devices have Location Services that provide rich functionality to the iOS experience.
  • Apple’s Opt-In Location Collection simply improves Location Services as stated in the F-Secure blog.
  • Apple is not playing big brother nor monitoring anyone.

This seems contrary to the WSJ article about iPhone location that just came out. That’s because no one is recognizing what they’re talking about – an iPhone. This mystical device has captured our attention since it’s inception in 2007 and since then, the public has had to continuously shift it’s view on what this device (and others like it – Android, etc.) are capable of. Whether we like it or not, the age of mobile is here, and in a big way. Mobile computing will continue it’s production of data – sensitive or not – and it’s up to us in the information security community to lead the public in the proper usage of these devices.

Critique Apple all you’d like. They’ve implemented Backup Encryption, Lock Passcodes, Sandbox’d Applications, and an Opt-In model that clearly doesn’t “big brother” users. These steps aren’t useless features, they’re there for a reason. If you’re not comfortable telling your Photos App your location, don’t allow it to. If you’re worried about your data (and you should be!) educate yourself and do something about it. Your data sits on your phone just as your credit card sits in your wallet. Protect it.

For better details on how to do this, please see my previous post.

3 New Thoughts on Mobile Location – A Follow up to Apple Location Tracking

Well, this has been an exciting 24 hours. I needed a break after a lack of sleep the previous night. I’m glad to see so much technical research being done now. Great work to everyone. I’ll begin linking interesting analysis through my Twitter feed as I come across it. You can follow me @alexlevinson.

Earlier today, Pete Warden and Alasdair Allan responded to me. I want to thank them for their words. I do not want to spill hate on them or their work. I wish them both the best in their ventures and would like to cross paths with them in San Francisco sometime. Fabulous work gentlemen.

What is Location Services?

I think several people have missed understanding how Location Services functions within an iOS device (and probably mobile in general) and the impact it has.The iPhone’s popularity comes from it’s human computer interaction. It’s smartphone capabilities are extended to the user in all sorts of unique and useful ways. Where am I? What’s Pizza Hut’s number? A smartphone has now replaced your fellow citizen for information. This data (or accessibility to determine) is fueling the movement in today’s digital world.One of those aspects as we know is Location Services, derived on the iPhone through this file, consolidated.db. This file acts as a hub of logging for geolocational information on the device. The various radios in the device log information to this database.When developing applications for an iOS device, you can access a series of libraries created by Apple to harness the capability of the phone. Why do you need these? Because your app runs in a “sandbox” — a jail cell for your application that prevents it from interacting with other applications or data on the device. This is a positive both for performance and security on the device. The “sandbox” design is not new to Apple. Unix and Linux systems have been jailing applications for years.

So unlike desktop computing, applications on the iPhone do not talk to one another or interact with system data. This is why third party applications do not have access to consolidated.db. They can manipulate and manage data within their own jailed directory, but nothing more.

Apple gives application developers access to programming frameworks that contain APIs to interact with data on the iPhone. This is how you can import your Facebook contacts, use Skype, or checkin on FourSquare. This allows an app to use the Core Location API to interact with location in a ways that expand the iOS user experience. CoreLocation uses consolidated.db to facilitate this transaction of data. So an app must use this API to begin interpreting location data. There is no other way. Given the static nature of these API functions, only certain data can be interpreted by apps, preventing them from simply harvesting all data within consolidated.db.

Apple previously used Skyhook Wireless for their determining the location of iPhones, but announced in 2010 they would be moving to their own location services starting in April of 2010 with iOS 3.2. This would confirm why there is data transmitted to Apple every 12 hours as reported by F-Secure yesterday. This explains my failure to see this unsolicited location data. My network traffic analysis of iOS devices has never spanned large time intervals. This still doesn’t translate into Apple tracking users behind their back.

I think overall, this location data is being used to further the iOS experience. An iPhone wouldn’t be an iPhone if you couldn’t press a button and have it tell me where the nearest Apple store or find a popular restaurant on Yelp. That doesn’t mean that this data is not sensitive though, which brings me to my next point.

The Advent of Mobile Data Security

Over the last century, we’ve seen data move from handwritten letters to typewriters and computers. Security measures have always existed to protect sensitive data. We’ve learned the painful mistakes of mismanaging digital information through the computer age and are beginning to create policy that helps ensure data “C – I – A.” – confidentiality, integrity, and availability. One of the outcomes of all this work is that there is no smoking gun to ensure privacy of data and a large responsibility lies on the person who owns the technology the data resides on.

In the last few years, mobile devices have swept the country as a new platform of computing – yes, computing. An iPhone is much more than a phone just as an iPod Touch is much more than a music player. These devices (including droids, blackberrys, etc.) are becoming a huge platform for computing on the go. Even with their boom, it’s a relatively new market and just as other technology markets, it’s still learning, especially about security.

The public perception of the mobile device as “just a phone” has led people to underestimate what these mini-computers are capable of. With this new breed of functionality, comes new blobs of data that is stored on these devices. Users should be treating their smart phones just as they do their computers – using passwords, following safety recommendations, encrypting data when possible. Your phone is now becoming your second wallet – a repository of data relating to you, the user, that could be harmful if exposed or improperly used. Although sensitive, it’s this data that drives the smart functionality of these devices.

Surely Apple has thought of this in the design of the iPhone. Through several avenues, Apple is attempting to protect your data on these devices. Following simple steps you can treat your phone like a digital fortress and protect the information thats on it. For example, there are simple and easy techniques that you can take to protect your data, including this location information.

1) Enable Backup Encryption & Use a Lock Passcode

After connecting your iPhone to your computer, you can set a “Backup Encryption Password” in iTunes. This does more than just encrypt your backups – it actually encrypts all data that could be retrieved from your phone physically. In the event you loose your phone, this can prevent someone from just downloading all the data off your device. Using a Lock Passcode can also stop anyone from simply viewing data on your phone who might grab it.

2) Use Caution Surfing the Web, E-Mailing

Just as you can get viruses through malicious websites and emails on your computer, the same is possible on your device. (That’s actually how works – it’s just exploiting flaws in your web browser). These devices are mini-computers, if a hacker can exploit the software running on your device, they can do virtually anything they want – including steal your sensitive data.

3) Be Careful Modifying Your Device (jailbreak)

Some will find this controversial, but I think it’s sound advice to all but expert users. The problem I have with the jailbreak system is it allows for anything to be run on your device. You break the walled garden approach of the App Store and in turn expose yourself to a world of malware you otherwise wouldn’t. This has shown in the Android Marketplace’s recent spike in malware. Also, system privileges are now changeable by a user and the software they install. This allows access to the filesystem on the phone – including consolidated.db. So while there may be an app to sanitize consolidated.db, their also might be an app that is secretly stealing your SMS or logging your usage in the jailbreak realm.

Mobile device security is only going to become a bigger issue. It’s in the best interest of all users to make sure they’re staying on top of this stuff. Finally, I want to start the discussion on some things that have been reported on lately.

Forensic Legal Issues

It has been reported on extensively that mobile devices are under fire from law enforcement. I think this is an interesting twist in the recent headlines of mobile data. In the world of forensics, we’ve known for some time that these devices contain lots of identifiable information. It’s troubling to hear accusations of law enforcement misusing forensic technology to incriminate or investigate. As someone who develops this technology, there are rules that Law Enforcement must abide by — the fourth amendment. It’s is in our EULA for Lantern that you may only use our software legally:

10. Illegal use.  You may not (and You covenant not to) use the Software for any illegal purpose; for example, you may not use the Software to access a computer or device on which you do not have authorized access.

Fortunately, through Katana Forensics interaction with law enforcement, I have not witnessed any such misuse. My interactions with law enforcement have led me to respect their efforts to make sure their efforts present legal validity – warrants, chain of custody, and what not.

It’s in the best interest of the public to make sure they understand the 4th amendment and what their rights are under it and also examine their state’s law on electronic search and seizure. Knowing the law in your current area can help you as a technology owner keep your data secure from illegal search and seizure.

Overall, I don’t think Apple has wronged the public. I think they’ve tried to incorporate security measures into an extremely interactive and data rich product. Using simple security techniques, users can secure their data while maintaining functionality. I’d love to hear your comments – feel free to tweet me @alexlevinson with your feedback.

3 Major Issues with the Latest iPhone Tracking “Discovery”


Today, two researchers for O’Reilly media published an article claiming discovery of a hidden tracking system on the iOS 4 operating system. Using simple techniques, Alasdair Allan and Pete Warden extracted data off of an iOS version 4 device and wrote an open source software utility to effectively graph this data onto a map. As a fellow researcher, I champion their creativity and their development. As an expert in this field, I have three points of argument to raise.

1) Apple is not collecting this data.

And to suggest otherwise is completely misrepresenting Apple. I quote:

Apple is gathering this data, but it’s clearly intentional, as the database is being restored across backups, and even device migrations.

Apple is not harvesting this data from your device. This is data on the device that you as the customer purchased and unless they can show concrete evidence supporting this claim – network traffic analysis of connections to Apple servers – I rebut this claim in full. Through my research in this field and all traffic analysis I have performed, not once have I seen this data traverse a network. As rich of data as this might be, it’s actually illegal under California state law:

(a) No person or entity in this state shall use an electronic tracking device to determine the location or movement of a person.

I don’t think that’s a legal battle Apple wants to face considering the sale of over 100 million iDevices worldwide. That raises the question – how is this data used? It’s used all the time by software running on the phone. Built-In applications such as Maps and Camera use this geolocational data to operate. Apple provides an API for access to location awareness called Core Location. Here is Apple’s description of this softare library:

The Core Location framework lets you determine the current location or heading associated with a device. The framework uses the available hardware to determine the user’s position and heading. You use the classes and protocols in this framework to configure and schedule the delivery of location and heading events. You can also use it to define geographic regions and monitor when the user crosses the boundaries of those regions.

Seems pretty clear. So now the question becomes why did this “hidden” file secretly appear in iOS 4?

2) This hidden file is neither new nor secret.

It’s just moved. Location services have been available to the Apple device for some time. Understand what this file is – a log generated by the various radios and sensors located within the device. This file is utilized by several operations on the device that actually is what makes this device pretty “smart”. This file existed in a different form prior to iOS 4, but not in form it is today.

Currently, consolidated.db lies within the “User Data Partition” on the device. This is a logical filesystem that maintains non-system level privileges and where most of the data is stored. When you perform an iOS Backup through iTunes, it is backing up this partition. Prior to iOS 4, a file called h-cells.plist actually existed in the /root/Library/caches/locationd folder, but with hidden access from other software and applications. h-cells.plist contained much of the same information regarding baseband radio locations as consolidated.db does now, but in Apple Property List format rather than sqlite3. Through my work with various law enforcement agencies, we’ve used h-cells.plist on devices older than iOS 4 to harvest geolocational evidence from iOS devices.

So lets recap.

h-cells.plist = Pre iOS 4 / Radio Logs including Geolocational Data / Hidden from Forensic Extraction (usually)

consolidated.db = iOS 4+ / Radio logs including geolocational Data / Easily acquired through simple forensic techniques

The change comes with a feature introduced in iOS 4 – Mutlitasking and Background Location Services. Apps now have to use Apple’s API to operate in the background – remember, this is not pure unix we’re dealing with – it is only a logical multitasking through Apple’s API. Because of these new APIs and the sandbox design of 3rd party applications, Apple had to move access to this data. Either way, it is not secret, malicious, or hidden. Users still have to approve location access to any application and have the ability to instantly turn off location services to applications inside the Settings menu on their device. That does not stop the generation of these logs, however, it simply prevents applications from utilizing the APIs to access the data.

3) This “discovery” was published months ago.

I understand that Mr. Allan and Mr. Warden are valued researchers for O’Reilly, but they have completely missed the boat on this one. In the spirit of academia, due diligence is a must to determine who else has done such research. Mr. Allan, Mr. Warden, and O’Reilly have overlooked and failed to cite an entire area of research that has already been done on this subject and claimed full authorship of it. Let’s break down my history:

Back in 2010 when the iPad first came out, I did a research project at the Rochester Institute of Technology on Apple forensics. Professor Bill Stackpole of the Networking, Security, & Systems Administration Department was teaching a computer forensics course and pitched the idea of doing forensic analysis on my recently acquired iPad. We purchased a few utilities and began studying the various components of apple mobile devices. We discovered three things:

  • Third Party Application data can contain usernames, passwords, and interpersonal communication data, usually in plain text.
  • Apple configurations and logs contain lots of network and communication related data.
  • Geolocational Artifacts were one of the single most important forensic vectors found on these devices.

After presenting that project to Professor Stackpole’s forensic class, I began work last summer with Sean Morrissey, managing director of Katana Forensics on it’s iOS Forensic Software utility, Lantern. While developing with Sean, I continued to work with Professor Stackpole an academic paper outlining our findings in the Apple Forensic field. This paper was accepted for publication into the Hawaii International Conference for System Sciences 44 and is now an IEEE Publication. I presented on it in January in Hawaii and during my presentation discussed consolidated.db and it’s contents with my audience – my paper was written prior to iOS 4 coming out, but my presentation was updated to include iOS 4 artifacts.

Throughout the summer, I worked extensively with Sean on both developing Lantern and writing custom software to interpret forensic data for customers of ours who needed better ways of searching for and interpreting data.

When the iPhone 4 came out, I was one of the first people in San Francisco to grab one (yes I waited to be in the front of that awful line).

Me in Line for the iPhone 4 in San Francisco

( Look for the RIT shirt )

Within 24 hours of the iPhone 4’s release, we had updated Lantern to support forensic analysis of iOS 4.0 devices. Within 36 hours, we had began writing code to investigate consolidated.db. Once a jailbreak came out for iOS 4, I wrote a small proof of concept application to harvest the contents of consolidated.db and feed it to a server for remote location tracking.

Ever since then, location artifacts have been a main area of interest for me. I’m now the Lead Engineer for Katana Forensics leading all technical research and development of both Lantern and private utilities. I travelled to Salt Lake City, UT in November for the Paraben Forensics Innovation Conference (PFIC) and presented with Sean on iOS Forensics including the content of consolidated.db. At that same conference, Sean and I announced the development of Lantern 2.0 which would fully support the interrogation of consolidated.db and other geolocational artifacts scattered throughout the device.

Sean and I even wrote a book detailing iOS forensics involving iOS 4 devices that came out on December 5th, 2010.

Sean Morrissey, Primary Author, Alex Levinson, Contributor

In the course of writing Chapter 10 – Network Forensics – I fully explain and detail the examination of consolidated.db and other network artifacts within the device!

Page 335 - Continued on page 336.

In February of 2011, Sean and I previewed Lantern 2.0 at the DoD Cyber Crimes Conference in Washington, DC including our geolocational features. Lantern 2.0 has been on the market for months now and performs the same functionality Mr. Warden’s utility does and much more. We correlate geolocational data embedded in images and third party application. We give you a geolocational timeline of events in list view showing much more than baseband logs within consolidated.db.

While forensics isn’t in the forefront of technology headlines these days, that doesn’t mean critical research isn’t being done surrounding areas such as mobile devices. I have no problem with what Mr. Warden and Mr. Allan have created or presented on, but I do take issue with them making erroneous claims and not citing previously published work. I’m all for creative development and research, as long as it’s honest.