Every developer hopes for huge user bases populated by large amounts of monthly users. People using an application to its potential in the thousands, if not hundreds of thousands, is a dream come true. Unfortunately for API services, with a greater deal of exposure comes a marked increase in vulnerability.

Key to this topic is the concept of reverse-engineering; the process of breaking apart a function to its base characteristics and discovering how to use them for other purposes. While these discussions are often theoretical, limited to thought experiments and permutations of other successes and failures, this discussion draws on a recent real-world example — Pokémon Go.

Now that the dust has settled, we’re going to discuss the launch of Pokémon Go, the process and effect of the reverse engineering it endured, and some lessons learned that can be used to prevent similar issues in other ecosystems.

Fanboy Rising

“The people that were playing around with the API… They weren’t hacktivists, they were actually game enthusiasts that knew about programming — they’re not really bad guys.”

Pokémon Go was an instant smash hit — something that’s not hard to believe given the Pokémon brand fame. While the game would eventually peak at 45 million concurrent users, issues began to arise even during the initial release week. For example, early versions of Pokémon Go had very little in the way of security for some basic core features, such as geolocation. Players caught wind of this, and immediately used it for their own gain.

Reverse engineering the geolocation service and serving it false geodata, players were able to “control” their character in-game, ignoring the app’s basic premise of capturing Pokémon in the player’s immediate vicinity and instead allowing players to travel across the world, hatching eggs in minutes rather than hours.

Additionally, players were able to reverse engineer the way the application handled the Pokedex, revealing all the current Pokémon and their spawn locations (and specifically which Pokémon were locked to specific regions).

This was largely driven by the fans, of course — many fans felt angered at a lack of clarity in how certain functions such as the Pokémon Proximity service worked and how accurate it actually was, and decided to figure out for themselves. They used simple man in the middle attacks, intercepting application communication, and began their work at digging through the application functions.

The fact that servers were not prepared to handle such large volumes of users was an issue as well — early issues with joining servers and the resulting slow load times and lack of accurate data kicked off discontentment quite early in the lifecycle.

Soon enough, hackers began to use automation in their process of reverse engineering, moving away from individual testing and into bulk command testing. This tipped off Niantic (in addition to the media coverage of the security faults), who responded quickly.

Their response was specific and targeted at these fans who were reverse engineering — first and foremost, Niantic implemented certificate pinning in an attempt to end man in the middle attacks, and then further disabled the Pokémon Proximity functionality in an attempt to break modified applications who used the function in unintended ways.

The damage in many ways was already done. Because of what the players learned from their reverse engineering before the patch was pushed, certificate pinning was broken almost immediately, and the proximity function, a function many players were angered at losing, was restored.

Not only was this functionality restored, in fact — it was expanded. Applications and websites began to pop up utilizing the feature to show exact GPS locations of nearby Pokémon, negating much of the apps purpose as a social platform and piggybacking onto the ad revenue generation.

Nitantic then responded with a further patch implementing “unknown6” encryption, supporting checksum authentication to ensure the application requests were indeed legitimate and coming from a known client. Again, however, the fans used information gathered in the initial reverse engineer to expose the API and figure out how the encryption was handled client side — because of this, the new encryption was broken in four days, circumventing the checksum protection.

Niantic responded with an urgent fix when the checksum was broken, implementing root checks and requiring Captcha completions. They followed this patch with a round of legal threats against well-known community engineers.

While the application is much more secure now, the effects of the numerous rounds of reverse engineering were impactful. Niantic lost brand image, largely because many fans saw their actions as being against their enjoyment — which many felt was unfair to Niantic, as the application was designed to be used a certain way, and the only people affected were those using them in unauthorized ways.

This of course only caused more disagreement and more brand damage.

There is then, of course, the loss in player happiness. User experience is key to the long-term happiness and success of an application, and losing the goodwill of the player base, whether justified or not, is never a good thing.

Finally, Niantic lost revenue both due to redirected revenue streams from third party implementations and the unplanned engineering effort and cost to mitigate the issue at hand.

4 Important Lessons Learned

So with that in mind, what can we deduce from the situation faced by Niantic?

First and foremost is the fact that communication is an incredibly important and vital currency in the world of mobile development. Much of the initial reverse engineering was in response to a perceived lack of clarity. Had the systems been explained or elaborated on, initial hacking might have been much more limited in scope or purpose.

Secondly is the fact that Niantic did not plan for success. Because of the sheer volume of users unleashed upon Pokémon Go, certain functionality did not work as intended. Server balancing often resulted in slow loading, and proximity functionality was not always accurate.

Third was the issue of continual updating in hotfix format to a known issue. When Niantic updated their security approaches, they were updating already exposed parts, and because they reused certain data and methodologies, their new functions were easily broken.

Fourth, Niantic committed what many consider a cardinal sin — they made their user base angry. While control over an application is incredibly important, by stifling initial exploration and not directly informing users as to functionality they alienated some users and embittered others. This led to the actions that so harmed them.

Finally, Niantic suffered from having too many attack vectors, and did not have a chance to properly respond before another issue was at the forefront. This severely dampened the effectiveness of their chosen solutions, and led to new issues with each fix.

Solutions

There are some solutions to these issues, thankfully. While there can be much discussion over whether or not Niantic deserved or was responsible for the issues they faced (this author believes they were not), the issues they faced can be used as the basis of new steps and procedures for a developer introducing an app into the mobile space.

Communication

Communication is vital — that cannot be stressed enough. While a certain amount of information should not be shared, Niantic could have made it clearer how certain functions worked. This range of communication could have been as formal as setting up a forum, creating a wiki site, or even issuing a complete guide on their site. Conversely, it could have been something as simple as an official statement on Twitter.

What really matters is the communication aspect. Stating that geodata was having accuracy issues in the proximity function because of overload, if that was indeed the issue, would have gone a long way towards convincing a lot of those who reverse engineered to leave the application alone.

The same goes for the response updates. If Niantic had communicated with tech outlets and fans directly, stating what the issues were, why they’ve implemented a new security feature, and what their plans are, they could have implemented their security features and assuaged fears that the features people were fighting to get back would indeed come back — and in a legit fashion.

A good deal of this was of course fans refusing to listen — this data was out there in one form or another, but it could have been, and in the future should be, shared in a more complete and prescient way.

Plan for Incredible Success

“The fact is that most companies don’t plan for success — they aren’t ready for success.”

Niantic was in many ways unprepared for just how successful Pokémon Go would be — and who could blame them? Initial numbers rocketed past expectations and into the lofty realm of virality, and stayed there for weeks.

The sheer number of requests became hard to deal with, causing issues in the community for server loading and stability. This caused anger, but also made the system much more exposed to memory faults and other such attacks.

The best approach here is to plan for incredible success. Take your wildest dreams of success, then triple them — that’s the gold standard you should develop for. Code in instancing. Subscribe to virtualization practices to spin up new servers where load balancing is needed. Implement proper volumetric filtering.

Most importantly, properly implement encryption and authentication from the get go, and obfuscate the API — the more users you get, the better the application does, but the more vulnerable it inevitably is.

Update, Not Hotfix

Hotfixing is a bandaid — and a bandaid is only good for temporary, small wounds. When such huge holes in security were discovered, the very first step Niantic took should have been the very last step they took in implementing security features to stamp out illicit use.

Instead, they were forced to use hotfixing. Hotfixes are not only a half-baked update, they are often only a temporary stopgap, and add extra developer effort and time that may be better spent on simply notifying users of the security issue, and developing a comprehensive responsive update.

While an argument could be made for plugging a dangerous hole quickly before waiting for a comprehensive update, hotfixes can introduce additional issues (as they did in this case), causing a feedback loop of hotfix upon hotfix.

Better yet, pre-empt these updates by simply utilizing the proper solution to begin with — just like with planning for success, plan for failure. Look at your worst case attack scenario, triple it, and prepare for the onslaught. If you prepare for a war with 1000 warriors, and only 200 opponents show up to the battlefield, you will win 100% of the time. Treat APIs the same way. Prepare for the worst, hope for the best.

Make The Fans, Fans

“If you get people, and annoy them […] they can turn against you. And when they turn against you — they can do some serious damage.”

People who use your application do so because they see value. Whether that value be in the form of entertainment or in attaching themselves to the revenue stream, users are using your app for a purpose. When Niantic updated their application to prevent abuse, they alienated many users who felt they were being treated unfairly.

Arguments could be made that this feeling was unjustified, but the simple fact is they felt it — and because of that, reverse engineer efforts doubled down simply to “stick it to Niantic.”

Make the fans, fans. Listen to them. Extend a hand. Instead of threatening legal action against reverse engineering fans, talk to them about why they’re doing it, how they’re doing it, and what can be done to stop it. Engage with users as equals, and do not keep them in the dark.

Your greatest enemy is a one time friend, and your greatest friend may be a one time enemy. Use their skill to your advantage, and see what their reasoning is. If proximity features had been refined and discussed with the fan base, much of what we’re talking about here today might never have happened.

Reduce Attack Surface

Finally, and perhaps most importantly, reduce your attack surface. The Pokémon Go app was riddled with avenues through which to reverse engineer, which led to each subsequent fix being broken almost immediately. By exposing so much to so many, Niantic was unfairly set up to struggle from the beginning.

Developers should make this process easy on themselves — failing to reduce attack surface by load balancing servers, obfuscating code, and hiding systems is simply making the battle harder on the engineers and developers who will have to respond.

This almost goes without saying — reduce, reduce, reduce. Anyone who’s ridden a motorcycle can tell you that you don’t sit perfectly straight up while riding down the highway, as your body creates far too much drag. Instead, you narrow yourself, and reduce your frame.

The same holds true here — reduces your attack surface, and be better off for it.

Conclusion

It’s unfortunate that the mobile space faces these sorts of issues as often as it does, but the fact is that the problems are here, and they’re not going away. By adhering to these basic principles, developers can come away from most releases unscathed, with optimism in their long-term support.

Philosopher, essayist, and poet George Santayana once said “those who cannot remember the past are condemned to repeat it.” Whether or not the issues faced by Niantic were justified or fair, we can use their experiences to learn and refine, and prevent it from happening again.

External Resources

• PokemonGO Current API Status (Reddit thread)
• Pokemon Go Update Cracks Down on Hackers (Inquisitr)
• Next Pokemon GO Update finally blocks cheaters (Slash Gear)