I've moved!

Well I decided to move and you can find me here! I am leaving these articles for legacy links but they are also located on my new site.

Writing High Performance Code on the iPhone

Over the last few months a lot of people have asked me how we write our code for the iPhone. Do we use straight C99? Or do we use C++? If so do we use exceptions? RTTI? STL? So I decided that instead of responding by email I would detail it in an article here.

Now please keep in mind that this isn't meant to be a bible on iPhone or mobile/embedded game development. You should always solve the problem with the right tools, and no two games are alike. Would you really try to develop a casual puzzle game using the exact same methods you would use to build a cutting edge shooter?

Also note that the majority of people that have been asking me these questions are PC/Mac developers who are either looking at or have already started iPhone development. So they are the target audience of this article. For the majority of developers who have worked on mobile/embedded devices and game consoles there will likely be nothing new here.

So let's get down to business.

What language did we use to build our iPhone engine?
We use a mix of C and C++ that I believe is commonly referred to as C with classes. The majority of the code would actually be classified as C, but we group objects together in a class (as well as using namespaces) rather than a struct and functions. The decision to allow classes was made simply because it was preferred by the development team, and we knew we would end up having to reimplement a lot of what C++ gives us with objects.

Exception Handling
We have exceptions disabled in the compiler settings for all projects. The current version of GCC used in the iPhone SDK has zero cost exceptions, but you should understand that this does not mean zero overhead. What it means is that there is no execution time overhead when an exception does not occur. Great we have free error handling? Not quite! The compiler will generate additional code and every stack frame will now contain added information required for stack unwinding. With an embedded device like the iPhone the unwind tables will take away resources from the precious little amount that we have. Meanwhile we don't lose much at all by disabling exceptions as there are very few errors that should actually be classified as exceptional on the iPhone. Considering you are shipping for an embedded device, your software has a single point of origin (App Store), and users are unable to modify the file system, the majority of errors will be developer errors. Of course there are some exceptional situations such as low memory, unable to allocate memory, network connection dropped, etc; but for the most part this is a very small group.

We use assertions very heavily in order to catch developer errors, and functions will return error codes in places where that functionality is needed. There are a lot of C++ programmers that heavily rely on exceptions, which is fine 99% of the time on the PC where you have near infinite resources (although this can be debatable), but on an embedded device I need a smaller footprint and all the resources I can get my hands on. To anyone interested in exception handling and wanting to know the cost and implementation details I encourage you to read the Exception Handling ABI for the ARM Architecture.

Virtual Functions, RTTI, and Multiple Inheritance
All of our projects have RTTI disabled in the compiler settings for a few reasons. The first being that while a failed dynamic cast on a pointer will return null, in the case of a reference cast it will throw an exception, which we have disabled. Another reason is that some compiler implementations will end up adding roughly 40 bytes per class of overhead (although many have much better implementations and I have not researched GCC to know exactly how it handles this). The base object model in our engine gives us the non-generic RTTI functionality that we need and only 4 bytes of overhead. Although we do not use this often because as a rule we make sure not to delay any type checking that can take place at compile time rather than run time. This way not only do we have a smaller footprint and faster execution time, but we also have earlier developer error detection.

I'm not going to say much regarding multiple inheritance / virtual inheritance other than we don't use this functionality on any platform and we design our code as not to need it.

Virtual functions are one of the great things that C++ gives us over C, but we use them extremely sparingly. Of course there are the usual rules of not using virtual functions in performance critical areas, tight loops, etc; but we also go a little further. As our engine runs on multiple platforms there is a need for different platform specific subsystems such as rendering. This is easily handled in C++ by creating an interface of virtual functions and then having your platform specific classes inherit from this. But at the same time this is not completely free and on some platforms the performance is extremely bad. You could end up disabling branch prediction and in the worst cases trashing the cache or flushing the instruction pipeline. Dereferencing the pointer may not be the main concern on some platforms, but the compiler skipping optimizations and inlining can be. What we do in these situations is to use platform #defines to include and typedef or inherit. Andre Weissflog from Radon Labs made a great post about this on his blog and how he switched to this method with the Nebula 3 engine. You are sacrificing the abstract class / virtual interface design in this way, but anyone with a PPC device (whether it be Mac, PS3, or 360) will tell you that there is definitely a cost with a missed branch or flush. I did not test the performance impact on an ARM device like the iPhone simply because a small benchmark will show you nothing, and at the time we made this decision we didn't have a large codebase to test with.

Memory Management
This is the one area that can actually differ from regular game console development (at least up until the 360/PS3 but even those systems have better memory gaurantees). In most cases you would do all of the memory management in your engine by allocating various heaps for different uses and making sure to minimize fragmentation. So what makes the iPhone any different?

While consoles have dedicated amounts of memory that you can count on (note that with the 360/PS3 you have to account for the background system usage and events such as the dashboard/XMB which use memory) the iPhone will not always give you what you ask for on application launch. On top of that the iPhoneOS will detect that it is running low on memory and send a warning to all of the current running applications (your game as well as the background applications such as phone, mail, etc) because in the case of a call or SMS there would be the possibility of failure. If you receive this notification you had better free up some resources as soon as possible otherwise the iPhone will force quit your process. With the iPod Touch and iPad this is much less of a concern as there are no phone interruptions, although they will still give you low memory warnings.

This is still an area that we are constantly working on and optimizing. Currently we are using nedmalloc as an allocator due to the extremely low memory fragmentation (we gained a performance boost just by using this rather than the default allocator). On top of that we have various pools for application, level, and frame lifetime allocations as well as pools for specific uses such as particle effects.

I will write another post strictly on this issue in the future once I have more time to analyze the exact performance impact and issues that we are running into.

C++ STL
When developing on the PC or Mac I love the STL because it is simple to use and gives you fully debugged and working containers and algorithms. Unfortunately because our engine and games and built for multiple platforms, the majority of which are embedded devices, we do not use it. Why? Well as I mentioned earlier we have exception handling and RTTI disabled in the compiler settings. Also note that memory management isn't the most fun thing in the world to watch when using the STL on a constrained platform. The STL is a great library for generic cases on a variety of platforms, but for us custom containers were needed.

We did not want to start from scratch and wanted to use containers that were written, debugged, tested, and preferably used in shipped titles. So where could we find this magical library? Again I will point you to the blog of Andre Weissflog where you can download the Nebula 3 SDK which is MIT licensed and use/modify the containers how you see fit. Sure we could have built our own container library from scratch instead, which would have also meant testing and debugging, but time is not infinite and we chose to concentrate on the engine, graphics, and gameplay. This library gave us the functionality we needed with very little modification.

Conclusion
We are using more of a C style approach adding in the convenience of C++ for working with objects. Please don't confuse this with "we used C++ for object oriented programming" because we could have accomplished the exact same design in straight C99. Objects would be structures instead of classes and functions would have a this pointer as the initial argument.

We sacrificed some common C++ idioms, designs, and functionality in order to gain maximum performance on the iPhone and other embedded devices because our games push the hardware to its limit.

I barely scratched the surface with this introductory article and really only covered the very basic details. There are still many more topics to cover such as SIMD using VFP and NEON (you will need this for animation and skinning performance whether you like it or not), various game loop structures on the different devices, optimizing mesh data and packing vertex data with your asset pipeline, memory alignment, using compressed PVRTC textures, and many more. These topics require an in depth explanation and implementation details / samples so I have saved them for a series of future articles.


Also here are some great reference points for console engine development, graphics, and the iPhone:

Oolong Engine - An MIT licensed iPhone game engine owned and managed by Wolfgang Engel, co-founder and CTO of Confetti Special Effects and former lead graphics programmer at Rockstar San Diego.

Diary of a Graphics Programmer - The personal blog of Wolfgang Engel containing extremely good information on graphics programming and iPhone development.

The Brain Dump - The personal blog of Andre Weissflog, co-founder and technical director at Radon Labs. Home of the MIT licensed Nebula 3 engine and a source of very good information on cross platform and console game engine development.

Gamedev.Net OpenGL Forums - Not only do these forums contain years worth of great information on GL and GLES but it is a great community to get help with problems you are unable to solve.

iPhone 3G Rendering Pipeline Video - A video by the lead iPhone developer Renaldas Zioma at Unity Technologies giving an in depth view of the iPhone GPU and drivers.

Good Coding!

Fighting piracy on the app store

It is hard to gauge the exact amount of piracy on the App Store and how it will affect your business model. Some companies have reported an 80-90% piracy rate on their titles, while others have claimed to have a much lower rate. Is there a viable way to deal with this issue? I believe there is within reason.

My company has categorized three types of potential players of our games:

User A
This user legally purchases their games directly from the App Store. They use the App Store featured and top listings to find games they would like, and possibly reading gaming sites such as touchArcade.

User B
This user will pirate some or most of their games from various sources, but they may also purchase titles if they are given further reason. A few of these users may be compelled to purchase your game if they really enjoy it, but you shouldn't count on any substantial revenue from that source. The majority of these users will only make a purchase in two cases:

1. If it is the only possible way to play (i.e. server based games such as MMOs where you control the environment).
   
2. If purchasing the game gives them an added benefit over playing a pirated copy.
   

User C

This user pirates all of their games and applications and refuses to make any purchases. The chance of this user purchasing your game is likely zero and is purely a lost cause.

So what can we do to lower piracy based on these user categories? Well both User A and User C have an easily defined course of action.

User A
In order to have this user purchase your game you need to make a high quality and polished title that appeals to them. Easier said than done! But at least you can rest assured that if you create an appealing game with high production value, this user will make a purchase. Keep in mind that the user needs to find your game somehow, which again is no easy task, but that is a subject for another post and has no direct relation to piracy.

User C
As I stated earlier this user will never purchase your game and therefore you should not concern yourself with them. If they want to pirate your game they will and there is literally nothing you can do about it. Focusing time on this user not only takes resources away from your key focus, but has always proved to be quite fruitless.

Ok. That sounds easy enough, but what about User B?

User B
For now let us assume that your game is not an MMO game, therefore you have no way of completely restricting piracy. It is a single or multi player game that sells on the App Store. How is it possible to convince User B to purchase your game rather than pirate?

What have people done in the past?

1. Detecting whether the game has been pirated (there are numerous methods, all of which can easily be tampered with) and then either exiting the game, disabling features, or popping a message box telling the user this is an illegal copy. This can deter some piracy but for the most part has only resulted in bad press for the game as the user tells their friends "Don't download game X because all it does is crash!". Also the IPA is normally hacked and those checks disabled unless you have littered your codebase with inline calls, etc.
   
   
2. Implemented "call home" functionality so the game needs to connect to a central server in order to authenticate. So you mean I can't play this game on the subway? (which is where most of my time playing iPhone games is spent) What if my internet connection goes out? What if your server goes down? I have yet to see a good system implemented based on this other than Steam and that system has fallbacks and a full support department.
   

Neither of these solutions appealed to me, so the two games my company currently has in production are based on the "value added" model. These were the criteria that I required:

1. The ability to play the single player version of the game wherever you are and at any time. This means a fully featured version with no "call home" functionality, no crashing, and the user experience is not changed by nagging dialogs. Honestly this was our first priority as we want the best experience for User A and we also don't want User C running to the blogs and forums (which happen to be the same sites that User A & B read to find their next purchases!) claiming that the game is buggy or broken.
   
   
2. Create an online aspect for the games that are restricted to legally purchased titles. When a user tries to use an online component our server checks for a valid receipt from Apple. At this point we display a message to the user explaining why they are restricted from using this functionality. Since this component is online we have complete control and it isn't something local that could be cracked.
   

The first point is pretty straight forward. For the single player mode of the games we do absolutely nothing to combat piracy, not even displaying a message box warning the user that we know their little secret. But what about the second point? Sure it sounds great but what could you possibly do? Restricting leaderboards and small features really haven't proven effective in the past.

Our idea was to create a fully integrated online experience rather than just tack on a few small features once the game was completed. My company hasn't yet announced or shown our games, so let's envision a hypothetical game in order to describe our system. As stated above any user (both legally purchased copies as well as pirated) can play the single player version of the game unhindered. One key reason for this is not only to avoid the bad reviews, but the thought that if User B really enjoys the game there is a good chance they will purchase it in order to access the "value added" components.

So what are these magical "value added" components I keep talking about? Here are a few of the things the system provides our games:

1. Online gameplay is either completely run on the server, or works peer to peer with the server acting as the man in the middle.
   
   
2. The game could have items and equipment that is only available in online play. This also opened up the ability to play the single player game in "online mode" with your items.
   
   
3. Each player can create an optional account that works across all of our games. This provides features such as leaderboards, statistics, achievements, friend lists, private messaging, forums, access to tournaments, etc.
   
   
4. The game could have specific online "extras" available to user accounts. Using our hypothetical game a user could upload their 3 hero party to the server and then access it from any device (iPhone, iPod Touch, iPad, Mac, PC). They could also decide to share their party layout with their friends, or even with the entire player base.
   
   
5. Game specific web presence and community. In our example we could have a user gallery where you can see your current party, any parties you have previously saved, and also look at the other player's configurations. People can show off their special items earned in online play, achievements, statistics, etc.
   

There are a lot more points that I will talk about in detail in the future, but currently some are a competitive advantage for us. We realized up front that the majority of these systems were generic enough and if we built them now we could use them in all of our future titles, whether they are App Store games, selling on Steam, or using any other digital distribution method. The cost of hosting the online components is unbelievably cheap these days by using services like Amazon EC2 and Windows Azure. Not to mention the cost scales based on the amount of users who have actually purchased the game.

Conclusion
In conclusion we decided to build a much better user experience for our paying customers where it was possible for us to control access. Even if every one of my assumptions turn out to be incorrect and we don't see an increase in sales from User B, I believe our time was spent wisely. Not only were we able to avoid wasting our time trying to forcefully fight piracy, but we ended up creating a really great online experience for the User A's of the world who have obtained the game legally. We also get the added value that our solution brings to the marketing campaigns and word of mouth.

Of course this model does not fit every single game type, and is most definitely the wrong solution for many different types of games. Although for many of the larger games implementing a model like this will not only increase your chances of higher revenue, but you end up with a much better overall user experience.

Be wary of the flash evangelists!

It's a little odd writing this topic as the first post on my new game development blog, but considering the flash developers are being extremely vocal and once again spreading misinformation or conveniently ignoring facts I thought I would clear some things up.

Ted Patrick (Adobe's platform evangelist) has been throwing out links and slamming Apple everywhere you look ever since Steve Jobs showed the world the iPad and that it didn't run Flash. It's obviously not going to stop anytime soon either considering this is what Adobe pays him for, but it turns out that much of the content he's linking to is not only bias (which I could obviously understand and would not hold against him) but blatant misinformation.

An example? Today he posted this on Twitter: "Dear Apple, 1984 Called, They Want Their Video Back": http://bit.ly/9uaYok

So I decided to read through the link to hear the argument. To your average reader it might seem like a semi-decent post built on facts, but if you analyze what is being said you can see right away that it is yet another "I didn't do my homework and/or decided to ignore the facts and now I want to rant in favour of my platform" posting.

Unfortunately the author of the blog chose not to approve my comment as the real facts would weaken her position. I realized why my comment was not going to be approved after clicking on "About Me":

"I am a Group Manager, Developer Relations for Adobe"

So let's walk through it here. The article states:

"Pushing as much content through the App Store as possible is great business for Apple; and honestly, I don’t blame them for wanting to build their App Store into a massive (and massively profitable) content juggernaut. It’s far friendlier to their margins than the hardware business, even with their premium pricing, so why not go for a platform play?"

Now I could understand stating a point like this if it were true, but it is horrendously false. The AppStore operates just above break even and gives Apple no meaningful revenue/profit in the foreseeable future. This is public information that is easily gained by listening to the quarterly investor relations conference call by Apple where it has been mentioned numerous times. Considering Apple streams the call live on their site as well as archives it for later listening there is absolutely no reason that anyone on the Internet could not obtain this information.

The AppStore merely exists as a support infrastructure for Apple's hardware that they are interested in selling at margins that retain profitability. So not only is claiming that the AppStore is massively profitable completely wrong; but claiming that the margins are higher than their hardware business borders insanity.

Next she states:

"Where I take exception — and developers should too — is with Apple’s “my way or the highway” approach to development. Adobe’s Flash is a very high-profile victim of this approach right now, but we’re not the first nor are we the only one."

Again she is completely ignoring the technologies and software that is not built the "Apple way" but is available on the AppStore. Novell has MonoTouch which is a .Net implementation allowing C# development on the iPhone. The game engine Unity embeds the Mono runtime to allow for C# development for games on the iPhone. Various other game engines embed LUA and use AOT compiling to run on the iPhone. Apple could have knocked off any one of these technologies or products but instead allows them. Flash can run on the iPhone without a problem just like the rest of these technologies as long as it is compiled to an IPA. So when the author is complaining about Flash it is really "her way or the highway" as she needs it in the browser and not how every other technology is implemented on the platform as an IPA.

But why no Flash browser plugin? First of all people need to understand that the most frequent causes of a crash in Mac OS X (as per the crash reports) was the Adobe Flash plugin, and this was announced at WWDC (you can read more about it here: http://daringfireball.net/2010/01/apple_adobe_flash). Not only has it been rife with bugs causing performance and stability issues over the years, but it has also turned into the new security worry with new vulnerabilities added quarterly and sometimes monthly. This is not only on Mac OS but also on the Windows platform. The worry is no longer what browser you are running, but what plugins have security issues.

On a consumer device or mobile device this is absolutely horrendous as you can't even control the security of your own device! If there are security issues people are not going to look to Adobe but they will be calling the Apple support lines as the average consumer doesn't understand this. I have no idea how other vendors are not considering this fact, although many of them seem to be grasping at straws and wanting to take any "competitive advantage" they can get.

She goes on to state that her 10 year old niece won't be able to play WebKinz and she won't want to play any other games. Although she forgets to mention the fact that as of Flash CS5 Adobe will allow retargeting to IPA, and therefore Ganz could very easily compile a native application and supply it as a free download. Any flash developer will be able to do this and are therefore not restricted from the platform.

So far what we've seen is a very vocal minority hitting the blogs and trying to incite your average consumer into demanding having the flash player on their platform. The fact is that the majority of consumers don't really care at all otherwise the iPhone wouldn't be such a hit (Consumers vote with their wallet not on blogs). Content providers are slowly switching over to HTML5/JS open standards (such as YouTube, Vimeo, and many others) and as long as Apple devices keep selling, more companies will have to offer sites based on open standards due to the growing percentage of people unable to view their content.

Considering these are two Adobe employees spreading misinformation like this do we even have reason to trust them at all? What do they lose if HTML5/JS were to take the place of Flash? Adobe would likely retarget Flash to supporting the HTML5/SVG/JS spec and so you would still get the great content creation tools they are known for. Although they would no longer control the platform which weakens their position immensely.

Disclaimer: I do not hold shares of either AAPL or ADBE. I do not work for either Apple or Adobe. I have no invested interest in either HTML5 or Flash.