Top-down view of the Aoren Software datacenter in the living room corner.

Why, you might ask? In my case the answer is simply because it is my only alternative. In development of my game Cannonade there quickly arose a need to be able to replay my user’s completed games and validate the results with exactness. In order to obtain that level of exactness, my games must execute with complete determinism. Unfortunately the implementation of floating-point match according to IEEE754 can actually vary somewhat between x86 and ARM processors. This means that if I replayed a game that two of my users played using iOS devices on an x86-based server, the results of the replay would very quickly diverge. Thus I am left with no choice but to set up a dedicated iOS device to wait for notifications of the matches that it needs to replay and validate (a process I call judging). An iOS server you might say.

The process that occurs when a match of Cannonade is completed goes something like this:

1. The last completed game state from one of the peers is uploaded to my x86-based server.
2. The server sends a push notification to one of any currently activate judges (a dedicated iOS server).
3. The judge replays the game and checks the results against every participating peer’s viewpoint.
4. The judge reports back to the x86-based server the results of the replay and validation.
5. The x86-based server then flags the game according to the judge’s verdict as being either “good” or “disputed“.

So I need to be able to set up dedicated iOS devices that run a special version of Cannonade 24/7 for the purpose of judging. Some problems arise because the current iOS and the hardware it runs on was never intended to be used in this manner. But that hasn’t ever stopped anyone from messing with iOS before.

Running The Hardware Numbers

I first gave a little thought as to what kind of devices should be used as the judge. Because the game replays are almost 100% bullet physics simulation, the computing power for a judge is locked almost linearly to CPU clock speed. Thus we can easily compute a computing power per dollar metric. Considering only armv7 devices that you can currently buy from Apple without any sort of contract, my analysis went something like this:

iOS Device	CPU Clock	Judging Power (JP)	Cost ($)	JP/$	Official SDK
16GB Wi-Fi 2nd Gen iPad	2x ~900Mhz	~1000?	$499	2.00	YES
16GB Wi-Fi 1st Gen iPad	1000Mhz	1000	$349	2.87	YES
8GB 4th Gen iPod Touch	800Mhz	800	$229	3.49	YES
8GB 3rd Gen iPod Touch	600Mhz	600	$149	4.03	YES
2nd Gen Apple TV	1000Mhz	1000	$99	10.10	NO

Boy, that Apple TV is looking like a pretty sweet option! However, I am not willing to spend the time at the moment to bring up a jailbroken toolchain for it and risk some unforeseen issue that might prevent Cannonade from running on it. So it looks like best judging bang for the buck would come from the 3rd generation iPod Touch. Fortunately a friend was able to lend me a 4th generation iPod Touch and so I was now in business. Now to just figure out how to get it to run in some server-like capacity.

The Three Laws Of An iOS Server

Broken down, the three main requirements that I have for setting up a judge are:

1. Be able to receive push notifications (so it can get it’s work)
2. Have it’s display turned off (to save energy and avoid things like screen burn-in/fatigue)
3. Require no human interaction (needs to be completely autonomous)

Hmm, the lack of screen on a 2nd generation Apple TV eliminates requirement #2 right off the bat! Maybe I should reconsider that jailbroken toolchain… Well realistically I need to consider a solution that fulfills all three requirements. Unfortunately these requirements conflict in various ways. Let me explain the ways that I approached these goals on iOS 4.3.3.

Delivery Boy

First of all, the easiest way to turn off the display is to hit the lock button. This powers down the LCD and the backlight, switches the “active” process to be the lock screen (which I am guessing is the Springboard process), it then “de-activates” the previously active process (- (void) applicationWillResignActive:). The process is still in the foreground and running though which is good. There is a problem with that though because when a process is in the foreground but not active it will not receive push notifications at all even though the notifications are displayed on the lock screen and even after a human swipes to unlock (bug?). So one avenue I looked at is to try to find a way to get the notifications to be delivered to my foreground server process even when the device is locked. I didn’t figure out any way to accomplish that and so I moved on to my second method of attack.

Blind As A Bat

The other avenue I had been investigating is to find a way to turn off the LCD and backlight while my process is in the foreground and active. Doing so would allow push notifications to be delivered directly to my server process. I found the private API -[UIApplication setBackLightLevel:] which will let me change the backlight level down to the same minimum amount that the system preference will allow. This, along with displaying a screen of all black pixels provides a result that appears in normal light to be completely off but when taken into a dark room it is clear that the backlight is still on and producing light. I have also poked around HIDDisplay stuff but reverse engineering that API proved to be more challenging than it was worth.

Epilogue

Another issue I ran into was that I would occasionally get killed by the watchdog timer. This is usually an issue with backgrounded apps but I found that in iOS 4.3.3 that the watchdog will still kill your foreground and active process if any one of your threads executes for more than 600 seconds without servicing a run loop (this includes background threads). This can also happen if the work it takes to clear all the execution blocks sitting in a dispatch_queue_t collectively take more than 600 seconds to complete (even if no one block takes more than 600 seconds on its own).

So where does this leave me? Well, running the process foreground and active with the backlight very low and all black pixels seems the best I can do on iOS 4.3.3. But! The first iOS 5.0 developer seed was distributed this week and it changes the game. Using new functionality found in the iOS 5.0 beta I have been able to fulfill all three of my above requirements! The iPod Touch now sits in corner with a completely powered down screen, waiting for push notifications and not needing any human interaction at all! I will publicly provide the details as soon as the following achievement is unlocked:

UPDATE: Now that the iOS 5 NDA has been lifted I can publish the solution.

UPDATE: Today my friend Jonathan made me aware of another alternative on iOS 4.3.3 called the VoiceOver “Screen Curtain”. When you turn on VoiceOver in Settings>General>Accessibility>VoiceOver it opens up some new gestures. One of these new gestures is the three-finger triple tap. This brings up the screen curtain which has been described as being for simulating the experience of a blind person for a developer or for granting blind iOS users privacy from onlookers. In the documentation the screen curtain is described as simply turning the screen off but this is not true. It is actually the exact same method that I programmatically employed above of turning the backlight level down to the lowest system preference level and displaying all black pixels (I have verified this in a dark room). So this appears to be a nice OS level solution for the problem in iOS 4 although you still need to go to iOS 5 for the real screen disabling solution.

This entry was posted on Saturday, June 11th, 2011 at 3:25 pm and is filed under Bullet3d, Cannonade, Computer Games, Game Center, Games, Hacking, Indie, iOS, Networking, Performance, Programming, Security/Cryptography, Servers, Startups, Systems. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response.