Tuesday, November 8, 2011

Devices for Testing iOS Software

For the next few months at least it's going to be a pain to test in iOS. Recently I had to test an app on 4 different devices and it behaved differently on all four. So, here is the set-up I used to try to get the best coverage.

Old Devices
The oldest devices I'm willing to support (or that my clients ask to support) are running iOS 3.1.3. These are second generation iPod touches as well as iPhone 3G devices. Thankfully, these are the last devices are the last ones that don't do OS signing with Apple, so it is pretty trivial to upgrade them to 4.2.1 (the highest OS they support) and then back to 3.1.3 (a common OS because it runs faster). For maximum coverage, I have one device at one OS version and one at the other. Switching OS versions is easy but time consuming. I have an iPod Touch and an iPhone 3G for this testing. I decide which one to set at which version depending on if I need the camera for my app or not. These devices require armv6 code.

Less Old Devices
Apple is currently selling iOS 5 installed on iPhone 3GS, iPhone 4 and iPhone 4S. All of these devices are armv7 So, I'm treating this hardware as interchangeable for now. I have an iPhone 4 running 4.3.5 and I have an iPhone 3GS and and iPhone 4 both running iOS 5. Apple has made it really hard to upgrade/downgrade without jailbreaking, so I have to have big stickers on my 4.3.5 device to make sure I never upgrade it by accident.

I have an iPad (1) and and iPad2 that are both running iOS 5. Because the iPad is a newish device, I have assumed that everyone has upgraded or is upgrading soon away from iOS 3.2. I test with 4.3 in the simulator and test on the devices in 5.

What else?
In my experience, the biggest change as far as the OS breaking things happened between iOS 4.2 and 4.3 which corresponds to the 3G devices and the 3GS and above. Apple made lots of changes to the video, and graphics libraries during that time. Because the 3G and the older iPod Touches only go up to 4.2.1 (and even then don't support multi-tasking) we need to keep testing with those devices for a while. If an application doesn't require the new features in the newer OS'es then it seems like a poor decision to cut out the older devices, those people still have iTunes accounts and can still give us money.

My Wish List
Google has a really nice chart of what OS versions in what volume are hitting its App Store each week. This can help a developer make good guesses about what OS should be the minimum. Apple should be able to provide this data. I wish they would. It's one thing for me to gather data about my current customers (which I can do using Flurry or similar analytics) but that information is only about my small part of the customer base, not about the entire population.