The Challenges of Mobile Testing for Jailbroken Devices
Many of the automation tools for mobile testing on the market today require the use of jailbroken or rooted devices, a controversial condition that some testing advocates believe violates mobile testing best practices.
Android rooting and iOS jailbreaking refer to the user’s ability to gain root access to the operating system on the device, allowing the user to manipulate the system outside the approved methods. Jailbreaking gives the tester the ability to extract data from low-level objects within the application being tested.
Advocates of jailbreaking extol the benefits of easy configuration and device management. Perfecto Mobile, developer of mobile testing solution UFT Mobile, says the ability to test application code “eliminates the need for compilation and intrusive operations, which could potentially pose a risk to quality.”
But the potential ramifications of jailbreaking and rooting are numerous. For one, jailbreaking the iPad is illegal, according to the latest Digital Millennium Copyright Act (DMCA) exemptions announced in October 2012. Apple says jailbreaking “has been a major source of instability, disruption of services, and other issues.” Warranties are typically voided on jailbroken devices.
Of course, that’s to be expected—Apple’s methods of tight control don’t mesh with hacker philosophy. That said, many in the tester community decry jailbreaking as well, citing legality, compromised security, and device instability as reasons to explore alternatives to rooting.
Perhaps most pertinent to the passionate tester is the concern over changing the operating environment to a state that doesn’t match a real user’s environment. Recent trading malfunctions in the stock market have demonstrated the dangers of releasing software that hasn’t been tested under real, “in-the-wild” conditions.
Jailbreaking a device fundamentally changes the OS and doesn’t reflect the production build unless the user has the same jailbreak. Brian Copeland argues that “anything you do to change your environment from what the actual operating environment will be like adds unacceptable risk to your testing.” Jailbreaking takes extra effort as well, as the device must be jailbroken again each time there is an OS upgrade or patch.
Since tablets were projected to outsell PCs as early as Q3 2012, developing a strategy for automating mobile tests while keeping risks low and generating high ROI is at the top of the to-do list for many test teams. I’ll leave you with this thought: To hack or not to hack to accomplish this mission? The testing community seems to still be figuring out that one.
