Research Challenges in Mobile Application Testing

Research Challenges in Mobile Application Testing

Niranjanamurthy M#1, Sai Prasad C #2, Manushree SP#3,Asharani A#4 Dr. Dharmendra Chahar #5

1 Research Scholar, Dept. of Computer Science Engg., JJTU, Rajasthan,, niruhsd@gmail.com

2Student of MCA, MSRIT, Bangalore, saiprasad0902@gmail.com 3Student of MCA, MSRIT, Bangalore, manushreesp.27@gmail.com 4Student of MCA, MSRIT, Bangalore, achar.asharani@gmail.com

5 HOD. Dept. of CS & IT, Seth G. B. Podar College, Nawalgarh (Jhunjhunu), Rajasthan

Abstract

Mobile application testing is mainly deals with functionality, Usability and consistency of that application. Mobile devices are everywhere. Businesses depend on them. Customers interact with them like never before. Their exponential growth is creating high demand and an even higher need for advanced functionality. But innovation and increased speed to market bring many testing challenges.Mobile applications either come pre- installed or can be installed from mobile software distribution platforms. Mobile devices have witnessed a phenomenal growth in the past few years. Device and platform diversity, short release cycles, lack of mature testing tools and the variety of network connectivity options result in frequent cost overruns and missed deadlines in todays mobile application testing environment. A comprehensive mobile testing strategy that includes device and network infrastructure, optimized selection of target devices, and an effective combination of manual and automated testing tools to cover both functional and non-functional testing is essential for getting your mobile applications to market on time and within budget.

Keywords Mobile application testing, Types of Mobile Application Testing, Challenges in Mobile App Testing, Customers Challenges in Mobile Testing, , Interruptions of the mobile application Testing, Mobility Testing Process, Security testing in Mobile application,

1. Introduction

   Mobile application testing (MAT) is a process by which application software developed for hand held mobile devices is tested for its functionality, usability and uniformity.Mobile Application Testing techniques are: Functional Testing, Laboratory Testing, Performance Testing, Memory Leakage Testing, Interrupt Testing, Usability testing, Installation testing, Certification Testing, Challenges in Mobile Application Testing are: Types of Mobile

   Devices, Diversity in Mobile Platforms/OS, Mobile network operators, Scripting.

   Mobile devices have witnessed a phenomenal growth in the recent years. A study conducted by the Yankee Group predicts generation of $4.2 billion in revenue by 2013 through 7 billion U.S. Smartphones app downloads. Mobile application testing involves evaluating functional & non-functional characteristics of application softwares developed for Hand held devices such as mobile phones, tablets etc. Testing a mobile application across multiple devices running on the same platform and every platform poses a unique challenge for testers.

   Testing mobile applications through:

   1. Devices.

   2. I phone — Simulator

   3. Android — Emulator

2. Aim of the Study

   Aims of this paper are:

   • To understand what is Mobile Application Testing

   • To Know the Types of Test techniques required for Mobile Application

   • Knowledge on Mobility Testing Process

   • To understand Key Challenges in Mobile Application Testing

   • To know Mobile Testing Tools and Tool Selection Criteria

   • To understand Customers Challenges in Mobile ApplicationTesting

   • To know how mobile testing isdifferent?

   • Knowledge on Interruptions of the mobile Application Testing

   • Understand the Security testing conditions in Mobile Application

3. Related Work

   The usability of mobile applications is criticalfor their adoption because of the relatively small screenand awkward (sometimes virtual) keyboard, despite therecent advances of Smartphones. Traditional laboratorybased usability testing is often tedious, expensive, and does not reflect real use cases. In this paper, we propose a toolkit that embeds into mobile applications the ability to automatically

   collect user interface (UI) events as the user interacts with the applications. The events are fine-grained and usefulfor quantified usability analysis. We have implementedthe toolkit on Android devices and we evaluated the toolkit with a real deployed Android application by comparing event analysis (state- machine based) with traditional laboratory testing (expert based). The results show that our toolkit is effective at capturing detailed UI events for accurate usability analysis.[1]

   Challenges in Mobile App Testing: Its been clear for a while that mobile devices are the currentmarket players, even more so that some experts have beencounting on them to take over the PCs and Desktops in nearfuture. But as with any emerging technology, developing andimplementing mobile applications can pose a number of

   Unique challenges. Mobile applications, although havelimited computing resources, are often built to be as agileand reliable as PC-based applications. In order to meet thechallenge, mobile application testing has evolved as a

   Separate stream of testing.The major challenge in Mobile App Testing is themultiplicity of mobile devices with different capabilities,features and restrictions. Devices may have differenttechnical capabilities such as amount of available memory,screen resolution, screen orientation and size of the display,network connectivity options, support for different standardsand interfaces [2]

   Figure 1.Mobile Testing

   There are multiple operating systems that are prevalent in the mobile space like Symbian, Android, iPhone OS, Windows, Linux, Blackberry OS, palm OS, Brew, etc. Each of the operating systems can have further versions for different types of devices which make platform testing complex and further challenging. Another challenge is that the developers need to focus on developing applications that are easy to use on a mobile and consume less power.[2]

4. Types of Mobile Application Testing

1. Functional Testing- Functional testing ensures that the application is working as per the requirements. Most of the test conducted for this is driven by the user interface and call flows.

2. Laboratory Testing- Laboratory testing, usually carried out by network carriers, is done by simulating the complete wireless network. This test is performed to find out any glitches when a mobile application uses voice and/or data connection to perform some functions.

3. Performance Testing- This testing process is undertaken to check the performance and behaviour of the application under certain conditions such as low battery, bad network coverage, low available memory, simultaneous access to applications server by several users and other conditions. Performance of an application can be affected from two sides: applications server side and clients side. Performance testing is carried out to check both.

4. Memory Leakage Testing- Memory leakage happens when a computer program or application is unable to manage the memory it is allocated resulting in poor performance of the application and the overall slowdown of the system. As mobile devices have significant constraints of available memory, memory leakage testing is crucial for the proper functioning of an application.

5. Interrupt Testin- An application while functioning may face several interruptions like incoming calls or network coverage outage and recovery.

   The different types of interruptions are:

   • Incoming and Outgoing SMS and MMS

   • Incoming and Outgoing calls

   • Incoming Notifications

   • Battery Removal

   • Cable Insertion and Removal for data transfer

   • Network outage and recovery

   • Media Player on/off

   • Device Power cycle

     An application should be able to handle these interruptions by going into a suspended state and resuming afterwards.

6. Usability testing- Usability testing is carried out to verify if the application is achieving its goals and getting a favourable response from users. This is important as the usability of an application is its key to commercial success (it is nothing but user friendly).

7. Installation testing- Certain mobile applications come pre-installed on the device whereas others have to be installed from the store. Installation testing verifies that the installation process goes smoothly without the user having to face any difficulty. This testing process covers installation, updating and uninstalling of an application.

8. Certification Testing- To get a certificate of compliance, each mobile device needs to be tested

against the guidelines set by different mobile platforms.

The Certified Mobile Application Tester popularly known as CMAT certification exam is offered by the Global Association for Quality Management (GAQM) via Pearson Vue Testing Centreworldwide to benefit the Mobile Application Testing Community. [4]

Figure 3. Testing Aspects

1. Mobility Testing Process

   A typical end-to-end mobile testing process should start from creating test cases of the application, performing user acceptance and finally device testing stage.The stages in mobile application testing process are as follows: [6]

   Figure 2. Testing Process

   Security and data privacy are of highest importance in todays scenario. Users are worried about their privacy.

2. Key Challenges in Mobile Application Testing

1. Variety of Mobile Devices- Mobile devices differ in screen sizes, input methods (QWERTY, touch, normal) with different hardware capabilities.

2. Diversity in Mobile Platforms/OS- There are different Mobile Operating Systems in the market. The major ones are Android, IOS, BREW, BREWMP, Symbian, Windows Phone, and BlackBerry (RIM). Each operating system has its own limitations. Testing a single application across multiple devices running on the same platform and every platform poses a unique challenge for testers.

3. Mobile network operators- There are over 400 mobile network operators in the world; [2] out of which some are CDMA, some GSM, whereas others use less common network standards like FOMA, and TD-SCDMA. Each network operator uses a different kind network infrastructure and this limits the flow of information.

4. Scripting- The variety of devices makes executing the test script (Scripting) a key challenge. As devices differ in keystrokes, input methods, menu structure and display properties single script does not function on every device. [4]

Figure 4. Test Cycles of Desktop and Mobile applications

1. Some Popular Mobile Testing Tools and Tool Selection Criteria

   Figure 5. Mobile Testing Tools

   Tool Selection Criteria:

   It is based on:

   • Cost Benefit Analysis

   • Timelinesfor the project

   • Regression Test Count & Complexity Scope based decisions:

     • Platforms support

     • Supported Versions of platforms

     • Types of automation supported.

     • Device variants within platforms (form factors) Feature based decisions:

     • Working on both simulator /device

     • Integration to test management

     • Ease of use and scripting supportInfrastructure Infrastructure decisions:

     • Stability

     • Dependency on Mac/desktops

     • Cloud based/stand alone

       Figure 6. Mobile Platforms

       Figure 7. Selecting Mobile Testing Tools

2. Customers Challenges in Mobile Testing

• Variety of mobile devices in market and multiple manufacturers

• Coping with the short lifecycle of the mobile application in market

• Huge variety of hardware capabilities

• Shorter duration of device life in the market

• Variety of network modes like 2G/3G/4G/Wi- Fi/Wi-Max

• Huge investment and high time frame to set up an test lab

• Most of the tools are image based comparison tool

• One test tool may not support all platforms versions

• Tools may need jailbreak/rooting that invokes security threat

• Testing on latest technologies like HTML5, NFC, etc.

• High rise of localized application capability

• Choice between simulators and real devices for testing

1. Mobile testing how is it different?

   1. DEVICE DIVERSITY

      • Multiple Platforms

      • Multiple Browsers

      • Rendering differences

      • Mobile devices have different application runtimes.

   2. NETWORK CHALLENGES

      • Multiple type of Networks(GSM / GPRS / Wi-Fi / Wi-Max etc )

      • Unpredictable time taken for data transfer

      • Different speed of connectivity across geographies

      • Multiple Network Operators with customized Network features

   3. HARDWARE CHALLENGES

      • Limitations in processing speed

      • Limitations of Memory size of mobile

      • Differences in Communication Protocols of devices WAP/ HTTP.

1. Interruptions of the mobile application Testing

   There are various events that can interrupt the flow of your application. Your application should be able to handle these and should be tested for the same.

   • Incoming Call

   • Text message

   • Other app notifications

   • No storage

   • Airplane mode

   • Intermittent connectivity

   • Home screen jump

   • Sleep mode

   • Storage low

   • Battery low

   • Battery dead

2. Security testing in Mobile application Security and data privacy are of highest importance in todays scenario. Users are worried about their data and recommendation being exposed through vulnerable applications.

   • Is your application storing payment information or credit card details?

   • Does your application use secure network protocols?

   • Can they be switched to insecure ones?

   • Does the application ask for more permissions than it needs?

   • Does your application use certificates?

   • Does your application use a Device ID as an identifier?

   • Does your application require a user to be authenticated before they are allowed to access their data?

   *Is there a maximum number of login attempts before they are locked out?

   Figure 8. Mobile Application Testing

3. Conclusion

   Mobile application testing is a process by which application software developed for hand held mobile devices is tested for its functionality, usability and uniformity.Challenges in Mobile App Testing: Its been clear for a while that mobile devices are the currentmarket players, even more so that some experts have beencounting on them to take over the PCs and Desktops in nearfuture.For testing Mobile Application Testing we need to follow the following test techniques: Functional Testing, Laboratory Testing, Performance Testing,Memory Leakage Testing,Interrupt Testing, Usability testing, Installation testing, Certification Testing.

   Tool Selection Criteria It is based on cost Benefit Analysis, Timelines for the projectRegression Test Count & Complexity and Customers Challenges in Mobile Testing, Mobile Testing How is it different? And different Interruptions and security of the mobile application Testing and platforms for mobile application testing topics are discussed.

4. Acknowledgement

   We thank Dr. T. V. Suresh Kumar, Prof. and Head, Dept. of MCA, MSRIT, Bangalore-54. For his continuous support andencouragement for completing this research paper and alsothanks to MSRIT management.

   We thank Mr. S. Jagannatha , Associate Professor. Dept. of MCA, MSRIT, Bangalore-54 for his constant support and motivation.

   (Jhunjhunu), Rajasthanfor his constant support and guidance.

   We thank Manish Kumar, Assistant Professor Dept. of MCA, MSRIT, Bangalore-54 for his constant support and suggestions.

5. References

1. Xiaoxiao Ma, Bo Yan Guanling Chen- "Design and Implementation of a Toolkit for Usability Testing of Mobile Apps"Published online: 21 November 2012 – Springer Science -Business Media New York 2012.

2. R RamchandraNimbalkar – "Mobile Application Testing and Challenges" – International Journal of Science and Research (IJSR), – Volume 2 Issue 7, July 2013 -India Online ISSN: 2319-7064

3. http:// www.tcs.com/resources/white_papers/Pages/Mobile_applic ation_testing.aspx [4]http://en.wikipedia.org/wiki/Mobile_application_testing [5]

http://www.infosys.com/flypp/resources/Documents/mobile

-application-testing.pdf [6]http://www.rapidvaluesolutions.com/mobile-application- testing-step-by-step-approach/

1. http://mindqsystems.com/Mobile_Testing.php

2. http://www.tutorialspoint.com/white-papers/335.pdf

3. Julian Harty, MahadevSatyanarayananA – Practical Guide to Testing Wireless Smartphones Applications (Synthesis Lectures on Mobile and Pervasive Computing)- 2009 – ISBN-10: 1608452530

4. Rex Black: Advanced Software Testing Vol. 1,

   Shroff Publishers, 2011

5. Srinivasan DesikanGopalaswamy: Software Testing Principles and Practices, 5

   th Edition , Pearson Education, 2007

6. http://www.qaac.org/wp-

content/uploads/2012/07/Mobile-Testing-QAAC.pdf