Multivariate Testing of Native Mobile Applications

A/B testing has a long history in web development and is used on a daily basis by many companies. Although it is a common test method for web pages, it is hardly used for native mobile applications. The reason seems to be that it is much more difficult to change the user interface of a mobile application, which has been downloaded from an app store, than that of a web page which is fetched from a server by request. In this paper, we present an approach for A/B testing of native mobile applications. Furthermore, it allows for the more exible multivariate testing, which is based on the same mechanisms as A/B testing, but compares a much higher number of variants by combining variations for dif- ferent sections of the user interface. Our proposed approach works without redeployment of the mobile application in the app store and thus allows for a seamless integration into the developer's workow with low effort for creating and deploy- ing new variants. We implemented a prototype solution for the Android platform and compared it against other A/B testing products. It shows that our solution requires less effort and is more convenient to use than related products. Moreover, it is the only one which allows for multivariate testing of native mobile applications.