Research on Mobile App Analysis and Testing
Background
Mobile apps are typical GUI-centered and event-driven software.
They are now ubiquitous and serving the needs of our daily
life in many different aspects. However, due to the
complex end-user environments (e.g., different OSes,
vendor devices and third-party libraries), ensuring app
reliability and correctness has thus become a longstanding
challenge in both academia and industry (see literature). Our research aims to tackle this challenge by developing novel, effective
and practical approaches and techniques to improve app quality, reliability
and usability.
Techniques, Tools and Dataset
To this end, we have devoted much research effort over the recent
years and developed several effective app analysis and testing techniques, including:
Stoat, a fully automated GUI fuzzing technique for finding crashing bugs;
Genie, SetDroid, Odin and RegDroid, fully automated GUI fuzzing techniques for finding non-crashing functional bugs (i.e., logic errors);
PBFDroid, a property-based testing tool for Android apps;
SetChecker, a static analysis tool for finding system setting related bugs;
Themis and DDroid, the first ground-truth benchmark for evaluating/analyzing automated GUI fuzzing tools;
DroidDefects/CrashAnalysis: the dataset of framework-specific exception bugs of Android apps.
Research Impact
In addition to successfully finding many bugs in open-source apps, our techniques have found and reported 100+ bugs in several highly-popular industrial apps
with billions of monthly-active users, many of which have been already fixed by the app vendors. For example:
TikTok (Douyin): 78 (confirmed) / 32 (fixed)
WeChat: 11 (confirmed) / 11 (fixed)
CapCut: 4 (confirmed) / 4 (fixed)
QQmail: 2 (confirmed) / 2 (fixed)
Google+: 2 (confirmed) / 2 (fixed)
AlipayHK: 2 (confirmed) / 2 (fixed)
Gmail: 1 (confirmed) / 1 (fixed)
Up to now:
- Stoat has become a representative model-based testing approach for Android (cited by 225+), and used/compared/extended by many work.
Specifically, Stoat has been included in GoalExplorer and TimeMachine and
inspired the design of FastBot (e.g., see this post from ByteDance's FastBot);
- SetDroid has been intergated into ByteDance's FastBot for daily testing (see this post from ByteDance's SE Lab);
- Themis has helped optimize/enhance FastBot (from ByteDance) and WCTester (from Wechat's team)
with several new GUI fuzzing & mutation strategies.
- FastBot has been fully open-sourced. Our research group has made several contributions in this process (see this post).
Selected Publications
General and Practical Property-based Testing for Android Apps
Yiheng Xiong, Ting Su, Jue Wang, Jingling Sun, Geguang Pu, Zhendong Su
39th ACM/IEEE International Conference on Automated Software Engineering
ASE 2024, pdf, tool.
Property-based Testing for Validating User Privacy-Related Functionalities in Social Media Apps
Jingling Sun, Ting Su, Jun Sun, Jianwen Li, Mengfei Wang, Geguang Pu
ACM International Conference on the Foundations of Software Engineering
FSE 2024 (industry track), pdf, tool
Automata-based Trace Analysis for Aiding Diagnosing GUI Testing Tools for Android
Enze Ma#, Shan Huang#, Weigang He, Ting Su, Jue Wang, Huiyu Liu, Geguang Pu, Zhendong Su
ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
FSE 2023, pdf, tool
(#Equal Contribution)
Property-based Fuzzing for Finding Data Manipulation Errors in Android Apps
Jingling Sun, Ting Su, Jiayi Jiang, Jue Wang, Geguang Pu, Zhendong Su
ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
FSE 2023, pdf, tool.
An Empirical Study of Functional Bugs in Android Apps
Yiheng Xiong, Mengqian Xu, Ting Su, Jingling Sun, Jue Wang, He Wen, Geguang Pu, Jifeng He, Zhendong Su
32th ACM SIGSOFT International Symposium on Software Testing and Analysis
ISSTA 2023, pdf, dataset.
ACM SIGSOFT Distinguished Paper Award
Characterizing and Finding System Setting-Related Defects in Android Apps
Jingling Sun, Ting Su, Kai Liu, Chao Peng, Zhao Zhang, Geguang Pu, Tao Xie, Zhendong Su
IEEE Transactions on Software Engineering (TSE), 2023, pdf.
Highlights: Our work (SetDroid and SetChecker) helped find 59 confirmed bugs (31 have already been fixed) in Douyin (TikTok). SetDroid has been integrated into ByteDance's official app testing infrastructure FastBot for daily testing.
Fastbot2: Reusable Automated Model-based GUI Testing for Android Enhanced by Reinforcement Learning
Zhengwei Lv, Chao Peng, Zhao Zhang, Ting Su, Kai Liu, Ping Yang
37th IEEE/ACM International Conference on Automated Software Engineering
ASE 2022 (industry track), pdf, FastBot.
Highlights: Fastbot2 has been deployed in the CI pipeline at ByteDance, and over 50% of the developer-fixed crash bugs were reported by Fastbot2.
Detecting Non-crashing Functional Bugs in Android Apps via Deep-State Differential Analysis
Jue Wang, Yanyan Jiang, Ting Su, Shaohua Li, Chang Xu, Jian Lu, Zhendong Su
ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
ESEC/FSE 2022, pdf.
Fully Automated Functional Fuzzing of Android Apps for Detecting Non-Crashing Logic Bugs
Ting Su, Yichen Yan, Jue Wang, Jingling Sun, Yiheng Xiong, Geguang Pu, Ke Wang, Zhendong Su
ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications
SPLASH/OOPSLA 2021, pdf, talk video, Genie.
Highlights: The first fully-automated GUI fuzzing technique to tackle the oracle problem in general for Andorid apps.
Benchmarking Automated GUI Testing for Android against Real-World Bugs
Ting Su, Jue Wang, Zhendong Su
29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
ESEC/FSE 2021, pdf, talk video, Themis.
Highlights: (1) The first ground-truth empirical evaluation of automated GUI testing for Android (after ten years' continuous research by our community since 2011). (2) Our artifact received the Available, Functional, Reusable badge.
Understanding and Finding System Setting-Related Defects in Android Apps
Jingling Sun, Ting Su, Junxin Li, Zhen Dong, Geguang Pu, Tao Xie, Zhendong Su
30th ACM SIGSOFT International Symposium on Software Testing and Analysis
ISSTA 2021, pdf, SetDroid.
Highlights: (1) Our technique has successfully detected 17 previously unknown bugs in WeChat, QQMail, TikTok, CapCut, and AlipayHK (all these apps have billions of monthly-active users). (2) Our artifact received the Available, Functional, Reusable badge.
Why My App Crashes? Understanding and Benchmarking Framework-specific Exceptions of Android apps
Ting Su, Lingling Fan, Sen Chen, Yang Liu, Lihua Xu, Geguang Pu, Zhendong Su
IEEE Transactions on Software Engineering
TSE 2020, pdf, DroidDefects.
Efficiently Manifesting Asynchronous Programming Errors in Android Apps
Lingling Fan, Ting Su, Sen Chen, Guozhu Meng, Yang Liu, Lihua Xu, Geguang Pu
The 33rd IEEE/ACM International Conference on Automated Software Engineering
ASE 2018, pdf.
Large-Scale Analysis of Framework-Specific Exceptions in Android Apps
Lingling Fan#, Ting Su#, Sen Chen, Guozhu Meng, Yang Liu, Lihua Xu, Geguang Pu and Zhendong Su
The 40th International Conference on Software Engineering
ICSE 2018, pdf, slides, CrashAnalysis.
(#Equal Contribution)
Highlights: (1) the largest and most comprehensive fault study: collected 8,243 framework-specific exceptions (crashes) from 2,486 open-source Android apps, and analyzed their characteristics, manifestation, and fixes. (2) motivated several follow-up research: bug detection, fault localization and patch generation.
ACM SIGSOFT Distinguished Paper Award
Guided, Stochastic Model-Based GUI Testing of Android Apps
Ting Su, Guozhu Meng, Yuting Chen, Ke Wu, Weiming Yang, Yao Yao, Geguang Pu, Yang Liu, Zhendong Su
The 11th joint meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering
ESEC/FSE 2017, pdf, slides, Stoat.
Highlights: Stoat has (1) contributted to these popular apps: WeChat (1 bug), Gmail (1 bug), and Google+ (2 bugs). All these bugs were reported and confirmed/fixed. (2) tested 6000+ open-source and industrial Android apps in the past one year, and detected 5800+ fatal crashes.
Best Research Prototype Tool Award (NASAC 2017 held by CCF)
Other Publications
SetDroid: Detecting User-configurable Setting Issues of Android Apps via Metamorphic Fuzzing
Jingling Sun
The 43th International Conference on Software Engineering
ICSE 2021, ACM Student Research Competition, pdf
Second Place of ACM Student Research Competition
FSMdroid: Guided GUI Testing of Android Apps
Ting Su
The 38th International Conference on Software Engineering
ICSE 2016, ACM Student Research Competition, pdf, Press
Golden Medal (First Place) of ACM Student Research Competition
Acknowledge: Our research has received generous funding support from Chinese NSF, Swiss NSF, Google, ByteDance, and NTUitive Gap Fund.
If you have some questions or want to know more, feel free to contact us (Ting Su).
last modified: 2023.8.22