Translating Video Recordings of Complex Mobile App UI Gestures into Replayable Scenarios
Translating Video Recordings of Complex Mobile App UI Gestures into Replayable Scenarios
复制标题
DOI：
10.1109/tse.2022.3192279
复制
发表时间：
2023-01
影响因子：
7.4
通讯作者：
Carlos Bernal-Cárdenas;Nathan Cooper;Madeleine Havranek;Kevin Moran;Oscar Chaparro;D. Poshyvanyk;Andrian Marcus
中科院分区：
文献类型：
--
作者：
Carlos Bernal-Cárdenas;Nathan Cooper;Madeleine Havranek;Kevin Moran;Oscar Chaparro;D. Poshyvanyk;Andrian Marcus
移动应用程序的屏幕记录易于获取，并捕获与软件开发人员有关的大量信息（例如，错误或功能请求），使其成为众包应用程序反馈的流行机制。管理。 ，与其他类型的（文本）工件相比。检测和图像分类以检测视频中捕获的用户手势，并将其转换为可重播的测试方案。评估V2S+涉及243个视频，描绘了从用户收集的4,028个基于GUI的动作，这些动作从练习功能和复制90多个流行本地和混合Android应用程序中的错误进行了复制。重现$ \ $〜$≈90.2％的顺序操作，记录在物理设备上的本机应用程序方案，$ \ $ \ $ \ $ \ $≈83％的仿真器中的混合应用程序中记录的顺序动作，均具有三个工业案例研究。合作伙伴从开发人员的角度说明了V2S+的潜在实用性。
Screen recordings of mobile applications are easy to obtain and capture a wealth of information pertinent to software developers (e.g., bugs or feature requests), making them a popular mechanism for crowdsourced app feedback. Thus, these videos are becoming a common artifact that developers must manage. In light of unique mobile development constraints, including swift release cycles and rapidly evolving platforms, automated techniques for analyzing all types of rich software artifacts provide benefit to mobile developers. Unfortunately, automatically analyzing screen recordings presents serious challenges, due to their graphical nature, compared to other types of (textual) artifacts. To address these challenges, this paper introduces V2S+, an automated approach for translating video recordings of Android app usages into replayable scenarios. V2S+ is based primarily on computer vision techniques and adapts recent solutions for object detection and image classification to detect and classify user gestures captured in a video, and convert these into a replayable test scenario. Given that V2S+ takes a computer vision-based approach, it is applicable to both hybrid and native Android applications. We performed an extensive evaluation of V2S+ involving 243 videos depicting 4,028 GUI-based actions collected from users exercising features and reproducing bugs from a collection of over 90 popular native and hybrid Android apps. Our results illustrate that V2S+ can accurately replay scenarios from screen recordings, and is capable of reproducing $\approx$≈ 90.2% of sequential actions recorded in native application scenarios on physical devices, and $\approx$≈ 83% of sequential actions recorded in hybrid application scenarios on emulators, both with low overhead. A case study with three industrial partners illustrates the potential usefulness of V2S+ from the viewpoint of developers.