US 9,811,352 B1
Replaying user input actions using screen capture images
Matthew Sharifi, Palo Alto, CA (US); and David Petrou, Brooklyn, NY (US)
Assigned to GOOGLE INC., Mountain View, CA (US)
Filed by GOOGLE INC., Mountain View, CA (US)
Filed on Aug. 4, 2014, as Appl. No. 14/451,401.
Claims priority of provisional application 62/023,736, filed on Jul. 11, 2014.
Int. Cl. G06F 17/30 (2006.01); G06F 9/44 (2006.01); G06F 3/0484 (2013.01)
CPC G06F 9/4443 (2013.01) [G06F 3/04842 (2013.01)] 22 Claims
OG exemplary drawing
 
1. A mobile device comprising:
at least one processor;
a data store storing:
a plurality of captured user input actions, each input action having a corresponding timestamp; and
a plurality of images, each image representing a screen captured on the mobile device, each image having a corresponding timestamp; and
memory storing instructions that, when executed by the at least one processor, cause the mobile device to:
receive a selection of a first image of the plurality of images,
determine a set of the captured user input actions, the set including user input actions occurring prior to a timestamp corresponding to the first image and after a timestamp corresponding to a reference image, the reference image being one of the plurality of images, and
replay the set of user input actions on the mobile device starting from a state corresponding to the reference image, wherein the replay includes:
performing the user input actions in the set that occur between the timestamp for the reference image and a timestamp for a subsequent image, the performance of the actions resulting in a current screen,
comparing the current screen with the subsequent image, and
responsive to the current screen matching the subsequent image, performing the user input actions in the set that occur between the timestamp for the subsequent image and a next subsequent image.