US 11,816,773 B2
Music reactive animation of human characters
Gurunandan Krishnan Gorumkonda, Seattle, WA (US); Hsin-Ying Lee, Sunnyvale, CA (US); and Jie Xu, Cambridge, MA (US)
Assigned to Snap Inc., Santa Monica, CA (US)
Filed by Snap Inc., Santa Monica, CA (US)
Filed on Sep. 28, 2021, as Appl. No. 17/487,558.
Claims priority of provisional application 63/085,755, filed on Sep. 30, 2020.
Prior Publication US 2022/0101586 A1, Mar. 31, 2022
Int. Cl. G06T 13/20 (2011.01); G06N 3/08 (2023.01); G06T 13/40 (2011.01); G06T 13/80 (2011.01); G06N 3/044 (2023.01); G06N 3/045 (2023.01)
CPC G06T 13/205 (2013.01) [G06N 3/044 (2023.01); G06N 3/045 (2023.01); G06N 3/08 (2013.01); G06T 13/40 (2013.01); G06T 13/80 (2013.01); G10H 2210/031 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A method for generating an animated character in dance poses to music, the method comprising:
generating, by at least one processor, a music input signal based on an acoustic signal associated with the music;
training, by the at least one processor, an encoding neural network to generate a model output signal including test previous pose data, the music input signal, and test current pose data;
receiving, by the at least one processor, the model output signal from the encoding neural network;
generating, by the at least one processor, current generated pose data using a decoding neural network, the current generated pose data being based on previous generated pose data of a previous generated pose, the music input signal, and the model output signal;
generating, by the at least one processor, an animated character based on the current generated pose data; and
causing, by the at least one processor, the animated character to be displayed by a display device.