US 11,704,568 B2
Method and system for hand activity sensing
Gierad Laput, Pittsburgh, PA (US); and Christopher Harrison, Pittsburgh, PA (US)
Assigned to CARNEGIE MELLON UNIVERSITY, Pittsburgh, PA (US)
Filed by Carnegie Mellon University, Pittsburgh, PA (US)
Filed on Oct. 9, 2019, as Appl. No. 16/597,404.
Claims priority of provisional application 62/766,396, filed on Oct. 16, 2018.
Prior Publication US 2020/0117889 A1, Apr. 16, 2020
Int. Cl. G06N 20/00 (2019.01); G06N 3/082 (2023.01); G06V 40/20 (2022.01); G06F 18/24 (2023.01)
CPC G06N 3/082 (2013.01) [G06F 18/24 (2023.01); G06N 20/00 (2019.01); G06V 40/28 (2022.01)] 17 Claims
OG exemplary drawing
 
1. A system, comprising:
a memory of a hand activity sensing component that stores computer executable components; and
a processor of the hand activity sensing component that executes computer executable components stored in the memory, wherein the computer executable components comprise:
an inertial measurement unit (IMU) component of the hand activity sensing component that obtains, from a wrist-worn computational device comprising the hand activity sensing component, hand activity data associated with a sustained series of hand motor actions in performance of a human task, wherein the human task is characterized by spatial-temporal relationships associated with the wrist-worn computational device over implementation of the human task that are encoded in the hand activity data for the sustained series of hand motor actions in the implementation of the human task, wherein the IMU component characterizes the spatial-temporal relationships based at least in part on an X-axis, a Y-axis, and a Z-axis of at least one accelerometer of the wrist-worn computational device, and wherein the IMU component generates first spectrogram data associated with an X-axis of a gyroscope of the wrist-worn computational device, second spectrogram data associated with a Y-axis of the gyroscope, and third spectrogram data associated with a Z-axis of the gyroscope; and
a machine learning component of the hand activity sensing component that employs a human task trained machine learning algorithm to determine classification data indicative of a classification for the human task, based at least in part on the spatial-temporal relationships, wherein the machine learning component employs data associated with the X-axis, the Y-axis, and the Z-axis of the at least one accelerometer as input for the machine learning algorithm to determine the classification data, wherein the machine learning component employs the first spectrogram data associated with the X-axis of the gyroscope, the second spectrogram data associated with the Y-axis of the gyroscope, and the third spectrogram data associated with the Z-axis of the gyroscope as input for the machine learning algorithm to determine the classification data, and wherein the machine learning component learns cross-axis relationships among the X-axis of the gyroscope, the Y-axis of the gyroscope, and the Z-axis of the gyroscope to determine the classification data.