US 11,818,510 B2
Monitoring adverse events in the background while displaying a higher resolution surgical video on a lower resolution display
Jagadish Venkataraman, Menlo Park, CA (US); Dave Scott, Oakland, CA (US); and Eric Johnson, Pacific Grove, CA (US)
Assigned to Verb Surgical Inc., Santa Clara, CA (US)
Filed by Verb Surgical Inc., Santa Clara, CA (US)
Filed on Aug. 8, 2022, as Appl. No. 17/883,311.
Application 17/883,311 is a continuation of application No. 17/340,942, filed on Jun. 7, 2021, granted, now 11,426,056.
Application 17/340,942 is a continuation of application No. 16/361,075, filed on Mar. 21, 2019, granted, now 11,026,561, issued on Jun. 8, 2021.
Prior Publication US 2022/0377373 A1, Nov. 24, 2022
This patent is subject to a terminal disclaimer.
Int. Cl. A61B 1/00 (2006.01); H04N 7/18 (2006.01); H04N 7/01 (2006.01); A61B 90/00 (2016.01); A61B 1/313 (2006.01); A61B 34/30 (2016.01); A61B 34/20 (2016.01); G06N 20/00 (2019.01); A61B 1/045 (2006.01); G06V 20/40 (2022.01); H04N 19/59 (2014.01); A61B 34/00 (2016.01)
CPC H04N 7/183 (2013.01) [A61B 1/0004 (2022.02); A61B 1/00009 (2013.01); A61B 1/00045 (2013.01); A61B 1/00055 (2013.01); A61B 1/000096 (2022.02); A61B 1/045 (2013.01); A61B 1/3132 (2013.01); A61B 34/20 (2016.02); A61B 34/25 (2016.02); A61B 34/30 (2016.02); A61B 90/37 (2016.02); G06N 20/00 (2019.01); G06V 20/41 (2022.01); H04N 7/0117 (2013.01); H04N 19/59 (2014.11); A61B 2034/2055 (2016.02); A61B 2034/2057 (2016.02); A61B 2090/373 (2016.02); G06V 20/44 (2022.01)] 19 Claims
OG exemplary drawing
 
1. A computer-implemented method comprising:
receiving an endoscope video of a surgical procedure corresponding to an endoscope view;
displaying a first portion of video images of the endoscope video on a display to assist a surgeon performing the surgical procedure, wherein the first portion of the video images represents an on-screen portion of the endoscope view;
while displaying the first portion of the video images, using one or more deep-learning models to monitor a second portion of the video images of the endoscope video not being displayed on the display, wherein the second portion of the video images represents an off-screen portion of the endoscope view, wherein each deep-learning model in the one or more deep-learning models is constructed to detect an adverse event among a set of known adverse events, and wherein the one or more deep-learning models include:
a first deep-learning model trained to detect a first tool off-screen risk event in a set of surgical-tool off-screen risk events, wherein the first tool off-screen event is when two jaws of a surgical tool are unintentionally engaged on a tissue; or
a second deep-learning model trained to detect a second tool off-screen risk event in the set of surgical-tool off-screen risk events, wherein the second tool off-screen event is when a sharp tip of a surgical tool is approaching a critical anatomy; and
in response to detecting the detected adverse event by the one or more deep-learning models in the second portion of the video images, notifying the surgeon of the detected adverse event to prompt an appropriate action to the detected adverse event.