US 11,682,165 B2
Methods and systems for augmenting depth data from a depth sensor, such as with data from a multiview camera system
Thomas Ivan Nonn, Kenmore, WA (US); David Julio Colmenares, Seattle, WA (US); and James Andrew Youngquist, Seattle, WA (US)
Assigned to Proprio, Inc., Seattle, WA (US)
Filed by Proprio, Inc., Seattle, WA (US)
Filed on Jan. 21, 2021, as Appl. No. 17/154,670.
Claims priority of provisional application 62/963,717, filed on Jan. 21, 2020.
Prior Publication US 2021/0225020 A1, Jul. 22, 2021
Int. Cl. G06T 17/00 (2006.01); G06T 7/557 (2017.01); G06T 17/20 (2006.01); H04N 13/282 (2018.01); G06V 10/80 (2022.01); G06V 20/20 (2022.01); G06V 20/64 (2022.01); H04N 13/00 (2018.01)
CPC G06T 17/00 (2013.01) [G06T 7/557 (2017.01); G06T 17/20 (2013.01); G06V 10/806 (2022.01); G06V 20/20 (2022.01); G06V 20/647 (2022.01); H04N 13/282 (2018.05); G06T 2207/10052 (2013.01); G06T 2207/20104 (2013.01); G06T 2210/41 (2013.01); H04N 2013/0081 (2013.01)] 14 Claims
OG exemplary drawing
 
1. A method of generating an output image of a portion of a scene, wherein the output image is from a perspective of a virtual camera having a field of view corresponding to the portion of the scene, the method comprising:
capturing depth data of the scene with a depth sensor;
capturing light field image data of the scene with a plurality of cameras, wherein the plurality of cameras each have a different position and orientation relative to the scene;
generating a point cloud representative of the scene based on the depth data;
identifying a missing region of the point cloud within the field of view of the virtual camera in which the point cloud includes no data or sparse data, wherein identifying the missing region of the point cloud includes determining that the missing region of the point cloud has fewer than a predetermined threshold number of data points;
generating depth data for the missing region based on the light field image data;
merging the depth data for the missing region with the depth data from the depth sensor to generate a merged point cloud representative of the scene;
processing the light field image data and the merged point cloud to synthesize the output image of the portion scene from the perspective of the virtual camera; and
transmitting the output image to a display for display to a user.