CPC G06T 17/00 (2013.01) [G06T 7/557 (2017.01); G06T 17/20 (2013.01); G06V 10/806 (2022.01); G06V 20/20 (2022.01); G06V 20/647 (2022.01); H04N 13/282 (2018.05); G06T 2207/10052 (2013.01); G06T 2207/20104 (2013.01); G06T 2210/41 (2013.01); H04N 2013/0081 (2013.01)] | 14 Claims |
1. A method of generating an output image of a portion of a scene, wherein the output image is from a perspective of a virtual camera having a field of view corresponding to the portion of the scene, the method comprising:
capturing depth data of the scene with a depth sensor;
capturing light field image data of the scene with a plurality of cameras, wherein the plurality of cameras each have a different position and orientation relative to the scene;
generating a point cloud representative of the scene based on the depth data;
identifying a missing region of the point cloud within the field of view of the virtual camera in which the point cloud includes no data or sparse data, wherein identifying the missing region of the point cloud includes determining that the missing region of the point cloud has fewer than a predetermined threshold number of data points;
generating depth data for the missing region based on the light field image data;
merging the depth data for the missing region with the depth data from the depth sensor to generate a merged point cloud representative of the scene;
processing the light field image data and the merged point cloud to synthesize the output image of the portion scene from the perspective of the virtual camera; and
transmitting the output image to a display for display to a user.
|