US 11,115,565 C1 (12,944th)
User feedback for real-time checking and improving quality of scanned image
Alexander Ilic, Zurich (CH)
Filed by ML Netherlands C.V., Amsterdam (NL)
Assigned to MAGIC LEAP, INC., Plantation, FL (US)
Reexamination Request No. 90/019,413, Feb. 13, 2024.
Reexamination Certificate for Patent 11,115,565, issued Sep. 7, 2021, Appl. No. 16/569,531, Sep. 12, 2019.
Application 90/019,413 is a continuation of application No. 15/173,355, filed on Jun. 3, 2016, granted, now 10,455,128.
Application 15/173,355 is a continuation of application No. PCT/EP2014/076469, filed on Dec. 3, 2014.
Claims priority of provisional application 61/911,039, filed on Dec. 3, 2013.
Ex Parte Reexamination Certificate issued on Jun. 24, 2025.
Int. Cl. G06T 3/4038 (2024.01); G06T 1/60 (2006.01); G06T 11/60 (2006.01); G06V 30/224 (2022.01); G06V 30/414 (2022.01); H04N 23/60 (2023.01); H04N 23/62 (2023.01); H04N 23/63 (2023.01); H04N 23/69 (2023.01); H04N 23/74 (2023.01); H04N 23/80 (2023.01); H04N 23/951 (2023.01)
CPC G06T 3/4038 (2013.01) [G06T 1/60 (2013.01); G06T 11/60 (2013.01); G06V 30/224 (2022.01); G06V 30/414 (2022.01); H04N 23/62 (2023.01); H04N 23/632 (2023.01); H04N 23/635 (2023.01); H04N 23/64 (2023.01); H04N 23/69 (2023.01); H04N 23/74 (2023.01); H04N 23/80 (2023.01); H04N 23/951 (2023.01)]
OG exemplary drawing
AS A RESULT OF REEXAMINATION, IT HAS BEEN DETERMINED THAT:
Claims 1-3, 12, 13, and 18 are determined to be patentable as amended.
Claims 4-11, 14-17, and 19-20, dependent on an amended claim, are determined to be patentable.
New claims 21-33 are added and determined to be patentable.
1. A method of forming a composite image of a scene using a portable electronic device, the method comprising:
capturing a stream of image frames of a scene with the portable electronic device;
extracting one or more image features from image frames of the stream of image frames;
associating sets of points with the one or more image features;
determining correspondences between sets of points associated with the one or more image features from multiple image frames of the stream of image frames;
sequentially incorporating image frames of the stream of image frames into a three dimensional point cloud based on the determined correspondences, wherein the incorporated image frames are incorporated into initial positions in the three dimensional point cloud; and
adjusting the points in the three dimensional point cloud based on a bundle adjustment for a plurality of the sets of points [ , wherein the points in the three dimensional point cloud are adjusted in parallel with capturing the stream of image frames] .
2. The method of claim 1, [ wherein determining correspondences between sets of points associated with the one or more image features from multiple image frames of the stream of image frames ] further comprising [ comprises ] associating the sets of points with a common frame of reference [ separate from the three dimensional point cloud] .
3. The method of claim 2 [ , ] wherein associating the sets of points with a common frame of reference comprises projecting the sets of points [ from the three dimensional point cloud ] into a common plane [ separate from the three dimensional point cloud and the stream of image frames] .
12. A portable electronic device, comprising:
computer memory;
a camera configured to capture a stream of image frames; and
a processor, in communication with the camera, configured to form a composite image of an object based on the stream of image frames by:
extracting one or more image features from image frames of the stream of image frames;
associating sets of points with the one or more image features;
determining correspondences between sets of points associated with the one or more image features from multiple image frames of the stream of image frames;
sequentially incorporating image frames of the stream of image frames into a three dimensional point cloud based on the determined correspondences [ , wherein the sequentially incorporating of the image frames of the stream of image frames into the three dimensional point cloud is performed in parallel with capturing the stream of image frames] ; and
reconstructing surfaces based on the three dimensional point cloud so as to provide in the computer memory a three dimensional representation of the object.
13. The portable electronic device of claim 12, wherein sequentially incorporating image frames of the stream of image frames into a three dimensional point cloud based on the determined correspondences comprises:
adjusting a three dimensional position associated with each set of points of the sets of points to achieve coincidence in a common plane [ , separate from the three dimensional point cloud and the stream of image frames, ] between points of the sets of points representing a same image feature.
18. At least one non-transitory, computer-readable storage medium having stored instructions for forming a composite image of an object from a stream of image frames captured by a camera of a portable electronic device, the instructions, when executed by an at least one processor [ of the portable electronic device] , cause the [ at least one ] processor to perform a method comprising steps of:
extracting one or more image features from image frames of the stream of image frames;
associating sets of points with the one or more image features;
determining correspondences between sets of points associated with the one or more image features from multiple image frames of the stream of image frames;
sequentially incorporating image frames of the stream of image frames into a three dimensional point cloud based on the determined correspondences [ , wherein the sequentially incorporating of the image frames of the stream of image frames into the three dimensional point cloud is performed in parallel with capturing the stream of image frames] ; and
rendering the composite image of the object based on the three dimensional point cloud.
[ 21. The method of claim 1, wherein sequentially incorporating image frames of the stream of image frames into a three dimensional point cloud based on the determined correspondences comprises:
as the stream of image frames of the scene is being captured, sequentially incorporating the image frames of the stream of image frames into the three dimensional point cloud at least in part by setting positions of the sets of points associated with the one or more image features in the three dimensional point cloud.]
[ 22. The portable electronic device of claim 12, wherein the processor is further configured to:
as the stream of image frames of the scene is being captured, sequentially incorporate the image frames of the stream of image frames into the three dimensional point cloud at least in part by setting positions of the sets of points associated with the one or more image features in the three dimensional point cloud.]
[ 23. The at least one non-transitory, computer-readable storage medium of claim 18, wherein sequentially incorporating image frames of the stream of image frames into a three-dimensional point cloud based on the determined correspondences comprises:
as the stream of image frames of the scene is being captured, sequentially incorporating the image frames of the stream of image frames into the three dimensional point cloud at least in part by setting positions of the sets of points associated with the one or more image features in the three dimensional point cloud.]
[ 24. The method of claim 1, further comprising:
stopping capturing of the stream of image frames after adjusting points in the three dimensional point cloud based on the bundle adjustment for the plurality of sets of points.]
[ 25. The method of claim 1, wherein:
adjusting points in the three dimensional point cloud based on the bundle adjustment comprises adjusting the initial points in the three dimensional point cloud determined from a first subset of image frames of the stream of image frames incorporated into the three dimensional point cloud prior to capturing a second subset of image frames of the stream of image frames that are incorporated into the three dimensional point cloud.]
[ 26. The method of claim 1, further comprising:
while sequentially incorporating image frames of the stream of image frames into the three dimensional point cloud, providing feedback to a user of the portable electronic device, the feedback indicating an adjustment to the portable electronic device for capturing the stream of images.]
[ 27. A method of forming a composite image of a scene using a portable electronic device, the method comprising:
capturing a stream of image frames of a scene with the portable electronic device;
extracting one or more image features from image frames of the stream of image frames;
associating sets of points with the one or more image features;
determining correspondences between sets of points associated with the one or more image features from multiple image frames of the stream of image frames;
sequentially incorporating image frames of the stream of image frames into a three dimensional point cloud based on the determined correspondences, wherein the incorporated image frames are incorporated into initial positions in the three dimensional point cloud; and
adjusting the points in the three dimensional point cloud based on a bundle adjustment for a plurality of the sets of points, wherein the method further comprises:
determining whether a threshold number of image frames have been captured by the portable electronic device; and
triggering the adjusting of the points in the three dimensional point cloud based on the bundle adjustment in response to determining that the threshold number of image frames has been captured.]
[ 28. A method of forming a composite image of a scene using a portable electronic device, the method comprising:
capturing a stream of image frames of a scene with the portable electronic device;
extracting one or more image features from image frames of the stream of image frames;
associating sets of points with the one or more image features;
determining correspondences between sets of points associated with the one or more image features from multiple image frames of the stream of image frames;
sequentially incorporating image frames of the stream of image frames into a three dimensional point cloud based on the determined correspondences, wherein the incorporated image frames are incorporated into initial positions in the three dimensional point cloud; and
adjusting the points in the three dimensional point cloud based on a bundle adjustment for a plurality of the sets of points, wherein the method further comprises:
detecting a loop closure in the stream of images of the scene; and
adjusting positions within the three dimensional point cloud of a subset of image frames within the detected loop based on detecting the loop closure.]
[ 29. The method of claim 28, wherein detecting the loop closure in the stream of images of the scene comprises:
determining whether one or more features of a captured image frame overlap with at least one image frame other than an immediately preceding image frame.]
[ 30. The method of claim 1, wherein the method is performed by the portable electronic device.]
[ 31. The method of claim 1, further comprising:
while capturing the stream of image frames:
determining whether a condition is met; and
triggering the adjusting of the points in the three dimensional point cloud based on the bundle adjustment in response to determining that the condition is met.]
[ 32. The method of claim 1, wherein the capturing of the stream of image frames of the scene with the portable electronic device is being performed in parallel with sequentially incorporating image frames of the stream of image frames into the three dimensional point cloud.]
[ 33. The method of claim 1, wherein the adjusting of the points in the three dimensional point cloud based on the bundle adjustment is performed in parallel with sequentially incorporating image frames of the stream of image frames into the three dimensional point cloud.]