US 11,682,098 B2
Determining biomarkers from histopathology slide images
Stephen Yip, Chicago, IL (US); Irvin Ho, Wilmette, IL (US); Lingdao Sha, Chicago, IL (US); and Boleslaw Osinski, Chicago, IL (US)
Assigned to TEMPUS LABS, INC., Chicago, IL (US)
Filed by TEMPUS LABS, INC., Chicago, IL (US)
Filed on Apr. 15, 2021, as Appl. No. 17/231,891.
Application 17/231,891 is a continuation of application No. 17/192,383, filed on Mar. 4, 2021.
Application 17/192,383 is a continuation of application No. 16/830,186, filed on Mar. 25, 2020, granted, now 10,957,041.
Application 16/830,186 is a continuation in part of application No. 16/732,242, filed on Dec. 31, 2019, granted, now 10,991,097.
Application 16/732,242 is a continuation in part of application No. 16/412,362, filed on May 14, 2019.
Claims priority of provisional application 62/983,524, filed on Feb. 28, 2020.
Claims priority of provisional application 62/889,521, filed on Aug. 20, 2019.
Claims priority of provisional application 62/824,039, filed on Mar. 26, 2019.
Claims priority of provisional application 62/787,047, filed on Dec. 31, 2018.
Claims priority of provisional application 62/671,300, filed on May 14, 2018.
Prior Publication US 2022/0101519 A1, Mar. 31, 2022
This patent is subject to a terminal disclaimer.
Int. Cl. G06T 1/20 (2006.01); G06T 7/00 (2017.01); G06T 7/11 (2017.01); G06T 11/00 (2006.01); G06F 18/21 (2023.01); G06F 18/2431 (2023.01); G06V 10/764 (2022.01); G06V 10/82 (2022.01); G06V 10/44 (2022.01)
CPC G06T 1/20 (2013.01) [G06F 18/21 (2023.01); G06F 18/2431 (2023.01); G06T 7/0012 (2013.01); G06T 7/11 (2017.01); G06T 11/00 (2013.01); G06V 10/44 (2022.01); G06V 10/764 (2022.01); G06V 10/82 (2022.01); G06T 2207/20081 (2013.01); G06T 2207/30024 (2013.01); G06T 2207/30096 (2013.01)] 28 Claims
OG exemplary drawing
 
1. A computer-implemented method of identifying biomarkers in a digital image of a hematoxylin and eosin (H&E) stained slide of target tissue, the method comprising:
receiving the digital image to an image-based biomarker prediction system having one or more processors;
performing an image tiling process, using the one or more processors, on the digital image by separating the digital image into a plurality of tile images, where each of the plurality of tile images contains a different portion of the digital image;
applying, using the one or more processors, the plurality of tile images to a multiscale deep learning framework comprising one or more trained deep learning multiscale classifier models, each trained deep learning multiscale classifier models being trained to classify a different tissue classification for each tile image and determining a tissue classification for each of the plurality of tile images, using the multiscale deep learning framework;
identifying, using the one or more processors, cells within each of the plurality of tile images using a trained cell segmentation model; and
identifying a predicted presence of one or more biomarkers associated with the digital image based at least in part on analyzing one or more tile images adjacent to the respective tile image and analyzing a larger tile image comprising a subset of the plurality of tile images, the subset containing the one or more tile images adjacent to the respective tile image,
wherein determining the tissue classification for each of the plurality of tile images comprises analyzing one or more tile images adjacent to the respective tile image and analyzing the larger tile image comprising the subset of the plurality of tile images and producing a 3D array of values that each represent the probability that a tissue class classification label matches a tissue classification in each tile image of the subset of the plurality of tile images.