US 9,811,765 B2
Image captioning with weak supervision
Zhaowen Wang, San Jose, CA (US); Quanzeng You, Rochester, NY (US); Hailin Jin, San Jose, CA (US); and Chen Fang, Santa Clara, CA (US)
Assigned to Adobe Systems Incorporated, San Jose, CA (US)
Filed by Adobe Systems Incorporated, San Jose, CA (US)
Filed on Jan. 13, 2016, as Appl. No. 14/995,032.
Prior Publication US 2017/0200065 A1, Jul. 13, 2017
This patent is subject to a terminal disclaimer.
Int. Cl. G06K 9/68 (2006.01); G06K 9/62 (2006.01); G06K 9/46 (2006.01); G06F 17/30 (2006.01); G06N 3/08 (2006.01); G06N 3/04 (2006.01); G06N 7/00 (2006.01)
CPC G06K 9/6269 (2013.01) [G06F 17/3028 (2013.01); G06F 17/30675 (2013.01); G06K 9/4604 (2013.01); G06K 9/6202 (2013.01); G06N 3/0445 (2013.01); G06N 3/08 (2013.01); G06N 7/005 (2013.01)] 20 Claims
OG exemplary drawing
 
1. In a digital media environment to facilitate management of image collections using at least one computing device, a method to automatically generate image captions using weak supervision data comprising:
obtaining, by the at least one computing device, a target image for caption analysis;
applying, by the at least one computing device, feature extraction to the target image to generate global concepts corresponding to the image;
comparing, by the at least one computing device, the target image to images from a source of weakly annotated images to identify visually similar images;
building, by the at least one computing device, a collection of keywords for the target image indicative of image details by extracting the keywords from the visually similar images; and
supplying, by the at least one computing device, the collection of keywords indicative of image details as the weak supervision data for caption generation along with the global concepts.