US 11,809,393 B2
Image and text data hierarchical classifiers
Andrew Stanton, San Francisco, CA (US); Nikhil Raghavan, Menlo Park, CA (US); Manjunath Rajashekhar, Menlo Park, CA (US); and Devin Guillory, San Francisco, CA (US)
Assigned to Etsy, Inc., Brooklyn, NY (US)
Filed by Etsy, Inc., Brooklyn, NY (US)
Filed on Feb. 24, 2020, as Appl. No. 16/799,292.
Application 16/799,292 is a continuation of application No. 14/866,554, filed on Sep. 25, 2015, abandoned.
Claims priority of provisional application 62/173,828, filed on Jun. 10, 2015.
Prior Publication US 2020/0210396 A1, Jul. 2, 2020
Int. Cl. G06F 16/22 (2019.01); G06F 16/951 (2019.01); G06F 16/2453 (2019.01); G06F 16/41 (2019.01); G06N 3/045 (2023.01); G06N 3/08 (2023.01); G06N 3/084 (2023.01); G06N 3/044 (2023.01)
CPC G06F 16/2228 (2019.01) [G06F 16/2453 (2019.01); G06F 16/41 (2019.01); G06F 16/951 (2019.01); G06N 3/045 (2023.01); G06N 3/08 (2013.01); G06N 3/044 (2023.01); G06N 3/084 (2013.01)] 20 Claims
OG exemplary drawing
 
1. A system, comprising:
a processor; and
a memory coupled with the processor, wherein the memory is configured to provide the processor with instructions which when executed cause the processor to perform operations comprising:
retrieving an image and text data that are both presented in a particular listing of a particular object;
generating, using one or more hierarchical classifiers, a first set of tags based on the image presented in the particular listing of the particular object;
generating a second set of tags that differ from the first set of tags based on the text data presented in the particular listing of the particular object;
determining that there is an inconsistency between (i) a particular tag in the first set of tags generated based on the image presented in the particular listing of the particular object and (ii) a different tag in the second set of tags generated based on the text data presented in the particular listing of the particular object based, at least in part, on a determination the different tag generated based on the text data describes a visual trait of the particular object in a different way than the particular tag describes how the visual trait of the particular object is visually depicted by the image presented in the particular listing;
resolving the inconsistency based on a comparison of the particular tag and the different tag to the text data and visual characteristics extracted from the image of the particular object;
generating a search index based at least on generated tags from the first set of tags and the second set of tags that remain following resolving the inconsistency;
receiving a search query comprising one or more search terms;
utilizing the search index to determine that a search result identifying the particular object is responsive to the search query based on the one or more search terms matching the generated tags that remain following resolving the inconsistency; and
returning a search result that presents the image and the text data, wherein:
determining that there is an inconsistency between (i) a particular tag in the first set of tags generated based on the image presented in the particular listing of the particular object and (ii) a different tag in the second set of tags generated based on the text data presented in the particular listing of the particular object comprises determining that the different tag describes the visual trait as having a first color and determining that the particular tag describes the visual trait as having a second color that differs from the first color.