Title: Automatic Annotation of Structured Facts in Images

Abstract: Motivated by the application of fact-level image understanding, we present an automatic method for data collection of structured visual facts from images with captions. Example structured facts include attributed objects (e.g., <flower, red>), actions (e.g., <baby, smile>), interactions (e.g., <man, walking, dog>), and positional information (e.g., <vase, on, table>). The collected annotations are in the form of fact-image pairs (e.g.,<man, walking, dog> and an image region containing this fact). With a language approach, the proposed method is able to collect hundreds of thousands of visual fact annotations with accuracy of 83\% according to human judgment. %that we obtained using both Amazon Mechanical Turk and volunteer scientists in our laboratory. Our method automatically collected more than 380,000 visual fact annotations and more than 110,000 unique visual facts from images with captions and localized them in images in less than one day of processing time on standard CPU platforms. %Our work enable large-scale fact-level understanding of images. %Based on human j evaluation shows that more than 83\% of the automatically collected annotation %We focus on collecting higher order visual facts annotations which include attributed objects (e.g. <car, black>, <flower, red>, actions <baby, smile>, and interactions <dog, riding, wave>.