Informatik, TU Wien

Can you see it? Annotating Image Regions based on Users' Gaze Information

Providing image annotations is a tedious task. This becomes even more cumbersome when distinct regions shall be annotated such as objects and people shown in the images.


Providing image annotations is a tedious task. This becomes even more cumbersome when distinct regions shall be annotated such as objects and people shown in the images. Such region-based annotations can be used in various ways like for similarity search and search based on the coherence of individual image regions. In the talk, we will present a three-step approach to improve the annotation of image regions based on the users' gaze information. In the first step, we investigate the principle idea of finding specific objects in images by conducting a controlled experiment where 20 subjects have viewed 50 image-tag-pairs with the task to decide whether a tag presented could be found in the image or not. The regions of the images are from LabelMe, a photo-sharing community with manually drawn in object polygons. We have compared 13 different fixation measures. The best performing measure is able to correctly identify 63% of the image-tag-pairs and significantly outperforms two baselines. In addition, we have investigated if different regions can be discriminated in the same image. Here, we are able to correctly identify two regions in the same image with an accuracy of 38%. In the second step, we have loosen the experiment restrictions by introducing a novel fixation measure that uses automatically computed segments instead of manually drawn in polygons.

The segmentation-based measure achieves a maximum average precision of 65% and best coverage of the segments with a F-measure of 35%. The novel segmentation measure significantly outperforms a baseline based on the golden ratio of photography. Finally, the last step is to investigate the assignment of tags to image regions in a free, interactive tagging application.


Ansgar Scherp is Juniorprofessor for Media Informatics / New Media in Information Systems Research at the University of Mannheim since August 2012. Prior to that he was working as Juniorprofessor for Semantic Web at the University of Koblenz-Landau in the Institute for Information Systems Research since April 2011 and lead the focus group on Interactive and Multimedia Web at the Institute for Web Science and Technologies (WeST) at the same university since May 2008. He has studied computer science at the University of Oldenburg, Germany and has received the Advancement Award for Outstanding Results in Studies from the Association for Electrical, Electronic & Information Technologies (VDE), Germany in 1998. He finished his PhD with the thesis title "A Component Framework for Personalized Multimedia Applications" at the University of Oldenburg, Germany with distinction in 2006.

Afterwards, Mr. Scherp has been EU Marie Curie Fellow with Prof. Ramesh Jain at the Donald Bren School of Information and Computer Sciences, University of California, Irvine, USA in Los Angeles between November 2006 to October 2007. He has lead the University of Koblenz-Landau's activities in the EU Integrated Project WeKnowIt from 2008 to 2011. Here, he has been leading the work packages on knowledge management and mass intelligence and has been member of the project management board and steering board committee. Mr. Scherp is scientific leader of the EU project SocialSensor, where the University of Koblenz-Landau is leading the work package on user modeling and presentation.

In December 2011, he has received his Venia Legendi (Habilitation) with the thesis title "Semantic Media Management: Process Innovation along the Value Chain of Media Companies" (in German) from the University of Koblenz-Landau, Germany. He has published over 60 peer-reviewed scientific publications including 12 journal articles, 21 conference papers, and 10 book chapters.


This lecture is organized by the Interactive Media Systems Group at the Institute of Software Technology & Interactive Systems.