Automatic tag generation and ranking for sensor-rich outdoor videos

ABSTRACT

Video tag annotations have become a useful and powerful feature to facilitate video search in many social media and web applications. The majority of tags assigned to videos are supplied by users - a task which is time consuming and may result in annotations that are subjective and lack precision. A number of studies have utilized content-based extraction techniques to automate tag generation. However, these methods are compute-intensive and challenging to apply across domains. Here, we describe a complementary approach for generating tags based on the geographic properties of videos. With today's sensor-equipped smartphones, the location and orientation of a camera can be continuously acquired in conjunction with the captured video stream. Our novel technique utilizes these sensor meta-data to automatically tag outdoor videos in a two step process. First, we model the viewable scenes of the video as geometric shapes by means of its accompanied sensor data and determine the geographic objects that are visible in the video by querying geo-information databases through the viewable scene descriptions. Subsequently we extract textual information about the visible objects to serve as tags. Second, we define six criteria to score the tag relevance and rank the obtained tags based on these scores. Then we associate the tags with the video and the accurately delimited segments of the video. To evaluate the proposed technique we implemented a prototype tag generator and conducted a user study. The results demonstrate significant benefits of our method in terms of automation and tag utility.

References

1. M. Ames and M. Naaman. Why We Tag: Motivations for Annotation in Mobile and Online Media. In CHI, 2007.
2. S. Arslan Ay, R. Zimmermann, and S. H. Kim. Viewable Scene Modeling for Geospatial Video Search. In ACM Multimedia, 2008.
3. L. Cao, J. Luo, and T. S. Huang. Annotating Photo Collections by Label Propagation According to Multiple Similarity Cues. In ACM Multimedia, 2008.
4. Y. Gao, J. Tang, R. Hong, Q. Dai, T. S. Chua, and R. Jain. W2Go: A Travel Guidance System by Automatic Landmark Ranking. In ACM Multimedia, 2010.
5. C. H. Graham, N. R. Bartlett, J. L. Brown, Y. Hsia, C. C. Mueller, and L. A. Riggs. Vision and Visual Perception. John Wiley & Sons, Inc., 1965.
6. R. Jain and P. Sinha. Content Without Context is Meaningless. In ACM Multimedia, 2010.
7. Y. G. Jiang, C. W. Ngo, and S. F. Chang. Semantic Context Transfer across Heterogeneous Sources for Domain Adaptive Video Search. In ACM Multimedia, 2009.
8. Y. Jin, M. Hu, H. Singh, D. Rule, M. Berlyant, and Z. Xie. MySpace Video Recommendation with Map-Reduce on Qizmt. In IEEE ICSC, 2010.
9. T. Judd, K. Ehinger, F. Durand, and A. Torralba. Learning to Predict Where Humans Look. In ICCV, 2009.
10. K. C. K. Lee, W.-C. Lee, and H. V. Leong. Nearest Surrounder Queries. IEEE TKDE, 2010.
11. K. Lerman and L. Jones. Social Browsing on Flickr. Arxiv preprint cs0612047, 2006.
12. X. Li, L. Guo, and Y. E. Zhao. Tag-based Social Interest Discovery. In WWW, 2008.
13. D. Liu, X. S. Hua, L. Yang, M. Wang, and H. J. Zhang. Tag Ranking. In WWW, 2009.
14. X. Liu, M. Corner, and P. Shenoy. SEVA: Sensor-Enhanced Video Annotation. In ACM Multimedia, 2005.
15. X. Lu, C. Wang, J. M. Yang, Y. Pang, and L. Zhang. Photo2Trip: Generating Travel Routes from Geo-Tagged Photos for Trip Planning. In ACM Multimedia, 2010.
16. M. Naaman, S. Harada, Q. Wang, H. G. Molina, and A. Paepcke. Context Data in GeoReferenced Digital Photo Collections. In ACM Multimedia, 2004.
17. M. Naphade, J. R. Smith, J. Tesic, S.-F. Chang, W. Hsu, L. Kennedy, A. Hauptmann, and J. Curtis. Large-Scale concept ontology for multimedia. IEEE Multimedia, 2006.
18. M. R. Naphade and J. R. Smith. On the Detection of Semantic Concepts at TRECVID. In ACM Multimedia, 2004.
19. A. Pigeau and M. Gelgon. Building and Tracking Hierarchical Geographical & Temporal Partitions for Image Collection Management on Mobile Devices. In ACM Multimedia, 2005.
20. G. J. Qi, X. S. Hua, Y. Rui, J. Tang, T. Mei, and H. J. Zhang. Correlative Multi-Label Video Annotation. In ACM Multimedia, 2007.
21. C. Shahabi, F. Banaei-Kashani, A. Khoshgozaran, L. Nocera, and S. Xing. GeoDec: A Framework to Effectively Visualize and Query Geospatial Data for Decision-Making. IEEE Multimedia, 2010.
22. S. Siersdorfer, J. San Pedro, and M. Sanderson. Automatic Video Tagging using Content Redundancy. In SIGIR, 2009.
23. B. Sigurbjörnsson and R. van Zwol. Flickr Tag Recommendation based on Collective Knowledge. In WWW, 2008.
24. F. M. Suchanek, M. Vojnovic, and D. Gunawardena. Social Tags: Meaning and Suggestions. In ACM CIKM, 2008.
25. K. Toyama, R. Logan, and A. Roseway. Geographic Location Tags on Digital Images. In ACM Multimedia, 2003.
26. M. Wang, X. S. Hua, X. Yuan, Y. Song, and L. R. Dai. Optimizing Multi-Graph Learning: Towards A Unified Video Annotation Scheme. In ACM Multimedia, 2007.
27. R. Yan, A. Natsev, and M. Campbell. A Learning-based Hybrid Tagging and Browsing Approach for Efficient Manual Image Annotation. In CVPR, 2008.
28. K. Yang, X. S. Hua, M. Wang, and H. J. Zhang. Tagging Tags. In ACM Multimedia, 2010.
29. P. A. Zandbergen. Accuracy of iPhone Locations: A Comparison of Assisted-GPS, WiFi and Cellular Positioning. Transactions in GIS, 2009.

Index Terms

• Information systems

  • Information retrieval