A scalable and extensible segment-event-object-based sports video retrieval system

Abstract

Sport video data is growing rapidly as a result of the maturing digital technologies that support digital video capture, faster data processing, and large storage. However, (1) semi-automatic content extraction and annotation, (2) scalable indexing model, and (3) effective retrieval and browsing, still pose the most challenging problems for maximizing the usage of large video databases. This article will present the findings from a comprehensive work that proposes a scalable and extensible sports video retrieval system with two major contributions in the area of sports video indexing and retrieval. The first contribution is a new sports video indexing model that utilizes semi-schema-based indexing scheme on top of an Object-Relationship approach. This indexing model is scalable and extensible as it enables gradual index construction which is supported by ongoing development of future content extraction algorithms. The second contribution is a set of novel queries which are based on XQuery to generate dynamic and user-oriented summaries and event structures. The proposed sports video retrieval system has been fully implemented and populated with soccer, tennis, swimming, and diving video. The system has been evaluated against 20 users to demonstrate and confirm its feasibility and benefits. The experimental sports genres were specifically selected to represent the four main categories of sports domain: period-, set-point-, time (race)-, and performance-based sports. Thus, the proposed system should be generic and robust for all types of sports.

References

1. Adali, S., Candan, K. S., Chen, S.-S., Erol, K., and Subrahmanian, V. S. 1996. The advanced video information system: data structures and query processing. Multimedia Syst. 4, 4, 172--186.
2. Adams, B., Dorai, C., and Venkatesh, S. 2002. Toward automatic extraction of expressive elements from motion pictures: tempo. IEEE Trans. Multimedia 4, 4, 472--481.
3. Assfalg, J., Bertini, M., Colombo, C., Del Bimbo, A., and Nunziati, W. 2003. Automatic extraction and annotation of soccer video highlights. In Proceedings of the International Conference on Image Processing. vol. 523.II, 527--530.
4. Assfalg, J., Bertini, M., Del Bimbo, A., Nunziati, W., and Pala, P. 2002. Detection and recognition of football highlights using HMM. In Proceedings of the 9th International Conference on Electronics, Circuits and Systems. vol. 1053, 1059--1062.
5. Assfalg, J., Bertini, M., Del Bimbo, A., Nunziati, W., and Pala, P. 2002. Soccer highlights detection and recognition using HMMs. In Proceedings of the IEEE International Conference on Multimedia and Expo. vol. 821, 825--828.
6. Babaguchi, N., Kawai, Y., and Kitabashi, T. 2002. Event based indexing of broadcasted sports video by intermodal collaboration. IEEE Trans Multimedia 4, 1, 68--75.
7. Babaguchi, N., Kawai, Y., Yasugi, Y., and Kitahashi, T. 2000. Linking live and replay scenes in broadcasted sports video. In Proceedings of the ACM Workshop on Multimedia. Los Angeles, CA. ACM Press, 205--208.
8. Babaguchi, N. and Nitto, N. 2003. Intermodal collaboration: A strategy for semantic content analysis for broadcasted sports video. In Proceedings of the International Conference on Image Processing. 13--16.
9. Babaguchi, N., Ohara, K., and Ogura, T. 2003. Effect of personalization on retrieval and summarization of sports video. In Proceedings of the Joint Conference of the 4th International Conference on Information, Communications and Signal Processing and the 4th Pacific Rim Conference on Multimedia.
10. Blaha, M. and Premerlani, W. 1998. Object-Oriented Modeling and Design for Database Applications. Prentice Hall, Upper Saddle River, NJ.
11. Boag, S., Chamberlin, D., Fernandez, M. P., Florescu, D., Robie, J., and Simeon, J. 2004. XQuery 1.0: An XML query language. W3C Working Draft. W3C.
12. Brundage, M. 2004. XQuery: The XML Query Language. Addison Wesley.
13. Chairsorn, L. and Chua, T.-S. 2002. The segmentation and classification of story boundaries in news video. In Proceedings of the 6th IFIP Working Conference on Visual Database Systems. Brisbane, Australia. Kluwer, 94--109.
14. Chang, S.-F., Sikora, T., and Purl, A. 2001. Overview of the MPEG-7 standard. IEEE Trans. Circ. Syst. Video Tech. 11, 6, 688--695.
15. Dimitrova, N., Rui, Y., and Sethi, I. 2001. Media content management. In Design Management of Multimedia Information Systems: Opportunities and Challenges. S. M. Rahman, Ed. Idea Group Publishing.
16. Dobbie, G., Xiaoying, W., Ling, T. W., and Lee, M. L. 2000. ORA-SS: An object-relationship-attribute model for semistructured data. Tech. rep., Department of Computer Science, National University of Singapore.
17. Ekin, A. and Tekalp, A. M. 2003. Generic play-break event detection for summarization and hierarchical sports video analysis. In Proceedings of the International Conference on Multimedia and Expo. 6--9.
18. Ekin, A. and Tekalp, M. 2003. Automatic soccer video analysis and summarization. IEEE Trans. Image Process, 12, 7, 796--807.
19. Han, M., Hua, W., Chen, T., and Gong, Y. 2003. Feature design in soccer video indexing. In Proceedings of the Joint Conference of the 4th International Conference on Information, Communications and Signal Processing and the 4th Pacific Rim Conference on Multimedia. 950--954.
20. Hanjalic, A. 2002. Shot-boundary detection: unraveled and resolved? IEEE Trans. Circ. Syst. Video Tech. 12, 2, 90--105.
21. Heng, W. J. and Ngan, K. N. 2002. Shot boundary refinement for long transition in digital video sequence. IEEE Trans. Multimedia 4, 434--445.
22. Kosch, H. 2004. Distributed Multimedia Database Technologies Supported by MPEG-7 and MPEG-21. CRC Press, Boca Raton, FL.
23. Lu, G. J. 1999. Multimedia Database Management Systems. Artech House, Boston, MA.
24. Manjunath, B. S., Salembier, P., and Sikora, T. 2002. Introduction to MPEG-7. John Wiley & Sons, New York, NY.
25. Meng, H. M., Tang, X., Hui, P. Y., Gao, X., and Li, Y. C. 2001. Speech retrieval with video parsing for television news programs. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing. vo1. 1403, 1401--1404.
26. Nepal, S., Srinivasan, U., and Reynolds, G. 2001. Automatic detection of ‘goal’ segments in basketball videos. In Proceedings of the ACM International Conference on Multimedia. Ottawa; Canada, 261--269.
27. Ngai, C. H., Chan, P. W., Yau, E., and Lyu, M. R. 2002. XVIP: an XML-based video information processing system. In Proceedings of the 26th Annual International Computer Software and Applications Conference (COMPSAC'02). 173--178.
28. Oh, J. and Hua, K. A. 2000. Efficient and cost-effective techniques for browsing and indexing large video databases. In Proceedings of the ACM SIGMOD International Conference on Management of Data. Dallas, TX, 415--426.
29. Oomoto, E. and Tanaka, K. 1997. Video database systems—recent trends in research and development activities. In The Handbook of Multimedia Information Management. William I. Grosky, Ed. Prentice Hall, Upper Saddle River, NJ, 405--448.
30. Pereira, F. 2001. MPEG-7 Requirements Document V.14. International Organisation For Standardisation, Coding of Moving Pictures and Audio ISOIIEC JTC 1/SC 29/WG I IIN4035, Singapore.
31. Ponceleon, D., Srinivasan, S., Amir, A., Petkovic, D., and Diklic, D. 1998. Key to effective video retrieval: Effective cataloging and browsing. In Proceedings of the IEEE International Workshop on Content-Based Image and Video Databases. Bombay, India. IEEE Computer Society, 99--107.
32. Rui, Y., Gupta, A., and Acero, A. 2000. Automatically extracting highlights for TV Baseball programs. In Proceedings of the ACM International Conference on Multimedia. Marina del Rey, CA, ACM, 105--115.
33. Sato, T., Kanade, T., Hughes, E. K., and Smith, M. A. 1998. Video OCR for digital news archive. In Proceedings of the IEEE International Workshop on Content-Based Access of Image and Video Database. 52--60.
34. Tjondronegoro, D., Chen, Y.-P. P., and Pham, B. 2004a. Integrating highlights to play-break sequences for more complete sport video summarization. IEEE Multimedia. 22--37.
35. Tjondronegoro, D., Chen, Y.-P. P., and Pham, B. 2004b. The Power of play-break for automatic detection and browsing of self-consumable sport video highlights. In Proceedings of the 6th International ACM Multimedia Information Retrieval Workshop. ACM, New York, NY.
36. Tjondronegoro, D., Chen, Y.-P. P., and Pham, B. 2003. Sports video summarization using highlights and play-breaks. In Proceedings of the 5th ACM SIGMM International Workshop on Workshop on Multimedia Information.
37. Tseng, B. L., Lin, C.-Y., and Smith, J. R. 2004. Using MPEG-7 and MPEG-21 for personalizing video. IEEE Multimedia 11, 1, 42--52.
38. Wu, C., Ma, Y.-F., Zhang, H.-J., and Zhong, Y.-Z. 2002. Events recognition by semantic inference for sports video. In Proceedings of the IEEE International Conference on Multimedia and Exp., 805--808.
39. Xie, L., Chang, S.-F., Divakaran, A., and Sun, H. 2002. Structure analysis of soccer video with hidden Markov models. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing. 4096--4099.
40. Xu, P., Xie, L., and Chang, S.-F. 1998. Algorithms and system for segmentation and structure analysis in soccer video. In Proceedings of the IEEE International Conference on Multimedia and Exp., Tokyo, Japan, IEEE.
41. Yu, X. 2003. Trajectory-based ball detection and tracking with applications to semantic analysis of broadcast soccer video. In Proceedings of the ACM Multimedia Conference. Berkeley, CA, ACM, 11--20.
42. Zeinik-Manor, L. and Irani, M. 2001. Event-based analysis of video. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 123--130.
43. Zhang, H. J. Ed. 1999. Content-Based Video Browsing and Retrieval. CRC Press, Boca Raton, FL.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

        • Video segmentation

      • Computer vision tasks

        • Video summarization

      • Image and video acquisition

• Information systems

  • Information retrieval

    • Document representation

    • Information retrieval query processing

    • Search engine architectures and scalability

      • Search engine indexing

• Theory of computation

  • Semantics and reasoning

    • Program reasoning

      • Abstraction