The participation payoff: challenges and opportunities for multimedia access in networked communities

ABSTRACT

Increasingly, multimedia collections are associated with networked communities consisting of interconnected groups of users who create, annotate, browse, search, share, view, critique and remix collection content. Information arises within networked communities via connections among users and in the course of interactions between users and content. Community-derived information can be exploited to improve user access to multimedia. This paper provides a survey of techniques that make use of a combination of three information sources: community-contributed information (e.g., tags and ratings), network structure and techniques for multimedia content analysis. This triple synergy offers a wide range of opportunities for improving access to multimedia in networked communities. We focus our survey on three areas important for multimedia access: annotation, distribution and retrieval. The picture that emerges is promising: information derived from the social community is remarkably effective in improving access to multimedia content, and participation in networked communities has a high payoff.

References

1. Boll, S. 2007. MultiTube - Where Web 2.0 and Multimedia Could Meet. IEEE MultiMedia 14, 1 (Jan. 2007), 9--13.
2. Trant, J. 2009. Studying social tagging and folksonomy: a review and framework. Journal of Digital Information 10,1.
3. de Jong, F. M. G., Westerveld, T., and de Vries, A.P. 2007. Multimedia search without visual analysis: the value of linguistic and contextual information. IEEE Transactions on Circuits and Systems for Video Technology 17, 3, 365--371.
4. Smeulders, A. W., Worring, M., Santini, S., Gupta, A., and Jain, R. 2000. Content-based image retrieval at the end of the early years. IEEE Trans. Pattern Anal. Mach. Intell. 22, 12 (Dec. 2000), 1349--1380.
5. Tsagkias, M., Larson, M., and de Rijke, M. 2008. Term clouds as surrogates for user generated speech. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Singapore, Singapore, July 20 - 24, 2008). SIGIR '08. ACM, New York, NY, 773--774.
6. Zweig, G., Makhoul, J., Stolke, A., August 2006. Introduction to the special section on rich transcription. IEEE Trans. On Audio, Speech, and Lang. Proc. 14, 5, 1490--1491.
7. Larson, M., Newman, E., Jones, G. 2009. Overview of VideoCLEF 2008: Automatic generation of topic-based feeds for dual language audio-visual content. In Peters, C. et al., (eds.) Evaluating Systems for Multilingual and Multimodal Information Access, CLEF 2008 Revised Selected Papers, LNCS 5706, 906--917.
8. Hauptmann, G., Christel, M. G., and Yan, R. 2008. Video retrieval based on semantic concepts. Proceedings of the IEEE 96, 4, (April 2008), 602--622.
9. Snoek, C. G. M. and Worring, M. 2009. Concept-based video retrieval. Foundations and Trends in Information Retrieval 2, 4, (2009), 215--322.
10. Mylonas, P., Spyrou, E., Avrithis, Y., and Kollias, S. 2009. Using visual context and region semantics for high-level concept detection. IEEE Trans. on Multimedia 1, 2 (Feb. 2009) 229--243.
11. Lin, Y.-R., Sundaram, H., De Choudhury, M., and Kelliher, A. 2009. Temporal patterns in social media streams: theme discovery and evolution using joint analysis of content and context. In Proceedings of the Int. Conf. on Multimedia and Expo (New York, NY, Jun./Jul. 2009) ICME 2009.
12. R. Glasberg, S. Schmiedeke, P. Kelm, T. Sikora. 2008. An automatic system for real-time video-genres detection using high-level descriptions and a set of classifiers. Proceedings of the 12th IEEE Int. Symp. on Consumer Electronics (Algarve, Portugal, April 2008) ISCE 2008.
13. Li, J. and Wang, J.Z. 2008. Real-time computerized annotation of pictures. IEEE Trans. on Pattern Analysis and Machine Intelligence 30, 6 (June 2008), 985--1002.
14. Cai, R., Lu, L., Hanjalic, A., Zhang, H.-J., and Cai, L.-H. 2006. A flexible framework for key audio effects detection and auditory context inference. IEEE Trans. on Audio, Speech, and Language Processing 14, 3, 1026--1039.
15. Wactlar, H. D., Hauptmann, A. G., Christel, M. G., Houghton, R. A., and Olligschlaeger, A. M. 2000. Complementary video and audio analysis for broadcast news archives. Comm. of the ACM 43, 2 (Feb. 2000), 42--47.
16. Hsu, W. H., Kennedy, L. S., and Chang, S. 2006. Video search reranking via information bottleneck principle. In Proceedings of the 14th Annual ACM International Conf. on Multimedia (Santa Barbara, CA, USA, October 23 - 27, 2006). MULTIMEDIA '06. ACM, New York, NY, 35--44.
17. Hanjalic, A. April 2006. Extracting moods from pictures and sounds: towards truly personalized TV. Signal Processing Magazine, IEEE 23, 2, 90--100.
18. Cavet, R., Volmer, S., Leopold, E., Kindermann, J., and Paass, G. 2004. Revealing the connoted visual code: a new approach to video classification. Computers & Graphics 28, 3, 361--369.
19. Sharma, A., Hua, G., Liu, Z. and Zhang, Z. 2008. Meta-tag propagation by co-training an ensemble classifier for improving image search relevance, Proceedings of the IEEE CVPR Workshop on Internet Vision (Anchorage AK, USA, June 2008). CVPR 2008.
20. Siersdorfer, S., San Pedro, J., and Sanderson, M. 2009. Automatic video tagging using content redundancy. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval (Boston, MA, USA, July 19 - 23, 2009). SIGIR '09. ACM, New York, NY, 395--402.
21. Li, X., Snoek, C. G. M. and Worring, M. 2009. Learning social tag relevance by neighbor voting, IEEE Trans. On Multimedia 11, 7 (Nov. 2009), 1310--1322.
22. Cattuto, C., Benz, D., Hotho, A., and Stumme, G. 2008. Semantic analysis of tag similarity measures in collaborative tagging systems. In Proc. of the 3rd Workshop on Ontology Learning and Population. (Patras, Greece, July 2008) OLP3.
23. Kliegr, T., Chandramouli, K., Nemrava, J., Svatek, V. and Izquierdo, E. 2008. Combining image captions and visual analysis for image concept classification. In Proceedings of the 9th Int. Workshop on Multimedia Data Mining. (Las Vegas, Aug 2008). MDM/KDD 2008.
24. Clements, M., de Vries, A. P., and Reinders, M. J. 2008. Detecting synonyms in social tagging systems to improve content retrieval. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Singapore, Singapore, July 20 - 24, 2008). SIGIR '08. ACM, New York, NY, 739--740.
25. Garg, N. and Weber, I. 2008. Personalized, interactive tag recommendation for Flickr. In Proceedings of the 2008 ACM Conference on Recommender Systems (Lausanne, Switzerland, October 23 - 25, 2008). RecSys '08. ACM, New York, NY, 67--74.
26. Sigurbjörnsson, B. and van Zwol, R. 2008. Flickr tag recommendation based on collective knowledge. In Proceeding of the 17th International Conference on World Wide Web (Beijing, China, April 21 - 25, 2008). WWW '08. ACM, New York, NY, 327--336.
27. Marlow, C., Naaman, M., Boyd, D., and Davis, M. 2006. Position Paper, Tagging, Taxonomy, Flickr, Article, To Read. In Proc. of the Collab. Web Tagging Workshop at WWW'06.
28. Von Ahn, L. and Dabbish, L. 2004. Labeling images with a computer game. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (New York, NY, USA). CHI '04. ACM Press, 319--326.
29. Von Ahn, L., Liu, R., and Blum, M. 2006. Peekaboom: a game for locating objects in images. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (New York, NY, USA). CHI '06. ACM, 55--64.
30. Von Ahn, Ginosar, S., Kedia, M., Liu, R. and Blum, M. 2006. Improving accessibility of the web with a computer game. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems (New York, NY, USA, 2006). CHI '06. ACM, 79--82.
31. Russell, B. C., Torralba, A., Murphy, K. P., and Freeman, W. T. 2008. LabelMe: A database and web-based tool for image annotation. Int. J. Comput. Vision 77, 1-3, 157--173.
32. Ogata, J. and Goto, M. 2009. PodCastle: Collaborative training of acoustic models on the basis of wisdom of crowds for podcast transcription. In Proceedings of Interspeech 2009 - 10th Annual Conference of the International Speech Communication Association, 1491--1494.
33. Petridis, S. and Pantic M. 2008. Audiovisual laughter detection based on temporal features. In Proceedings of the Int. Conf. on Multimodal Interfaces, (Chania, Crete, Greece, October 2008), 37--44.
34. Soleymani, M., Chanel, G., Kierkels J., and Pun, T. 2008. Affective characterization of movie scenes based on multimedia content analysis and user's physiological emotional responses. In Proceedings of the Int. Symposium on Multimedia (Berkeley, CA, Dec. 15-17, 2008).
35. Yazdani, A., Lee, J.-S., and Ebrahimi, T. 2009. Implicit emotional tagging of multimedia using EEG signals and brain computer interface. In Proceedings of the ACM Multimedia Workshop on Social Media. (Beijing, China).
36. Koelstra, S., Muehl, C., and Patras, I. 2009. EEG analysis for implicit tagging of video data. In Proc. of the Workshop on Affective Brain-Computer Interfaces, Proc. ACII, 27--32.
37. Gallagher, A. and Chen, T. 2009. Understanding Images of Groups of People, IEEE Conference on Computer Vision and Pattern Recognition 2009.
38. Melenhorst, M., Grootveld, M., van Setten, M., and Veenstra, M. 2008. Tag-based information retrieval of video content. In Proceeding of the 1st international Conference on Designing interactive User Experiences For TV and Video (Silicon Valley, CA, Oct. 22-24, 2008). UXTV '08, vol. 291.
39. Nguyen, T., Kolazhi, K. Kamath, R., Cheung, S. S. and Tran, D. A. 2008. Efficient multimedia distribution in source constraint networks. IEEE Trans. on Multimedia 10, 3 (Apr. 2008), 523--537.
40. Androutsellis-Theotokis, S. and Spinellis, D. 2004. A survey of peer-to-peer content distribution technologies. ACM Computing Surveys 36, 4 (2004) 335--371.
41. Ramzan, N., Zgaljic, T., and Izquierdo, E. 2009. An efficient optimisation scheme for scalable surveillance centric video communications. Signal Processing: Image Communication 24, 6 (July 2009), 510--523.
42. Ramzan N. and Izquierdo, E. 2009. Scalable video coding as basis for networked media internet. In Proceedings of the European Commission NEM Summit Towards Future Media Internet. (Saint-Malo, France, Sept. 2009).
43. Ramzan, N. and Izquierdo, E. 2009. An optimised error protection scheme for wavelet based scalable video. In Proc. of Latin American Conference on Networked Electronic Media (Bogota, Columbia, Aug. 2009). LACNEM.
44. Baccichet, P., Schierl, T., Wiegand T., and Girod, B. 2007. Low-delay peer-to-peer streaming using scalable video coding. In Proceedings of the 16th International Packet Video Workshop (Lausanne, Switzerland, Nov. 2007).
45. Li, H. and Chen, G. 2007. Data persistence in structured P2P networks with redundancy schemes. In Proceedings of the Sixth International Conference of Grid and Cooperative Computing (Los Alamitos, CA, 2007). GCC.
46. Palomar, E., Estevez-Tapiador, J. M., Hernandez-Castro, J. C., and Ribagorda, A. 2006. A protocol for secure content distribution in pure P2P networks. In Proceedings of the 17th International Conference on Database and Expert Systems Applications. DEXA'06. 712--716.
47. De Simone, F., Naccari, M., Tagliasacchi, M., Dufaux, F., Tubaro, S., and Ebrahimi, T. 2009. Subjective assessment of H.264/AVC video sequences transmitted over a noisy channel. In Proceedings of the First International Workshop on Quality of Multimedia Experience (2009). QoMEX.
48. Mathieu, B. 2008. A network and QoE aware P2P video streaming solution. In Proc. of the 8th Int. Conference on New Technologies in Distributed Systems (Lyon, France, June 23 - 27, 2008). NOTERE '08. ACM, NY, NY, 1--9.
49. Smith, M., Barash, V., Getoor, L., and Lauw, H. W. 2008. Leveraging social context for searching social media. In Proceedings of the 2008 ACM Workshop on Search in Social Media (Napa Valley, California, USA, October 26-30, 2008). SSM '08. ACM, New York, NY, 91--94.
50. Shirky, C., 2008. Here Comes Everybody: The Power of Organizing Without Organizations. The Penguin Press HC.
51. Evans, B. M. and Chi, E. H. 2008. Towards a model of understanding social search. In Proceedings of the ACM 2008 Conference on Computer Supported Cooperative Work (San Diego, CA, USA, November 08 - 12, 2008). CSCW '08. ACM, New York, NY, 485--494.
52. Agichtein, E., Hearst, M. A., and Soboro, I. 2009. The Search and Social Media Workshop at SIGIR 2009. SIGIR Forum 43, 2 (Dec. 2009), 53--56.
53. Marchand-Maillet, S., Vries, A. P., and Naaman, M. 2009. Workshop on Information Retrieval over Social Networks. In Proceedings of the 31th European Conference on IR Research on Advances in Information Retrieval (Toulouse, France, April 06 - 09, 2009) 803.
54. Boll, S., Hoi, S. C., Luo, J., Jin, R., King, I., and Xu, D. 2009. First ACM SIGMM International Workshop on Social Media. WSM'09. In Proceedings of the 17th ACM International Conference on Multimedia (Beijing, China, October 19 - 24, 2009), MM '09. ACM, New York, NY, 1161--1162.
55. Haythornthwaite, C. 1996. Social network analysis: An approach and technique for the study of information exchange. Library & Information Science Research. 18, 4 (Autumn 1996), 323--342.
56. Newman, M. E. J. 2003. The structure and function of complex networks, SIAM Review 45, 167--256.
57. Leskovec, J., Lang, K. J., Dasgupta, A., and Mahoney, M. W. 2008. Statistical properties of community structure in large social and information networks. In Proceedings of the 17th International Conference on World Wide Web (Beijing, China, April 21 - 25, 2008). WWW '08. ACM, New York, NY, 695--704.
58. Li, C. and Bernoff, J. 2008. Groundswell: Winning in a World Transformed by Social Technologies. 1st Edition. Harvard Business School Press.
59. Agichtein, E., Brill, E., and Dumais, S. 2006. Improving web search ranking by incorporating user behavior information. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Seattle, Washington, USA, August 06 - 11, 2006). SIGIR '06. ACM, New York, NY, 19--26.
60. Xue, G., Zeng, H., Chen, Z., Yu, Y., Ma, W., Xi, W., and Fan, W. 2004. Optimizing web search using web click-through data. In Proceedings of the Thirteenth ACM International Conference on Information and Knowledge Management (Washington, D.C., USA, November 08 - 13, 2004). CIKM '04. ACM, New York, NY, 118--126.
61. Craswell, N. and Szummer, M. 2007. Random walks on the click graph. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Amsterdam, Netherlands, July 23 - 27, 2007). SIGIR '07. ACM, New York, NY, 239--246.
62. Müller, H., Müller, W., Squire, D., Marchand-Maillet, S., and Pun, T. 2000. Long-term learning from user behavior in content-based image retrieval. Tech. report. University of Geneva 2000.
63. Marchand-Maillet, S. and Bruno, E. 2003. Exploiting user interaction for semantic content-based image retrieval. Tech. report. Computer Vision and Multimedia Laboratory, Computing Centre, University of Geneva.
64. Huurnink, B., Hollink, L., van den Heuvel, W., and de Rijke. M. 2010. The Search Behavior of Media Professionals at an Audiovisual Archive: A Transaction Log Analysis. Journal of the American Society for Information Science and Technology. 2010.
65. Hopfgartner, F., Vallet, D., Halvey, M., and Jose, J. 2008. Search trails using user feedback to improve video search. In Proceedings of the 16th ACM International Conference on Multimedia (Vancouver, British Columbia, Canada, October 26 - 31, 2008). MM '08. ACM, New York, NY, 339--348.
66. Davis, S. J., Ritz, C. H., and Burnett, I. S. 2009. Using social networking and collections to enable video semantics acquisition. IEEE MultiMedia 16, 4, 52--61.
67. Yamamoto, D., Masuda, T., Ohira, S., and Nagao, K. 2008. Video scene annotation based on web social activities. IEEE MultiMedia 15, 3 (Jul. 2008), 22--32.
68. Shardanand U. and Maes P. 1995. Social information filtering: algorithms for automating 'word of mouth'. In Proceedings of SIGCHI, (Denver, Colorado, 1995), 210--217.
69. Kazai, G. and Milic-Frayling, N. 2008. Trust, authority and popularity in social information retrieval. In Proceedings of the 17th ACM Conference on Information and Knowledge Management (Napa Valley, California, USA, October 26 - 30, 2008). CIKM '08. ACM, New York, NY, 1503--1504.
70. Golbeck, J. and Hendler, J. 2006. Inferring binary trust relationships in web-based social networks. ACM Trans. Internet Technol. 6, 4 (Nov. 2006), 497--529.
71. Daylamani Zad, D. and Agius, H. 2009. Multimedia Metadata 2.0: Challenges of collaborative content modeling. Advances in Semantic Media Adaptation and Personalization 2, M.C. Angelides, M. Wallace and P. Mylonas, Eds. CRC Press, Boca Raton, FL, USA, 1--20.
72. Resnick P., Iacovou N., Suchak M., Bergstorm P., and Riedl J. Grouplens: An open architecture for collaborative filtering of netnews. In the Proceedings of ACM 1994 Conference on Computer Supported Cooperative Work, (Chapel Hill, North, Carolina, 1994) ACM, 175--186.
73. Adomavicius, G. and Tuzhilin, A. 2005. Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Transactions on Knowledge and Data Engineering (Jun. 2005), 734--749.
74. Linden, G., Smith, B., and York, J. 2003. Amazon.com recommendations: item-to-item collaborative filtering. IEEE Internet Computing 7, 1 (Jan./Feb. 2003), 76--80.
75. Yoshii, K., Goto, M., Komatani, K., Ogata, T., and Okuno, H. G. 2008. An efficient hybrid music recommender system using an incrementally trainable probabilistic generative model. Audio, Speech, and Language Processing, IEEE Transactions on. 16, 2, 435--447.
76. Wang J., Clements M., Yang, J., de Vries, A. P. and Reinders, M. J. T. 2010. Personalization of tagging systems. Information Processing and Management 46, 58--70.
77. Marchand-Maillet, S., Szekely, E., and Bruno, E. 2009. Optimizing strategies for the exploration of social-networks and associated data collections. In Proceedings of the International Workshop on Image Analysis for Multimedia Interactive Services. (London, UK). WIAMIS'09.
78. Clements, M., de Vries, A. P., and Reinders, M. J. T. 2008. Optimizing single term queries using a personalized Markov random walk over the social graph. Proceedings of the Workshop on Exploiting Semantic Annotations in Information Retrieval (March 2008). ESAIR.
79. Hardman, L., Obrenović, Ž., Nack, F., Kerhervé, B., Piersol, K. 2008. Canonical processes of semantically annotated media production. Multimedia Systems, 14, 6, (December 2008) 327--340.
80. Kennedy, L., Naaman, M., Ahern, S., Nair, R., and Rattenbury, T. 2007. How Flickr helps us make sense of the world: context and content in community-contributed media collections. In Proceedings of the 15th International Conference on Multimedia (Augsburg, Germany, September 25 - 29, 2007). MULTIMEDIA '07. ACM, New York, NY, 631--640.
81. Serdyukov, P., Murdock, V., and van Zwol, R. 2009. Placing Flickr photos on a map. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval (Boston, MA, USA, July 19 - 23, 2009). SIGIR '09. ACM, New York, NY, 484--491.
82. Kirsch, S. M., Gnasa, M., Won, M., and Cremers, A. B. 2007. From page rank to social rank: Authority-based retrieval in social information spaces. In Social information retrieval systems: Emerging technologies and applications for searching the web effectively. D. H.Gohand S. Foo (eds.), Idea Group Inc., 2007.
83. Bian, J., Liu, Y., Agichtein, E., and Zha, H. 2008. A few bad votes too many?: Towards robust ranking in social media. In Proceedings of the 4th International Workshop on Adversarial Information Retrieval on the Web (Beijing, China, April 22 - 22, 2008). C. Castillo, K. Chellapilla, and D. Fetterly, (eds.) AIRWeb '08, 295. ACM, NY, NY, 53--60.
84. Bian, J., Liu, Y., Zhou, D., Agichtein, E., and Zha, H. 2009. Learning to recognize reliable users and content in social media with coupled mutual reinforcement. In Proceedings of the 18th International Conference on World Wide Web (Madrid, Spain, April 20 - 24, 2009). WWW '09. ACM, New York, NY, 51--60.
85. Lew, M. S., Sebe, N., Djeraba, C., and Jain, R. 2006. Content-based multimedia information retrieval: state of the art and challenges. ACM Trans. Multimedia Comput. Commun. Appl. 2, 1, 1--19.
86. Kankanhalli, M. S., and Rui, Y. March 2008. Application potential of multimedia information retrieval. Proceedings of the IEEE 96, 4, 712--720.
87. Jain, R. 2008. Multimedia information retrieval: watershed events. In Proceeding of the 1st ACM International Conference on Multimedia Information Retrieval (MIR '08) ACM, New York, NY, USA, 229--236.
88. Halvey, M. J. and Keane, M. T. 2007. Exploring social dynamics in online media sharing. In Proceedings of the 16th International Conference on World Wide Web (Banff, Alberta, Canada, May 08 - 12, 2007). WWW '07. ACM, New York, NY, 1273--1274.

Index Terms

• Information systems

  • Information retrieval

    • Document representation

    • Search engine architectures and scalability

      • Distributed retrieval

      • Peer-to-peer retrieval

      • Search engine indexing

  • Information storage systems

    • Storage architectures

      • Distributed storage