Evaluating a content based image retrieval system

ABSTRACT

Content Based Image Retrieval (CBIR) presents special challenges in terms of how image data is indexed, accessed, and how end systems are evaluated. This paper discusses the design of a CBIR system that uses global colour as the primary indexing key, and a user centered evaluation of the systems visual search tools. The results indicate that users are able to make use of a range of visual search tools, and that different tools are used at different points in the search process. The results also show that the provision of a structured navigation and browsing tool can support image retrieval, particularly in situations in which the user does not have a target image in mind. The results are discussed in terms of their implications for the design of visual search tools, and their implications for the use of user-centered evaluation for CBIR systems.

References

1. 1.Arnheim, R. Art and Visual Perception: A Psychology of the Creative Eye. The New Version (The expanded and revised edition of the original publication of 1954). University of California Press, Berkeley and Los Angeles, California, 1974.]]
2. 2.Bach, J.R., Fuller, C., Gupta, A., et el. The Virage Image Search Engine: An Open Framework for Image Management. In: Storage and Retrieval for Still Image and Video Databases IV. 1-2 February 1996. SPIE Proceedings, Vol. 2670. pp. 76-87.]]
3. 3.Batley, S. Visual information retrieval browsing strategies in pictoral data. PhD thesis, University of Aberdeen, Aberdeen, UK. 1988.]]
4. 4.Berlin, B., Kay, P. Basic Colour Terms: Their Universality and Evolution. The Paperback Edition (The first publication: 1969). University of California Press, California, 1991.]]
5. 5.Cleverdon, C. W., Mills, J., Keen, E. M. Factors Determining the Performance of Indexing Systems. Aslib Cranfield Research Project, College of Aeronautics, Cranfield, 1966.]]
6. 6.De Marsicoi, M., Cinque, L., Levialdi, S. Indexing Pictorial Documents by Their Content: A Survey of Current Techniques. Image and Vision Computing. 15(2), 1997. pp. 119-141.]]
7. 7.Draper, S. Evaluation in HCI and IR. In: Dunlop, M. (ed.) Proceedings of the Second Mira Workshop. Glasgow University Research Report, 1996. Available at http://www.dcs.gla.ac.uk/mira/workshops/padua_procs]]
8. 8.Dunlop, M. The Problems with Precision and Recall. In: Dunlop, M. (ed.) Proceedings of the Second Mira Workshop. Glasgow University Research Report, 1996.]]
9. 9.Dunlop, M. Reflections on Mira: Interactive evaluation in information retrieval. Journal of the American Society for Information Science, 51 (14) 126-1274, 2000.]]
10. 10.Furnas, G. Effective View Navigation. Conference Proceedings on Human Factors in Computer Systems, 307- 374, 1997.]]
11. 11.Harper, D., Hendry, D. Evaluation light. In: Dunlop, M. (ed.) Proceedings of the Second Mira Workshop. Glasgow University Research Report, 1996.]]
12. 12.Holt, B., Weiss, K., Niblack, W., et al. The QBIC Project in the Department of Art and Art History at UC Davis. In: Proceedings of the ASIS Annual Meeting. Vol. 34, 1997. pp. 189-195.]]
13. 13.Jose, J., Furner, J., Harper, D. Spatial Querying For Images Retrieval: A User-Oriented Evaluation. In: ACM SIGIR'98, August 24-28, 1998. pp. 232-240.]]
14. 14.Large, A., & Behshti, J. OPACs: a research review. Library & Information Science Research 19(2), 111-133 (1997).]]
15. 15.Lai, T. S., Tait, J.I., McDonald, S. (1999) Image Browsing and Navigation Using Hierarchical Classification The Challenge of Image Retrieval 1999 conference, CIR'99]]
16. 16.Lai, T.-S. CHROMA: A Photographic Image Retrieval System. Unpublished Ph.D. Thesis, University of Sunderland, UK, January 2000. Available at http://osiris.sund.ac.uk/~cs0sla/thesis/]]
17. 17.Lai , T-S, Tait, J and McDonald, S. A user centred evaluation of visual search methods for CBIR. Proceedings of CIR2000, Third UK conference on Image Retrieval. John P Eakins and Peter Enser (eds). University of Brighton. May 2000.]]
18. 18.Markkula, M, and Sormunen, E. Searching for photos - Journalist's practices in pictorial IR. Proceedings of CIR1998, First UK conference on Image Retrieval. Feb 1998.]]
19. 19.McDonald, S., and Stevenson, R.J. Disorientation in hypertext: the effects of three text structures on navigation performance. Applied Ergonomics, Vol 27(1), 1996. pp 61-68.]]
20. 20.Nordlie, R. "User revealment" - a comparison of initial queries and ensuing question development in online searching and in human reference interactions. . In: ACM SIGIR'99, August, 1999. pp. 11-18.]]
21. 21.Rodden, K, Basalaj, W, Sinclair, D, Wood, K. Does organisation by similarity assist image browsing? Proceedings of the SIG-CHI conference on Human Factors in Computing Systems. March 31 - April 5 2001, Seattle, WA USA.]]
22. 22.Rui, Y., Huang, T., Chang, S.-F. Image Retrieval: Current Techniques, Promising Directions, and Open Issues. Journal of Visual Communication and Image Representation. 10(1), 1999. pp. 39-32.]]
23. 23.Rui, Y., Huang, T.S., Mehrotra, S. Relevance Feedback Techniques in Interactive Content-Based Image Retrieval. In: Proceedings of Storage and Retrieval for Image and Video Databases VI. SPIE Proceedings, Vol. 3312, 1998. pp. 25-36.]]
24. 24.Santini, S and Jain, R. Integrated browsing and query for image databases. IEEE Multimedia Vol 7 (3) (2000) 26-39.]]
25. 25.Spark Jones, K. Automatic language and information processing: rethinking evaluation. Natural Language Engineering 7 (1), (in press).]]
26. 26.Spark Jones, K, and Galliers, J. Evaluating natural language processing systems. Lecture Notes in Artificial Intelligence. 1996. Springer-Verlag]]
27. 27.Tait, J.I., "Exploring Information Spaces: Information Retrieval Strategies combining querying, relevance feedback and browsing". Abstracts of Invited Papers of the Ninth International Colloquium on Numerical Analysis and Computer Science with Applications, Plovdiv, Bulgaria, August 12-17, 2000]]

Index Terms

• Information systems

  • Information retrieval

    • Information retrieval query processing