Patent image retrieval: a survey

ABSTRACT

Drawings are an important component of patents, and many search tasks in the intellectual property domain rely on the comparison of patent drawings. In this paper, we begin with a review of algorithms developed for the automated retrieval of similar images in the patent domain. There is however a larger body of research dedicated to analysis and retrieval of images found in technical documents that is also applicable to the images found in patents. We present an overview of this research for technical drawings, diagrams, charts, plots and chemical structures. Finally, we discuss the evaluation of image retrieval for patents, including short descriptions of the new patent image evaluation tasks in the CLEF-IP and TREC-CHEM evaluation campaigns in 2011.

References

1. S. Adams. Electronic non-text material in patent applications - some questions for patent offices, applicants and searchers. World Patent Information, 27(2):99--103, 2005.
2. A. Antonacopoulos and D. Kennedy. Information extraction from complex circular charts. Document Analysis and Recognition, International Conference on, 0:784--787, 2001.
3. D. V. Bakergem. Image collections in the design studio. The electronic design studio, pages 261--271, 1990.
4. A. Barta and I. Vajk. Document image analysis by probabilistic network and circuit diagram extraction. Informatica, An International Journal of Computing and Informatics, 29:291--301, 2005.
5. D. Blostein, E. Lank, and R. Zanibbi. Treatment of diagrams in document image analysis. In Proceedings of the First International Conference on Theory and Application of Diagrams, pages 330--344, London, UK, 2000. Springer-Verlag.
6. W. Browuer, S. Kataria, S. Das, P. Mitra, and C. L. Giles. Segregating and extracting overlapping data points in two-dimensional plots. In JCDL, pages 276--279, 2008.
7. W. H. Clayton M. Enhancing the sketchbook. Proceedings of the Association for Computer Aided Design in Architecture (ACADIA 91), 1991.
8. M. Das, R. Manmatha, and E. M. Riseman. Indexing flower patent images using domain knowledge. IEEE Intelligent Systems, 14(5):24--33, 1999.
9. R. Datta, D. Joshi, J. Li, and J. Z. Wang. Image retrieval: Ideas, influences, and trends of the new age. ACM Computing Surveys, 40(2):1--60, 2008.
10. G. Feng, C. Viard-Gaudin, and Z. Sun. On-line hand-drawn electric circuit diagram recognition using 2d dynamic programming. Pattern Recognition, 42(12):3215--3223, 2009.
11. I. V. Filippov and M. C. Nicklaus. Optical structure recognition software to recover chemical information: Osra, an open source solution. Journal of Chemical Information and Modeling, 49(3):740--743, 2009.
12. L. A. Fletcher and R. Kasturi. A robust algorithm for text string separation from mixed text/graphics images. IEEE Trans. Pattern Anal. Mach. Intell., 10(6):910--918, 1988.
13. W. Huang, C. L. Tan, and W. K. Leow. Model-based chart image recognition. In GREC, pages 87--99, 2003.
14. B. Huet, G. Guarascio, N. J. Kern, and B. Mérialdo. Relational skeletons for retrieval in patent drawings. In ICIP (2), pages 737--740, 2001.
15. S. Kataria, W. Browuer, P. Mitra, and C. L. Giles. Automatic extraction of data points and text blocks from 2-dimensional plots in digital documents. In AAAI, pages 1169--1174, 2008.
16. J. List. How drawings could enhance retrieval in mechanical and device patent searching. World Patent Information, 29(3):210--218, 2007.
17. X. Lu, S. Kataria, W. J. Brouwer, J. Z. Wang, P. Mitra, and C. L. Giles. Automated analysis of images in documents for intelligent document search. Int. J. Doc. Anal. Recognit.(IJDAR), 12(2):65--81, 2009.
18. X. Lu, P. Mitra, J. Z. Wang, and C. L. Giles. Automatic categorization of figures in scientific documents. In JCDL, pages 129--138. ACM, 2006.
19. M. Lupu, J. Huang, J. Zhu, and J. Tait. TREC Chemical Information Retrieval - An Evaluation Effort for Chemical IR Systems. World Patent Information, 2011.
20. L. Najman, O. Gibot, and S. Berche. Indexing technical drawings using title block structure recognition. In Proc. International Conference on Document Analysis and Recognition (ICDAR), pages 587--591, 2001.
21. J. R. Osborn. Theory pictures as trails: diagrams and the navigation of theoretical narratives. Cognitive Science Online, 3.2:14--44, 2005.
22. P. Sidiropoulos, S. Vrochidis, and I. Kompatsiaris. Content-based binary image retrieval using the adaptive hierarchical density histogram. Pattern Recognition, 44(4):739--750, 2011.
23. A. W. M. Smeulders, M. Worring, S. Santini, A. Gupta, and R. Jain. Content-based image retrieval at the end of the early years. IEEE Trans. Pattern Anal. Mach. Intell., 22(12):1349--1380, 2000.
24. A. Tiwari and V. Bansal. Patseek: Content based image retrieval system for patent database. In ICEB, pages 1167--1171, 2004.
25. K. Tombre. Analysis of engineering drawings: State of the art and challenges. In GREC, pages 257--264, 1998.
26. S. Vrochidis, S. Papadopoulos, A. Moumtzidou, P. Sidiropoulos, E. Pianta, and I. Kompatsiaris. Towards content-based patent image retrieval: A framework perspective. World Patent Information, 32(2):94--106, 2010.
27. N. Weber and A. Henrich. Retrieval of technical drawings in dxf format - concepts and problems. In LWA, pages 213--220, 2007.
28. Z. Zhiyuan, Z. Juan, and X. Bin. An outward-appearance patent-image retrieval approach based on the contour-description matrix. In FCST, pages 86--89, Washington, DC, USA, 2007. IEEE Computer Society.
29. Y. Zhou and C. L. Tan. Chart analysis and recognition in document images. In ICDAR, page 1055, Washington, DC, USA, 2001. IEEE Computer Society.
30. Y. P. Zhou and C. L. Tan. Hough technique for bar charts detection and recognition in document images. In International Conference on Image Processing, ICIP 2000, pages 494--497, 2000.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision tasks

• Information systems

  • Information retrieval

    • Document representation