Learning to simplify: fully convolutional networks for rough sketch cleanup

Abstract

In this paper, we present a novel technique to simplify sketch drawings based on learning a series of convolution operators. In contrast to existing approaches that require vector images as input, we allow the more general and challenging input of rough raster sketches such as those obtained from scanning pencil sketches. We convert the rough sketch into a simplified version which is then amendable for vectorization. This is all done in a fully automatic way without user intervention. Our model consists of a fully convolutional neural network which, unlike most existing convolutional neural networks, is able to process images of any dimensions and aspect ratio as input, and outputs a simplified sketch which has the same dimensions as the input image. In order to teach our model to simplify, we present a new dataset of pairs of rough and simplified sketch drawings. By leveraging convolution operators in combination with efficient use of our proposed dataset, we are able to train our sketch simplification model. Our approach naturally overcomes the limitations of existing methods, e.g., vector images as input and long computation time; and we show that meaningful simplifications can be obtained for many different test cases. Finally, we validate our results with a user study in which we greatly outperform similar approaches and establish the state of the art in sketch simplification of raster images.

References

1. Bae, S.-H., Balakrishnan, R., and Singh, K. 2008. Ilovesketch: As-natural-as-possible sketching system for creating 3d curve models. In ACM Symposium on User Interface Software and Technology, 151--160.
2. Barla, P., Thollot, J., and Sillion, F. X. 2005. Geometric clustering for line drawing simplification. In ACM SIGGRAPH 2005 Sketches.
3. Bartolo, A., Camilleri, K. P., Fabri, S. G., Borg, J. C., and Farrugia, P. J. 2007. Scribbles to vectors: Preparation of scribble drawings for cad interpretation. In Eurographics Workshop on Sketch-based Interfaces and Modeling, 123--130.
4. Baudel, T. 1994. A mark-based interaction paradigm for freehand drawing. In ACM Symposium on User Interface Software and Technology, 185--192.
5. Chang, H.-H., and Yan, H. 1998. Vectorization of hand-drawn image using piecewise cubic bézier curves fitting. Pattern Recognition 31, 11, 1747--1755.
6. Chen, J., Guennebaud, G., Barla, P., and Granier, X. 2013. Non-oriented mls gradient fields. Computer Graphics Forum 32, 8, 98--109.
7. Cole, F., DeCarlo, D., Finkelstein, A., Kin, K., Morley, K., and Santella, A. 2006. Directing gaze in 3d models with stylized focus. In Eurographics Conference on Rendering Techniques, 377--387.
8. Deussen, O., and Strothotte, T. 2000. Computer-generated pen-and-ink illustration of trees. In Conference on Computer Graphics and Interactive Techniques, 13--18.
9. Dong, C., Loy, C. C., He, K., and Tang, X. 2016. Image super-resolution using deep convolutional networks. PAMI 38, 2, 295--307.
10. Dosovitskiy, A., Springenberg, J. T., and Brox, T. 2015. Learning to generate chairs with convolutional neural networks. In CVPR.
11. Fischer, P., Dosovitskiy, A., Ilg, E., Häusser, P., Hazirbas, C., Golkov, V., van der Smagt, P., Cremers, D., and Brox, T. 2015. Flownet: Learning optical flow with convolutional networks.
12. Fišer, J., Asente, P., and Sýkora, D. 2015. Shipshape: A drawing beautification assistant. In Workshop on Sketch-Based Interfaces and Modeling, 49--57.
13. Freeman, H. 1974. Computer processing of line-drawing images. ACM Comput. Surv. 6, 1, 57--97.
14. Fukushima, K. 1988. Neocognitron: A hierarchical neural network capable of visual pattern recognition. Neural networks 1, 2, 119--130.
15. Grabli, S., Durand, F., and Sillion, F. 2004. Density measure for line-drawing simplification. In Pacific Conference on Computer Graphics and Applications, 2004, 309--318.
16. Grimm, C., and Joshi, P. 2012. Just drawit: A 3d sketching system. In nternational Symposium on Sketch-Based Interfaces and Modeling, 121--130.
17. Hilaire, X., and Tombre, K. 2006. Robust and accurate vectorization of line drawings. PAMI 28, 6, 890--904.
18. Igarashi, T., Matsuoka, S., Kawachiya, S., and Tanaka, H. 1997. Interactive beautification: A technique for rapid geometric design. In ACM Symposium on User Interface Software and Technology, 105--114.
19. Ioffe, S., and Szegedy, C. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML.
20. Janssen, R. D., and Vossepoel, A. M. 1997. Adaptive vectorization of line drawing images. Computer Vision and Image Understanding 65, 1, 38--56.
21. Krizhevsky, A., Sutskever, I., and Hinton, G. E. 2012. Imagenet classification with deep convolutional neural networks. In NIPS.
22. LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE 86, 11, 2278--2324.
23. Lindlbauer, D., Haller, M., Hancock, M. S., Scott, S. D., and Stuerzlinger, W. 2013. Perceptual grouping: selection assistance for digital sketching. In International Conference on Interactive Tabletops and Surfaces, 51--60.
24. Liu, X., Wong, T.-T., and Heng, P.-A. 2015. Closure-aware sketch simplification. ACM Trans. Graph. 34, 6, 168:1--168:10.
25. Long, J., Shelhamer, E., and Darrell, T. 2015. Fully convolutional networks for semantic segmentation. In CVPR.
26. Nair, V., and Hinton, G. E. 2010. Rectified linear units improve restricted boltzmann machines. In ICML, 807--814.
27. Noh, H., Hong, S., and Han, B. 2015. Learning deconvolution network for semantic segmentation. In ICCV.
28. Noris, G., Hornung, A., Sumner, R. W., Simmons, M., and Gross, M. 2013. Topology-driven vectorization of clean line drawings. ACM Trans. Graph. 32, 1, 4:1--4:11.
29. Orbay, G., and Kara, L. 2011. Beautification of design sketches using trainable stroke clustering and curve fitting. IEEE Trans. on Visualization and Computer Graphics 17, 5, 694--708.
30. Preim, B., and Strothotte, T. 1995. Tuning rendered line-drawings. In Winter School in Computer Graphics, 227--237.
31. Pusch, R., Samavati, F., Nasri, A., and Wyvill, B. 2007. Improving the sketch-based interface. The Visual Computer 23, 9-11, 955--962.
32. Rosin, P. L. 1994. Grouping curved lines. In Machine Graphics and Vision 7, 625--644.
33. Rumelhart, D., Hinton, G., and Williams, R. 1986. Learning representations by back-propagating errors. In Nature.
34. Selinger, P. 2003. Potrace: a polygon-based tracing algorithm. Potrace (online), http://potrace. sourceforge. net/potrace.pdf (2009-07-01).
35. Shen, W., Wang, X., Wang, Y., Bai, X., and Zhang, Z. 2015. Deepcontour: A deep convolutional feature learned by positive-sharing loss for contour detection. In CVPR.
36. Shesh, A., and Chen, B. 2008. Efficient and dynamic simplification of line drawings. Computer Graphics Forum 27, 2, 537--545.
37. Simonyan, K., and Zisserman, A. 2015. Very deep convolutional networks for large-scale image recognition. In ICLR.
38. Springenberg, J. T., Dosovitskiy, A., Brox, T., and Riedmiller, M. A. 2015. Striving for simplicity: The all convolutional net. In ICLR Workshop Track.
39. Wilson, B., and Ma, K.-L. 2004. Rendering complexity in computer-generated pen-and-ink illustrations. In International Symposium on Non-photorealistic Animation and Rendering, 129--137.
40. Zeiler, M. D., and Fergus, R. 2014. Visualizing and understanding convolutional networks. In ECCV.
41. Zeiler, M. D. 2012. ADADELTA: an adaptive learning rate method. CoRR abs/1212.5701.
42. Zhang, T. Y., and Suen, C. Y. 1984. A fast parallel algorithm for thinning digital patterns. Commun. ACM 27, 3, 236--239.

Index Terms

• Applied computing

  • Arts and humanities

    • Fine arts

• Computing methodologies

  • Machine learning

    • Machine learning approaches

      • Neural networks