Augmented reality instruction for object assembly based on markerless tracking

ABSTRACT

Conventional object assembly instructions are usually written or illustrated in a paper manual. Users have to associate these static instructions with real objects in 3D space. In this paper, a novel augmented reality system is presented for a user to interact with objects and instructions. While most related methods pasted obvious markers onto objects for tracking and constrained their orientations or shapes, we adopt a markerless strategy for more intuitive interaction. Based on live information from an off-the-shelf RGB-D camera, the proposed tracking procedure identifies components in a scene, tracks their 3D positions and orientations, and evaluates whether there are combinations of components. According to the detected events and poses, our indication procedure then dynamically displays indication lines, circular arrows and other hints to guide a user to manipulate the components into correct poses. The experiment shows that the proposed system can robustly track the components and respond intuitive instructions at an interactive rate. Most of users in evaluation are interested and willing to use this novel technique for object assembly.

References

1. Alvarez, H., Aguinaga, I., and Borro, D. 2011. Providing guidance for maintenance operations using automatic markerless augmented reality system. In Proc. IEEE Intl. Symp. Mixed and Augmented Reality, 181--190.
2. ASUSTek Computer Inc. Xtion pro live. https://www.asus.com/3D-Sensor/Xtion_PRO/.
3. Autodesk Inc. 123d catch. http://www.123dapp.com/catch.
4. Avrahami, D., Wobbrock, J. O., and Izadi, S. 2011. Portico: tangible interaction on and around a tablet. In Proc. ACM Symp. User Interface Software and Technology, 347--356.
5. Besl, P. J., and McKay, N. D. 1992. A method for registration of 3-D shapes. IEEE Trans. Pattern Analysis and Machine Intelligence 14, 2, 239--256.
6. Douadi, L., Aldon, M.-J., and Crosnier, A. 2006. Pair-wise registration of 3d/color data sets with icp. In Proc. IEEE/RSJ Intl. Conf. Intelligent Robots and Systems, 663--668.
7. Gupta, A., Fox, D., Curless, B., and Cohen, M. 2012. DuploTrack: a real-time system for authoring and guiding duplo block assembly. In Proc. ACM Symp. User Interface Software and Technology, 389--402.
8. Held, R. T., Gupta, A., Curless, B., and Agrawala, M. 2012. 3D puppetry: a kinect-based interface for 3D animation. In Proc. ACM Symp. User Interface Software and Technology, 423--433.
9. Henderson, S., and Feiner, S. 2011. Exploring the benefits of augmented reality documentation for maintenance and repair. IEEE Trans. Visualization and Computer Graphics 17, 10, 1355--1368.
10. Henderson, S., and Feiner, S. K. 2011. Augmented reality in the psychomotor phase of a procedural task. In Proc. IEEE Intl. Symp. Mixed and Augmented Reality, 191--200.
11. Hinterstoisser, S., Cagniart, C., Ilic, S., Sturm, P., Navab, N., Fua, P., and Lepetit, V. 2012. Gradient response maps for real-time detection of textureless objects. IEEE Trans. Pattern Analysis and Machine Intelligence 34, 5, 876--888.
12. Hinterstoisser, S., Lepetit, V., Ilic, S., Holzer, S., Bradski, G., Konolige, K., and Navab, N. 2012. Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes. In Proc. Asian Conf. Computer Vision, vol. 7724, 548--562.
13. Kapadia, M., Falk, J., Zünd, F., Marti, M., and Gross, M. 2015. Computer-assisted authoring of interactive narratives. In Proc. ACM SIGGRAPH Symp. Interactive 3D Graphics and Games, 85--92.
14. Khuong, B. M., Kiyokawa, K., Miller, A., LaViola Jr., J. J., Mashita, T., and Takemura, H. 2014. The effectiveness of an ar-based context-aware assembly support system in object assembly. In Proc. IEEE Virtual Reality, 57--62.
15. Kyriazis, N., and Argyros, A. 2013. Physically plausible 3d scene tracking: The single actor hypothesis. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, 9--16.
16. Li, W., Agrawala, M., Curless, B., and Salesin, D. 2008. Automated generation of interactive 3d exploded view diagrams. ACM Trans. Graphics 27, 3, 101:1--101:7.
17. Liang, R. H., Cheng, K. Y., Chan, L., Peng, C. X., Chen, M. Y., Liang, R. H., Yang, D. N., and Chen, B. Y. 2013. Gaussbits: magnetic tangible bits for portable and occlusion-free near-surface interactions. In Proc. SIGCHI Conf. Human Factors in Computing Systems, 1391--1400.
18. Lowe, D. G. 2004. Distinctive image features from scale-invariant keypoints. Intl J. Computer Vision 60, 91--110.
19. Men, H., Gebre, B., and Pochiraju, K. 2011. Color point cloud registration with 4d icp algorithm. In Proc. IEEE Intl. Conf. Robotics and Automation, 1511--1516.
20. Oculus VR. Oculus rift. https://www.oculus.com/.
21. Reiners, D., Stricker, D., Klinker, G., and Müller, S. 1998. Augmented reality for construction tasks: doorlock assembly. In Proc. Intl. Workshop on Augmented reality, 31--46.
22. Ren, Z., Mehra, R., Coposky, J., and Lin, M. C. 2012. Tabletop ensemble: touch-enabled virtual percussion instruments. In Proc. ACM SIGGRAPH Symp. Interactive 3D Graphics and Games, 7--14.
23. Tang, A., Owen, C., Biocca, F., and Mou, W. 2003. Comparative effectiveness of augmented reality in object assembly. In Proc. SIGCHI Conf. Human Factors in Computing Systems, 73--80.
24. Yang, J., Li, H., and Jia, Y. 2013. Go-icp: Solving 3d registration efficiently and globally optimally. In Proc. IEEE Intl. Conf. Computer Vision, 1457--1464.
25. Zauner, J., Haller, M., Brandl, A., and Hartman, W. 2003. Authoring of a mixed reality assembly instructor for hierarchical structures. In Proc. IEEE/ACM Intl. Symp. Mixed and Augmented Reality, 237--246.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

        • Tracking

  • Computer graphics

    • Graphics systems and interfaces

      • Mixed / augmented reality

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction techniques