Zooids: Building Blocks for Swarm User Interfaces

ABSTRACT

This paper introduces swarm user interfaces, a new class of human-computer interfaces comprised of many autonomous robots that handle both display and interaction. We describe the design of Zooids, an open-source open-hardware platform for developing tabletop swarm interfaces. The platform consists of a collection of custom-designed wheeled micro robots each 2.6 cm in diameter, a radio base-station, a high-speed DLP structured light projector for optical tracking, and a software framework for application development and control. We illustrate the potential of tabletop swarm user interfaces through a set of application scenarios developed with Zooids, and discuss general design considerations unique to swarm user interfaces.

References

1. Ahlberg, C., Williamson, C., and Shneiderman, B. Dynamic queries for information exploration: An implementation and evaluation. In Proceedings of the SIGCHI conference on Human factors in computing systems, ACM (1992), 619--626.
2. Alonso-Mora, J., Breitenmoser, A., Rufli, M., Siegwart, R., and Beardsley, P. Multi-robot system for artistic pattern formation. In Robotics and Automation (ICRA), 2011 IEEE International Conference on, IEEE (2011), 4512--4517.
3. Alonso-Mora, J., Lohaus, S. H., Leemann, P., Siegwart, R., and Beardsley, P. Gesture based human-multi-robot swarm interaction and its application to an interactive display. In 2015 IEEE International Conference on Robotics and Automation (ICRA), IEEE (2015), 5948--5953.
4. Amano, K., and Yamamoto, A. Tangible interactions on a flat panel display using actuated paper sheets. In Proceedings of the 2012 ACM international conference on Interactive tabletops and surfaces, ACM (2012), 351--354.
5. Bennett, E., and Stevens, B. The effect that touching a projection augmented model has on object-presence. In Information Visualisation, 2005. Proceedings. Ninth International Conference on, IEEE (2005), 790--795.
6. Brave, S., Ishii, H., and Dahley, A. Tangible interfaces for remote collaboration and communication. In Proceedings of the 1998 ACM conference on Computer supported cooperative work, ACM (1998), 169--178.
7. Chang, B.-W., and Ungar, D. Animation: from cartoons to the user interface.
8. Cucu, L., Rubenstein, M., and Nagpal, R. Towards self-assembled structures with mobile climbing robots. In Robotics and Automation (ICRA), 2015 IEEE International Conference on, IEEE (2015), 1955--1961.
9. Ducatelle, F., Di Caro, G., Pinciroli, C., and Gambardella, L. Self-organized cooperation between robotic swarms. Swarm Intelligence Journal 5, 2 (2011), 73--96.
10. Dudek, G., Jenkin, M., Milios, E., and Wilkes, D. A taxonomy for swarm robots. In Intelligent Robots and Systems'93, IROS'93. Proceedings of the 1993 IEEE/RSJ International Conference on, vol. 1, IEEE (1993), 441--447.
11. Felton, S. M., Tolley, M. T., Onal, C. D., Rus, D., and Wood, R. J. Robot self-assembly by folding: A printed inchworm robot. In Robotics and Automation (ICRA), 2013 IEEE International Conference on, IEEE (2013), 277--282.
12. Fishkin, K. P. A taxonomy for and analysis of tangible interfaces. Personal Ubiquitous Comput. 8 (September 2004), 347--358.
13. Fitzmaurice, G. W., and Buxton, W. An empirical evaluation of graspable user interfaces: towards specialized, space-multiplexed input. In Proc. CHI 1997, 43--50.
14. Follmer, S., Leithinger, D., Olwal, A., Cheng, N., and Ishii, H. Jamming User Interfaces: Programmable Particle Stiffness and Sensing for Malleable and Shape-Changing Devices. In ACM Symposium on User Interface Software and Technology (2012), 519--528.
15. Follmer, S., Leithinger, D., Olwal, A., Hogge, A., and Ishii, H. inform: Dynamic physical affordances and constraints through shape and object actuation. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology, UIST '13, ACM (New York, NY, USA, 2013), 417--426.
16. Futurelab, A. E. Drone 100 -- the world record for intel 2015. http://tinyurl.com/drone100, 2016.
17. Gervais, R. Interaction and introspection with tangible augmented objects. Phd dissertation, Université de Bordeaux, Dec. 2015.
18. Goldstein, S. C., Campbell, J. D., and Mowry, T. C. Programmable matter. Computer 38, 6 (2005), 99--101.
19. Greenberg, S., and Fitchett, C. Phidgets: easy development of physical interfaces through physical widgets. In Proceedings of the 14th annual ACM symposium on User interface software and technology, ACM (2001), 209--218.
20. Grieder, R., Alonso-Mora, J., Bloechlinger, C., Siegwart, R., and Beardsley, P. Multi-robot control and interaction with a hand-held tablet. In Workshop Proc. Int. Conf. Robotics and Automation, vol. 131, Citeseer (2014).
21. Hauri, S., Alonso-Mora, J., Breitenmoser, A., Siegwart, R., and Beardsley, P. Multi-robot formation control via a real-time drawing interface. In Field and Service Robotics, Springer (2014), 175--189.
22. Heer, J., and Robertson, G. G. Animated transitions in statistical data graphics. Visualization and Computer Graphics, IEEE Transactions on 13, 6 (2007), 1240--1247.
23. Horn, M. S., Solovey, E. T., Crouser, R. J., and Jacob, R. J. Comparing the use of tangible and graphical programming languages for informal science education. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '09, 975--984.
24. Hornecker, E., and Buur, J. Getting a grip on tangible interaction: a framework on physical space and social interaction. In Proceedings of the SIGCHI conference on Human Factors in computing systems, ACM (2006), 437--446.
25. Huron, S., Jansen, Y., and Carpendale, S. Constructing visual representations: Investigating the use of tangible tokens. Visualization and Computer Graphics, IEEE Transactions on 20, 12 (2014), 2102--2111.
26. Ishii, H., Lakatos, D., Bonanni, L., and Labrune, J.-B. Radical atoms: beyond tangible bits, toward transformable materials. interactions 19, 1 (Jan. 2012), 38--51.
27. Jansen, Y., Dragicevic, P., and Fekete, J.-D. Evaluating the efficiency of physical visualizations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM (2013), 2593--2602.
28. Jansen, Y., Dragicevic, P., Isenberg, P., Alexander, J., Karnik, A., Kildal, J., Subramanian, S., and Hornbæk, K. Opportunities and challenges for data physicalization. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, ACM (2015), 3227--3236.
29. Kira, Z., and Potter, M. A. Exerting human control over decentralized robot swarms. In Autonomous Robots and Agents, 2009. ICARA 2009. 4th International Conference on, IEEE (2009), 566--571.
30. Kojima, M., Sugimoto, M., Nakaruma, A., Tomita, M., Inami, M., and Nii, H. Augmented coliseum: An augmented game environment with small vehicles. Horizontal Interactive Human-Computer Systems, International Workshop on 0 (2006), 3--8.
31. Kolling, A., Nunnally, S., and Lewis, M. Towards human control of robot swarms. In Proceedings of the seventh annual ACM/IEEE international conference on human-robot interaction, ACM (2012), 89--96.
32. Kuhn, H. W. The hungarian method for the assignment problem. Naval research logistics quarterly 2, 1--2 (1955), 83--97.
33. Kushleyev, A., Mellinger, D., Powers, C., and Kumar, V. Towards a swarm of agile micro quadrotors. Autonomous Robots 35, 4 (2013), 287--300.
34. Le Goc, M., Dragicevic, P., Huron, S., Boy, J., and Fekete, J.-D. Smarttokens: Embedding motion and grip sensing in small tangible objects. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology, ACM (2015), 357--362.
35. Le Goc, M., Dragicevic, P., Huron, S., Boy, J., and Fekete, J.-D. A better grasp on pictures under glass: Comparing touch and tangible object manipulation using physical proxies. In Proceedings of the International Working Conference on Advanced Visual Interfaces, ACM (2016), 76--83.
36. Lederman, S. J., and Campbell, J. I. Tangible graphs for the blind. Human Factors: The Journal of the Human Factors and Ergonomics Society 24, 1 (1982), 85--100.
37. Lee, J. C., Hudson, S. E., Summet, J. W., and Dietz, P. H. Moveable interactive projected displays using projector based tracking. In Proceedings of the 18th annual ACM symposium on User interface software and technology, ACM (2005), 63--72.
38. Lee, N., Kim, J., Lee, J., Shin, M., and Lee, W. Molebot: mole in a table. In ACM SIGGRAPH 2011 Emerging Technologies, ACM (2011), 9.
39. Leithinger, D., and Ishii, H. Relief: a scalable actuated shape display. In Proceedings of the fourth international conference on Tangible, embedded, and embodied interaction, ACM (2010), 221--222.
40. Lifton, J., Broxton, M., and Paradiso, J. A. Experiences and directions in pushpin computing. In IPSN 2005. Fourth International Symposium on Information Processing in Sensor Networks, 2005., IEEE (2005), 416--421.
41. Marquardt, N., Nacenta, M. A., Young, J. E., Carpendale, S., Greenberg, S., and Sharlin, E. The haptic tabletop puck: tactile feedback for interactive tabletops. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces, ACM (2009), 85--92.
42. Marshall, M., Carter, T., Alexander, J., and Subramanian, S. Ultra-tangibles: creating movable tangible objects on interactive tables. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM (2012), 2185--2188.
43. Mi, H., and Sugimoto, M. Hats: interact using height-adjustable tangibles in tabletop interfaces. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces, ACM (2011), 71--74.
44. Microsoft. Sanddance: Visually explore, understand, and present data. Online. http://research.microsoft.com/en-us/projects/sanddance/, 2016.
45. Moere, A. V. Beyond the tyranny of the pixel: Exploring the physicality of information visualization. In Information Visualisation, 2008. IV'08. 12th International Conference, IEEE (2008), 469--474.
46. Nielsen, J. Usability engineering.
47. Nowacka, D., Ladha, K., Hammerla, N. Y., Jackson, D., Ladha, C., Rukzio, E., and Olivier, P. Touchbugs: Actuated tangibles on multi-touch tables. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM (2013), 759--762.
48. Pangaro, G., Maynes-Aminzade, D., and Ishii, H. The actuated workbench: Computer-controlled actuation in tabletop tangible interfaces. In Proceedings of the 15th Annual ACM Symposium on User Interface Software and Technology, UIST '02, 181--190.
49. Patten, J. Thumbles - robotic tabletop user interface platform. TED.com (2014).
50. Patten, J., and Ishii, H. Mechanical constraints as computational constraints in tabletop tangible interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '07, ACM (New York, NY, USA, 2007), 809--818.
51. Patten, J., Ishii, H., Hines, J., and Pangaro, G. Sensetable: A wireless object tracking platform for tangible user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '01, ACM (New York, NY, USA, 2001), 253--260.
52. Patten, J., Recht, B., and Ishii, H. Audiopad: A tag-based interface for musical performance. In Proceedings of the 2002 Conference on New Interfaces for Musical Expression, NIME '02, National University of Singapore (Singapore, Singapore, 2002), 1--6.
53. Pedersen, E. W., and Hornbæk, K. Tangible bots: interaction with active tangibles in tabletop interfaces. In Proc. CHI, ACM (2011), 2975--2984.
54. Poupyrev, I., Nashida, T., Maruyama, S., Rekimoto, J., and Yamaji, Y. Lumen: Interactive visual and shape display for calm computing. In ACM SIGGRAPH 2004 Emerging Technologies, SIGGRAPH '04, ACM (New York, NY, USA, 2004), 17--.
55. Poupyrev, I., Nashida, T., and Okabe, M. Actuation and tangible user interfaces: the vaucanson duck, robots, and shape displays. In TEI '07, 205--212.
56. Rasmussen, M. K., Pedersen, E. W., Petersen, M. G., and Hornbaek, K. Shape-changing interfaces: a review of the design space and open research questions. In CHI '12, 735--744.
57. Reznik, D., and Canny, J. A flat rigid plate is a universal planar manipulator. In IEEE ICRA 1998, vol. 2, IEEE (1998), 1471--1477.
58. Richter, J., Thomas, B. H., Sugimoto, M., and Inami, M. Remote active tangible interactions. In Proceedings of the 1st international conference on Tangible and embedded interaction, ACM (2007), 39--42.
59. Romanishin, J. W., Gilpin, K., and Rus, D. M-blocks: Momentum-driven, magnetic modular robots. In Intelligent Robots and Systems (IROS), 2013 IEEE/RSJ International Conference on, IEEE (2013), 4288--4295.
60. Rosenfeld, D., Zawadzki, M., Sudol, J., and Perlin, K. Physical objects as bidirectional user interface elements. Computer Graphics and Applications, IEEE 24, 1 (2004), 44--49.
61. Roudaut, A., Karnik, A., Löchtefeld, M., and Subramanian, S. Morphees: toward high shape resolution in self-actuated flexible mobile devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM (2013), 593--602.
62. Rubens, C., Braley, S., Gomes, A., Goc, D., Zhang, X., Carrascal, J. P., and Vertegaal, R. Bitdrones: Towards levitating programmable matter using interactive 3d quadcopter displays. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology, ACM (2015), 57--58.
63. Rubenstein, M., Ahler, C., and Nagpal, R. Kilobot: A low cost scalable robot system for collective behaviors. In Robotics and Automation (ICRA), 2012 IEEE International Conference on, IEEE (2012), 3293--3298.
64. Rus, D. Programmable matter with self-reconfiguring robots. In Proceedings of the 7th ACM international conference on Computing frontiers, CF '10, 51--52.
65. Sahai, R., Avadhanula, S., Groff, R., Steltz, E., Wood, R., and Fearing, R. S. Towards a 3g crawling robot through the integration of microrobot technologies. In Robotics and Automation, 2006. ICRA 2006. Proceedings 2006 IEEE International Conference on, IEEE (2006), 296--302.
66. Seah, S. A., Drinkwater, B. W., Carter, T., Malkin, R., and Subramanian, S. Dexterous ultrasonic levitation of millimeter-sized objects in air. IEEE transactions on ultrasonics, ferroelectrics, and frequency control 61, 7 (2014), 1233--1236.
67. Snape, J., van den Berg, J., Guy, S. J., and Manocha, D. The hybrid reciprocal velocity obstacle. Robotics, IEEE Transactions on 27, 4 (2011), 696--706.
68. Snape, J., van den Berg, J. P., Guy, S. J., and Manocha, D. Independent navigation of multiple mobile robots with hybrid reciprocal velocity obstacles. In IROS (2009), 5917--5922.
69. Steltz, E., Seeman, M., Avadhanula, S., and Fearing, R. S. Power electronics design choice for piezoelectric microrobots. In Intelligent Robots and Systems, 2006 IEEE/RSJ International Conference on, IEEE (2006), 1322--1328.
70. Sutherland, I. E. The ultimate display, 1965.
71. Taher, F., Hardy, J., Karnik, A., Weichel, C., Jansen, Y., Hornbæk, K., and Alexander, J. Exploring interactions with physically dynamic bar charts. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, ACM (2015), 3237--3246.
72. Ullmer, B., and Ishii, H. The metadesk: Models and prototypes for tangible user interfaces. In Proceedings of the 10th Annual ACM Symposium on User Interface Software and Technology, UIST '97, ACM (New York, NY, USA, 1997), 223--232.
73. Ullmer, B., Ishii, H., and Jacob, R. J. K. Token+constraint systems for tangible interaction with digital information. ACM Trans. Comput.-Hum. Interact. 12, 1 (Mar. 2005), 81--118.
74. Underkoffier, J., and Ishii, H. Urp: A luminous-tangible workbench for urban planning and design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '99, 386--393.
75. Victor, B. A brief rant on the future of interaction design. http://tinyurl.com/bvrant, 2011.
76. Wakita, A., Nakano, A., and Kobayashi, N. Programmable blobs: a rheologic interface for organic shape design. In Proceedings of the fifth international conference on Tangible, embedded, and embodied interaction, ACM (2011), 273--276.
77. Weiser, M. Some computer science issues in ubiquitous computing. Communications of the ACM 36, 7 (1993), 75--84.
78. Weiss, M., Schwarz, F., Jakubowski, S., and Borchers, J. Madgets: Actuating widgets on interactive tabletops. In Proceedings of the 23Nd Annual ACM Symposium on User Interface Software and Technology, UIST '10, 293--302.
79. Werfel, J., Petersen, K., and Nagpal, R. Designing collective behavior in a termite-inspired robot construction team. Science 343, 6172 (2014), 754--758.
80. Yamamoto, A., Tsuruta, S., and Higuchi, T. Planar 3-dof paper sheet manipulation using electrostatic induction. In Industrial Electronics (ISIE), 2010 IEEE International Symposium on, IEEE (2010), 493--498.
81. Yamanaka, S., and Miyashita, H. Vibkinesis: notification by direct tap and-dying message-using vibronic movement controllable smartphones. In Proceedings of the 27th annual ACM symposium on User interface software and technology, ACM (2014), 535--540.
82. Yao, L., Niiyama, R., Ou, J., Follmer, S., Della Silva, C., and Ishii, H. Pneui: Pneumatically actuated soft composite materials for shape changing interfaces. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology, UIST '13, 13--22.
83. Yi, J. S., Melton, R., Stasko, J., and Jacko, J. A. Dust & magnet: multivariate information visualization using a magnet metaphor. Information Visualization 4, 4 (2005), 239--256.
84. Zhao, J., and Moere, A. V. Embodiment in data sculpture: a model of the physical visualization of information. In Proceedings of the 3rd international conference on Digital Interactive Media in Entertainment and Arts, ACM (2008), 343--350.

Index Terms

• Human-centered computing