Calendar.help: Designing a Workflow-Based Scheduling Agent with Humans in the Loop

ABSTRACT

Although we may complain about meetings, they are an essential part of an information worker's work life. Consequently, busy people spend a significant amount of time scheduling meetings. We present Calendar.help, a system that provides fast, efficient scheduling through structured workflows. Users interact with the system via email, delegating their scheduling needs to the system as if it were a human personal assistant. Common scheduling scenarios are broken down using well-defined workflows and completed as a series of microtasks that are automated when possible and executed by a human otherwise. Unusual scenarios fall back to a trained human assistant executing an unstructured macrotask. We describe the iterative approach we used to develop Calendar.help, and share the lessons learned from scheduling thousands of meetings during a year of real-world deployments. Our findings provide insight into how complex information tasks can be broken down into repeatable components that can be executed efficiently to improve productivity.

References

1. Gabor Angeli, Christopher D Manning, and Daniel Jurafsky. 2012. Parsing time: Learning to interpret time expressions. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 446--455.
2. Yoav Artzi, Nicholas FitzGerald, and Luke S Zettlemoyer. 2013. Semantic Parsing with Combinatory Categorial Grammars. ACL (Tutorial Abstracts) 3 (2013).
3. Jakob E. Bardram and Claus Bossen. 2005. A web of coordinative artifacts: collaborative work at a hospital ward. In Proceedings of the 2005 International ACM SIGGROUP Conference on Supporting Group Work. ACM, 168--176. http://dx.doi.org/10.1145/1099203.1099235
4. Julian Birkinshaw and Jordan Cohen. 2013. Make time for the work that matters. Harvard Business Review (01 September 2013). https://hbr.org/2013/09/ make-time-for-the-work-that-matters.
5. Mike Brzozowski, Kendra Carattini, Scott R. Klemmer, Patrick Mihelich, Jiang Hu, and Andrew Y. Ng. 2006. groupTime: Preference based group scheduling. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1047--1056. http://dx.doi.org/10.1145/1124772.1124929
6. Angel X. Chang and Christopher D. Manning. 2012. SUTime: A library for recognizing and normalizing time expressions. In LREC. 3735--3740.
7. Lydia B. Chilton, Greg Little, Darren Edge, Daniel S. Weld, and James A. Landay. 2013. Cascade: Crowdsourcing taxonomy creation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1999--2008. http://dx.doi.org/10.1145/2470654.2466265
8. Nicolas Ducheneaut and Victoria Bellotti. 2001. E-mail as habitat: an exploration of embedded personal information management. Interactions 8, 5 (2001), 30--38. http://dl.acm.org/citation.cfm?doid=382899.383305
9. Susan F. Ehrlich. 1987a. Social and psychological factors influencing the design of office communication systems. In Proceedings of the SIGCHI/GI Conference on Human Factors in Computing Systems and Graphics Interface (CHI '87). 323--329. http://dx.doi.org/10.1145/29933.275651
10. Susan F. Ehrlich. 1987b. Strategies for encouraging successful adoption of office communication systems. ACM Transactions on Information Systems (TOIS) 5, 4 (1987), 340--357. http://dx.doi.org/10.1145/42196.42198
11. Oren Etzioni, Michael Cafarella, Doug Downey, Ana-Maria Popescu, Tal Shaked, Stephen Soderland, Daniel S Weld, and Alexander Yates. 2005. Unsupervised named-entity extraction from the web: An experimental study. Artificial intelligence 165, 1 (2005), 91--134.
12. Jonathan Grudin. 1988. Why CSCW applications fail: Problems in the design and evaluation of organizational interfaces. In Proceedings of the 1988 ACM Conference on Computer-Supported Cooperative Work (CSCW '88). 85--93. http://dx.doi.org/10.1145/62266.62273
13. Jessi Hempel. 2015. Facebook launches M, its bold answer to Siri and Cortana. Wired (2015). https://www.wired.com/2015/08/ facebook-launches-m-new-kind-virtual-assistant/.
14. Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 159--166. http://dx.doi.org/10.1145/302979.303030
15. Ellen Huet. 2016. The humans hiding behind the chatbots. Bloomberg Technology (2016). http://www.bloomberg.com/news/articles/2016-04--18/ the-humans-hiding-behind-the-chatbots.
16. Shamsi T. Iqbal Justin Cheng, Jaime Teevan and Michael S. Bernstein. 2015. Break it down: A comparison of macro- and microtasks.. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). 4061--4064. http://dx.doi.org/10.1145/2702123.2702146
17. Ece Kamar. 2016. Hybrid Intelligence and the Future of Work. In Productivity Decomposed: Getting Big Things Done with Little Microtasks Workshop (CHI 2016). http://research.microsoft.com/en-us/um/people/eckamar/ papers/HybridIntelligence.pdf
18. Ece Kamar, Severin Hacker, and Eric Horvitz. 2012. Combining human and machine intelligence in large-scale crowdsourcing. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems-Volume 1. International Foundation for Autonomous Agents and Multiagent Systems, 467--474.
19. Abraham Bernstein Michael Naf Katharina Reinecke, Minh Khoa Nguyen and Krzysztof Z. Gajos. 2013. Doodle around the world: Online scheduling behavior reflects cultural differences in time perception and group decision-making. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work. 45--54. http://dx.doi.org/10.1145/2441776.2441784
20. Benjamin Kellermann and Rainer Bohme. 2009. Privacy-Enhanced Event Scheduling. IEEE Computer Society. 52--59 pages.
21. John F. Kelley. 1984. An iterative design methodology for user-friendly natural language office information applications. ACM Transactions on Information Systems (TOIS) 2, 1 (1984), 26--41. http://dx.doi.org/10.1145/357417.357420
22. Aniket Kittur, Jeffrey V. Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The Future of Crowd Work. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work (CSCW '13). ACM, New York, NY, USA, 1301--1318. DOI: http://dx.doi.org/10.1145/2441776.2441923
23. Walter S. Lasecki, Mitchell Gordon, Jaime Teevan, Ece Kamar, and Jeffrey P Bigham. 2015. Preserving Privacy in Crowd-Powered Systems. (2015).
24. Walter S. Lasecki, Rachel Wesley, Jeffrey Nichols, Anand Kulkarni, James F Allen, and Jeffrey P. Bigham. 2013. Chorus: a crowd-powered conversational assistant. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology. ACM, 151--162. http://dx.doi.org/10.1145/2501988.2502057
25. Pattie Maes and others. 1994. Agents that reduce work and information overload. Commun. ACM 37, 7 (1994), 30--40. http://dx.doi.org/10.1145/176789.176792
26. Cade Metz. 2015. AI helps humans best when humans help the AI. Wired (2015). https://www.wired.com/2015/ 09/ai-helps-humans-best-humans-help-ai/.
27. Tom M. Mitchell, Rich Caruana, Dayne Freitag, John McDermott, David Zabowski, and others. 1994. Experience with a learning personal assistant. Commun. ACM 37, 7 (1994), 80--91. http://dx.doi.org/10.1145/176789.176798
28. Karen Myers, Pauline Berry, Jim Blythe, Ken Conley, Melinda Gervasio, Deborah L McGuinness, David Morley, Avi Pfeffer, Martha Pollack, and Milind Tambe. 2007. An intelligent personal assistant for task and time management. AI Magazine 28, 2 (2007), 47.
29. Daniela Retelny, Sébastien Robaszkiewicz, Alexandra To, Walter S. Lasecki, Jay Patel, Negar Rahmati, Tulsee Doshi, Melissa Valentine, and Michael S. Bernstein. 2014. Expert Crowdsourcing with Flash Teams. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). ACM, New York, NY, USA, 75--85. DOI: http://dx.doi.org/10.1145/2642918.2647409
30. Sandip Sen and Edmund H. Durfee. 1998. A formal study of distributed meeting scheduling. Group Decision and Negotiation 7, 3 (1998), 265--289. https://doi.org/10.1145/122831.122837
31. John C. Tang, Chen Zhao, Xiang Cao, and Kori Inkpen. 2011. Your time zone or mine?: a study of globally time zone-shifted collaboration. In Proceedings of the ACM 2011 Conference on Computer-Supported Cooperative Work. ACM, 235--244. http://dx.doi.org/10.1145/1958824.1958860
32. Jaime Teevan. 2016. The future of microwork. XRDS: Crossroads 23, 2 (2016), 26--29. https://doi.org/10.1145/3019600
33. Jaime Teevan, Shamsi T. Iqbal, and Curtis von Veh. 2016. Supporting Collaborative Writing with Microtasks. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 2657--2668. DOI: http://dx.doi.org/10.1145/2858036.2858108
34. Jaime Teevan, Daniel J. Liebling, and Walter S. Lasecki. 2014. Selfsourcing personal tasks. In CHI'14 Extended Abstracts on Human Factors in Computing Systems. ACM, 2527--2532. http://dx.doi.org/10.1145/2559206.2581181
35. Wendy A. Kellogg Thomas Erickson, Catalina M. Danis and Mary E. Helander. 2008. Assistance: The work practices of human administrative assistants and their implications for it and organizations. In Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work (CSCW '08). 608--618. http://dx.doi.org/10.1145/1460563.1460658
36. Mikko Raatikainen Terho Norja Varvana Myllärniemi, Olli Korjus and Tomi Männistö. 2014. Meeting scheduling across heterogeneous calendars and organizations utilizing mobile devices and cloud services. In Proceedings of the 13th International Conference on Mobile and Ubiquitous Multimedia (MUM '14). 224--227. http://dx.doi.org/10.1145/2677972.2678002
37. Puyang Xu and Ruhi Sarikaya. 2013. Convolutional neural network based triangular crf for joint intent detection and slot filling. In Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on. IEEE, 78--83.
38. Haoqi Zhang, Edith Law, Rob Miller, Krzysztof Gajos, David Parkes, and Eric Horvitz. 2012. Human computation tasks with global constraints. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 217--226. http://dx.doi.org/10.1145/2207676.2207708
39. Alejandro Zunino and Marcelo Campo. 2009. Chronos: A multi-agent system for distributed automatic meeting scheduling. Expert Systems with Applications 36, 3 (2009), 7011--7018.

Index Terms

• Human-centered computing