Abstract
People behave differently in different situations. With the advances in ubiquitous sensing technologies, it is now easier to capture human behavior across multiple situations automatically and unobtrusively. We investigate human behavior across two situations that are ubiquitous in hospitality (job interview and reception desk) with the objective of inferring performance on the job. Utilizing a dataset of 338 dyadic interactions, played by students from a hospitality management school, we first study the connections between automatically extracted nonverbal cues, linguistic content, and various perceived variables of soft skills and performance in these two situations. A correlation analysis reveals connection between perceived variables and nonverbal cues displayed during job interviews, and perceived performance on the job. We then propose a computational framework, with nonverbal cues and linguistic style from the two interactions as features, to infer the perceived performance and soft skills in the reception desk situation as a regression task. The best inference performance, with R2 = 0.40, is achieved using a combination of nonverbal cues extracted from the reception desk setting and the human-rated interview scores. We observe that some behavioral cues (greater speaking turn duration and head nods) are positively correlated to higher ratings for all perceived variables across both situations. The best performance using verbal content is achieved by fusion of LIWC and Doc2Vec features with R2 = 0.25 for perceived performance. Our work has implications for the creation of behavioral training systems with focus on specific behaviors for hospitality students.
- Alexander T Adams, Jean Costa, Malte F Jung, and Tanzeem Choudhury. 2015. Mindless computing: designing technologies to subtly influence behavior. In Proc. ACM UBIComp. ACM, 719--730. Google ScholarDigital Library
- Nalini Ambady, Mary Anne Krabbenhoft, and Daniel Hogan. 2006. The 30-sec sale: Using thin-slice judgments to evaluate sales effectiveness. J. Consumer Psychology 16, 1 (2006), 4--13.Google ScholarCross Ref
- Nalini Ambady and Robert Rosenthal. 1992. Thin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis. Psychological Bulletin 111, 2 (1992).Google Scholar
- Nalini Ambady and Robert Rosenthal. 1993. Half a minute: Predicting teacher evaluations from thin slices of nonverbal behavior and physical attractiveness. J. Personality and Social Psychology 64, 3 (1993), 431.Google ScholarCross Ref
- Nalini Ambady and John Joseph Skowronski. 2008. First impressions. Guilford Press.Google Scholar
- Neil Anderson and Viv Shackleton. 1990. Decision making in the graduate selection interview: A field study. J. Occupational Psychology 63, 1 (1990), 63--76.Google ScholarCross Ref
- Ligia Maria Batrinca, Nadia Mana, Bruno Lepri, Fabio Pianesi, and Nicu Sebe. 2011. Please, tell me about yourself: automatic personality assessment using short self-presentations. In Proc. ACM ICMI. Google ScholarDigital Library
- Marjorie A Bayes. 1972. Behavioral cues of interpersonal warmth. J. Consulting and Clinical Psychology 39, 2 (1972), 333.Google ScholarCross Ref
- Joan-Isaac Biel and Daniel Gatica-Perez. 2013. The YouTube lens: Crowdsourced personality impressions and audiovisual analysis of vlogs. IEEE Trans. on Multimedia 15, 1 (2013). Google ScholarDigital Library
- Joan-Isaac Biel, Vagia Tsiminaki, John Dines, and Daniel Gatica-Perez. 2013. Hi youtube!: Personality impressions and verbal content in social video. In Proc. 15th ACM ICMI. ACM, 119--126. Google ScholarDigital Library
- Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. O'Reilly Media, Inc. Google ScholarDigital Library
- Silvia Bonaccio, Jane O'Reilly, Sharon L O'Sullivan, and François Chiocchio. 2016. Nonverbal behavior and communication in the workplace: A review and an agenda for research. J. Management 42, 5 (2016), 1044--1074.Google Scholar
- Joyce E Bono and Remus Ilies. 2006. Charisma, positive emotions and mood contagion. The Leadership Quarterly 17, 4 (2006), 317--334.Google ScholarCross Ref
- Leo Breiman. 2001. Random forests. Machine learning 45, 1 (2001), 5--32. Google ScholarDigital Library
- Judee K Burgoon, Thomas Birk, and Michael Pfau. 1990. Nonverbal behaviors, persuasion, and credibility. Human Communication Research 17, 1 (1990), 140--169.Google ScholarCross Ref
- Ron Caneel. 2005. Social signaling in decision making. Ph.D. Dissertation. Massachusetts Institute of Technology.Google Scholar
- Lei Chen, Gary Feng, Chee Wee Leong, Blair Lehman, Michelle Martin-Raugh, Harrison Kell, Chong Min Lee, and Su-Youn Yoon. 2016. Automated scoring of interview videos using Doc2Vec multimodal feature extraction paradigm. In Proc. 18th ACM ICMI. ACM, 161--168. Google ScholarDigital Library
- Yiqiang Chen, Yu Yu, and Jean-Marc Odobez. 2015. Head Nod Detection from a Full 3D Model. In Proc. IEEE ICCV Workshops. Google ScholarDigital Library
- Cindy Chung and James W Pennebaker. 2007. The psychological functions of function words. Social Communication (2007), 343--359.Google Scholar
- Microsoft Cognitive Services. {n. d.}. Azure Emotion API. https://azure.microsoft.com/en-us/services/cognitive-services/emotion/Google Scholar
- Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine learning 20, 3 (1995), 273--297. Google ScholarDigital Library
- Jared R Curhan and Alex Pentland. 2007. Thin slices of negotiation: predicting outcomes from conversational dynamics within the first 5 minutes. J. Applied Psychology 92, 3 (2007).Google ScholarCross Ref
- Timothy DeGroot and Janaki Gooty. 2009. Can nonverbal cues be used to make meaningful personality attributions in employment interviews? J. Business and Psychology 24, 2 (2009).Google ScholarCross Ref
- Timothy DeGroot and Stephan J Motowidlo. 1999. Why visual and vocal interview cues can affect interviewers' judgments and predict job performance. J. Applied Psychology 84, 6 (1999), 986.Google ScholarCross Ref
- Christian M End and Katherine Saunders. 2013. Short Communication: Powerless and Jobless? Comparing the Effects of Powerless Speech and Speech Disorders on an Applicant's Employability. Frontiers 2, 1 (2013).Google Scholar
- Ray J Forbes and Paul R Jackson. 1980. nonverbal behaviour and the outcome of selection interviews. J. Occupational Psychology 53, 1 (1980), 65--72.Google ScholarCross Ref
- David C Funder. 2006. Towards a resolution of the personality triad: Persons, situations, and behaviors. J. Research in Personality 40, 1 (2006), 21--34.Google ScholarCross Ref
- Michelle Fung, Yina Jin, RuJie Zhao, and Mohammed Ehsan Hoque. 2015. ROC speak: semi-automated personalized feedback on nonverbal behavior from recorded videos. In Proc. ACM UBIComp. ACM, 1167--1178. Google ScholarDigital Library
- Mark Gabbott and Gillian Hogg. 2000. An empirical investigation of the impact of nonverbal communication on service evaluation. European J. Marketing 34, 3/4 (2000), 384--398.Google ScholarCross Ref
- Venkata Rama Kiran Garimella, Abdulrahman Alfayad, and Ingmar Weber. 2016. Social media image analysis for public health. In Proc. 2016 CHI Conf. on Human Factors in Computing Systems. ACM, 5543--5547. Google ScholarDigital Library
- Daniel Gatica-Perez. 2009. Automatic nonverbal analysis of social interaction in small groups: A review. Image and Vision Computing 27, 12 (2009). Google ScholarDigital Library
- Gerald J Gorn, Marvin E Goldberg, and Kunal Basu. 1993. Mood, awareness, and product evaluation. J. Consumer Psychology 2, 3 (1993), 237--256.Google ScholarCross Ref
- Judith A Hall, Debra L Roter, and Cynthia S Rand. 1981. Communication of affect between patient and physician. J. Health and Social Behavior (1981), 18--30.Google Scholar
- Marvin A Hecht and Marianne LaFrance. 1995. How (Fast) Can I Help You? Tone of Voice and Telephone Operator Efficiency in Interactions1. J. Applied Social Psychology 25, 23 (1995), 2086--2098.Google ScholarCross Ref
- James G Hollandsworth, Robert C Glazeski, and Mary Edith Dressel. 1978. Use of social-skills training in the treatment of extreme anxiety and deficient verbal skills in the job-interview setting. J. Applied Behaviour Analysis 11, 2 (1978).Google ScholarCross Ref
- James G Hollandsworth, Richard Kazelskis, Joanne Stevens, and Mary Edith Dressel. 1979. Relative contributions of verbal, articulative, and nonverbal communication to employment decisions in the job interview setting. J. Personnel Psychology 32, 2 (1979).Google Scholar
- Mohammed Ehsan Hoque, Matthieu Courgeon, Jean-Claude Martin, Bilge Mutlu, and Rosalind W Picard. 2013. Mach: My automated conversation coach. In Proc. ACM UBICOMP. Google ScholarDigital Library
- Jack L Howard and Gerald R Ferris. 1996. The employment interview context: Social and situational influences on interviewer decisions. J. Applied Social Psychology 26, 2 (1996), 112--136.Google ScholarCross Ref
- Allen I Huffcutt, James M Conway, Philip L Roth, and Nancy J Stone. 2001. Identification and meta-analytic assessment of psychological constructs measured in employment interviews. J. Applied Psychology 86, 5 (2001).Google ScholarCross Ref
- Hayley Hung, Yan Huang, Gerald Friedland, and Daniel Gatica-Perez. 2011. Estimating dominance in multi-party meetings using speaker diarization. IEEE Transactions on Audio, Speech, and Language Processing 19, 4 (2011), 847--860. Google ScholarDigital Library
- Andrew S Imada and Milton D Hakel. 1977. Influence of nonverbal communication and rater proximity on impressions and decisions in simulated employment interviews. J. Applied Psychology 62, 3 (1977).Google ScholarCross Ref
- Hyo Sun Jung and Hye Hyun Yoon. 2011. The effects of nonverbal communication of employees in the family restaurant upon customer's emotional responses and customer satisfaction. Int. J. Hospitality Management 30, 3 (2011), 542--550.Google ScholarCross Ref
- Juhee Kang and Sunghyup Sean Hyun. 2012. Effective communication styles for the customer-oriented service employee: Inducing dedicational behaviors in luxury restaurant patrons. Int. J. Hospitality Management 31, 3 (2012), 772--785.Google ScholarCross Ref
- Douglas T Kenrick and David C Funder. 1988. Profiting from controversy: Lessons from the person-situation debate. American psychologist 43, 1 (1988), 23.Google Scholar
- Mark Knapp, Judith Hall, and Terrence Horgan. 2013. Nonverbal communication in human interaction. Cengage Learning.Google Scholar
- Max Kuhn. 2016. A Short Introduction to the caret Package. (2016).Google Scholar
- Nathan R Kuncel, Sarah A Hezlett, and Deniz S Ones. 2004. Academic performance, career potential, creativity, and job performance: Can one construct predict them all?Google Scholar
- Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In Proc. 31st ICML. 1188--1196. Google ScholarDigital Library
- Bruno Lepri, Nadia Mana, Alessandro Cappelletti, and Fabio Pianesi. 2009. Automatic prediction of individual performance from thin slices of social behavior. In Proc. 17th ACM MMProc. ACM, 733--736. Google ScholarDigital Library
- Haim Mano and Richard L Oliver. 1993. Assessing the dimensionality and structure of the consumption experience: evaluation, feeling, and satisfaction. J. Consumer research 20, 3 (1993), 451--466.Google Scholar
- Thomas V McGovern and Howard EA Tinsley. 1978. Interviewer evaluations of interviewee nonverbal behavior. J. Vocational Behavior 13, 2 (1978).Google ScholarCross Ref
- Matthias R Mehl, Megan L Robbins, and Fenne große Deters. 2012. Naturalistic observation of health-relevant social processes: the electronically activated recorder (EAR) methodology in psychosomatics. Psychosomatic Medicine 74, 4 (2012), 410.Google ScholarCross Ref
- Stephan J Motowidlo and Jennifer R Burnett. 1995. Aural and visual sources of validity in structured employment interviews. Organizational Behavior and Human Decision Processes 61, 3 (1995), 239--249.Google ScholarCross Ref
- Skanda Muralidhar and Daniel Gatica-Perez. 2017. Examining Linguistic Content and Skill Impression Structure for Job Interview Analytics in Hospitality. In Proc. 16th ACM MUM. Google ScholarDigital Library
- Skanda Muralidhar, Laurent Son Nguyen, Denise Frauendorfer, Jean-Marc Odobez, Marianne Schimd-Mast, and Daniel Gatica-Perez. 2016. Training on the Job: Behavioral Analysis of Job Interviews in Hospitality. In Proc. 18th ACM ICMI. 84--91. Google ScholarDigital Library
- Skanda Muralidhar, Marianne Schimd-Mast, and Daniel Gatica-Perez. 2017. How May I Help You? Behavior and Impressions in Hospitality Service Encounters. In Proc. 19th ACM ICMI. Google ScholarDigital Library
- Iftekhar Naim, M Iftekhar Tanveer, Daniel Gildea, and Mohammed Ehsan Hoque. 2015. Automated prediction and analysis of job interview performance: The role of what you say and how you say it. Proc. IEEE FG (2015).Google ScholarCross Ref
- Laurent Son Nguyen, Denise Frauendorfer, Marianne Schmid Mast, and Daniel Gatica-Perez. 2014. Hire me: Computational inference of hirability in employment interviews based on nonverbal behavior. IEEE Trans. on Multimedia 16, 4 (2014). Google ScholarDigital Library
- NIST NIST. 2012. Machine translation evaluation official results.Google Scholar
- Christopher Y Olivola, Dawn L Eubanks, and Jeffrey B Lovelace. 2014. The many (distinctive) faces of leadership: Inferring leadership domain from facial appearance. The Leadership Quarterly 25, 5 (2014), 817--834.Google ScholarCross Ref
- Maja Pantic, Alex Pentland, Anton Nijholt, and Thomas S Huang. 2007. Human computing and machine understanding of human behavior: A survey. In Artifical Intelligence for Human Computing. Springer, 47--71. Google ScholarDigital Library
- Charles K Parsons and Robert C Liden. 1984. Interviewer perceptions of applicant qualifications: A multivariate field study of demographic characteristics and nonverbal cues. J. Applied Psychology 69, 4 (1984), 557.Google ScholarCross Ref
- Karl Pearson. 1901. LIII. On lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and J. Science 2, 11 (1901), 559--572.Google Scholar
- James W Pennebaker and Laura A King. 1999. Linguistic styles: language use as an individual difference. J. Personality and Social Psychology 77, 6 (1999), 1296.Google ScholarCross Ref
- Alex Pentland. 2004. Social dynamics: Signals and behavior. In Int. Conf. on Developmental Learning, Vol. 5.Google Scholar
- Alex Pentland and Tracy Heibeck. 2010. Honest signals: how they shape our world. MIT press. Google ScholarDigital Library
- Bogdan Raducanu, Jordi Vitria, and Daniel Gatica-Perez. 2009. You are fired! nonverbal role analysis in competitive meetings. In Acoustics, Speech and Signal Processing, 2009. ICASSP 2009. IEEE Int. Conf. on. IEEE, 1949--1952. Google ScholarDigital Library
- Keith G Rasmussen. 1984. Nonverbal behavior, verbal behavior, resumé credentials, and selection interview outcomes. J. Applied Psychology 69, 4 (1984), 551.Google ScholarCross Ref
- Radim Řehůřek and Petr Sojka. {n. d.}. Software Framework for Topic Modelling with Large Corpora. In Proc. LREC 2010 Workshop on New Challenges for NLP Frameworks. ELRA.Google Scholar
- Dairazalia Sanchez-Cortes, Oya Aran, Marianne Schmid Mast, and Daniel Gatica-Perez. 2012. A nonverbal behavior approach to identify emergent leaders in small groups. Multimedia, IEEE Trans. on 14, 3 (2012), 816--832. Google ScholarDigital Library
- Dairazalia Sanchez-Cortes, Petr Motlicek, and Daniel Gatica-Perez. 2012. Assessing the impact of language style on emergent leadership perception from ubiquitous audio. In Proc. 11th Int. Conf. on MUM. ACM, 33. Google ScholarDigital Library
- Klaus R Scherer. 1982. Methods of research on vocal communication: Paradigms and parameters. Handbook of Methods in Nonverbal Behavior Research (1982), 136--198.Google Scholar
- Albrecht Schmidt. 2016. Cloud-based AI for pervasive applications. IEEE Pervasive Computing 15, 1 (2016), 14--18. Google ScholarDigital Library
- Patrick E Shrout and Joseph L Fleiss. 1979. Intraclass correlations: uses in assessing rater reliability. Psychological bulletin 86, 2 (1979).Google Scholar
- Aron W Siegman. 1987. The telltale voice: Nonverbal messages of verbal communication. (1987).Google Scholar
- Vivek K Singh, Souvick Ghosh, and Christin Jose. 2017. Toward Multimodal Cyberbullying Detection. In Proc. 2017 CHI Conf. Extended Abstracts on Human Factors in Computing Systems. ACM, 2090--2099. Google ScholarDigital Library
- Del A Sole. 2018. Getting Started with the Computer Vision API. Apress, Berkeley, CA.Google Scholar
- Del A Sole. 2018. Introducing Microsoft Cognitive Services. Apress, Berkeley, CA.Google Scholar
- Brian R Spisak, Allen E Grabo, Richard D Arvey, and Mark van Vugt. 2014. The age of exploration and exploitation: Younger-looking leaders endorsed for change and older-looking leaders endorsed for stability. The Leadership Quarterly 25, 5 (2014), 805--816.Google ScholarCross Ref
- DS Sundaram and Cynthia Webster. 2000. The role of nonverbal communication in service encounters. J. Services Marketing 14, 5 (2000), 378--391.Google ScholarCross Ref
- Yla R Tausczik and James W Pennebaker. 2010. The psychological meaning of words: LIWC and computerized text analysis methods. J. Language and Social Psychology 29, 1 (2010), 24--54.Google ScholarCross Ref
- Cornelis J Van Rijsbergen. 1986. A new theoretical framework for information retrieval. In ACM SIGIR Forum, Vol. 21. ACM. Google ScholarDigital Library
- Chockalingam Viswesvaran and Deniz S Ones. 2000. Perspectives on models of job performance. Int. J. Selection and Assessment 8, 4 (2000), 216--226.Google ScholarCross Ref
- Mark Weiser. 2002. The computer for the 21st century. IEEE Pervasive Computing 1, 1 (2002), 19--25. Google ScholarDigital Library
Index Terms
- A Tale of Two Interactions: Inferring Performance in Hospitality Encounters from Cross-Situation Social Sensing
Recommendations
Facing Employers and Customers: What Do Gaze and Expressions Tell About Soft Skills?
MUM '18: Proceedings of the 17th International Conference on Mobile and Ubiquitous MultimediaEye gaze and facial expressions are central to face-to-face social interactions. These behavioral cues and their connections to first impressions have been widely studied in psychology and computing literature, but limited to a single situation. ...
Training on the job: behavioral analysis of job interviews in hospitality
ICMI '16: Proceedings of the 18th ACM International Conference on Multimodal InteractionFirst impressions play a critical role in the hospitality industry and have been shown to be closely linked to the behavior of the person being judged.In this work, we implemented a behavioral training framework for hospitality students with the goal of ...
Examining linguistic content and skill impression structure for job interview analytics in hospitality
MUM '17: Proceedings of the 16th International Conference on Mobile and Ubiquitous MultimediaFirst impressions are critical to professional interactions especially in the context of employment interviews. This work investigates connections between linguistic content and first impressions in job interviews and the structure of ten soft skills ...
Comments