Benjamin M. Gorman
David R. Flatla
ABSTRACT
At least 360 million people worldwide have disabling hearing loss that frequently causes difficulties in day-to-day conversations. Traditional technology (e.g., hearing aids) often fails to offer enough value, has low adoption rates, and can result in social stigma. Speechreading can dramatically improve conversational understanding, but speechreading is a skill that can be challenging to learn. To address this, we developed a novel speechreading acquisition framework that can be used to design Speechreading Acquisition Tools (SATs) - a new type of technology to improve speechreading acquisition. We interviewed seven speechreading tutors and used thematic analysis to identify and organise the key elements of our framework. We then evaluated our framework by using it to: 1) categorise every tutor-identified speechreading teaching technique, 2) critically evaluate existing conversational aids, and 3) design three new SATs. Through the use of SATs designed using our framework, the speechreading abilities of people with hearing loss around the world should be enhanced, thereby improving the conversational foundation of their day-to-day lives.
Supplemental Material
Available for Download
- Action On Hearing Loss 2011. Facts and figures on deafness and tinnitus. (2011).Google Scholar
- Action On Hearing Loss 2015. Hearing Matters. https://www.actiononhearingloss.org.uk/supporting-you/policy-research-and-influencing/research/hearing-matters.aspx. (2015).Google Scholar
- Action On Hearing Loss 2016. Describing Deafness. https://www.actiononhearingloss.org.uk/your-hearing/about-deafness-and-hearing-loss/deafness/describing-deafness.aspx. (2016).Google Scholar
- David Armstrong, Ann Gosling, John Weinman, and Theresa Marteau. 1997. The place of inter-rater reliability in qualitative research: an empirical study. Sociology 31, 3 (1997), 597--606. Google ScholarCross Ref
- L. Armstrong. 2015. On everybody's lips. http://www.scotlipreading.org.uk/files/1914/2686/1587/On_everybodys_lips_-_report.pdf. (2015).Google Scholar
- C. Benoît, M. T. Lallouache, T. Mohamadi, and C. Abry. 1992. A set of French visemes for visual speech synthesis. In Talking Machines: Theories, Models and Designs, G. Bailly and C. Benoît (Eds.). Elsevier Science Publishers B. V., North-Holland, Amsterdam, 485--504.Google Scholar
- H. Russell Bernard. 2011. Research methods in anthropology: Qualitative and quantitative approaches. Rowman Altamira, Lanham, Maryland.Google Scholar
- C. A. Binnie. 1977. Attitude changes following speechreading training. Scandinavian Audiology 6, 1 (1977), 13--19. Google ScholarCross Ref
- Carl A. Binnie, Allen A. Montgomery, and Pamela L. Jackson. 1974. Auditory and visual contributions to the perception of consonants. Journal of speech, language, and hearing research 17, 4 (1974), 619--630. Google ScholarCross Ref
- Dan G. Blazer, Sarah Domnitz, and Catharyn T. Liverman (Eds.). 2016. Hearing Health Care for Adults: Priorities for Improving Access and Affordability. The National Academies Press, Washington, D.C.Google Scholar
- Jan Blustein and Barbara E. Weinstein. 2016. Opening the Market for Lower Cost Hearing Aids: Regulatory Change Can Improve the Health of Older Americans. American journal of public health 106, 6 (2016), 1032--1035. Google ScholarCross Ref
- Arthur Boothroyd, Liat Kishon-Rabin, and Robin Waldstein. 1995. Studies of tactile speechreading enhancement in deaf adults. In Seminars in Hearing, Vol. 16. Thieme Medical Publishers, Inc., New York, NY, 328--340. Google ScholarCross Ref
- Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77--101. Google ScholarCross Ref
- Christoph Bregler and Yochai Konig. 1994. "Eigenlips" for robust speech recognition. In Acoustics, Speech, and Signal Processing ICASSP-94., Vol. 2. IEEE, Piscataway, NJ, II--669. Google ScholarCross Ref
- Martha Emma Bruhn. 1949. The Mueller-Walle Method of Lipreading for The Hard of Hearing. Volta Bureau, Washington, D.C.Google Scholar
- Alan Bryman. 2015. Social research methods. Oxford University Press, Oxford, United Kingdom.Google Scholar
- Anna Mae Bunger. 1961. Speech reading, Jena method: a textbook with lesson plans in full development for hard of hearing adults and discussion of adaptations for hard of hearing and deaf children. The Interstate, Chicago, IL.Google Scholar
- Ruth Campbell, Barbara Dodd, and Denis K. Burnham. 1998. Hearing by eye II. Vol. 2. Psychology Press, Hove, United Kingdom.Google Scholar
- Richard Orin Cornett. 1967. Cued speech. Am. Ann. Deaf. 112, 1 (1967), 3--13.Google Scholar
- Gary S. Dell, Franklin Chang, and Zenzi M. Griffin. 1999. Connectionist models of language production: Lexical access and grammatical encoding. Cognitive Science 23, 4 (1999), 517--542. Google ScholarCross Ref
- D. Ebrahimi and H. Kunov. 1991. Peripheral vision lipreading aid. IEEE transactions on biomedical engineering 38, 10 (1991), 944--952. Google ScholarCross Ref
- Melisa Echalier. 2010. In it together: the impact of hearing loss on personal relationships. (2010).Google Scholar
- Cletus G. Fisher. 1968. Confusions among visually perceived consonants. Journal of Speech, Language, and Hearing Research 11, 4 (1968), 796--804. Google ScholarCross Ref
- Marilyn French-St. George and Richard G. Stoker. 1988. Speechreading: An historical perspective. The Volta Review 90, 5 (1988), 17--21.Google Scholar
- Stuart Gatehouse, Graham Naylor, and Clous Elberling. 2003. Benefits from hearing aids in relation to the interaction between the user and the environment. International Journal of Audiology 42, sup1 (2003), 77--85.Google ScholarCross Ref
- E. Goldstein. 2013. Sensation and perception. Cengage Learning, Independence, KY.Google Scholar
- Benjamin M. Gorman. 2016. Reducing viseme confusion in speech-reading. ACM SIGACCESS Accessibility and Computing 114 (2016), 36--43. Google ScholarDigital Library
- Beth G. Greene, David B. Pisoni, and Thomas D. Carrell. 1984. Recognition of speech spectrograms. JASA 76, 1 (1984), 32--43. Google ScholarCross Ref
- Joshua Hailpern, Karrie Karahalios, and James Halle. 2009. Creating a spoken impact: encouraging vocalization through audio visual feedback in children with ASD. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, ACM, New York, NY, 453--462. Google ScholarDigital Library
- Louise Hickson, Carly Meyer, Karen Lovelock, Michelle Lampert, and Asad Khan. 2014. Factors associated with success with hearing aids in older adults. International journal of audiology 53, sup1 (2014), S18--S27.Google ScholarCross Ref
- Shizuo Hiki and Yumiko Fukuda. 1997. Negative Effect of Homophones on Speechreading in Japanese. In Audio-Visual Speech Processing: Computational & Cognitive Science Approaches. ISCA, Baxias, France, 9--12.Google Scholar
- Janet Jeffers and Margaret Barley. 1980. Speechreading (lipreading). Charles C. Thomas Publisher, Springfield, IL.Google Scholar
- Carl J. Jensema, Ramalinga Sarma Danturthi, and Robert Burch. 2000. Time spent viewing captions on television programs. Am. Ann. Deaf. 145, 5 (2000), 464--468. Google ScholarCross Ref
- Harriet Kaplan, Scott J. Bally, and Carol Garretson. 1985. Speechreading: A way to improve understanding. Gallaudet University Press, Chicago, IL.Google Scholar
- C. E. Kinzie and R. Kinzie. 1920. The Kinzie method of speech reading. Volta Review 22 (1920), 609--19.Google Scholar
- Patrick Lucey, Terrence Martin, and Sridha Sridharan. 2004. Confusability of phonemes grouped according to their viseme classes in noisy environments. In Proc. of Australian Int. Conf. on Speech Science & Tech. Australasian Speech Science and Technology Association (ASSTA), Canberra City, ACT, Australia, 265--270.Google Scholar
- David F. Marks and Lucy Yardley. 2004. Research methods for clinical and health psychology. Sage, Thousand Oaks, CA, USA. Google ScholarCross Ref
- Dominic W. Massaro, Michael M. Cohen, Walter Schwartz, Sam Vanderhyden, and Heidi Meyer. 2013. Facilitating Speech Understanding for Hearing-Challenged Perceivers in Face-to-Face Conversation and Spoken Presentations. ICTHP (2013).Google Scholar
- .Laura Matthews. 2011. Unlimited potential? A research report into hearing loss in the workplace. (2011).Google Scholar
- Abby McCormack and Heather Fortnum. 2013. Why do people fitted with hearing aids not wear them? International Journal of Audiology 52, 5 (2013), 360--368. Google ScholarCross Ref
- Edward Bartlett Nitchie. 1919. Lip-reading Principles and Practise: A Hand-book for Teachers and for Self Instruction. Frederick A. Stokes Company, New York, NY.Google Scholar
- Catherine Pope, Sue Ziebland, and Nicholas Mays. 2000. Analysing qualitative data. Bmj 320, 7227 (2000), 114--116. Google ScholarCross Ref
- Zoe Roxburgh, James M Scobbie, and Joanne Cleland. 2015. Articulation therapy for children with cleft palate using visual articulatory models and ultrasound biofeedback. Proceedings of the 18th ICPhS, Glasgow 0858 (2015).Google Scholar
- Oliver Sacks. 2009. Seeing voices: A journey into the world of the deaf. Pan Macmillan, London, United Kingdom.Google Scholar
- Conrad Sanderson and Brian C. Lovell. 2009. Multi-region probabilistic histograms for robust and scalable identity inference. In International Conference on Biometrics. Springer, New York, NY, 199--208. Google ScholarDigital Library
- June E. Shoup. 1980. Phonological aspects of speech recognition. In Trends in speech recognition. Prentice Hall, Englewood Cliffs, NJ, 125--138.Google Scholar
- Donald G. Sims, C. Dorn, C. Clark, L. Bryant, and B. Mumford. 2002. New developments in computer assisted speechreading and auditory training. Paper presented at the American Speech-Language Hearing Association convention (2002).Google Scholar
- Nancy Tye-Murray. 2002. Conversation made easy: Speechreading and conversation training for individuals who have hearing loss (adults and teenagers). St. Louis: Central Institute for the Deaf (2002).Google Scholar
- Nancy Tye-Murray. 2014. Foundations of aural rehabilitation: Children, adults, and their family members. Nelson Education, Scarborough, ON, Canada.Google Scholar
- University of Auckland, School of Psychology 2016. Why don't we advocate multiple-coders and inter-rater reliability for TA? https://www.psych.auckland.ac.nz/en/about/our-research/research-groups/thematic-analysis/frequently-asked-questions-8.html#c83c77d6d1c625135085e489bd66e765. (2016).Google Scholar
- Hubert W Upton. 1968. Wearable eyeglass speechreading aid. American Annals of the Deaf 113, 2 (1968), 222--229.Google Scholar
- Akira Watanabe, Shingo Tomishige, and Masahiro Nakatake. 2000. Speech visualization by integrating features for the hearing impaired. IEEE Trans. Speech Audio Process. 8, 4 (2000), 454--466. Google ScholarCross Ref
- WHO 2015. Deafness and hearing loss, Fact sheet N.300. http://www.who.int/mediacentre/factsheets/fs300/en/. (2015).Google Scholar
- Lei Xie, Yi Wang, and Zhi-Qiang Liu. 2006. Lip Assistant: Visualize Speech for Hearing Impaired People in Multimedia Services. In Proc. SMC'06, Vol. 5. IEEE, Piscataway, NJ, 4331--4336. Google ScholarCross Ref
- Hanfeng Yuan, Charlotte M. Reed, and Nathaniel I. Durlach. 2005. Tactual display of consonant voicing as a supplement to lipreading. The Journal of the Acoustical society of America 118, 2 (2005), 1003--1015. Google ScholarCross Ref
Index Terms
- A Framework for Speechreading Acquisition Tools
Recommendations
MirrorMirror: A Mobile Application to Improve Speechreading Acquisition
CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing SystemsMany people around the world have difficulties in day-to-day conversation due to hearing loss. Hearing aids often fail to offer enough benefits and have low adoption rates. However, people with hearing loss find that speechreading can improve their ...
LIPPS - a virtual teacher for speechreading based on a dialog-controlled talking-head
ICCHP'10: Proceedings of the 12th international conference on Computers helping people with special needs: Part IPeople with decreasing hearing ability are more dependent on alternative personal communication channels. To read the visible articulator movements of the conversation partner is one possible solution for understanding verbal statements. After giving an ...
Harnessing AI for Speech Reconstruction using Multi-view Silent Video Feed
MM '18: Proceedings of the 26th ACM international conference on MultimediaSpeechreading or lipreading is the technique of understanding and getting phonetic features from a speaker's visual features such as movement of lips, face, teeth and tongue. It has a wide range of multimedia applications such as in surveillance, ...
Comments