Matt Huenerfauth
Liming Zhao
Abstract
There are many important factors in the design of evaluation studies for systems that generate animations of American Sign Language (ASL) sentences, and techniques for evaluating natural language generation of written texts are not easily adapted to ASL. When conducting user-based evaluations, several cultural and linguistic characteristics of members of the American Deaf community must be taken into account so as to ensure the accuracy of evaluations involving these users. This article describes an implementation and user-based evaluation (by native ASL signers) of a prototype ASL natural language generation system that produces sentences containing classifier predicates, which are frequent and complex spatial phenomena that previous ASL generators have not produced. Native signers preferred the system's output to Signed English animations -- scoring it higher in grammaticality, understandability, and naturalness of movement. They were also more successful at a comprehension task after viewing the system's classifier predicate animations.
- Badler, N., Allbeck, J., Lee, S. J., Rabbitz, R., Broderick, T., and Mulkern, K. 2005. New behavioral paradigms for virtual human models. In Proceedings of the SAE Digital Human Modeling Conference.]]Google Scholar
- Badler, N., Bindiganavale, R., Allbeck, J., Schuler, W., Zhao, L., Lee, S., Shin, H., and Palmer, M. 2000. Parameterized action representation & natural language instructions for dynamic behavior modification of embodied agents. In Proceedings of the AAAI Spring Symposium.]]Google Scholar
- Bangalore, S., Rambow, O., and Whittaker, S. 2000. Evaluation Metrics for Generation. In Proceedings of the First International Conference on Natural Language Generation.]] Google ScholarDigital Library
- Coyne, R. and Sproat, R. 2001. WordsEye: An automatic text-to-scene conversion system. In Proceedings of the 28th International Conference and Exhibition on Computer Graphics and Interactive Techniques (SIGGRAPH-2001) (Los Angeles, CA). ACM, New York.]] Google ScholarDigital Library
- Emmorey, K., (ed.). 2003. Perspectives on Classifier Constructions in Sign Languages. Lawrence Erlbaum Associates, Mahwah, NJ.]]Google Scholar
- Holt, J. A. 1993. Demographic, stanford achievement test - 8th edition for deaf and hard of hearing students: Reading comprehension subgroup results. Amer. Annals Deaf. 138, 172--175.]]Google ScholarCross Ref
- Huenerfauth, M. 2006a. Representing coordination and non-coordination in American Sign Language animations. Behaviour & Information Technology, 25:4.]]Google ScholarCross Ref
- Huenerfauth, M. 2006b. Generating American Sign Language classifier predicates for English-to-ASL machine translation. Dissertation, Department of Computer and Information Science, University of Pennsylvania.]] Google ScholarDigital Library
- Huenerfauth, M. 2008. Animations from Article in ACM Transactions on Accessible Computing (TACCESS). Linguistic and Assistive Technologies Laboratory website. (Accessed on 28 March 2008.) http://latlab.cs.qc.cuny.edu/taccess2008/.]]Google Scholar
- Huenerfauth, M. In Press. Representing ASL Classifier Predicates Using Spatially Parameterized Planning Templates. In Generalization of Knowledge: Multidisciplinary Perspectives. M. T. Banich and D. Caccamise, Eds., Psychology Press, New York, NY.]]Google Scholar
- Huenerfauth, M., Zhao, L., Gu, E., and Allbeck, J. 2007a. Design and evaluation of an American Sign Language generator. In Proceedings of the Workshop on Embodied Language Processing, 45th Annual Meeting of the Association for Computational Linguistics. Prague, Czech Republic.]] Google ScholarDigital Library
- Huenerfauth, M., Zhao, L., Gu, E., and Allbeck, J. 2007b. Evaluation of American Sign Language generation through the participation of native ASL signers. In Proceedings of the 9th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS-2007) (Tempe, AZ). ACM, New York.]] Google ScholarDigital Library
- Liddell, S. 2003. Grammar, Gesture, and Meaning in American Sign Language. Cambridge University Press, Cambridge, UK.]]Google Scholar
- Lucas, C. and Valli, C. 1992. Language Contact in the American Deaf Community. Academic Press, San Diego, CA.]]Google Scholar
- Mitchell, R. E., Young, T. A., Bachleda, B., and Karchmer, M. A. 2006. How many people use ASL in the United States? Why estimates need updating. Sign Language Studies, 6:3.]]Google ScholarCross Ref
- Morrissey, S. and Way, A. 2006. Lost in translation: The problems of using mainstream MT evaluation metrics for sign language translation. In Proceedings of the 5th SALTMIL Workshop on Minority Languages, Language Resources and Evaluation Conference (LREC-2006).]]Google Scholar
- Neidle, C., Kegl, J., MacLaughlin, D., Bahan, B., and Lee, R. G. 2000. The Syntax of American Sign Language: Functional Categories and Hierarchical Structure. The MIT Press, Cambridge, MA.]]Google Scholar
- Padden, C. and Humphries, T. 2005. Inside Deaf Culture. Harvard University Press, Cambridge, MA.]]Google Scholar
- Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of Association for Computational Linguistics.]] Google ScholarDigital Library
- Pasquariello, S. and Pelachaud, C. 2001. Greta: A simple facial animation engine. In Proceedings of the 6th Online World Conference on Soft Computing in Industrial Applications.]]Google Scholar
- Sáfár, É. and Marshall, I. 2001. The architecture of an English-text-to-Sign-Languages translation system. In Recent Advances in Natural Language Processing.]]Google Scholar
- Sandler, W. and Lillo-Martin, D. 2006. Sign Language and Linguistic Universals. Cambridge University Press, Cambridge, UK.]]Google Scholar
- Strong, M. and Prinz, P. 1997. A study of the relationship between American Sign Language and English literacy. J. Deaf Stud. Deaf Educat. 2, 1, 37--46.]]Google ScholarCross Ref
- Supalla, T. 1978. Morphology of verbs of motion and location. In Proceedings of the 2nd National Symposium on Sign Language Research and Teaching, F. Caccamise and D. Hicks (eds.), National Association for the Deaf, Silver Spring, MD. pp. 27--45.]]Google Scholar
- Zhao, L., Kipper, K., Schuler, W., Vogler, C., Badler, N. I., and Palmer, M. 2000. Machine translation system from English to American Sign Language. In Proceedings of the Meeting of the Association for Machine Translation in the Americas.]] Google ScholarDigital Library
- Zhao, L., Liu, Y., and Badler, N. I. 2005. Applying empirical data on upper torso movement to real-time collision-free reach tasks. In Proceedings of the 2005 SAE Digital Human Modeling for Design and Engineering Conference and Exhibition.]]Google Scholar
Index Terms
- Evaluation of American Sign Language Generation by Native ASL Signers
Recommendations
Evaluating American Sign Language generation through the participation of native ASL signers
Assets '07: Proceedings of the 9th international ACM SIGACCESS conference on Computers and accessibilityWe discuss important factors in the design of evaluation studies for systems that generate animations of American Sign Language (ASL) sentences. In particular, we outline how some cultural and linguistic characteristics of members of the American Deaf ...
A Linguistically Motivated Model for Speed and Pausing in Animations of American Sign Language
Many deaf adults in the United States have difficulty reading written English text; computer animations of American Sign Language (ASL) can improve these individuals’ access to information, communication, and services. Planning and scripting the ...
Evaluation of a psycholinguistically motivated timing model for animations of american sign language
Assets '08: Proceedings of the 10th international ACM SIGACCESS conference on Computers and accessibilityUsing results in the psycholinguistics literature on the speed and timing of American Sign Language (ASL), we built algorithms to calculate the time-duration of signs and the location/length of pauses during an ASL animation. We conducted a study in ...
Comments