Liu Sicong
Skip Abstract Section
Abstract
Non-speech sound-awareness is important to improve the quality of life for the deaf and hard-of-hearing (DHH) people. DHH people, especially the young, are not always satisfied with their hearing aids. According to the interviews with 60 young hard-of-hearing students, a ubiquitous sound-awareness tool for emergency and social events that works in diverse environments is desired. In this paper, we design UbiEar, a smartphone-based acoustic event sensing and notification system. Core techniques in UbiEar are a light-weight deep convolution neural network to enable location-independent acoustic event recognition on commodity smartphons, and a set of mechanisms for prompt and energy-efficient acoustic sensing. We conducted both controlled experiments and user studies with 86 DHH students and showed that UbiEar can assist the young DHH students in awareness of important acoustic events in their daily life.
- Audio Analytic. 2017. https://www.audioanalytic.com/. (2017).Google Scholar
- James Bergstra, Norman Casagrande, Dumitru Erhan, Douglas Eck, and Balázs Kégl. 2006. Aggregate features and AdaBoost for music classification. Machine Learning 65, 2-3 (2006), 473--484. Google ScholarDigital Library
- Sourav Bhattacharya and Nicholas D Lane. 2016. Sparsification and Separation of Deep Learning Layers for Constrained Resource Inference on Wearables. In Proc. SenSys. ACM, 176--189. Google ScholarDigital Library
- Danielle Bragg, Nicholas Huynh, and Richard E Ladner. 2016. A Personalizable Mobile Sound Detector App Design for Deaf and Hard-of-Hearing Users. In Proc. ASSETS. ACM, 3--13. Google ScholarDigital Library
- J-F Cardoso. 1998. Multidimensional independent component analysis. In Proc. ICASSP, Vol. 4. IEEE, 1941--1944.Google ScholarCross Ref
- Andrew Collette. 2015. HDF5 for Python. http://www.h5py.org/. (2015).Google Scholar
- Simon Dixon. 2006. Onset detection revisited. In Proc. DAFx.Google Scholar
- Google. 2016. android.util.LruCache. https://developer.android.com/reference/android/util/LruCache.html. (2016).Google Scholar
- Benjamin M Gorman. 2014. VisAural:: a wearable sound-localisation device for people with impaired hearing. In Proc. ASSETS. ACM, 337--338. Google ScholarDigital Library
- Song Han, Huizi Mao, and William J Dally. 2016. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. In Proc. ICLR.Google Scholar
- Toni Heittola, Annamaria Mesaros, Tuomas Virtanen, and Antti Eronen. 2011. Sound event detection in multisource environments using source separation. In Proc. CHiME. 36--40.Google Scholar
- F Ho-Ching, Jennifer Mankoff, and James A Landay. 2003. Can you see what i hear?: the design and evaluation of a peripheral sound display for the deaf. In Proc. CHI. ACM, 161--168. Google ScholarDigital Library
- Forrest N Iandola, Matthew W Moskewicz, Khalid Ashraf, Song Han, William J Dally, and Kurt Keutzer. 2016. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and< 1MB model size. arXiv preprint arXiv:1602.07360 (2016).Google Scholar
- Keisuke Imoto and Nobutaka Ono. 2015. Acoustic scene analysis from acoustic event sequence with intermittent missing event. In Proc. ICASSP. IEEE, 156--160.Google ScholarCross Ref
- Leeo Inc. 2017. Meet the Leeo Smart Alert Nightlight. https://www.leeo.com/meet-the-leeo-smart-alert-nightlight/. (2017).Google Scholar
- Dhruv Jain, Leah Findlater, Jamie Gilkeson, Benjamin Holland, Ramani Duraiswami, Dmitry Zotkin, Christian Vogler, and Jon E Froehlich. 2015. Head-Mounted Display Visualizations to Support Sound Awareness for the Deaf and Hard of Hearing. In Proc. CHI. ACM, 241--250. Google ScholarDigital Library
- Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional architecture for fast feature embedding. In Proc. MM. ACM, 675--678. Google ScholarDigital Library
- Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. 2014. Large-scale video classification with convolutional neural networks. In Proc. CVPR. IEEE, 1725--1732. Google ScholarDigital Library
- Keras. 2016. Keras: Deep Learning library for Theano and TensorFlow. https://keras.io/. (2016).Google Scholar
- Hamed Ketabdar and Tim Polzehl. 2009. Tactile and Visual Alerts for Deaf People by Mobile Phones. In Proc. ASSETS. ACM, 253--254. Google ScholarDigital Library
- Adel Khalil, James Sun, Yu Zhang, and Gordon Poole. 2014. RTM noise attenuation and image enhancement using time-shift gathers. In Proc. EAGE Conference and Exhibition.Google Scholar
- George Kimura. 2017. Ambient Noise Database. http://www.ntt-at.com/product/noise-DB/. (2017).Google Scholar
- Sergei Kochkin. 2000. MarkeTrak V: Why my hearing aids are in the drawer: The consumers’ perspective. The Hearing Journal 53, 2 (2000), 34--36.Google ScholarCross Ref
- Gregoire Lafay. 2017. IEEE DCASE 2016 Challenge-Task 2-Train/Development Datasets. https://archive.org/details/dcase2016_task2_train_dev. (2017). Published February 10, 2016.Google Scholar
- Nicholas D Lane, Sourav Bhattacharya, Petko Georgiev, Claudio Forlivesi, Lei Jiao, Lorena Qendro, and Fahim Kawsar. 2016. DeepX: A software accelerator for low-power deep learning inference on mobile devices. In Proc. IPSN. ACM, 1--12. Google ScholarDigital Library
- Nicholas D Lane, Petko Georgiev, and Lorena Qendro. 2015. DeepEar: robust smartphone audio sensing in unconstrained acoustic environments using deep learning. In Proc. UbiComp. ACM, 283--294. Google ScholarDigital Library
- Joseph Lee Rodgers and W Alan Nicewander. 1988. Thirteen ways to look at the correlation coefficient. The American Statistician (1988), 59--66.Google Scholar
- Andy Liaw and Matthew Wiener. 2002. Classification and regression by randomForest. R news 2, 3 (2002), 18--22.Google Scholar
- Min Lin, Qiang Chen, and Shuicheng Yan. 2013. Network in network. arXiv preprint arXiv:1312.4400 (2013).Google Scholar
- Baoyuan Liu, Min Wang, Hassan Foroosh, Marshall Tappen, and Marianna Pensky. 2015. Sparse convolutional neural networks. In Proc. CVPR. IEEE, 806--814.Google Scholar
- Sicong Liu and Junzhao Du. 2016. Poster: MobiEar-Building an Environment-independent Acoustic Sensing Platform for the Deaf using Deep Learning. In Proc. MobiSys. ACM, 50--50. Google ScholarDigital Library
- Nitin N Lokhande, Navnath S Nehe, and Pratap S Vikhe. 2012. Voice activity detection algorithm for speech recognition applications. In Proc. ICCIA. IJCA.Google Scholar
- Hong Lu, Wei Pan, Nicholas D Lane, Tanzeem Choudhury, and Andrew T Campbell. 2009. SoundSense: scalable sound sensing for people-centric applications on mobile phones. In Proc. MobiSys. ACM, 165--178. Google ScholarDigital Library
- Hong Lu, Jun Yang, Zhigang Liu, Nicholas D Lane, Tanzeem Choudhury, and Andrew T Campbell. 2010. The Jigsaw continuous sensing engine for mobile phone applications. In Proc. SenSys. ACM, 71--84. Google ScholarDigital Library
- Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research Nov (2008), 2579--2605.Google Scholar
- Tara Matthews, Janette Fong, and Jennifer Mankoff. 2005. Visualizing non-speech sounds for the deaf. In Proc. ASSETS. ACM, 52--59. Google ScholarDigital Library
- Abby McCormack and Heather Fortnum. 2013. Why do people fitted with hearing aids not wear them? International Journal of Audiology 52, 5 (2013), 360--368.Google ScholarCross Ref
- Annamaria Mesaros, Toni Heittola, Antti Eronen, and Tuomas Virtanen. 2010. Acoustic event detection in real life recordings. In Proc. EUSIPCO. IEEE, 1267--1271.Google Scholar
- Matthias Mielke and Rainer Brueck. 2015. Design and evaluation of a smartphone application for non-speech sound awareness for people with hearing loss. In Proc. EMBC. IEEE, 5008--5011.Google ScholarCross Ref
- OtoSense. 2017. https://www.otosense.com/. (2017).Google Scholar
- Jouni Paulus and Tuomas Virtanen. 2005. Drum transcription with non-negative spectrogram factorisation. In Proc. EUSIPCO. IEEE, 1--4.Google Scholar
- Karol J Piczak. 2015. Environmental sound classification with convolutional neural networks. In Proc. MLSP. IEEE, 1--6.Google ScholarCross Ref
- Ilyas Potamitis, Stavros Ntalampiras, Olaf Jahn, and Klaus Riede. 2014. Automatic bird sound detection in long real-field recordings: Applications and tools. Applied Acoustics 80 (2014), 1--9.Google ScholarCross Ref
- Tauhidur Rahman, Alexander Travis Adams, Mi Zhang, Erin Cherry, Bobby Zhou, Huaishu Peng, and Tanzeem Choudhury. 2014. BodyBeat: a mobile system for sensing non-speech body sounds.. In Proc. MobiSys. ACM, 2--13. Google ScholarDigital Library
- Ann Mette Rekkedal. 2012. Assistive hearing technologies among students with hearing impairment: Factors that promote satisfaction. Journal of Deaf Studies and Deaf Education 17, 4 (2012), 499--517.Google ScholarCross Ref
- Android Studio. 2017. Battery Historian Charts. https://developer.android.com/studio/profile/battery-historian-charts.html. (2017).Google Scholar
- Denis Tomé, Luca Bondi, Luca Baroffio, Stefano Tubaro, Emanuele Plebani, and Danilo Pau. 2016. Reduced memory region based deep Convolutional Neural Network detection. In Proc. ICCE-Berlin. IEEE, 15--19.Google ScholarCross Ref
- Emmanuel Vincent, Nancy Bertin, Rémi Gribonval, and Frédéric Bimbot. 2014. From blind to guided audio source separation: How models and side information can improve the separation of sound. Signal Processing Magazine 31, 3 (2014), 107--115.Google ScholarCross Ref
- In-Chul Yoo and Dongsuk Yook. 2008. Automatic sound recognition for the hearing impaired. Transactions on Consumer Electronics (2008), 2029--2036. Google ScholarDigital Library
Index Terms
- UbiEar: Bringing Location-independent Sound Awareness to the Hard-of-hearing People with Smartphones
Recommendations
Tender: Smartphone Application for Braille Reading
ACIT '19: Proceedings of the 7th ACIS International Conference on Applied Computing and Information TechnologyVisually impaired people use Braille in their daily life. Braille text is printed in many ways like a handrail of stairs, guide maps, and so on. Visually impaired people read these Braille guides to navigate themselves. However, misprinted Braille may ...
ShadowRine: Accessible game for blind users, and accessible action RPG for visually impaired gamers
2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC)Though some games for visually impaired persons have been developed, most of games that use only auditory information present challenges for sighted persons. Moreover, unfortunately, it is still difficult for visually impaired persons to play the same ...
Comments