Educational violin transcription by fusing multimedia streams

ABSTRACT

Computer-assisted violin tutoring requires accurate violin transcription. For pitched non-percussive (PNP) sounds such as from the violin, note segmentation is a much more difficult task than pitch detection. This issue is accentuated when the audio is recorded during an instrument practice session at home which is acoustically inferior to a professional recording studio. This paper presents a new approach to the problem by using the correlation between different media streams for e-learning applications. We design a capture mechanism to record one audio and two video streams simultaneously, and exploit the relationships among them for enhanced transcription. State-of-the-art audio methods for note segmentation and pitch estimation are implemented as the audio-only baseline. Two web-cameras are employed to track the right hand (bowing) and the left hand's four fingers (fingering) on the fingerboard, respectively. The audio and visual information is then fused in the feature space. Our new approach is evaluated with an audio-visual violin music database containing 16 complete music pieces of different styles with 2157 notes in total. Experimental results show that our multimodal approach achieves a 10% increase in true positives, and a 8% reduction in false positives of overall transcription performance in comparison with the audio-only baseline.

References

1. Yin J., Wang Y. and Hsu D., Digital Violin Tutor: An Integrated System for Beginning Violin Learners, ACM Multimedia Conf., 2005.
2. Perkins, D., Smart Schools: Better Thinking and Learning for Every Child, The Free Press, New York, 1992.
3. Collins, N., A Comparison of Sound Onset Detection Algorithms with Emphasis on Psycho-Acoustically Motivated Detection Functions, Journal of the Audio Engineering Society, 2005.
4. Bello, J. B., Daudet, L., Samer, A., Duxbury, C., Davies, M. and Sandler, M. B., A Tutorial on Onset Detection in Music Signals, IEEE Trans. on Speech and Audio Processing, pp: 1035--1047, 2005.
5. Gillet O. and Richard G., Automatic Transcription of Drum Sequences using Audiovisual Features. Proc. IEEE Int. Conf. on Acoustics, Speech, and Signal Processing, 2005.
6. Baader A. P., Kazennikov O., and Wiesendanger M., Coordination of bowing and fingering in violin playing, Cognitive Brain Research, pp:436--1C443, 2005.
7. Nakamura, S., Statistcal Multimodal Integration for Audio-Visual Speech Processing, IEEE Trans. on Neural Networks, pp:854--866, 2002.
8. Garg, A., Potamianos, G., Neti, C., and Huang, T. S., Frame-dependent multi-stream reliability indicators for audio-visual speech recognition, Proc. IEEE Int. Conf. Multimedia and Expo (ICME03), pp:605--608, 2003.
9. Kaynak, M. N., Qi Z., Cheok, A. D., Sengupta, K., Zhang J., Ko C. C., Analysis of lip geometric features for audio-visual speech recognition, IEEE Trans. Systems, Man, and Cybernetics, pp:564--570, 2004.
10. Fragopanagos, N. and Taylor, J. G., Emotion recognition in human-computer interaction, Neural Network, pp:389--405, 2005.
11. Foote, J., Automatic Audio Segmentation Using a Measure of Audio Novelty, Proc. IEEE Int. Conf. Multimedia and Expo (ICME00), pp:452--455, 2000.
12. Collins, N., Using a Pitch Detector for Onset Detection, Proc. of ISMIR2005, 2005.
13. Boo W., Wang Y., Loscos A., A Violin Music Transcriber for Personalized Learning, IEEE Inter. Conf. on Multimedia Expo, 2006.
14. Klapuri, A., A perceptually motivated multiple-f0 estimation method, Proc. IEEE Workshop on Applications of Audio Signal Processing to Audio and Acoustics, pp: 291--294, 2005.
15. Klapuri, A., Multiple Fundamental Frequency Estimation by Summing Harmonic Amplitudes, Proc. of ISMIR2006, pp:216--221, 2006.
16. Loscos A., Wang Y., Boo W., Low Level Descriptors for Automatic Violin Transcription, Proc. of ISMIR2006, 2006.
17. Flesch C., Art of Violin Playing: Book One, Carl Fischer Music Dist, 2000.
18. Letessier J. and Brard F., Visual tracking of bare fingers for interactive surfaces, Seventeenth Annual ACM Symposium on User Interface Software and Technology, pp:119--122, 2004.
19. Burns, A. and Wanderley, Visual methods for the retrieval of guitarist fingering, Proc. of the 2006 Conf. on New interfaces For Musical Expression, pp:196--199, 2006.
20. Wu Y., Lin J., Huang T. S., Analyzing and Capturing Articulated Hand Motion in Image Sequences, IEEE Trans. Pattern Anal. Mach. Intell, pp:1910--1922, 2005.
21. Leveau, P., Daudet, L., Richard G., Methodology and Tools for the Evaluation of Automatic Onset Detection Algorithms in Music, Proc. of ISMIR2004, pp:72--75, 2004.

Index Terms

• Applied computing

  • Arts and humanities

    • Sound and music computing

• Hardware

  • Communication hardware, interfaces and storage

    • Signal processing systems

  • Robustness

    • Hardware reliability

      • Signal integrity and noise analysis