Impact of Using Tools in an Undergraduate Software Testing Course Supported by WReSTT

Abstract

Software continues to affect a major part of our daily lives, including the way we use our phones, home appliances, medical devices, and cars. The pervasiveness of software has led to a growing demand for software developers over the next decade. To ensure the high quality of software developed in industry, students being trained in software engineering also need to be trained on how to use testing techniques and supporting tools effectively at all levels of development.

In this article, we investigate how testing tools are used in the software project of an undergraduate testing course. We also investigate how a cyberlearning environment—the Web-Based Repository of Software Testing Tutorials (WReSTT)—is used to supplement the learning materials presented in class, particularly the tutorials on different software testing tools. The results of a study spanning three semesters of the undergraduate course suggest that (1) the use of code coverage tools motivates students to improve their test suites; (2) the number of bugs found when using coverage tools slightly increased, which is similar to the results found in the research literature; and (3) students find WReSTT to be a useful resource for learning about software testing techniques and the use of code coverage tools.

References

1. AAAS and NSF. 2013. Transforming Undergraduate Education in STEM: Making and Measuring Impacts. In Proceedings of the 2013 TUES Principal Investigators (Pls) Conference. http://ccliconference.org/files/2013/01/2013-Tues-Conference-Program.pdf
2. Kalle Aaltonen, Petri Ihantola, and Otto Seppälä. 2010. Mutation analysis vs. code coverage in automated assessment of students’ testing skills. In Proceedings of the ACM International Conference Companion on Object Oriented Programming Systems Languages and Applications Companion (OOPSLA’10). ACM, New York, NY, 153--160. DOI:http://dx.doi.org/10.1145/1869542.1869567
3. ACM/IEEE-CS Interim Review Task Force. 2008. Computer Science Curriculum 2008: An Interim Revision of CS 2001. Retrieved June 12, 2017, from http://www.acm.org/education/curricula/ComputerScience 2008.pdf.
4. Paul Ammann and Jeff Offutt. 2008. Introduction to Software Testing. Cambridge University Press, New York, NY.
5. Robert V. Binder. 1999. Testing Object-Oriented Systems: Models, Patterns, and Tools. Addison Wesley Longman, Boston, MA.
6. P. Bourque and R. Dupuis. 2004. Guide to the Software Engineering Body of Knowledge 2004 Version. IEEE, Los Alamitos, California.
7. Tom Britton, Lisa Jeng, Graham Carver, and Paul Cheak. 2013. Reversible Debugging Software “Quantify the Time and Cost Saved Using Reversible Debuggers.” Available at http://citeseerx.ist.psu.edu/viewdoc/summary? doi=10.1.1.370.9611
8. Bernd Bruegge and Allen H. Dutoit. 2009. Object-Oriented Software Engineering Using UML, Patterns, and Java (3rd ed.). Prentice Hall, Upper Saddle River, NJ.
9. J. C. Carver and N. A. Kraft. 2011. Evaluating the testing ability of senior-level computer science students. In Proceedings of the 24th IEEE-CS Conference on CSEET. IEEE, Los Alamitos, CA, 169--178. DOI:http://dx.doi.org/10.1109/CSEET.2011.5876084
10. Peter J. Clarke, Debra Davis, Raymond Chang-Lau, and Tariq King. 2014a. Observations on student use of tools in an undergraduate testing class. In 121st American Society for Engineering Education (ASEE)—Software Engineering Constituent Committee Division Track (SWECC). Paper No. 10123. ASEE, Washington DC.
11. Peter J. Clarke, Debra Davis, Tariq M. King, Jairo Pava, and Edward L. Jones. 2014b. Integrating testing into software engineering courses supported by a collaborative learning environment. Transactions on Computing Education 14, 3, Article No. 18. DOI:http://dx.doi.org/10.1145/2648787
12. Peter J. Clarke, Jairo Pava, Debra Davis, and Tariq M. King. 2012. Using WReSTT in SE courses: An empirical study. In Proceedings of the 43rd SIGCSE Conference. ACM, New York, NY, 307--312.
13. CodeCover Team. 2011. CodeCover Home Page. Retrieved June 12, 2017, from http://codecover.org/
14. Jacob Cohen. 1988. Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Lawrence Erlbaum, NJ.
15. John W. Creswell. 2004. Educational Research: Planning, Conducting, and Evaluating Quantitative and Qualitative Research (2nd ed.). Prentice Hall, Upper Saddle River, NJ.
16. Sebastian Deterding, Dan Dixon, Rilla Khaled, and Lennart Nacke. 2011. From game design elements to gamefulness: Defining “gamification.” In Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments (MindTrek’11). ACM, New York, NY, 9--15. DOI:http://dx.doi.org/10.1145/2181037.2181040
17. E. W. Dijkstra. 1970. Software Engineering Techniques. Report on a conference sponsored by the NATO Science Committee, Rome, Italy, October 1969.
18. T. Dvornik, D. S. Janzen, J. Clements, and O. Dekhtyar. 2011. Supporting introductory test-driven labs with WebIDE. In Proceedings of the 24th IEEE-CS Software Engineering Education and Training Conference (CSEET’11). IEEE, Los Alamitos, CA, 51--60.
19. Stephen Edwards. 2015. Web-CAT: The Web-Based Center for Automated Testing. http://web-cat.org/.
20. Stephen H. Edwards. 2004. Using software testing to move students from trial-and-error to reflection-in-action. In Proceedings of the 35th SIGCSE Technical Symposium on Computer Science Education (SIGCSE’04). ACM, New York, NY, 26--30. DOI:http://dx.doi.org/10.1145/971300.971312
21. Stephen H. Edwards and Zalia Shams. 2014. Comparing test quality measures for assessing student-written tests. In Companion Proceedings of the 36th International Conference on Software Engineering (ICSE Companion’14). ACM, New York, NY, 354--363. DOI:http://dx.doi.org/10.1145/2591062.2591164
22. FitNesse Team. 2015. FitNesse Home Page. Retrieved June 12, 2017, from http://www.fitnesse.org/.
23. P. G. Frankl and E. J. Weyuker. 1993. A formal analysis of the fault-detecting ability of testing methods. IEEE Transactions on Software Engineering 19, 3, 202--213. http://dx.doi.org/10.1109/32.221133
24. S. Frezza. 2002. Integrating testing and design methods for undergraduates: Teaching software testing in the context of software design. In Proceedings of the 32nd Annual Conference on Frontiers in Education (FIE’02). Vol. 3. IEEE, Los Alamitos, CA.
25. E. Gamma and K. Beck. 2015. JUnit Home Page. Retrieved June 12, 2017, from http://www.junit.org/.
26. V. Garousi. 2011. Incorporating real-world industrial testing projects in software testing courses: Opportunities, challenges, and lessons learned. In Proceedings of the 24th IEEE-CS Conference on CSEET. IEEE, Los Alamitos, CA, 396--400. DOI:http://dx.doi.org/10.1109/CSEET.2011.5876112
27. Michael Hackett. 2013. The changing landscape of software testing. LogiGear Magazine 7, 1, 7--15. http://www.logigear.com/magazine/wp-content/uploads/2013/02/LogiGear-Magazine-The-Rapidly-Changing-Testing-Landscape1.pdf.
28. Joachim Hofer. 2010. eCobertura Home Page. (August2010). Retrieved June 12, 2017, from http://ecobertura.johoop.de/.
29. Marc R. Hoffmann, Brock Janiczak, and Evgeny Mandrikov. 2015. EclEmma Home Page. Retrieved June 12, 2017, from http://www.eclemma.org/.
30. IBM. 2015a. Rational Functional Tester. Retrieved June 12, 2017, from http://www-01.ibm.com/software/awdtools/tester/functional/.
31. IBM. 2015b. Rational Test RealTime. Retrieved June 12, 2017, from http://www-03.ibm.com/software/products/en/realtime.
32. Laura Inozemtseva and Reid Holmes. 2014. Coverage is not strongly correlated with test suite effectiveness. In Proceedings of the 36th International Conference on Software Engineering (ICSE’14). ACM, New York, NY, 435--445. DOI:http://dx.doi.org/10.1145/2568225.2568271
33. David S. Janzen and Hossein Saiedian. 2006. Test-driven learning: Intrinsic integration of testing into the CS/SE curriculum. ACM SIGCSE Bulletin 38, 1, 254--258. DOI:http://dx.doi.org/10.1145/1124706.1121419
34. Edward L. Jones. 2000. Software testing in the computer science curriculum—a holistic approach. In Proceedings of the Australasian Conference on Computing Education (ACSE’00). ACM, New York, NY, 153--157.
35. C. Kaner and S. Padmanabhan. 2007. Practice and transfer of learning in the teaching of software testing. In Proceedings of the 20th Conference on Software Engineering Education Training (CSEET’07). IEEE, Los Alamitos, CA, 157--166. DOI:http://dx.doi.org/10.1109/CSEET.2007.38
36. Alex Koohang. 2004. Creating learning objects in collaborative e-learning settings. Issues in Information Systems 4, 2, 584--590. http://www.iacis.org/iis/2004/Koohang.pdf
37. Ilaria Liccardi, Asma Ounnas, Reena Pau, Elizabeth Massey, Päivi Kinnunen, Sarah Lewthwaite, Marie-Anne Midy, and Chandan Sarkar. 2007. The role of social networks in students’ learning experiences. ACM SIGCSE Bulletin 39, 4, 224--237. http://doi.acm.org/10.1145/1345375.1345442
38. Thomas W. Malone. 1980. What makes things fun to learn? Heuristics for designing instructional computer games. In Proceedings of the 3rd ACM SIGSMALL Symposium and the 1st SIGPC Symposium on Small Systems (SIGSMALL’80). ACM, New York, NY, 162--169. DOI:http://dx.doi.org/10.1145/800088.802839
39. Aditya P. Mathur. 2008. Foundations of Software Testing. Pearson Education, Delhi, India.
40. John D. McGregor and David A. Sykes. 2001. A Practical Guide to Testing Object-Oriented Software. Addison Wesley Longman, Boston, MA.
41. NUnit Team. 2015. NUnit Home Page. Retrieved June 12, 2017, from http://www.nunit.org/.
42. Selenium Development Team. 2014. SeleniumHQ Home Page. Retrieved June 12, 2017, from http://www.seleniumhq.org/.
43. Charlie Y. Shim, Mina Choi, and Jung Yeop Kim. 2009. Promoting collaborative learning in software engineering by adapting the PBL strategy. In Proceedings of the WASET International Conference on Computer and Information Technology (ICCIT’09). IEEE, Los Alamitos, CA, 1167--1170.
44. Barbara Leigh Smith and Jean T. MacGregor. 1992. What is collaborative learning? In Collaborative Learning: A Sourcebook for Higher Education, A. S. Goodsell, M. R. Maher, and V. Tinto (Eds.). National Center on Postsecondary Teaching, Learning, and Assessment, University Park, PA.
45. Rachel S. Smith. 2004. Guidelines for Authors of Learning Objects. New Media Consortium. Available at https://www.nmc.org/publication/guidelines-for-authors-of-learning-objects.
46. Jean Tesser. 2010. Dependency Finder. Retrieved June 12, 2017, from http://depfind.sourceforge.net.
47. Wikipedia. 2013. Category:Software testing tools. Retrieved June 12, 2017, from http://en.wikipedia.org/wiki/Category:Software_testing_tools.
48. WReSTT Team. 2015. WReSTT: Web-Based Repository for Software Testing Tutorials. Retrieved June 12, 2017, from http://wrestt.cis.fiu.edu/.
49. Qian Yang, J. Jenny Li, and David Weiss. 2007. A survey of coverage-based testing tools. Computer Journal 52, 5, 589--597.
50. Hong Zhu. 1996. A formal analysis of the subsume relation between software test adequacy criteria. IEEE Transactions on Software Engineering 22, 4, 248--255. DOI:http://dx.doi.org/10.1109/32.491648
51. Hong Zhu, Patrick A. V. Hall, and John H. R. May. 1997. Software unit test coverage and adequacy. ACM Computing Surveys 29, 366--427.

Index Terms

• Applied computing

  • Education

    • Collaborative learning

• Social and professional topics

  • Professional topics

    • Computing education

      • Computing education programs

        • Software engineering education

• Software and its engineering

  • Software creation and management

    • Software verification and validation

      • Software defect analysis

        • Software testing and debugging