Filippo Ricca
Marco Torchiano
Skip Abstract Section
Skip Supplemental Material SectionAbstract
Over the last few years, the software engineering community has proposed a number of modeling methods to represent functional requirements. Among them, use cases are recognized as an easy to use and intuitive way to capture and define such requirements. Screen mockups (also called user-interface sketches or user interface-mockups) have been proposed as a complement to use cases for improving the comprehension of functional requirements. In this article, we aim at quantifying the benefits achievable by augmenting use cases with screen mockups in the comprehension of functional requirements with respect to effectiveness, effort, and efficiency. For this purpose, we conducted a family of four controlled experiments, involving 139 participants having different profiles. The experiments involved comprehension tasks performed on the requirements documents of two desktop applications. Independently from the participants' profile, we found a statistically significant large effect of the presence of screen mockups on both comprehension effectiveness and comprehension task efficiency. No significant effect was observed on the effort to complete tasks. The main pragmatic lesson is that the screen mockups addition to use cases is able to almost double the efficiency of comprehension tasks.
Supplemental Material
Available for Download
zip
ricca.zip (229.1 KB)
Supplemental movie, appendix, image and software files for, Assessing the Effect of Screen Mockups on the Comprehension of Functional Requirements
- S. Abrahão, C. Gravino, E. Insfran, G. Scanniello, and G. Tortora. 2013. Assessing the effectiveness of sequence diagrams in the comprehension of functional requirements: Results from a family of five experiments. IEEE Trans. Softw. Eng. 29, 3, 327--342. Google Scholar
Digital Library
- S. Adolph, A. Cockburn, and P. Bramble. 2002. Patterns for Effective Use Cases. Addison-Wesley Longman Publishing Co., Inc., Boston, MA. Google ScholarDigital Library
- A. Agresti. 2007. An Introduction to Categorical Data Analysis. Wiley-Interscience.Google Scholar
- B. Anda, D. I. K. Sjøberg, and M. Jørgensen. 2001. Quality and understandability of use case models. In Proceedings of the European Conference on Object-Oriented Programming. Berlin, Springer-Verlag, 402--428. Google ScholarDigital Library
- G. Andrew and P. Drew. 2009. Use case diagrams in support of use case modeling: Deriving understanding from the picture. J. Datab. Manage. 20, 1, 1--24.Google ScholarCross Ref
- J. Aranda, N. Ernst, J. Horkoff, and S. Easterbrook. 2007. A framework for empirical evaluation of model comprehensibility. In Proceedings of the ICSE Workshop on Modeling in Software Engineering. IEEE, 7--13. Google ScholarDigital Library
- E. Astesiano, M. Cerioli, G. Reggio, and F. Ricca. 2007. Phased highly-interactive approach to teaching UML-based software development. In Proceedings of the Symposium at MODELS 2007. 9--19.Google Scholar
- E. Astesiano and G. Reggio. 2012. A disciplined style for use case based requirement specification. Tech. Rep. DISI-TR-12-04. DISI University of Genova, Italy. http://softeng.disi.unige.it/TR/Disciplined Requirements.pdf.Google Scholar
- V. Basili, G. Caldiera, and D. H. Rombach. 1994. The Goal Question Metric Paradigm, Encyclopedia of Software Engineering. Wiley.Google Scholar
- V. Basili, F. Shull, and F. Lanubile. 1999. Building knowledge through families of experiments. IEEE Trans. Softw. Eng. IEEE, 456--473. Google ScholarDigital Library
- L. Bratthall and C. Wohlin. 2002. Is it possible to decorate graphical software design and architecture models with qualitative information?-An Experiment. IEEE Trans. Softw. Eng. 28, 12, 1181--1193. Google ScholarDigital Library
- B. Bruegge and A. H. Dutoit. 2003. Object-Oriented Software Engineering: Using UML, Patterns and Java, 2nd Ed. Prentice-Hall. Google ScholarDigital Library
- J. Carver, L. Jaccheri, S. Morasca, and F. Shull. 2003. Issues in using students in empirical studies in software engineering education. In Proceedings of the International Symposium on Software Metrics. IEEE Computer Society, Los Alamitos, CA, 239--250. Google ScholarDigital Library
- M. Ceccato, P. Di Penta, P. Falcarin, F. Ricca, M. Torchiano, and P. Tonella. 2014. A family of experiments to assess the effectiveness and efficiency of source code obfuscation techniques. Empir. Softw. Eng. 19, 4, 1040--1074. Google ScholarDigital Library
- M. Ciolkowski, D. Muthig, and J. Rech. 2004. Using academic courses for empirical validation of software development processes. In Proceedings of the EUROMICRO Conference. IEEE Computer Society, Los Alamitos, CA, 354--361. Google ScholarDigital Library
- N. Cliff. 1993. Dominance statistics: Ordinal analyses to answer ordinal questions. Psychol. Bull. 114, 3, 494--509.Google ScholarCross Ref
- A. Cockburn. 2000. Writing Effective Use Cases. Addison-Wesley, Reading, MA. Google ScholarDigital Library
- J. Conallen. 2002. Building Web Applications with UML, Second Edition. Addison-Wesley, Reading, MA. Google ScholarDigital Library
- N. Condori-Fernandez, M. Daneva, K. Sikkel, R. Wieringa, O. Dieste, and O. Pastor. 2009. Research findings on empirical evaluation of requirements specifications approaches. In Proceedings of the Workshop on Requirements Engineering. Valparaiso University Press, 121--128.Google Scholar
- O. J. Dunn. 1961. Multiple comparisons among means. J. ASA, 56, 52--64.Google Scholar
- J. Ferreira, J. Noble, and R. Biddle. 2007. Agile Development Iterations and UI Design. In Proceedings of the Agile Conference (AGILE). 50--58. Google ScholarDigital Library
- M. Fowler. 2003. UML Distilled: A Brief Guide to the Standard Object Modeling Language, 3rd Ed. Addison-Wesley Longman Publishing Co., Inc., Boston, MA. Google ScholarDigital Library
- C. Gravino, G. Scanniello, and G. Tortora. 2008. An empirical investigation on dynamic modeling in requirements engineering. In Proceedings of the Conference on Model Driven Engineering Languages and Systems. IEEE Computer Society, Los Alamitos, CA, 615--629. Google ScholarDigital Library
- J. Hannay and M. Jørgensen. 2008. The role of deliberate artificial design elements in software engineering experiments. IEEE Trans. Softw. Eng. 34, 2, 242--259. Google ScholarDigital Library
- H. R. Hartson and E. C. Smith. 1991. Rapid prototyping in human-computer interface development. Interacting with Computers 3, 1, 51--91. Google ScholarDigital Library
- D. Hendrix, J. Cross, and S. Maghsoodloo. 2002. The effectiveness of control structure diagrams in source code comprehension activities. IEEE Trans. Softw. Eng. 28, 5, 463--477. Google ScholarDigital Library
- J. J. Higgins. 2004. An Introduction to Modern Nonparametric Statistics. Brooks/Cole.Google Scholar
- L. Hochstein, V. R. Basili, M. V. Zelkowitz, J. K. Hollingsworth, and J. Carver. 2005. Combining self-reported and automatic data to improve programming effort measurement. ACM SIGSOFT Softw. Eng. Notes 30, 5, 356--365. Google ScholarDigital Library
- K. Hogarty and J. Kromrey. 2001. We've been reporting some effect sizes: Can you guess what they mean? In Proceedings of the Annual Meeting of the American Educational Research Association.Google Scholar
- L. Huang and M. Holcombe. 2009. Empirical investigation towards the effectiveness of test first programming. Inf. Softw. Tech. 51, 1, 182--194. Google ScholarDigital Library
- W. S. Humphrey. 1995. A Discipline for Software Engineering. Addison-Wesley Longman Publishing Co., Inc., Boston, MA. Google ScholarDigital Library
- ISO. 1991. Information technology--software product evaluation: Quality characteristics and guidelines for their use, ISO/IEC IS 9126, ISO, Geneva.Google Scholar
- ISO. 2000. Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 9: Requirements for non-keyboard input devices. ISO 9241-11, ISO, Geneva, Switzerland.Google Scholar
- ISO. 2011. Systems and software engineering--Systems and software quality requirements and Evaluation (SQuaRE)--System and software quality models. ISO/IEC 25010, ISO, Geneva, Switzerland.Google Scholar
- I. Jacobson. 2004. Object-Oriented Software Engineering: A Use Case Driven Approach. Addison-Wesley Longman Publishing Co., Inc., Redwood City, CA. Google ScholarDigital Library
- N. Juristo and A. M. Moreno. 2001. Basics of Software Engineering Experimentation. Kluwer Academic Publishers, Englewood Cliffs, NJ. Google ScholarDigital Library
- R. I. Kabacoff. 2011. R in Action -- Data Analysis and Graphics with R. Manning Publications Co., Shelter Island, NY. Google ScholarDigital Library
- E. Kamsties, A. von Knethen, and R. Reussner. 2003. A controlled experiment to evaluate how styles affect the understandability of requirements specifications. Inf. Softw. Tech. 45, 14, 955--965.Google ScholarCross Ref
- B. Kitchenham, H. Al-Khilidar, M. Babar, M. Berry, K. Cox, J. Keung, F. Kurniawati, M. Staples, H. Zhang, and L. Zhu. 2008. Evaluating guidelines for reporting empirical software engineering studies. Empir. Softw. Eng. 13, 97--121. Google ScholarDigital Library
- B. Kitchenham, S. Pfleeger, L. Pickard, P. Jones, D. Hoaglin, K. El Emam, and J. Rosenberg. 2002. Preliminary guidelines for empirical research in software engineering. IEEE Trans. Softw. Eng. 28, 8, 721--734. Google ScholarDigital Library
- J. A. Landay and B. A. Myers. 1995. Interactive sketching for the early stages of user interface design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM Press/Addison-Wesley Publishing Co., New York, NY, 43--50. Google ScholarDigital Library
- S. Lauesen. 2002. Software Requirements: Styles and Techniques. Addison-Wesley. Google ScholarDigital Library
- M. Lindvall, I. Rus, F. Shull, M. V. Zelkowitz, P. Donzelli, A. M. Memon, V. R. Basili, P. Costa, R. T. Tvedt, L. Hochstein, S. Asgari, C. Ackermann, and D. Pech. 2005. An evolutionary testbed for software technology evaluation. Innov. Syst. Softw. Eng. 1, 1, 3--11.Google ScholarCross Ref
- M. G. Mendonça, J. C. Maldonado, M. C. F. de Oliviera et al. 2008. A framework for software engineering experimental replications. In Proceedings of the 13th IEEE International Conference on Engineering of Complex Computer Systems (ICECCS'08). IEEE Computer Society, Los Alamitos, CA, 203--212. Google ScholarDigital Library
- B. Meyer. 1985. On formalism in specification. IEEE Softw. 3, 1, 6--25. Google ScholarDigital Library
- H. Motulsky. 2010. Intuitive Biostatistics: A Non-Mathematical Guide to Statistical Thinking. Oxford University Press.Google Scholar
- R. Mugridge and W. Cunningham. 2005. Fit for Developing Software: Framework for Integrated Tests. Prentice-Hall. Google ScholarDigital Library
- M. O'Docherty. 2005. Object-Oriented Analysis and Design: Understanding System Development with UML 2.0 1st Ed. Wiley.Google Scholar
- A. N. Oppenheim. 1992. Questionnaire Design, Interviewing and Attitude Measurement. Pinter, London, UK.Google Scholar
- S. L. Pfleeger and W. Menezes. 2000. Marketing technology to software practitioners. IEEE Softw. 17, 1, 27--33. Google ScholarDigital Library
- K. T. Phalp, J. Vincent, and K. Cox. 2007. Improving the quality of use case descriptions: Empirical assessment of writing guidelines. Softw. Qual. Cont. 15, 4, 383--399. Google ScholarDigital Library
- R Core Team. 2013. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria.Google Scholar
- F. Ricca, M. Di Penta, M. Torchiano, P. Tonella, and M. Ceccato. 2010a. How developers' experience and ability influence web application comprehension tasks supported by UML stereotypes: A Series of Four Experiments. IEEE Trans. Softw. Eng. 36, 96--118. Google ScholarDigital Library
- F. Ricca, G. Scanniello, M. Torchiano, G. Reggio, and E. Astesiano. 2010b. On the effectiveness of screen mockups in requirements engineering: results from an internal replication. In Proceedings of the International Symposium on Empirical Software Engineering and Measurement. ACM, New York, NY. Google ScholarDigital Library
- F. Ricca, G. Scanniello, M. Torchiano, G. Reggio, and E. Astesiano. 2010c. On the effort of augmenting use cases with screen mockups: results from a preliminary empirical study. In Proceedings of the International Symposium on Empirical Software Engineering and Measurement. ACM, New York. Google ScholarDigital Library
- F. Ricca, M. Torchiano, M. Di Penta, M. Ceccato, and P. Tonella. 2009. Using acceptance tests as a support for clarifying requirements: A series of experiments. Inf. Softw. Tech. 51, 2, 270--283. Google ScholarDigital Library
- J. Romano, J. D. Kromrey, J. Coraggio, and J. Skowronek. 2006. Appropriate statistics for ordinal level data: Should we really be using t-test and Cohen's d for evaluating group differences on the NSSE and other surveys? In Proceedings of the Annual Meeting of the Florida Association of Institutional Research.Google Scholar
- G. Scanniello, C. Gravino, M. Genero, J. A. Cruz-Lemus, and G. Tortora. 2014. On the impact of UML analysis models on source code comprehensibility and modifiability. Trans. Softw. Eng. Meth. 23, 2, 13:1--13:26. Google ScholarDigital Library
- G. Scanniello, C. Gravino, and G. Tortora. 2010. Investigating the role of UML in the software modeling and maintenance - A preliminary industrial survey. In Proceedings of ICEIS. 141--148.Google Scholar
- G. Scanniello, F. Ricca, M. Torchiano, C. Gravino, and G. Reggio. 2013. Estimating the effort to develop screen mockups. In Proceedings of the EUROMICRO Conference on Software Engineering and Advanced Applications. 341--348. Google ScholarDigital Library
- S. Shapiro and M. Wilk. 1965. An analysis of variance test for normality. Biometrika 52, 3--4, 591--611.Google ScholarCross Ref
- F. Shull, J. C. Carver, S. Vegas, and N. J. Juzgado. 2008. The role of replications in empirical software engineering. Empir. Softw. Eng. 13, 2, 211--218. Google ScholarDigital Library
- M. Staron, L. Kuzniarz, and C. Wohlin. 2006. Empirical assessment of using stereotypes to improve comprehension of UML models: A set of experiments. J. Syst. Softw. 79, 5, 727--742. Google ScholarDigital Library
- M. Torchiano. 2014. Effsize: Efficientc Effects Size Computation. R package version 1.2-2.Google Scholar
- M. Torchiano, F. C. A. Tomassetti, F. Ricca, A. Tiso, and G. Reggio. 2013. Relevance, benefits, and problems of software modelling and model driven techniques--A survey in the Italian industry. J. Syst. Softw. 86, 8, 2110--2126. Google ScholarDigital Library
- H. van der Linden, G. Boers, H. Tange, J. Talmon, and A. Hasman. 2003. Proper: A multi disciplinary EPR system. Int. J. Med. Inf. 70, 2--3, 149--160.Google ScholarCross Ref
- B. Wheeler. 2010. lmPerm: Permutation Tests for Linear Models. R package version 1.1-2.Google Scholar
- C. Wohlin, P. Runeson, M. Höst, M. Ohlsson, B. Regnell, and A. Wesslén. 2012. Experimentation in Software Engineering. Springer. Google ScholarDigital Library
- R. Young. 2001. Effective Requirements Practice. Addison-Wesley, Boston, MA. Google ScholarDigital Library
- T. Yue, L. C. Briand, and Y. Labiche. 2009. A use case modeling approach to facilitate the transition towards analysis models: Concepts and empirical evaluation. In Proceedings of the 12th International Conference on Model Driven Engineering Languages and Systems (MODELS'09). Springer-Verlag, Berlin, 484--498. Google ScholarDigital Library
- M. K. Zimmerman, K. Lundqvist, and N. Leveson. 2002. Investigating the readability of state-based formal requirements specification languages. In Proceedings of the International Conference on Software Engineering. ACM, New York, 33--43. Google ScholarDigital Library
Index Terms
- Assessing the Effect of Screen Mockups on the Comprehension of Functional Requirements
Recommendations
On the impact of UML analysis models on source-code comprehensibility and modifiability
We carried out a family of experiments to investigate whether the use of UML models produced in the requirements analysis process helps in the comprehensibility and modifiability of source code. The family consists of a controlled experiment and 3 ...
On the effort of augmenting use cases with screen mockups: results from a preliminary empirical study
ESEM '10: Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and MeasurementIn order to increase stakeholders' comprehension on software requirements, Use Cases can be enhanced with screen mock-ups (i.e., GUI prototypes sketched with a special conceived graphical tool). However, the effort to write Use Cases augmented with ...
On the effectiveness of screen mockups in requirements engineering: results from an internal replication
ESEM '10: Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and MeasurementIn this paper, we present and discuss the results of an internal replication of a controlled experiment for assessing the effectiveness of including screen mockups when adopting Use Cases. The results of the original experiment indicate a clear ...
Comments