The ABC of Software Engineering Research

Abstract

A variety of research methods and techniques are available to SE researchers, and while several overviews exist, there is consistency neither in the research methods covered nor in the terminology used. Furthermore, research is sometimes critically reviewed for characteristics inherent to the methods. We adopt a taxonomy from the social sciences, termed here the ABC framework for SE research, which offers a holistic view of eight archetypal research strategies. ABC refers to the research goal that strives for generalizability over Actors (A) and precise measurement of their Behavior (B), in a realistic Context (C). The ABC framework uses two dimensions widely considered to be key in research design: the level of obtrusiveness of the research and the generalizability of research findings. We discuss metaphors for each strategy and their inherent limitations and potential strengths. We illustrate these research strategies in two key SE domains, global software engineering and requirements engineering, and apply the framework on a sample of 75 articles. Finally, we discuss six ways in which the framework can advance SE research.

References

1. S. Adolph, P. Kruchten, and W. Hall. 2012. Reconciling perspectives: A grounded theory of how people manage the process of software development. J. Syst. Software 85 (2012), 1269--1286.
2. P. J. Ågerfalk and B. Fitzgerald. 2006. Flexible and distributed software processes: Old petunias in New Bowls? Commun. ACM 49, 10 (2006), 27--34.
3. E. Alégroth and R. Feldt. 2017. On the long-term use of visual GUI testing in industrial practice: A case study. Empir. Software Eng. 22, 6 (2017), 2937--2971.
4. V. Antinyan, M. Staron, and A. Sandberg. 2017. Evaluating code complexity triggers, use of complexity measures and the influence of code complexity on maintenance time. Empir. Software Eng. 22, 6 (2017), 3057--3087.
5. E. Arisholm, H. Gallis, T. Dybå, and D. I. K. Sjøberg. 2007. Evaluating pair programming with respect to system complexity and programmer expertise. IEEE Trans. Softw. Eng. 33, 2 (2007), 65--86.
6. W. K. G. Assunção, R. E. Lopez-Herrejon, L. Linsbauer, S. R. Vergilio, and A. Egyed. 2017. Multi-objective reverse engineering of variability-safe feature models based on code dependencies of system variants. Empir. Software Eng. 22, 4 (2017), 1763--1794.
7. D. Avison, F. Lau, M. Myers, and P. A. Nielsen. 1999. Action research. Commun. ACM 42, 1 (1999), 94--97.
8. M. A. Babar, B. Kitchenham, and R. Jeffery. 2008. Comparing distributed and face-to-face meetings for software architecture evaluation: A controlled experiment. Empir. Software Eng. 13, 1 (2008), 39--62.
9. M. Bano, D. Zowghi, and F. da Rimini. 2017. User satisfaction and system success: An empirical exploration of user involvement in software development. Empir. Software Eng. 22, 5 (2017), 2339--2372.
10. L. Bao, J. Li, Z. Xing, X. Wang, X. Xia, and B. Zhou. 2017. Extracting and analyzing time-series HCI data from screen-captured task videos. Empir. Software Eng. 22, 1 (2017), 134--174.
11. V. R. Basili, R. W. Selby, and D. H. Hutchens. 1986. Experimentation in software engineering. IEEE Trans. Softw. Eng. 12, 7 (1986), 733--743.
12. V. R. Basili and M. V. Zelkowitz. 2007. Empirical studies to build a science of computer science. Commun. ACM 50, 11 (2007), 33--37.
13. B. C. D. Anda, D. I. K. Sjøberg, and A. Mockus. 2009. Variability and reproducibility in software engineering: A study of four companies that developed the same system. IEEE Trans. Softw. Eng. 35, 3 (2009), 407--429.
14. S. Beecham, N. Baddoo, T. Hall, H. Robinson, and H. Sharp. 2008. Motivation in software engineering: A systematic literature review. Inform. Software Tech. 50, 9--10 (2008), 860--878.
15. P. Behnamghader, D. M. Le, J. Garcia, D. Link, A. Shahbazian, and N. Medvidovic. 2017. A large-scale study of architectural evolution in open-source software systems. Empir. Software Eng. 22, 3 (2017), 1146--1193.
16. M. Beller, A. Zaidman, A. Karpov, and R. A. Zwaan. 2017. The last line effect explained. Empir. Software Eng. 22, 3 (2017), 1508--1536.
17. D. M. Berry and W. F. Tichy. 2003. Comments on “Formal methods application: An empirical tale of software development.” IEEE Trans. Software Eng. 29, 6 (2003), 567--571.
18. C.-P. Bezemer, S. McIntosh, B. Adams, D. M. German, and A. E. Hassan. 2017. An empirical study of unspecified dependencies in make-based build systems. Empir. Software Eng. 22, 6 (2017), 3117--3148.
19. N. Bos, N. Sadat Shami, J. S. Olson, A. Cheshin, and N. Nan. 2004. In-group/out-group effects in distributed teams: An experimental simulation. In Proc. International Conference on Computer-Supported Cooperative Work and Social Computing (CSCW’04). ACM, New York, 429--436.
20. S. S. Brilliant and J. C. Knight. 1999. Empirical research in software engineering: A workshop. ACM SIGSOFT Software Eng. Notes 24, 3 (1999), 44--52.
21. A. Bryant. 2000. “It’s engineering jim … but not as we know it”: Software engineering - solution to the software crisis, or part of the problem? In Proc. International Conference on Software Engineering, 77--86.
22. M. Caneill, D. M. Germán, and S. Zacchiroli. 2017. The debsources dataset: Two decades of free and open source software. Empir. Software Eng. 22, 3 (2017), 1405--1437.
23. E. Capra, C. Francalanci, and F. Merlo. 2008. An empirical study on the relationship among software design quality, development effort, and governance in open source projects. IEEE Trans. Softw. Eng. 34, 6 (2008), 765--782.
24. E. Carmel. 1999. Global Software Teams. Prentice Hall.
25. A. Charpentier, J.-R. Falleri, F. Morandat, E. Ben Hadj Yahia, and L. Réveillère. 2017. Raters’ reliability in clone benchmarks construction. Empir. Software Eng. 22, 1 (2017), 235--258.
26. B. Chen and Z. M. Jiang. 2017. Characterizing logging practices in Java-based open source software projects -- A replication study in Apache Software Foundation. Empir. Software Eng. 22, 1 (2017), 330--374.
27. M. Choetkiertikul, H. K. Dam, T. Tran, and A. Ghose. 2017. Predicting the delay of issues with due dates in software projects. Empir. Software Eng. 22, 3 (2017), 1223--1263.
28. M. Ó Cinnéide, I. Hemati Moghadam, M. Harman, S. Counsell, and L. Tratt. 2017. An experimental search-based approach to cohesion metric evaluation. Empir. Software Eng. 22, 1 (2017), 292--329.
29. R. Coelho, L. Almeida, G. Gousios, A. van Deursen, and C. Treude. 2017. Exception handling bug hazards in Android. Empir. Software Eng. 22, 3 (2017), 1264--1304.
30. K. Conboy and B. Fitzgerald. 2010. Method and developer characteristics for effective agile method tailoring: A study of XP expert opinion. ACM Trans. Softw. Eng. Methodol. 20, 1 (2010), 1--30.
31. D. S. Cruzes and T. Dybå. 2011. Research synthesis in software engineering: A tertiary study. Inform. Software Tech. 53 (2011), 440--455.
32. B. Curtis. 1980. Measurement and experimentation in software engineering. Proc. IEEE 68, 9 (1980), 1144--1157.
33. B. Curtis. 1984. Fifteen years of psychology in software engineering: Individual differences and cognitive science. In Proc. 7th International Conference on Software Engineering (ICSE’84). IEEE Press, 97--106.
34. B. Curtis. 2009. Point/counterpoint: Are rigorous experiments realistic for software engineering? IEEE Software 26, 6 (2009), 56--59.
35. B. Curtis, E. M. Soloway, R. E. Brooks, J. B. Black, K. Ehrlich, and H. R. Ramsey. 1986. Software psychology: The need for an interdisciplinary program. Proc. IEEE 74, 8 (1986), 1092--1106.
36. N. Dalkey and O. Helmer. 1963. An experimental application of the delphi method to the use of experts. Manage. Sci. 9, 3 (1963), 458--467.
37. D. E. Damian and D. Zowghi. 2003. RE challenges in multi-site software development organisations. Requir. Eng. 8 (2003), 149--160.
38. M. Daneva. 2011. Engineering the coordination requirements in cross-organizational ERP projects: A package of good practices. In Enterprise Information Systems: Concepts, Methodologies, Tools and Applications, I. Management Association (Ed.). Hershey, PA: IGI Global. 1941--1959.
39. M. Daneva and N. Ahituv. 2010. A focus group study on inter-organizational ERP requirements engineering practices. In Proc. 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement. ACM.
40. C. De Souza, Y. Dittrich, H. Sharp, and J. Singer. 2009. Cooperative and human aspects of software engineering (CHASE 2009). In Proc. International Conference on Software Engineering (Companion Volume). 451--452.
41. O. Dieste, A. M. Aranda, F. Uyaguari, B. Turhan, A. Tosun, D. Fucci, M. Oivo, and N. Juristo. 2017. Empirical evaluation of the effects of experience on code quality and programmer productivity: An exploratory study. Empir. Software Eng. 22, 5 (2017), 2457--2542.
42. Y. Dittrich. 2000. Beg, borrow, and steal—But what, and what for? In Workshop: Beg, Borrow, or Steal: Using Multidisciplinary Approaches in Empirical Software Engineering Research (co-located with ICSE’00).
43. Y. Dittrich, M. John, J. Singer, and B. Tessem. 2007. For the special issue on qualitative software engineering research. Inform. Software Tech. 49, 6 (2007), 531--539.
44. L. Dobrica and E. Niemelä. 2005. A survey on software architecture analysis methods. IEEE Trans. Software Eng. 28, 7 (2005), 638--653.
45. C. H. C. Duarte. 2017. Productivity paradoxes revisited. Empir. Software Eng. 22, 2 (2017), 818--847.
46. T. Dybå, R. Prikladnicki, K. Rönkkö, C. Seaman, and J. Sillito. 2011. Qualitative research in software engineering. Empir. Software Eng. 16 (2011), 425--429.
47. T. Dybå, D. I. K. Sjøberg, and D. S. Cruzes. 2012. What works for whom, where, when, and why? On the role of context in empirical software engineering. In Proc. International Symposium on Empirical Software Engineering and Measurement (ESEM’12). ACM.
48. S. Easterbrook, J. Singer, M.-A. Storey, and D. Damian. 2008. Selecting empirical methods for software engineering research. In Guide to Advanced Software Engineering, Forest Shull, Janice Singer, and Dag I. K. Sjøberg (Eds.). Springer-Verlag London Limited.
49. C. Ebert, C. H. Parro, R. Suttels, and H. Kolarczyk. 2001. Better validation in a world-wide development environment. In Proc. 7th International Software Metrics Symposium (METRICS’01).
50. A. C. Edmondson and S. E. McManus. 2007. Methodological fit in management field research. Acad. Manage. Rev. 32, 4 (2007), 1155--1179.
51. H. Edwards, S. McDonald, and M. Young. 2009. The repertory grid technique: Its place in empirical software engineering research. Inform. Software Tech. 51, 4 (2009), 785--798.
52. J. A. Espinosa and E. Carmel. 2003. The impact of time separation on coordination in global software teams: A conceptual foundation. Software Process. Improve. Pract. 8, 4 (2003), 249--266.
53. D. Falessi, M. Di Penta, G. Canfora, and G. Cantone. 2017. Estimating the number of remaining links in traceability recovery. Empir. Software Eng. 22, 3 (2017), 996--1027.
54. N. Fenton and S. L. Pfleeger. 1997. Software Metrics: A Rigorous and Practical Approach (2nd (revised printing) ed.). PWS Publishing Company.
55. D. Méndez Fernández, S. Wagner, M. Kalinowski, M. Felderer, P. Mafra, A. Vetrò, T. Conte, M.-T. Christiansson, D. Greer, C. Lassenius, T. Männistö, M. Nayabi, M. Oivo, B. Penzenstadler, D. Pfahl, R. Prikladnicki, G. Ruhe, A. Schekelmann, S. Sen, R. Spinola, A. Tuzcu, J. L. de la Vara, and R. Wieringa. 2017. Naming the pain in requirements engineering. Empir. Software Eng. 22, 5 (2017), 2298--2338.
56. B. Fitzgerald and D. Howcroft. 1998. Towards dissolution of the IS research debate: From polarization to polarity. J. Inform. Technol. 13, 4 (1998), 313--326.
57. M. Gharehyazie and V. Filkov. 2017. Tracing distributed collaborative development in apache software foundation projects. Empir. Software Eng. 22, 4 (2017), 1795--1830.
58. Y. Gil and G. Lalouche. 2017. On the correlation between size and metric validity. Empir. Software Eng. 22, 5 (2017), 2585--2611.
59. B. G. Glaser. 1978. Theoretical Sensitivity. Sociology Press.
60. B. G. Glaser and A. L. Strauss. 1967. The Discovery of Grounded Theory. AldineTransaction.
61. R. L. Glass. 1994. The software-research crisis. IEEE Software 11, 6 (1994), 42--47.
62. R. L. Glass. 2002. Facts and Fallacies of Software Engineering. Addison Wesley.
63. R. L. Glass, I. Vessey, and V. Ramesh. 2002. Research in software engineering: An analysis of the literature. Inform. Software Tech. 44 (2002), 491--506.
64. D. Graziotin, X. Wang, and P. Abrahamsson. 2014. Happy software developers solve problems better: Psychological measurements in empirical software engineering. PeerJ 2, e289 (2014), 1--23.
65. S. Gregor. 2006. The nature of theory in information systems. MIS Quart. 30, 3 (2006), 611--642.
66. J. Guo, M. Gibiec, and J. Cleland-Huang. 2017. Tackling the term-mismatch problem in automated trace retrieval. Empir. Software Eng. 22, 3 (Nov. 2017), 1103--1142.
67. J. E. Hannay and M. Jørgensen. 2008. The role of deliberate artificial design elements in software engineering experiments. IEEE Trans. Software Eng. 34, 2 (2008), 242--259.
68. R. Harrison, N. Badoo, E. Barry, S. Biffl, A. Parra, B. Winter, and J. Wuest. 1999. Directions and methodologies for empirical software engineering research. Empir. Software Eng. 4, 4 (1999), 405--410.
69. S. Hassan, W. Shang, and A. E. Hassan. 2017. An empirical study of emergency updates for top android mobile apps. Empir. Software Eng. 22, 1 (2017), 505--546.
70. V. T. Heikkilä, M. Paasivaara, C. Lasssenius, D. Damian, and C. Engblom. 2017. Managing the requirements flow from strategy to release in large-scale agile development: A case study at Ericsson. Empir. Software Eng. 22, 6 (2017), 2892--2936.
71. S. Herbold, A. Trautsch, and J. Grabowski. 2017. Global vs. local models for cross-project defect prediction. Empir. Software Eng. 22, 4 (2017), 1866--1902.
72. J. D. Herbsleb and R. E. Grinter. 1999. Architectures, coordination, and distance: Conway’s law and beyond. IEEE Software 16, 5 (1999), 63--70.
73. J. D. Herbsleb and R. E. Grinter. 1999. Splitting the organization and integrating the code: Conway’s law revisited. In Proc. International Conference on Software Engineering. 85--95.
74. R. Hoda, J. Noble, and S. Marshall. 2013. Self-organizing roles on agile software development teams. IEEE Trans. Software Eng. 39, 3 (2013), 422--444.
75. A. Höfer and W. Tichy. 2007. Status of empirical research in software engineering. In Empirical Software Engineering Issues, LNCS 4336. 10--19.
76. G. Hofstede, B. Neuijen, D. D. Ohayv, and G. Sanders. 1990. Measuring organizational cultures: A qualitative and quantitative study across twenty cases. Admin. Sci. Quar. 35, 2 (1990), 286--316.
77. M. Höst, B. Regnell, J. N. och Dag, J. Nedstam, and C. Nyberg. 2001. Exploring bottlenecks in market-driven requirements management processes with discrete event simulation. J. Sys. Software 59, 3 (2001), 323--332.
78. C. L. Iacovou and R. Nakatsu. 2008. A risk profile of offshore-outsourced development projects. Commun. ACM 51, 6 (2008), 89--94.
79. M. Ivarsson and T. Gorschek. 2011. A method for evaluating rigor and industrial relevance of technology evaluations. Empir. Software Eng. 16, 3 (2011), 365--395.
80. J. Tisseau. 2008. In vivo, in vitro, in silico, in virtuo. In 1st Workshop on SMA in Biology at Meso or Macroscopic Scales.
81. S. Jain, M. A. Babar, and J. Fernandez. 2013. Conducting empirical studies in industry: Balancing rigor and relevance. In Proc. International Workshop on Conducting Empirical Studies in Industry (CESI’13).
82. H. Jansen. 2010. The logic of qualitative survey research and its position in the field of social research methods. Forum: Qual. Social Res. 11, 2 (2010).
83. A. Jbara and D. G. Feitelson. 2017. How programmers read regular code: A controlled experiment using eye tracking. Empir. Software Eng. 22, 3 (2017), 1440--1477.
84. D. R. Jeffery and L. G. Votta. 1999. Guest editor’s special section introduction. IEEE Trans. Software Eng. 25, 4 (1999), 435--437.
85. J. Jiang, D. Lo, J. He, X. Xia, P. S. Kochhar, and L. Zhang. 2017. Why and how developers fork what from whom in GitHub. Empir. Software Eng. 22, 1 (2017), 547--578.
86. S. Jiang, C. McMillan, and R. Santelices. 2017. Do programmers do change impact analysis in debugging? Empir. Software Eng. 22, 2 (2017), 631--669.
87. M. Joblin, S. Apel, and W. Mauerer. 2017. Evolutionary trends of developer coordination: A network approach. Empir. Software Eng. 22, 4 (2017), 2050--2094.
88. A. N. Johanson and W. Hasselbring. 2017. Effectiveness and efficiency of a domain-specific language for high-performance marine ecosystem simulation: A controlled experiment. Empir. Software Eng. 22, 4 (2017), 2206--2236.
89. R. Jongeling, P. Sarkar, S. Datta, and A. Serebrenik. 2017. On negative results when using sentiment analysis tools for software engineering research. Empir. Software Eng. 22, 5 (2017), 2543--2584.
90. H. Jordan, S. Beecham, and G. Botterweck. 2014. Modelling software engineering research with RSML. In Proc. 18th International Conference on Evaluation and Assessment in Software Engineering.
91. N. Juristo and A. M. Moreno. 2001. Basics of Software Engineering Experimentation. Springer Science+Business Media.
92. E. Kalliamvakou, G. Gousios, K. Blincoe, L. Singer, D. M. German, and D. Damian. 2014. The promises and perils of mining GitHub. In Proc. 11th Working Conference on Mining Software Repositories.
93. R. Kazman, M. Klein, and P. Clements. 2000. ATAM: Method for Architecture Evaluation. Carnegie Mellon Software Engineering Institute, Technical Report CMU/SEI-2000-TR-004.
94. M. Kessentini, U. Mansoor, M. Wimmer, A. Ouni, and K. Deb. 2017. Search-based detection of model level changes. Empir. Software Eng. 22, 2 (2017), 670--715.
95. F. M. Kifetew, R. Tiella, and P. Tonella. 2017. Generating valid grammar-based test inputs by means of genetic programming and annotated grammars. Empir. Software Eng. 22, 2 (2017), 928--961.
96. J. King, J. Stallings, M. Riaz, and L. Williams. 2017. To log, or not to log: Using heuristics to identify mandatory log events -- A controlled experiment. Empir. Software Eng. 22, 5 (2017), 2684--2717.
97. B. A. Kitchenham and S. L. Pfleeger. 2002. Principles of survey research Part 2: Designing a survey. ACM Software Eng. Notes 27, 1 (2002), 18--20.
98. B. A. Kitchenham and S. L. Pfleeger. 2002. Principles of survey research Part 3: Constructing a survey instrument. ACM Software Eng. Notes 27, 2 (2002), 20--24.
99. B. A. Kitchenham and S. L. Pfleeger. 2002. Principles of survey research Part 4: Questionnaire evaluation. ACM Software Eng. Notes 27, 3 (2002), 20--23.
100. B. A. Kitchenham and S. L. Pfleeger. 2002. Principles of survey research Part 5: Populations and samples. ACM Software Eng. Notes 27, 5 (2002), 17--20.
101. B. A. Kitchenham and S. L. Pfleeger. 2003. Principles of survey research Part 6: Data analysis. ACM Software Eng. Notes 28, 2 (2003), 24--27.
102. B. A. Kitchenham, S. L. Pfleeger, L. M. P. Pickard, P. W. Jones, D. C. Hoaglin, K. El Emam, and J. Rosenberg. 2008. Preliminary guidelines for empirical research in software engineering. IEEE Trans. Softw. Eng. 28, 2 (2008), 721--734.
103. A. J. Ko, B. A. Myers, M. J. Coblenz, and H. H. Aung. 2006. An exploratory study of how developers seek, relate, and collect relevant information during software maintenance tasks. IEEE Trans. Software Eng. 32, 12 (2006), 971--987.
104. J. Kontio, J. Bragge, and L. Lehtola. 2008. The focus group method as an empirical tool in software engineering. In Guide to Advanced Empirical Software Engineering. Springer.
105. M. Krafft, K. Stol, and B. Fitzgerald. 2016. How do free/open source developers pick their tools? A Delphi study of the Debian project. In Proc. 38th International Conference on Software Engineering (Companion Volume), 232--241.
106. P. Kruchten, R. L. Nord, and I. Ozkaya. 2012. Technical debt: From metaphor to theory and practice. IEEE Software 29, 6 (2012), 18--21.
107. I. Kwan, A. Schröter, and D. Damian. 2011. Does socio-technical congruence have an effect on software build success? A study of coordination in a software project. IEEE Trans. Software Eng. 37, 3 (2011), 307--324.
108. K. Labunets, F. Massacci, F. Paci, S. Marczak, and F. M. de Oliveira. 2017. Model comprehension for security risk assessment: An empirical comparison of tabular vs. graphical representations. Empir. Software Eng. 22, 6 (2017), 3017--3056.
109. F. Lanubile. 1997. Empirical evaluation of software maintenance technologies. Empir. Software Eng. 2 (1997), 97--108.
110. T. D. LaToza, M. Chen, L. Jiang, M. Zhao, and A. van der Hoek. 2015. Borrowing from the crowd: A study of recombination in software design competitions. In Proc. International Conf. Software Engineering.
111. S. Lauesen and O. Vinter. 2001. Preventing requirement defects: An experiment in process improvement. Requir. Eng. 6 (2001), 37--50.
112. T.-D. B. Le, F. Thung, and D. Lo. 2017. Will this localization tool be effective for this bug? Mitigating the impact of unreliability of information retrieval based bug localization tools. Empir. Software Eng. 22, 4 (2017), 2237--2279.
113. T. O. A. Lehtinen, J. Itkonen, and C. Lassenius. 2017. Recurring opinions or productive improvements—What agile teams actually discuss in retrospectives. Empir. Software Eng. 22, 5 (2017), 2409--2452.
114. P. Lenberg, R. Feldt, and L. G. Wallgren. 2015. Behavioral software engineering: A definition and systematic literature review. J. Syst. Software 107 (2015), 15--37.
115. P. Lenberg, L. G. W. Tengberg, and R. Feldt. 2017. An initial analysis of software engineers’ attitudes towards organizational change. Empir. Software Eng. 22, 4 (2017), 2179--2205.
116. F. J. Lerch, D. J. Ballou, and D. E. Harter. 1997. Using simulation-based experiments for software requirements engineering. Ann. Software Eng. 3, 1 (1997), 345--366.
117. T. C. Lethbridge, S. E. Sim, and J. Singer. 2005. Studying software engineers: Data collection techniques for software field studies. Empir. Software Eng. 10 (2005), 311--341.
118. H. Li, W. Shang, and A. E. Hassan. 2017. Which log level should developers choose for a new logging statement? Empir. Software Eng. 22, 4 (2017), 1684--1716.
119. H. Li, W. Shang, Y. Zou, and A. E. Hassan. 2017. Towards just-in-time suggestions for log changes. Empir. Software Eng. 22, 4 (2017), 1831--1865.
120. D. Lin, C.-P. Bezemer, and A. E. Hassan. 2017. Studying the urgent updates of popular games on the steam platform. Empir. Software Eng. 22, 4 (2017), 2095--2126.
121. H. A. Linstone and M. Turoff (Eds.). 2002. The Delphi Method Techniques and Applications. Addison-Wesley.
122. C. Lokan and E. Mendes. 2017. Investigating the use of moving windows to improve software effort prediction: A replicated study. Empir. Software Eng. 22, 2 (2017), 716--767.
123. L. Lopez-Fernandez, G. Robles, and J. M. Gonzalez-Barahona. 2004. Applying social network analysis to the information in CVS repositories. In Proc. 1st Workshop on Mining Software Repositories (MSR’04). 101--105.
124. Q. Luo, A. Nair, M. Grechanik, and D. Poshyvanyk. 2017. FOREPOST: Finding performance problems automatically with feedback-directed learning software testing. Empir. Software Eng. 22, 1 (2017), 6--56.
125. J. Ma, J. Li, W. Chen, R. Conradi, J. Ji, and C. Liu. 2008. A state-of-the-practice study on communication and coordination between Chinese software suppliers and their global outsourcers. Software Process Improve. Pract. 13, 3 (2008), 233--247.
126. L. MacLeod, A. Bergen, and M.-A. Storey. 2017. Documenting and sharing software knowledge using screencasts. Empir. Software Eng. 22, 3 (2017), 1478--1507.
127. A. Mahmoud and G. Bradshaw. 2017. Semantic topic models for source code analysis. Empir. Software Eng. 22, 4 (2017), 1965--2000.
128. R. Malhotra and M. Khanna. 2017. An empirical study for software change prediction using imbalanced data. Empir. Software Eng. 22, 6 (2017), 2806--2851.
129. M. Martinez, T. Durieux, R. Sommerard, J. Xuan, and M. Monperrus. 2017. Automatic repair of real bugs in Java: A large-scale experiment on the defects4j dataset. Empir. Software Eng. 22, 4 (2017), 1936--1964.
130. J. E. McGrath. 1964. Social Psychology: A Brief Introduction. Holt, Rinehart and Winston.
131. J. E. McGrath. 1964. Towards a “Theory of Method” for research on organizations. In New Perspectives in Organization Research, W. Cooper, H. Leavitt, and M. Shelly (Eds.). John Wiley 8 Sons, New York, 533--556.
132. J. E. McGrath. 1981. Dilemmatics: The study of research choices and dilemmas. Am. Behav. Sci. 25, 2 (1981), 179--210.
133. J. E. McGrath. 1984. Groups: Interaction and Performance. Prentice-Hall.
134. J. E. McGrath. 1994. Methodology matters: Doing research in the behavioral and social sciences. In Readings in Human-Computer Interaction: Toward the Year 2000, Ronald M. Baecker (Ed.). Morgan Kaufmann, 152--169.
135. E. R. McLean. 1973. Comments on empirical studies of management information systems by Richard L. Van Horn. Data Base 5, 2 (1973), 181--182.
136. N. Medvidović and R. N. Taylor. 2000. A classification and comparison framework for software architecture description languages. IEEE Trans. Software Eng. 26, 1 (2000), 70--93.
137. T. Menzies, W. Nichols, F. Shull, and L. Layman. 2017. Are delayed issues harder to resolve? Revisiting cost-to-fix of defects throughout the lifecycle. Empir. Software Eng. 22, 4 (2017), 1903--1935.
138. T. Menzies, Y. Yang, G. Mathew, B. Boehm, and J. Hihn. 2017. Negative results for software effort estimation. Empir. Software Eng. 22, 5 (2017), 2658--2683.
139. B. Meyer, H. Gall, M. Harman, and G. Succi. 2013. Empirical answers to fundamental software engineering problems (panel). In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering (ESEC/FSE’13). ACM, 14--18.
140. M. W. Mkaouer, M. Kessentini, M. Ó. Cinnéide, S. Hayashi, and K. Deb. 2017. A robust multi-objective approach to balance severity and importance of refactoring opportunities. Empir. Software Eng. 22, 2 (2017), 894--927.
141. A. Mockus, R. T. Fielding, and J. D. Herbsleb. 2000. A case study of open source software development: The Apache server. In Proc. International Conf. Software Engineering.
142. A. Mockus, R. T. Fielding, and J. D. Herbsleb. 2002. Two case studies of open source software development: Apache and Mozilla. ACM Trans. Software Eng. Methodol. 11, 3 (2002), 309--346.
143. M. Montesi and P. Lago. 2008. Software engineering article types: An analysis of the literature. J. Syst. Software 81, 10 (2008), 1694--1714.
144. M. Müller and D. Pfahl. 2008. Simulation methods. In Guide to Advanced Software Engineering, F. Shull, J. Singer, and D. I. K. Sjøberg (Eds.). Springer-Verlag London Limited.
145. N. Munaiah, S. Kroh, C. Cabrey, and M. Nagappan. 2017. Curating GitHub for engineered software projects. Empir. Software Eng. 22, 6 (2017), 3219--3253.
146. C. J. Neill and P. Laplante. 2003. Requirements engineering: The state of the practice. IEEE Software 20, 6 (2003), 40--45.
147. L. Nguyen and G. Shanks. 2008. A framework for understanding creativity in requirements engineering. Inform. Software Tech. 51, 3 (2008), 655--662.
148. A. Niknafs and D. Berry. 2017. The impact of domain knowledge on the effectiveness of requirements engineering activities. Empir. Software Eng. 22, 1 (2017), 80--133.
149. H. Niu, I. Keivanloo, and Y. Zou. 2017. Learning to rank code examples for code search engines. Empir. Software Eng. 22, 1 (2017), 259--291.
150. E. Noei, M. D. Syer, Y. Zou, A. E. Hassan, and I. Keivanloo. 2017. A study of the relation of mobile device attributes with the user-perceived quality of Android apps. Empir. Software Eng. 22, 6 (2017), 3088--3116.
151. P. Ovaska, M. Rossi, and P. Marttiin. 2003. Architecture as a coordination tool in multi-site software development. Software Process Improve. Pract. 8 (2003), 233--247.
152. C. Palomares, C. Quer, and X. Franch. 2017. Requirements reuse and requirement patterns: A state of the practice survey. Empir. Software Eng. 22, 6 (2017), 2719--2762.
153. J. Park, M. Kim, and D.-H. Bae. 2017. An empirical study of supplementary patches in open source projects. Empir. Software Eng. 22, 1 (2017), 436--473.
154. D. L. Parnas. 2009. Point/counterpoint: Empirical research in software engineering: A critical view. IEEE Software 26, 6 (2009), 56--59.
155. D. E. Perry, A. E. Porter, and L. G. Votta. 2000. Empirical studies of software engineering: A roadmap. In Future of Software Engineering.
156. D. E. Perry, N. A. Staudenmayer, and L. G. Votta. 1994. People, organizations, and process improvement. IEEE Software 11, 4 (1994), 36--45.
157. K. Petersen and C. Wohlin. 2009. Context in industrial software engineering research. In Proc. 3rd International Symposium on Empirical Software Engineering and Measurement.
158. S. L. Pfleeger and B. A. Kitchenham. 2001. Principles of survey research: Part 1: Turning lemons into lemonade. ACM SIGSOFT Software Eng. Notes 26, 6 (2001), 16--18.
159. A. A. Porter, L. G. Votta, and V. R. Basili. 1995. Comparing detection methods for software requirements inspections: A replicated experiment. IEEE Trans. Software Eng. 21, 6 (1995), 563--575.
160. D. Raffo and S.-O. Setamanit. 2005. A simulation model for global software development project. In Proc. 6th International Workshop on Software Process Simulation and Modeling (ProSim’05). Fraunhofer IRB Verlag.
161. P. Ralph. 2018. Toward methodological guidelines for process theories and taxonomies in software engineering. IEEE Trans. Software Eng. In press (2018). https://ieeexplore.ieee.org/document/8267085/.
162. B. Ray, D. Posnett, V. Filkov, and P. Devanbu. 2014. A large scale study of programming languages and code quality in github. In Proc. 22nd ACM SIGSOFT International Sym. Foundations of Software Engineering.
163. M. Riaz, J. King, J. Slankas, L. Williams, F. Massacci, C. Quesada-López, and M. Jenkins. 2017. Identifying the implied: Findings from three differentiated replications on the use of security requirements templates. Empir. Software Eng. 22, 4 (2017), 2127--2178.
164. H. Robinson, J. Segal, and H. Sharp. 2007. Ethnographically-informed empirical studies of software practice. Inform. Software Tech. 49, 6 (2007), 540--551.
165. J. Miguel Rojas, M. Vivanti, A. Arcuri, and G. Fraser. 2017. A detailed investigation of the effectiveness of whole test suite generation. Empir. Software Eng. 22, 2 (2017), 852--893.
166. D. Rosenblum and E. Weyuker. 1996. Lessons learned from a regression testing case study. In Proc. International Workshop on Empirical Studies of Software Maintenance (WESS’96).
167. D. T. Ross. 1977. Guest editorial: Reflections on requirements. IEEE Trans. Software Eng. 3, 1 (1977), 2--5.
168. P. Runeson and M. Höst. 2009. Guidelines for conducting and reporting case study research in software engineering. Empir. Software Eng. 14 (2009), 131--164.
169. P. Runeson, M. Höst, A. Rainer, and B. Regnell. 2012. Case Study Research in Software Engineering: Guidelines and Examples. Wiley.
170. P. J. Runkel and J. E. McGrath. 1972. Research on Human Behavior: A Systematic Guide to Method. Holt, Rinehart and Winston.
171. A. Sabané, Y.-G. Guéhéneuc, V. Arnaoudova, and G. Antoniol. 2017. Fragile base-class problem, problem? Empir. Software Eng. 22, 5 (2017), 2612--2657.
172. V. Sakhnini, L. Mich, and D. M. Berry. 2017. Group versus individual use of power-only EPMcreate as a creativity enhancement technique for requirements elicitation. Empir. Software Eng. 22, 4 (2017), 2001--2049.
173. A. Ashok Sawant and A. Bacchelli. 2017. fine-GRAPE: Fine-grained APi usage extractor -- An approach and dataset to investigate API usage. Empir. Software Eng. 22, 3 (2017), 1348--1371.
174. C. B. Seaman. 1999. Qualitative methods in empirical studies of software engineering. IEEE Trans. Software Eng. 24, 4 (1999), 557--572.
175. J. Segal. 2003. Some parallels between empirical software engineering and research in human-computer interaction. In Proc. 15th Workshop of the Psychology of Programming Interest Group.
176. S.-O. Setamanit. 2007. A Software Process Simulation Model of Global Software Development (GSD) Projects. Ph.D. Dissertation. Portland State University.
177. S.-O. Setamanit, W. Wakeland, and D. Raffo. 2007. Using simulation to evaluate global software development task allocation strategies. Software Process Improve. Pract. 12 (2007), 491--503.
178. B. Sharif, J. Meinken, T. Shaffer, and H. Kagdi. 2017. Eye movements in software traceability link recovery. Empir. Software Eng. 22, 3 (2017), 1063--1102.
179. H. Sharp, Y. Dittrich, and C. R. B. de Souza. 2016. The role of ethnographic studies in empirical software engineering. IEEE Trans. Software Eng. 42, 8 (2016), 786--804.
180. H. Sharp and H. Robinson. 2004. An ethnographic study of XP practice. Empir. Software Eng. 9, 4 (2004), 353--375.
181. H. Sharp, M. Woodman, and F. Hovenden. 2005. Using metaphor to analyse qualitative data: Vulcans and humans in software development. Empir. Software Eng. 10, 3 (2005), 343--365.
182. M. Shaw. 2002. What makes good research in software engineering? Int. J. Software Tools Technol. Transf. 4, 1 (2002), 1--7.
183. M. Shaw. 2003. Writing good software engineering research papers. In Proc. 25th International Conf. Software Engineering. 726--736.
184. Y. Shi, M. Li, S. Arndt, and C. Smidts. 2017. Metric-based software reliability prediction approach and its application. Empir. Software Eng. 22, 4 (2017), 1579--1633.
185. F. Shull, J. Singer, and D. I. K. Sjøberg (Eds.). 2008. Guide to Advanced Empirical Software Engineering. Springer.
186. J. Siegmund, N. Siegmund, and S. Apel. 2015. Views on internal and external validity in empirical software engineering. In Proc. 37th International Conference on Software Engineering. IEEE.
187. S. E. Sim, J. Singer, and M.-A. Storey. 2001. Beg, borrow, or steal: Using multidisciplinary approaches in empirical software engineering research. Empir. Software Eng. 6, 1 (2001), 85--93.
188. D. I. K. Sjøberg, B. Anda, E. Arisholm, T. Dybå, M. Jørgensen, A. Karahasanovic, E. F. Koren, and M. Vokác. 2002. Conducting realistic experiments in software engineering. In Proc. International Symposium on Empirical Software Engineering (ISESE’02).
189. D. I. K. Sjøberg, T. Dybå, B. C. D. Anda, and J. E. Hannay. 2008. Building theories in software engineering. In Guide to Advanced Empirical Software Engineering, Forrest Shull, Janice Singer, and Dag I. K. Sjøberg (Eds.). Springer-Verlag London Limited.
190. D. I. K. Sjøberg, T. Dybå, and M. Jørgensen. 2007. The future of empirical methods in software engineering research. In Future of Software Engineering. IEEE Computer Society.
191. D. I. K. Sjøberg, J. E. Hannay, O. Hansen, V. B. Kampenes, A. Karahansanovic, N.-K. Liborg, and A. C. Rekdal. 2005. A survey of controlled experiments in software engineering. IEEE Trans. Software Eng. 31, 9 (2005), 733--753.
192. A. E. Kelley Sobel and M. R. Clarkson. 2002. Formal methods application: An empirical tale of software development. IEEE Trans. Software Eng. 28, 3 (2002), 308--320.
193. A. E. Kelley Sobel and M. R. Clarkson. 2003. Response to “Comments on ‘Formal methods application: An empirical tale of software development.”’ IEEE Trans. Software Eng. 29, 6 (2003), 572--575.
194. D. Spinellis. 2017. A repository of Unix history and evolution. Empir. Software Eng. 22, 3 (2017), 1372--1404.
195. D. Ståhl, K. Hallén, and J. Bosch. 2017. Achieving traceability in large scale continuous integration and delivery deployment, usage and validation of the Eiffel framework. Empir. Software Eng. 22, 3 (2017), 967--995.
196. I. Stavropoulou, M. Grigoriou, and K. Kontogiannis. 2017. Case study on which relations to use for clustering-based software architecture recovery. Empir. Software Eng. 22, 4 (2017), 1717--1762.
197. K. Stol, B. Caglayan, and B. Fitzgerald. 2018. Competition-based crowdsourcing software development: A multi-method study from a customer perspective. IEEE Trans. Software Eng. In press (2018).
198. K. Stol and B. Fitzgerald. 2014. Two’s company, three’s a crowd: A case study of crowdsourcing software development. In Proc. 36th International Conference on Software Engineering. 187--198.
199. K. Stol and B. Fitzgerald. 2015. A holistic overview of software engineering research strategies. In 3rd International Workshop on Conducting Empirical Studies in Industry (CESI’15). ACM.
200. K. Stol and B. Fitzgerald. 2015. Theory-oriented software engineering. Sci. Computer Program. 101 (2015), 79--98.
201. K. Stol, M. Goedicke, and I. Jacobson. 2016. Introduction to the special section—General theories of software engineering: New advances and implications for research. Inform. Software Tech. 70 (2016), 176--180.
202. K. Stol, P. Ralph, and B. Fitzgerald. 2016. Grounded theory in software engineering research: A critical review and guidelines. In Proc. 38th International Conference on Software Engineering. ACM, 120--131.
203. M.-A. Storey, L. Singer, F. F. Filho, A. Zagalsky, and D. M. German. 2017. How social and communication channels shape and challenge a participatory culture in software development. IEEE Trans. Software Eng. 43, 2 (2017), 185--204.
204. P. Thongtanunam, S. McIntosh, A. E. Hassan, and H. Iida. 2017. Review participation in modern code review. Empir. Software Eng. 22, 2 (2017), 768--817.
205. N. M. Tichy, M. L. Tushman, and C. Fombrun. 1979. Social network analysis for organizations. Acad. Manag. Rev. 4, 4 (1979), 507--519.
206. W. F. Tichy. 1998. Should computer scientists experiment more? Computer 31, 5 (1998), 32--40.
207. W. F. Tichy. 2000. Hints for reviewing empirical work in software engineering. Empir. Software Eng. 5 (2000), 309--312.
208. J. Tisseau. 2001. Virtual Reality: In virtuo autonomy. University of Rennes I.
209. A. Tosun, O. Dieste, D. Fucci, S. Vegas, B. Turhan, H. Erdogmus, A. Santos, M. Oivo, K. Toro, J. Jarvinen, and N. Juristo. 2017. An industry experiment on the effects of test-driven development on external quality and productivity. Empir. Software Eng. 22, 6 (2017), 2763--2805.
210. G. H. Travassos and M. de Oliveira Barros. 2003. Contributions of in virtuo and in silico experiments for the future of empirical studies in software engineering. In Proc. 2nd Workshop on Empirical Software Engineering.
211. C. Tsigkanos, L. Pasquale, C. Menghi, C. Ghezzi, and B. Nuseibeh. 2014. Engineering topology aware adaptive security: Preventing requirements violations at runtime. In Proc. 22nd IEEE International Requirements Engineering Conference. 203--212.
212. R. L. van Horn. 1973. Empirical studies of management information systems. ACM SIGMIS Database: DATABASE Adv. Inf. Syst. 5 (1973), 172--182.
213. J. van Maanen. 1982. Fieldwork on the beat. In Varieties of Qualitative Research, J. van Maanen, J. M. Dabbs, and R. R. Faulkner (Eds.). Sage Publications.
214. C. Vendome, G. Bavota, M. Di Penta, M. Linares-Vásquez, D. German, and D. Poshyvanyk. 2017. License usage and changes: A large-scale study on gitHub. Empir. Software Eng. 22, 3 (2017), 1537--1577.
215. P. Vitharana. 2017. Defect propagation at the project-level: Results and a post-hoc analysis on inspection efficiency. Empir. Software Eng. 22, 1 (2017), 57--79.
216. L. Votta. 1995. By the way, has anyone studied any real programmers, yet? In Proc. 9th International Software Process Workshop.
217. E. J. Webb, D. T. Campbell, R. D. Schwartz, and L. Sechrest. 1966. Unobtrusive Measures: Nonreactive Research in the Social Sciences. Rand-McNally.
218. G. M. Weinberg. 1971. The Psychology of Computer Programming. Van Nostrand Reinhold New York.
219. R. Wieringa. 2009. Design science as nested problem solving. In Proc. DESRIST.
220. R. Wieringa and M. G. Heerkens. 2006. The methodological soundness of requirements engineering papers: A conceptual framework and two case studies. Requir. Eng. 11 (2006), 295--307.
221. R. Wieringa, N. Maiden, N. Mead, and C. Rolland. 2006. Requirements engineering paper classification and evaluation criteria: A proposal and a discussion. Requir. Eng. 11 (2006), 102--107.
222. C. Wohlin and A. Aurum. 2015. Towards a decision-making structure for selecting a research design in empirical software engineering. Empir. Software Eng. 20, 6 (2015), 1427--1455.
223. C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslén. 2000. Experimentation in Software Engineering. Kluwer Academic Publishers.
224. C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslén. 2012. Experimentation in Software Engineering (2nd ed.). Springer.
225. C. Wohlin, D. S̆mite, and N. B. Moe. 2015. A general theory of software engineering: Balancing human, social and organizational capitals. J. Syst. Software 109 (2015), 229--242.
226. J. Wu, S. Ali, T. Yue, J. Tian, and C. Liu. 2017. Assessing the quality of industrial avionics software: An extensive empirical evaluation. Empir. Software Eng. 22, 4 (2017), 1634--1683.
227. X. Xia, L. Bao, D. Lo, P. S. Kochhar, A. E. Hassan, and Z. Xing. 2017. What do developers search for on the web? Empir. Software Eng. 22, 6 (2017), 3149--3185.
228. D. Ye, Z. Xing, and N. Kapre. 2017. The structure and dynamics of knowledge network in domain-specific Q&A sites: A case study of stack overflow. Empir. Software Eng. 22, 1 (2017), 375--406.
229. R. K. Yin. 2003. Case Study Research: Design and Methods. Sage.
230. C. Zannier, G. Melnik, and F. Maurer. 2006. On the success of empirical studies in the international conference on software engineering. In Proc. International Conf. Software Engineering. 341--350.
231. M. V. Zelkowitz. 2007. Techniques for empirical validation. Empir. Software Eng. Issues LNCS 4336 (2007), 4--9.
232. M. V. Zelkowitz and D. R. Wallace. 1998. Experimental models for validating technology. Computer 31, 5 (1998), 23--31.
233. W. Zogaan, I. Mujhid, J. C. S. Santos, D. Gonzalez, and M. Mirakhorli. 2017. Automated training-set creation for software architecture traceability problem. Empir. Software Eng. 22, 3 (2017), 1028--1062.

Index Terms

• General and reference

  • Cross-computing tools and techniques

    • Empirical studies

  • Document types

    • General literature

    • Surveys and overviews