Assessing information technology personnel: toward a behavioral rating scale

Abstract

Performance ratings are used as a means to measure and record performance of individuals on any particular job. Results can then be used for promotion, increases in pay and other incentives. They, therefore, can have a large impact on a person's career and are viewed by both employers and employees as extremely important. Performance ratings vary widely and typically include a variety of measures - some of which the employee can control, and others over which the employee has no direct control. Overall, the various types of performance ratings are not perfect; each has its flaws along with its advantages. In this study, one particular type of scale - a behavioral rating scale - is proposed by taking the results of a previous study performed by Green (1989) and empirically testing a set of critical skills for use as a performance assessment instrument for information technology (IT) personnel. The instrument is provided for researchers who wish to incorporate a behavior-oriented measure in their research and for practitioners who may be looking for a behavior-oriented scale to evaluate IT personnel. The psychometric properties of the instrument as observed in this study are also presented.

References

1. Anderson, J. C., and Gerbing, D. W. (1988). "Structural equation modeling in practice: A review and recommended two-step approach," Psychological Bulletin, Vol. 103, pp. 411--423.
2. Ang, S., and Slaughter, S.A. (2001). "Work outcomes and job design for contract versus permanent information systems professionals on software development teams," MIS Quarterly, Vol. 25, No. 3, pp. 321--350.
3. Arvey, R. D., and Hoyle, J. C. (1974). "A Guttman approach to the development of behaviorally based rating scales for systems analysts and programmer/analysts," Journal of Applied Psychology, Vol. 59, No. 1),pp. 61--65.
4. Arvey, R. D., and Renz, G. L. (1992). "Fairness in the selection of employees," Journal of Business Ethics, Vol. 11, pp.331--340.
5. Bartol, K.M., and Martin, D.C. (1982). "Managing information systems personnel: A review of the literature and managerial implications," MIS Quarterly, Vol. 6, Special Issue, pp. 49--70.
6. Beatty, R.W., Schneier, C.E., and Beatty, J.R. (1977). "An empirical investigation of perceptions of ratee behavior frequency and ratee behavior change using behavioral expectation scales," Personnel Psychology, Vol. 30, No. 4, pp. 647--658.
7. Bernard, H.R. (2000). Social research methods: qualitative and quantitative approaches. Sage Publications, Inc., Thousand Oaks, CA.
8. Bernardin, H.J., and Smith, P.C. (1981). "A clarification of some issues regarding the development and use of behaviorally anchored rating scales (BARS)," Journal of Applied Psychology, Vol. 66, No. 4, pp. 458--463.
9. Bollen, K. A. (1989). Structural Equations with Latent Variables. John Wiley and Sons, New York, NY.
10. Brett, J.F., and Atwater, L.E. (2001). "360 degree feedback: Accuracy, reactions, and perceptions of usefulness," Journal of Applied Psychology, Vol. 86, No. 5, pp. 930--942.
11. Browne, M.W. and Cudeck, R. (1989). "Single sample cross-validation indices for covariance structures," Multivariate Behavioral Research, Vol. 24, No. 4, pp. 445--455.
12. Browne, M.W. and Cudeck, R. (1993). "Alternative ways of assessing model fit," in Testing Structural Equation Models, Bollen, K.A. and Long, J.S. (eds.), Sage Publications, Newbury Park, CA, pp.136--162.
13. Campbell, J. P. (1990). "Modeling the performance prediction problem in industrial and organizational psychology," in Handbook of Industrial and Organizational Psychology, M. D. Dunnette and L. M. Hough (eds.), Consulting Psychologists Press, Palo Alto, CA, pp. 687--732.
14. Campbell, J.P. (1994). "Models of measurement and their implications for research on the linkages between individual and organizational productivity," in Organizational Linkages: Understanding the Productivity Paradox, Harris, D.H., (ed.), National Academies Press, Washington, D.C., pp. 193--213.
15. Campbell, J. P., Dunnette, M. D., Arvey, R. D., and Hellervik, L. V. (1973). "The development and evaluation of behaviorally based rating scales," Journal of Applied Psychology Vol. 57, No. 4, pp.15--22.
16. Carmines, E. G., and Zeller, R. A. (1979). Reliability and Validity Assessment, Sage Publications, Thousand Oaks, CA.
17. Cudeck, R. and Browne, M.W. (1982). "Cross-validation of covariance structures," Multivariate Behavioral Research, Vol. 18, pp. 147--167.
18. Ferratt, T.W., and Agarwal, R. (1994). "Guided self-development: An innovative approach to management education and development applied to information systems supervisors," Computer Personnel, Vol. 15, No. 3, pp. 3--20.
19. Ferratt, T. W., and Short, L. E. (1988). "Are information systems people different? An investigation of how they are and should be managed, "MIS Quarterly, Vol.12, No.3, pp. 427--443.
20. Gomez-Mejia, L.R. (1988). "Evaluating employee performance: Does the appraisal instrument make a difference," Journal of Organizational Behavior Management, Vol. 9, No.2, pp. 155--172.
21. Green, G. I. (1989). "Perceived importance of system analysts' job skills, roles, and non-salary incentives," MIS Quarterly, Vol. 13, No. 2, pp. 114--133.
22. Greenhaus, J. H., Parasuraman, S., and Wormley, W. M. (1990). "Effects of race on organizational experiences, job performance evaluations, and career outcomes," Academy of Management Journal, Vol.33, No. 1, pp. 64--86.
23. Hatcher, L. (1994). A Step-by-step Approach to Using the SAS® System for Factor Analysis and Structural Equation Modeling, SAS® Institute, Cary, NC.
24. Hesketh, B., and Neal, A. (1999). "Technology and performance," in Implications for Staffing, Motivation, and Development, Daniel Ilgen and Elain Pulakos, (eds), Jossey-Bass, San Francisco, CA, pp. 21--55.
25. Igbaria, M., and Baroudi, J. (1995). "The impact of job performance evaluations on career advancement prospects: An examination of gender differences in the IS workplace," MIS Quarterly, Vol.19, No. 1, pp.107--123.
26. Igbaria, M., and Greenhaus, J.H. (1992). "The career advancement prospects of managers and professionals: Are MIS employees unique?" Decision Sciences, Vol. 23, No. 2, pp. 478--499.
27. Igbaria, M., and Wormley, W. (1992). "Organizational experiences and career success of MIS professionals and managers: An examination of race differences," MIS Quarterly, Vol. 16, No. 4, pp. 507--529.
28. Jiang, J.J., Sobol, M.G., and Klein, G. (2000). "Performance ratings and importance of performance measures for IS staff: The different perceptions of IS users and IS staff," IEEE Transactions on Engineering Management, Vol. 47, No. 4, pp. 424--434.
29. Joreskog, K. & Sorbom, D. (1993). LISREL 8: Structural Equation Modeling with the SIMPLIS Command Language. L. Erlbaum Associates, Hillsdale, NJ.
30. Khan, M.B., and Kukalis, S. (1990). "MIS professionals: Education and performance," Information and Management, Vol. 19, pp. 249--255.
31. Kavanagh, M.J. (1971). "The content issue in performance appraisal: A review," Personnel Psychology, Vol. 24, No. 4, pp. 653--668.
32. Ketler, K., and Smith, R.D. (1993). "Placement, performance and turnover of information systems professionals: Implications for HRIS," Proceedings of the 1993 ACM SIGCPR Conference, April 1-3, St. Louis, MO, ACM Press, New York, NY, pp. 131--138.
33. Killingworth, B.L., Hayden, M.B., Crawford, D., and Schellenberger, R. (2001). "A model for motivating and measuring quality performance in information systems staff," Information Systems Management, Vol. 18, No.2, pp. 8--14.
34. Landy, F.J., and Farr, J.L. (1980). "Performance rating," Psychological Bulletin, Vol. 87, No.1, pp. 72--107.
35. Latham, G.P., and Wexley, K.N. (1977). "Behavioral observation scales for performance appraisal purposes," Personnel Psychology, Vol. 30, No. 2, pp. 255--268.
36. Lee, D. M. S., Trauth, E. M., and Farwell, D. (1995). "Critical skills and knowledge requirements of IS professionals: A joint academic/industry investigation," MIS Quarterly, Vol. 19, No. 3, pp. 313--340.
37. Leitheiser, R. L. (1992). "MIS skills for the 1990s: A survey of MIS managers' perceptions," Journal of Management Information Systems, Vol. 9, No. 1, pp.69--91.
38. Murphy, K.R., and Constans, J.I. (1987). "Behavioral anchors as a source of bias in rating," Journal of Applied Psychology, Vol. 72, No. 4, pp. 573--577
39. Murphy, K.R., Balzer, W.K., Lockhart, M.C., and Eisenman, E.J. (1985). "Effects of previous performance on evaluations of present performance," Journal of Applied Psychology, Vol. 70, no. 1, pp. 72--84.
40. Murphy, K.R., Martin, C., and Garcia, M. (1982). "Do behavioral observation scales measure observation?" Journal of Applied Psychology, Vol. 67, No. 5, pp. 562--567.
41. Pedhazur, E. J., and Schmelkin, L. P. (1991). Measurement, Design, and Analysis: An Integrated Approach, Lawrence Erlbaum Associates, Hillsdale, NJ.
42. Rasch, R.H., and Tosi, H.L. (1992). "Factors affecting software developers' performance: An integrated approach," MIS Quarterly, Vol. 16, No. 3, pp. 395--413.
43. Ruch, W. A. (1994). "Managing and measuring individual productivity," in Organizational Linkages: Understanding the Productivity Paradox, Harris, D.H., (ed.), National Academies Press, Washington, D.C., pp. 105--130.
44. Smith, P. C., and Kendall, L. M. (1963). "Retranslation of expectations: an approach to the construction of unambiguous anchors for rating scales," Journal of Applied Psychology, Vol. 47, No. 2, pp. 149--155.
45. Todd, P.A., McKeen, J.D., Gallupe, R.B. (1995). "The evolution of IS job skills: A content analysis of IS job advertisements from 1970 to 1990," MIS Quarterly, Vol. 19, No. 1, pp. 1--27.
46. Touliatos, J., Bedeian, A. G., Mossholder, K. W., and Barkman, A. I. (1984). "Job related perceptions of male and female government, industrial, and public accountants," Social Behavior and Personality, Vol. 12, No. 1, pp. 61--68.
47. Tziner, A., Joanis, C., and Murphy, K.R. (2000). "A comparison of three methods of performance appraisal with regard to goal properties, goal perception, and ratee satisfaction," Group and Organization Management, Vol. 25, No. 2, pp. 175--190.
48. Van Dyne, L. (1993). In-Role and Extra-Role Behaviors: Cross Level and Longitudinal Effects of Individual Similarity to Other Group Members, Unpublished Ph.D. Dissertation, University of Minnesota, Minneapolis.
49. Vitalari, N.P. (1985). "Knowledge as a basis for expertise in systems analysis: An empirical study," MIS Quarterly, Vol. 9, No. 3, pp. 221--241.
50. Wiersma, U.J., and Latham, G.P. (1986). "The practicality of behavioral observation scales, behavioral expectation scales, and trait scales," Personnel Psychology, Vol. 39, No. 3, pp. 619--628.
51. Wiersma, U.J., Van Den Berg, P.T., and Latham, G.P. (1995). "Dutch reactions to behavioral observation, behavioral expectation, and trait scales," Group and Organization Management, Vol. 20, No. 3, pp. 297--309.

Index Terms

• Social and professional topics

  • Professional topics

    • Computing industry

    • Management of computing and information systems

      • Project and people management

• Software and its engineering

  • Software creation and management

    • Software development process management

  • Software notations and tools

    • Software configuration management and version control systems