Melody Y. Ivory
Skip Abstract Section
Abstract
The Web enables broad dissemination of information and services; however, the ways in which sites are designed can either facilitate or impede users' benefit from these resources. We present a longitudinal study of web site design from 2000 to 2003. We analyze over 150 quantitative measures of interface aspects (e.g., amount of text on pages, numbers and types of links, consistency, accessibility, etc.) for 22,000 pages and over 1,500 sites that received ratings from Internet professionals. We examine characteristics of highly rated sites and provide three perspectives on the evolution of web site design patterns: (1) descriptions of design patterns during each time period; (2) changes in design patterns across the three time periods; and (3) comparisons of design patterns to those that are recommended in the relevant literature (i.e., texts by recognized experts and user studies). We illustrate how design practices conform to or deviate from recommended practices and the consequent implications. We show that the most glaring deficiency of web sites, even for sites that are highly rated, is their inadequate accessibility, in particular for browser scripts, tables, and form elements.
- Ambuhler, R. and Lindenmeyer, J. 1999. Measuring accessibility. In Proceedings of the Eighth International World Wide Web Conference. Foretec Seminars, Inc., Toronto, Canada, 48--49. http://www.weboffice.ethz.ch/www8.Google Scholar
- Barry, C. and Lang, M. 2001. A survey of multimedia and web development techniques and methodology usage. IEEE MultiMedia 8, 3, 52--60. Google Scholar
- Bernard, M., Liao, C. H., and Mills, M. 2001. The effects of font type and size on the legibility and reading time of online text by older adults. In Proceedings of the Conference on Human Factors in Computing Systems. Vol. 2. Seattle, WA, 175--176. Google Scholar
- Bernard, M. and Mills, M. 2000. So, what size and type of font should I use on my website? Usability News Summer. http://wsupsy.psy.twsu.edu/surl/usabilitynews/2S/font.htm.Google Scholar
- Borges, J. A., Morales, I., and Rodriguez, N. J. 1996. Guidelines for designing usable World Wide Web pages. In Proceedings of the Conference on Human Factors in Computing Systems. Vol. 2. ACM Press, Vancouver, Canada, 277--278. Google Scholar
- Bowers, N. 1996. Weblint: quality assurance for the World Wide Web. In Proceedings of the Fifth International World Wide Web Conference. Elsevier Science Publishers, Paris, France. http://www5conf.inria.fr/fich_html/papers/P34/Overview.html. Google Scholar
- Brewington, B. E. and Cybenko, G. 2000. How dynamic is the web? In Proceedings of the 9th international World Wide Web conference on Computer networks: the international journal of computer and telecommunications netowrking. North-Holland Publishing Co., Amsterdam, The Netherlands, 257--276. Google Scholar
- Brin, S. and Page, L. 1998. The anatomy of a large-scale hypertextual web search engine. Computer Networks and ISDN Systems 30, 1--7, 107--117. Google Scholar
- Brinck, T., Gergle, D., and Wood, S. D. 2001. Usability for the Web: Designing Web Sites That Work. Morgan Kaufmann Publishers, San Francisco. Google Scholar
- Chevalier, A. and Ivory, M. Y. 2003a. Can novice designers apply usability criteria and recommendations to make web sites easier to use? In Proceedings of the 10th International Conference on Human-Computer Interaction. Theory and Practice (Part I), vol. 1. Crete, Greece, 58--62. http://ubit.ischool.washington.edu/pubs/hcii03/hcii03a.pdf.Google Scholar
- Chevalier, A. and Ivory, M. Y. 2003b. Web site designs: Influences of designer's experience and design constraints. Int. J. Hum. Comput. Studies 58, 1, 57--87. http://ubit.ischool.washington.edu/pubs/ijhcs03/designers.pdf. Google Scholar
- Chi, E. H., Pirolli, P., and Pitkow, J. 2000. The scent of a site: A system for analyzing and predicting information scent, usage, and usability of a web site. In Proceedings of the Conference on Human Factors in Computing Systems. ACM Press, The Hague, The Netherlands, 161--168. Google Scholar
- Cho, J. and Garcia-Molina, H. 2000. The evolution of the web and implications for an incremental crawler. In Proceedings of the 26th International Conference on Very Large Data Bases. Morgan Kaufmann Publishers Inc., 200--209. Google Scholar
- Cho, J. and Garcia-Molina, H. 2003. Estimating frequency of change. ACM Trans. Inter. Tech. 3, 3, 256--290. Google Scholar
- Clark, D. and Dardailler, D. 1999. Accessibility on the web: Evaluation & repair tools to make it possible. In Proceedings of the CSUN Technology and Persons with Disabilities Conference. Los Angeles, CA. http://www.dinf.org/csun_99/session0030.html.Google Scholar
- Comber, T. 1995. Building usable web pages: An HCI perspective. In Proceedings of the First Australian World Wide Web Conference, R. Debreceny and A. Ellis, Eds. Norsearch, Ballina, Australia, 119--124. http://www.scu.edu.au/sponsored/ausweb/ausweb95/papers/hypertext/comber/.Google Scholar
- Computer Science and Telecommunications Board. 1997. More Than Screen Deep: Toward Every-Citizen Interfaces to the Nations Information Infrastructure. National Academy Press, Washington, DC. Google Scholar
- Cooper, M. 1999. Universal design of a web site. In Proceedings of the CSUN Technology and Persons with Disabilities Conference. Los Angeles, CA. http://www.dinf.org/csun_99/session0030.html.Google Scholar
- Curry, S. 2002. Marketing---e-marketing evolution---some say banner ads are a waste of time and money, but contrary to popular belief, online marketing is finally growing up take a look at how three companies have used electronic marketing campaigns to raise revenues---and profits. Sales & Marketing Management 154, 6, 32.Google Scholar
- Dahlen, M. 2001. Banner ads through a new lens. Journal of Advertising Research 41, 4, 8.Google Scholar
- Donatelli, B. 2003. Bandwagon---online adviser: Hunting where the ducks are: Online banner ads in 2002. Campaigns & Elections 24, 4, 3.Google Scholar
- Easton, V. J. and McColl, J. H. 1997. Statistics glossary v1.1. http://www.stats.gla.ac.uk/steps/glossary/index.html.Google Scholar
- Fetterly, D., Manasse, M., Najork, M., and Wiener, J. 2003. A large-scale study of the evolution of web pages. In Proceedings of the 12th International Conference on World Wide Web. ACM Press, Budapest, Hungary, 669--678. Google Scholar
- Fetterly, D., Manasse, M., Najork, M., and Wiener, J. L. 2004. A large-scale study of the evolution of web pages. Softw. Pract. Exper. 34, 2, 213--237. Google Scholar
- Flanders, V. and Willis, M. 1998. Web Pages That Suck: Learn Good Design by Looking at Bad Design. SYBEX, San Francisco, CA. Google Scholar
- Fleming, J. 1998. Web Navigation: Designing the User Experience. O'Reilly & Associates, Sebastopol, CA. Google Scholar
- Fogg, B., Marshall, J., Osipovich, A., Varma, C., Laraki, O., Fang, N., Paul, J., Rangnekar, A., Shon, J., Swani, P., and Treinen, M. 2000. Elements that affect web credibility: Early results from a self-report study. In Proceedings of the Conference on Human Factors in Computing Systems. ACM Press, The Hague, The Netherlands, 287--288. http://www.webcredibility.org/WebCredEarlyResults.ppt. Google Scholar
- Fogg, B. J., Marshall, J., Laraki, O., Osipovich, A., Varma, C., Fang, N., Paul, J., Rangnekar, A., Shon, J., Swani, P., and Treinen, M. 2001. What makes web sites credible?: a report on a large quantitative study. In Proceedings of the ACM CHI'01 Conference on Human Factors in Computing Systems. Vol. 1. Seattle, WA, 61--68. Google Scholar
- Forrester Research. 1999. Why most web sites fail. http://www.forrester.com/Research/ReportExcerpt/0,1082,1285,00.html.Google Scholar
- Furnas, G. W. 1997. Effective view navigation. In Proceedings of Conference on Human Factors in Computing Systems. Vol. 1. ACM Press, Atlanta, GA, 367--374. Google Scholar
- Harrison, A. and Shin, Y.-M. 2003. Using document image analysis to evaluate web page design. In Proceedings of the Sixth Annual UW Undergraduate Research Symposium. Poster Session. University of Washington. http://webtango.ischool.washington.edu/pages/showAbstract.php?absid=urp03&abstype=Publication.Google Scholar
- Harwood, P. and Rainie, L. 2004. Use of the Internet in places other than home or work: A Pew Internet project data memo. http://www.pewinternet.org/pdfs/PIP_Other_Places.pdf.Google Scholar
- Ivory, M. Y. 2001. An empirical foundation for automated web interface evaluation. Ph.D. thesis, University of California, Berkeley. Ph.D. thesis. Google Scholar
- Ivory, M. Y. 2003a. Automated Web Site Evaluation: Researchers' and Practitioners' Perspectives. Human-Computer Interaction Series, vol. 4. Kluwer Academic Publishers, Dordrecht, The Netherlands.Google Scholar
- Ivory, M. Y. 2003b. Characteristics of web site designs: Reality vs. recommendations. In Proceedings of the 10th International Conference on Human-Computer Interaction. Theory and Practice (Part I), vol. 1. Crete, Greece, 773--777.Google Scholar
- Ivory, M. Y. 2004. SmartSites toolkit for evaluating course web sites. Tech. Rep. IS-TR-2004-12-02, Information School, University of Washington. http://hdl.handle.net/1773/2030.Google Scholar
- Ivory, M. Y. and Chevalier, A. 2002. A study of automated web site evaluation tools. Tech. Rep. 02-10-01, University of Washington, Department of Computer Science and Engineering. http://ubit.ischool.washington.edu/pubs/tr02/toolstudy.pdf.Google Scholar
- Ivory, M. Y. and Hearst, M. A. 2002a. Improving web site design. IEEE Internet Computing 6, 2, 56--63. http://ubit.ischool.washington.edu/pubs/ieee02/ieee-final.pdf. Google Scholar
- Ivory, M. Y. and Hearst, M. A. 2002b. Statistical profiles of highly-rated web site interfaces. In Proceedings of the Conference on Human Factors in Computing Systems. CHI Letters, vol. 4. Minneapolis, MN, 367--374. http://ubit.ischool.washington.edu/pubs/chi02/chi2002.pdf. Google Scholar
- Ivory, M. Y., Mankoff, J., and Le, A. 2003. Using automated tools to improve web site usage by users with diverse abilities. IT&Society 1, 3, 195--236. http://www.stanford.edu/group/siqss/itandsociety/v01i03/v01i03a11.pdf.Google Scholar
- Ivory, M. Y., Sinha, R. R., and Hearst, M. A. 2000. Preliminary findings on quantitative measures for distinguishing highly rated information-centric web pages. In Proceedings of the 6th Conference on Human Factors & the Web. Austin, TX. http://ubit.ischool.washington.edu/pubs/hfw00/hfw00.pdf.Google Scholar
- Ivory, M. Y., Sinha, R. R., and Hearst, M. A. 2001. Empirically validated web page design metrics. In Proceedings of the Conference on Human Factors in Computing Systems. Vol. 1. Seattle, WA, 53--60. http://ubit.ischool.washington.edu/pubs/chi01/chi2001.pdf. Google Scholar
- Jackson, A. 1999. Web page design: A study of three genres. Master's paper, University of North Carolina - Chapel Hill.Google Scholar
- Jackson-Sanborn, E., Odess-Harnish, K., and Warren, N. 2002. Website accessibility: A study of ADA compliance. Tech. Rep. TR-2001-05, University of North Carolina - Chapel Hill, School of Information and Library Science. http://ils.unc.edu/ils/research/reports/accessibility.pdf.Google Scholar
- Jain, R. 1991. The Art of Computer Systems Performance Analysis. Wiley-Interscience, New York.Google Scholar
- Kaiser, J. 1998. Browser-safe colors. Web Design June 15. http://webdesign.about.com/compute/webdesign/library/weekly/aa061598.htm.Google Scholar
- Kim, N. and Fogg, B. J. 1999. World Wide Web credibility: What effects do advertisements and typos have on the perceived credibility of web page information? Ph.D. thesis, Stanford University.Google Scholar
- Klee, M. and Schroeder, W. 2000. Report 2: How business goals affect site design. In Designing Information-Rich Web Sites. User Interface Engineering, Bradford, MA.Google Scholar
- Koyani, S., Allison, S., Bailey, R., Chaparro, B., Ivory, M., and Wheeler, S. 2003a. Use of research-based guidelines in the development of websites. In Proceedings of the Conference on Human Factors in Computing Systems. Extended Abstracts. New York, Fort Lauderdale, FL, 696--697. http://ubit.ischool.washington.edu/pubs/chi03/sigchi03.pdf. Google Scholar
- Koyani, S. J., Bailey, R. W., and Nall, J. R. 2003b. Research-Based Web Design & Usability Guidelines. National Institutes of Health.Google Scholar
- Landesman, L. and Schroeder, W. 2000. Report 5: Organizing links. In Designing Information-Rich Web Sites. User Interface Engineering, Bradford, MA.Google Scholar
- Lin, J. and Landay, J. A. 2002. Damask: A tool for early-stage design and prototyping of multi-device user interfaces. In Proceedings of The 8th International Conference on Distributed Multimedia Systems. San Francisco, CA, 573--580.Google Scholar
- Lowgren, J. and Nordqvist, T. 1992. Knowledge-based evaluation as design support for graphical user interfaces. In Proceedings of the Conference on Human Factors in Computing Systems. ACM Press, Monterey, CA, 181--188. Google Scholar
- Madden, M. and Rainie, L. 2003. America's online pursuits: The changing picture of who's online and what they do. http://www.pewinternet.org/pdfs/PIP_Online_Pursuits_Final.PDF.Google Scholar
- Mahajan, R. and Shneiderman, B. 1997. Visual and textual consistency checking tools for graphical user interfaces. IEEE Trans. Soft. Eng. 23, 11, 722--735. Google Scholar
- Murch, G. M. 1985. Colour graphics---blessing or ballyhoo? Computer Graphics Forum 4, 2, 127--135.Google Scholar
- National Cancer Institute. 2001. Research-based web design & usability guidelines. http://usability.gov/guidelines/.Google Scholar
- Nielsen, J. 1998. Web usability: Why and how. Users First! September 14. http://www.zdnet.com/devhead/stories/articles/0,4413,2137433,00.html.Google Scholar
- Nielsen, J. 1999. User interface directions for the web. Comm. ACM 42, 1, 65--72. http://www.acm.org:80/pubs/citations/journals/cacm/1999-42-1/p65-nielsen/. Google Scholar
- Nielsen, J. 2000. Designing Web Usability: The Practice of Simplicity. New Riders Publishing, Indianapolis, IN. Google Scholar
- Ntoulas, A., Cho, J., and Olston, C. 2004. What's new on the web?: the evolution of the web from a search engine perspective. In Proceedings of the 13th international conference on World Wide Web. ACM Press, New York, NY, USA, 1--12. Google Scholar
- Ratner, J., Grose, E. M., and Forsythe, C. 1996. Characterization and assessment of HTML style guides. In Proceedings of the Conference on Human Factors in Computing Systems. Vol. 2. ACM Press, Vancouver, Canada, 115--116. Google Scholar
- Rosenfeld, L. and Morville, P. 1998. Information Architecture for the World Wide Web. O'Reilly & Associates, Sebastopol, CA. Google Scholar
- Sano, D. 1996. Designing Large-Scale Web Sites: A Visual Design Methodology. Wiley Computer Publishing, John Wiley & Sons, Inc., New York, NY. Google Scholar
- Sawyer, P., Danca, R., and Schroeder, W. 2000. Report 6: Myths of page layout. In Designing Information-Rich Web Sites. User Interface Engineering, Bradford, MA.Google Scholar
- Sawyer, P. and Schroeder, W. 2000. Report 4: Links that give off scent. In Designing Information-Rich Web Sites. User Interface Engineering, Bradford, MA.Google Scholar
- Scanlon, T. and Schroeder, W. 2000a. Report 1: What people do with web sites. In Designing Information-Rich Web Sites. User Interface Engineering, Bradford, MA.Google Scholar
- Scanlon, T. and Schroeder, W. 2000b. Report 7: Designing graphics with a purpose. In Designing Information-Rich Web Sites. User Interface Engineering, Bradford, MA.Google Scholar
- Schmetzke, A. 2004a. Web accessibility survey homepage. http://library.uwsp.edu/aschmetz/Accessible/websurveys.htm.Google Scholar
- Schmetzke, A. 2004b. Web page accessibility on University of Wisconsin campuses: 2004 survey and six-year trend data. http://library.uwsp.edu/aschmetz/Accessible/UW-Campuses/Survey2004/contents2004.htm.Google Scholar
- Schriver, K. A. 1997. Dynamics in Document Design. Wiley Computer Publishing, John Wiley & Sons, Inc., New York, NY. Google Scholar
- Shedroff, N. 1999. Recipe for a successful web site. http://www.nathan.com/thoughts/recipe.Google Scholar
- Shneiderman, B. 1997. Designing information-abundant web sites: Issues and recommendations. Int. J. Hum.-Comput. Studies 47, 1, 5--29. Google Scholar
- Smith, S. L. 1986. Standards versus guidelines for designing user interface software. Behaviour and Information Technology 5, 1, 47--61.Google Scholar
- Souza, F. d. and Bevan, N. 1990. The use of guidelines in menu interface design: Evaluation of a draft standard. In Proceedings of the Third IFIP TC13 Conference on Human-Computer Interaction, G. Cockton, D. Diaper, and B. Shackel, Eds. Elsevier Science Publishers, Cambridge, UK, 435--440. Google Scholar
- Spool, J. M., Klee, M., and Schroeder, W. 2000. Report 3: Designing for scent. In Designing Information-Rich Web Sites. User Interface Engineering, Bradford, MA.Google Scholar
- Spool, J. M., Scanlon, T., Schroeder, W., Snyder, C., and DeAngelo, T. 1999. Web Site Usability: A Designer's Guide. Morgan Kaufmann Publishers, Inc., San Francisco, CA. Google Scholar
- Stein, L. D. 1997. The rating game. http://stein.cshl.org/~lstein/rater/.Google Scholar
- The International Academy of Arts and Sciences. 2000. The webby awards 2000 judging criteria. http://www.webbyawards.com/judging/criteria.html.Google Scholar
- The Poynter Institute. 2004. Poynter online - design/graphics. http://www.poynter.org/subject.asp?id=11.Google Scholar
- van Duyne, D. K., Landay, J. A., and Hong, J. I. 2002. The Design of Sites: Patterns, Principles, and Processes for Crafting a Customer-Centered Web Experience. Addison-Wesley, Boston. Google Scholar
- Vora, P. R. 1998. Design/methods & tools: Designing for the web: A survey. interactions 5, 3, 13--30. Google Scholar
- W3C. 1999. Web content accessibility guidelines 1.0. http://www.w3.org/TR/WAI-WEBCONTENT/.Google Scholar
- WatchFire. 2002. Welcome to bobby worldwide. http://bobby.watchfire.com/bobby/html/en/index.jsp.Google Scholar
- Witten, I. H. and Frank, E. 1999. Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations. Morgan Kaufmann, San Francisco. Google Scholar
Index Terms
- Evolution of web site design patterns
Recommendations
Empirically validated web page design metrics
CHI '01: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsA quantitative analysis of a large collection of expert-rated web sites reveals that page-level metrics can accurately predict if a site will be highly rated. The analysis also provides empirical evidence that important metrics, including page ...
Statistical profiles of highly-rated web sites
CHI '02: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsWe are creating an interactive tool to help non-professional web site builders create high quality designs. We have previously reported that quantitative measures of web page structure can predict whether a site will be highly or poorly rated by experts,...
Reviews
Mercedes Martinez Gonzalez
The design of Web sites influences their usability, accessibility, and acceptance by users. Hence, the evaluation of Web sites measures their quality in terms of these criteria. This paper contains a survey of the evolution of Web design patterns. The study uses 150 quantitative measures and statistical models, and covers three years (2000, 2002, and 2003). It compares results from each year to obtain conclusions about the evolution during this period of time. In this period, some of the most current components of Web pages became popular (scripting, animation, multimedia, and so on), guaranteeing the timeliness of the study. The paper starts with an overview of the current state of Web design studies. A presentation of the methodology, the most relevant results, and their interpretation follows. General conclusions about what these results mean in terms of accessibility, usability, and design guidelines are also part of the study. There are other types of Web site evaluations, but this methodology is original in two ways: its use of quantitative measures and empirical data, and its use of profiles as a basis for comparison (these profiles are calculated from experts' ratings of Web pages). The paper is clear and well organized. It will be as interesting to experts in Web design and evaluation as it will be to nonexpert readers, who are just curious about what makes a Web site more attractive to its potential visitors. The expert reader may have to consult some additional references and papers by the authors (included in the references section) to obtain a technically exhaustive description of this evaluation, the methodology, and the measures used. Online Computing Reviews Service
Access critical reviews of Computing literature here
Become a reviewer for Computing Reviews.
Comments