Accessibility testing of popular Web sites

MediaLT has tested 12 useful and / or popular Web sites and Windows Live for Delta centre, the centre of expertise in accessibility at the Norwegian Directorate of Health. It comes as no surprise to us that we find some outrageously bad results!

Story by: Morten Tollefsen - 11.02.2009

The Web sites were selected by Delta centre. The testing is part of a larger project: Aids or Mainstreaming. The final report from the Health Directorate will be published at a later date.

Testing with usual assistive technologies for the disabled has been the most important approach. To compare the sites we also conducted quantitative testing (with scores). Strictly speaking usability and achieved scores cannot be directly correlated, but since disabled testers in the quantitative testing were also using different assistive technologies, the results do give an indication of the accessibility of the Web sites. In the final report we have given an overall score as a combined assessment of quantitative and qualitative testing.

The maximum score was 35. Nettby came out worst with 10 points and Yr, with 24 points, was the best of the sites where we could carry out all the planned tasks. NAV (Norwegian Labour and Welfare Administration) had the best total with 26 points, but we could not test all functionality here due to log in. Feedback from disabled users of NAV indicates that there are many advanced services (job search, CV posting etc.) that do not work optimally for these users. nav.no was the only site with a home page which meets WCAG 1.0 A requirements (none of the sites meet the AA or AAA requirements).

Disabled users try very hard to cope with bad technology!

In connection with the testing, disabled users were encouraged to share their experiences of selected Web sites. We received many comments, that demonstrate to an even greater extent than our testing how poor the usability of these sites is, even with highly motivated users!

"I really want to do what the others can do, but when I go into what should be fun, I don't understand how to do things. I get angry, frustrated and very sad." (Blind / motor disabled woman)

There can be no doubt that severely visually impaired and other disabled users are struggling to use new online services. This is due to insufficient labelling (e.g.. graphical links without alternative text), the complexity of the pages, and procedures with poor usability. This response from a boy in his teens summarizes the situation quite well:

"I seem to use most Web sites, but that's because I use lots of time and do not give up. Often I have to choose links without having control, the screen reader doesn't always show buttons and Flash is bad. If you are doing this as a blind person you have to know a lot about computing, have loads of patience, and sometimes even this is not enough! "

And a girl in her twenties says:

 "I think it is important to investigate and try them out. Then you are not so "different" just because you cannot join Facebook. This is important for me because "everyone" of my age, and also many others are on Facebook, and I would feel like an outsider if I wasn't there. So I'm trying to find good solutions as a Braille display user. "

Some sites are extremely poor, for example Filmweb which had an overall score of 1:

 "I feel sick when I use Filmweb. There is blinking and movement all over the place. The contrast is poor, and it is impossible to find what I want. When you use magnification the dynamic menus are completely hopeless. I could not make a booking. It took too long and then I got timeout. " (visually impaired man)

Scores in the quantitative test

The score calculations are mainly based on the technical criteria described in the Web Content Accessibility Guidelines (WCAG) 1.0, but to allow for WCAG 2.0 and known accessibility problems we set up our own test system. All tests were performed before the release of WCAG 2.0 in December 2008.

One problem common to almost all Web sites are missing alternative texts for graphics. We have not taken away points for lack of labelling of decorative graphics, but if graphics with links do not have alternative texts, then links will not have meaningful names for screen readers / magnifiers. Only yr.no gets a top score of 3, vgb.no, nettby.no and Norwegian.no score 0! In practice, this means that these sites are difficult or impossible to use for visually impaired users, and others who use synthetic speech or do not display graphics. Alternative text for audio is assessed at the same time. We found, however, no problem with this on these Web sites, nor have we received feedback about problems from hearing impaired users.

Lack of use of headings, i.e.. <h1>, <h2> etc in the HTML-code is another problem which we find repeatedly. It is important to structure a page if many assistive devices are to be used effectively. Proper use of headings is also important when using search engines etc.. All the pages we tested can improve the use of headings (in some cases, the headings can be hidden). Hidden headings can be very useful for main menus, submenus, etc. Sites that had no heading points are actually among the losers in the quantitative test, and this perhaps suggests consistently inadequate / poor coding. Data tables are lacking header <th> tags (some sites do not lose points here because they do not use these tables).

Web sites that use frames have forgotten to give them proper names. This means that screen reader users get less information about the contents of the frame and it is therefore more difficult to navigate on the page.

Just about all the Web sites have documents or other information / functionality that is not based on W3C technology. This in itself need not be a major accessibility problem. There are often PDF files which should also be structured so that everyone can read the contents. Many of the techniques used for HTML also apply to PDF. We have not checked whether this is done as part of our testing. Random tests suggest, however, that almost all the PDF is found lacking with regard to accessibility.

Many Web sites use very small buttons / icons. This can be problematic for both motor impaired users (to hit with the mouse pointer) and visually impaired users. It has been difficult to give scores here because screen resolution, eye / hand control, location, etc. also has significance. We have given 0 points if test simulations with the head mouse / joystick mouse have shown that the objects are obviously too small (a score of either 0 or 1 is given).

Almost all the non-public Web sites use Flash-based advertisements. Flash that is inaccessible for screen readers also creates problems for the rest of the page. The reason for this is that Flash often causes screen readers to lose focus. Facilitation of this type of Flash is quite straightforward, and would ensure that the visually impaired are also exposed to advertising.

There certainly is potential for improvement in these Web sites, and it does not require much to bring about very great improvements!

Universal design

There has been increasing focus on the concept of Universal design. This concept is used in the new Norwegian Anti-Discrimination Act, and will undoubtedly be important for the design of technology.

The principles of universal design probably most important for ICT are:

• Equal opportunities for use
• Flexible in use
• Simple and intuitive to use
• Perceptible information
• Tolerance for error

The principle of usability (easy in use) is in many ways the most important. Simple to use for everyone requires for example flexibility and perceptible information. The Web sites we have tested are perhaps "difficult to use for everyone"? They are at least not "easy to use for everyone", and users with severe visual impairments, cognitive disabilities, and many other disabilities are either unable to use the sites, or they must be very determined to work with very poor user interfaces.

There are not any measurable criteria for what is a universally designed Web site. Measurement of this concept is just as important during development as in the final product, and in all the Web sites it is clear that universal design has not been a serious focus during the development process. The Web sites offering the least functionality come out best with respect to universal design. This is partly because there has been some focus on accessibility and that these Web sites initially contain mostly text. Common to the more interactive sites is that for people using assistive technologies, they work far worse than is necessary!

Usable Web sites are accessible, but accessible Web sites are not necessarily usable. This is clearly demonstrated by our tests. The Web sites that score low in the quantitative test are also the least usable. However, a high score is not necessarily enough for a Web site to be called universally designed. Even if the Web site is based on graphical links without alternative text it may still get a high score, but it will not be usable for anyone who does not show or see the graphics. The quantitative tests shows that all Web sites can gain more points. Perhaps a real review of universal design is only appropriate when a Web site has first achieved top score? This is not unreasonable because if criteria are not met, then some users will have problems. That none of the tested Web sites can be said to be universally designed is an obvious conclusion! Even with a top score it is not certain that some of the Web sites had been approved for flexibility, simplicity, equal access and the other universal design principles.

Some prejudices

It is not particularly difficult to create pages that can be used by everyone. However, it is not entirely unproblematic either. In my work with development of Web sites, lectures and evaluation of Web sites with respect to accessibility / universal design I meet different points of view every day.

Some argue that they are experts because they have looked a little at WAI's guidelines, but they have never used the Web with for example a screen reader or a switch system. Yes, there are many experts, and now that the Anti-Discrimination Law has been passed, there are suddenly many people who know a lot about accessibility. I do not think it is possible for them to completely understand how a blind person perceives a Web site without being blind! I do not think it is possible to understand how a disabled person with very different solutions for input or output uses the technology, without having studied and worked with this. In my opinion, it is also a necessity to have access to the necessary test facilities. Technology for the disabled is constantly evolving, and accessibility experts need to keep informed about these developments. It is not enough to have a little knowledge of some general guidelines. This is about the same as believing that you can drive a car just because you have a racing game for Nintendo DS.

I also meet prejudices that say it is impossible to make Internet accessible. This is wrong! It is all about having a serious approach. It is about appreciating that accessibility is a serious subject. Web sites in this test have not used real experts - of that I am sure! Those who have done anything at all have perhaps had some contact with a random user? Talking to " a random user with some disability or other" is well above average in this test?

News archive