Web Accessibility Evaluation of Government Websites for People with Disabilities in Turkey

Home , Bobby (software), SortSite, Validator

Journal of Advanced Management Science Vol. 4, No. 3, May 2016

Yakup AKGÜL Dumlupınar University, Simav Vocational School Email: [email protected]

Kemal VATANSEVER Pamukkale University, Applied Science School Email: [email protected]

Abstract—The Web is a progressively more important being encouraged to adapting with the coming future. resource in many aspects of life: government, commerce Turkey has declared the “Vision 2023”, which targets and more. As governments to continue to provide businesses establishment of a resourceful and modern country by and citizens with new value-added e-services, citizens with 2023 through effective use of information and disabilities are still being deprived from taking full communication technology [3]. The government of advantage of these services. While the proportion of people with disabilities (visual impairment, hearing impairment, Turkey has realized the importance of ICT to improve the cognitive disability etc.) in society has been rapidly delivery of information and services to disabled citizens. increasing due to the demographic trends long documented And now the changes are being seen in government by many researchers, governmental leaders have paid little initiatives [4]. The websites of all the ministries and attention to their needs when planning and implementing divisions are developed under the technical assistance of Web projects. Therefore, it is essential that all citizens must Republic of Turkey E-Transformation Turkey Prime have equal accessible opportunities to all e-government Ministry State Planning Organization Interoperability recourses. This research evaluates the accessibility of each Principles Project Guide and are in working for the last of the 25 e-Government websites in Turkey by people five years [5]. disabilities based on the Web Content Accessibility Guidelines (WCAG) 1.0 and 2.0 and using automated A considerable number of users of the Web have testing tools. The results of study indicate that the prevalent various types of disabilities such as vision, hearing, priority-1 accessibility barriers identified in this study were motor and cognitive impairments [6]. Studies show that related to the absence of text equivalents for non-text presently most of the government websites are elements, and the failure of the static equivalents for inaccessible for the impaired users [7]. However, more dynamic content to get updated when the dynamic content than one billion people in the world are disable and this  changes. number is increasing day by day as the population increases [8]. Turkey has an estimated population of 76 Index Terms—web accessibility, disability, e-government million, out of which about 8, 5 million are disable [9].

The accessibility of these web sites, especially by the people with disabilities, has not been evaluated to date. I. INTRODUCTION This has motivated us to assess the accessibility of e- The rapid growth of new Information and government web sites for people with disabilities using Communication Technologies (ICTs) imposes the automatic testing tools for checking of target websites. adoption of these technologies in different parts of the The purpose of this study is limited to the accessibility modern life, including the governmental side. assessment of the central government websites and to EGovernment can be defined as the process by which the find out whether the web based public services are government can deliver services and information to its provided in equitable manner to all the citizens. citizens via the internet [1]. Access to government The rest of the paper is organized in six sections: In through Web interfaces has become commonplace in Section 2 presents we review the relevant works. Section recent times as a consequence of pervasive use of the 3 presents W3C standards and guidelines. Section 4 Internet for access to information and services [2]. The describes web accessibility evaluation tools. Section 5 use of information and communication technology (lCT) describes the adopted methodology to make the complete has been playing a vital role in the 21st century due to analysis of selected websites of government. Section 6 globalization and the governments of the countries are presents the results and their detailed description. Section 7 presents limitations and future work. Section 8 Manuscript received September 1, 2014; revised December 21, concludes the paper with recommendation. 2014.

II. RELATED WORKS automated testing tools and identified many usability and accessibility issues related to Malaysia e-Government Many studies have been carried out in the field of Web websites. Al-Radaideh et al. [20] evaluated the accessibility. These studies used different techniques and accessibility of major E-government Websites in Jordan different measures for assessing the accessibility of by people with disabilities with conformance to Web different Websites, especially the government ones. Also Content Accessibility Guidelines (WCAG) 1.0. Results these studies found that large percentage of Websites showed that all tested Websites did not address the issue have serious problems in their accessibility. In this of disability-accessibility and they have many Web section we briefly mention some work that has been done accessibility problems. Abu-Doush et al. [21] evaluated a in the field of Web accessibility. Definition of set of Jordan e-government websites using 20 blind and accessibility is “making web content available to all visually impaired volunteers and at the same time individuals, regardless of any disabilities or conducted a survey on e-government websites developers. environmental constraints they experience” [10]. The Al Mourad and Kamoun [22] evaluated the accessibility provision of physical access to appropriate hardware and of each of the 21 Dubai e-Government websites based on software to enable access to the web; it can mean the the Web Content Accessibility Guidelines (WCAG) 1.0 provision of add-on technologies to widen access to the and using automated testing tools. The results of research web, for example through the use of assistive revealed that many Dubai e-Government sites did not technologies such as screen reading software, screen meet the minimum W 3C accessibility conformance level. magnification, alternative mouse and keyboard devices, AbuAli et al. [23] evaluated Jordan E-Government alternative pointing devices, refreshable Braille displays Website from the accessibility perspective. The results and voice input [11], [6]. from the evaluation process showed that Jordan E- Abdul Latif and Masrek [12] undertaken with the Government Website lacks accessibility and needs purpose of identifying the accessibility of Malaysian e- further improvements to improve its quality. Luj´an- government websites based on the World Wide Web Mora et al. [24] analysed the accessibility of a group of Consortium (W3C). The result of the analysis indicated e-government websites of all South American countries that there were no single Malaysian e-government and Spain. The results of research showed that the websites that passed the W3C Priority 1 accessibility majority of e-government websites do not provide checkpoints. Dominic et al. [13] have used diagnostic to adequate levels of web accessibility. evaluate the Asian egovernment websites in terms of Some similar studies on accessibility of web sites and technical aspects such as loading time, page rank, web contents were also conducted by Mankoff et.al [10], frequency of update, traffic, mark-up validation, Lazar et.al [6], Venter et.al [25], Choudrie et al. [26], Shi accessibility errors, etc. Baowaly and Bhuiyan [14] [27], Potter [28], Abanumy et al. [29], Rowena Cullen concentrated on mainly two things; firstly, it briefly and Caroline Houghton [30], Salon et al. [31], Kuzma examined accessibility guidelines, evaluation methods [32], Kuzma et al. [33], Kuzma et al. [34], Hong et al. and analysis tools. Secondly, it analyzed and evaluated [35], Hong et al.[36], Basdekis et al.[37], Kurniawan and the web accessibility of e-Government websites of Zaphiris [38], Choi et al.[39], Johnson and Kent [40], Bangladesh according to the “W3C Web Content Evans-Cowley [41], Freire et al. [42], Paris [43], Goette Accessibility Guidelines”. Baowaly et al. [15] analyzed et al. [44], Jaeger [45], Jaeger [46], Shi [47], Rabaiah and and evaluated accessibility of government websites’ in Vandijck [48], Huang [49], Jati and Dominic [50], perspective of developing countries. They taked Loiacono et al. [51], Mehmood [52], Baguma ve Lubega Bangladesh as a case study. Bakhsh and Mehmood [16] [53], Baguma et al. [54] and give suggestions for evaluated the websites of central government in Pakistan improvements. including all ministries and divisionsusing accessibility evaluation tools based on World Wide Web Consortium's III. WEB ACCESSIBILITY AND GUIDELINES (W3C) web accessibility standards. The results showed that most of the web sites were not developed according Web accessibility can be defined as the degree to to the accessibility standards for disabled persons. which a site is accessible to the largest possible range of Kuzma [17] assessed the accessibility of e-Government people. The more people are able to access a website, the websites for 12 developing and developed countries. She more accessible is the site. At its core, Web accessibility identified serious accessibility issues for the tested e- emphasizes making website accessible to persons with Government sites, even for websites belonging to disabilities and involves removing potential barriers to governments who stated adherence to W3C accessibility access caused by inconsiderate website designs [55]. standards and UN legislations. Goodwin et al. [18] Web accessibility primarily benefits people with conducted a global web accessibility analysis of e- disabilities. However, as an accessible website is Government websites from the United Nations member designed to meet different user needs, preferences, skills states. The study revealed that, with few exceptions, and situations, this flexibility can also benefit people government websites of developed countries are more without disabilities in certain situations, “such as people accessible than those of developing countries. The study using a slow Internet connection, people with temporary also found that e-Government websites that are disabilities such as a broken arm, and people with recognized as mature and of high quality are more likely changing abilities due to aging” [56]. The World Wide to be accessible. Isa et al. [19] who used several Web Consortium (W3C) is an international organization

dedicated to the standardization of the World Wide Web TABLE I. WAI CONFORMANCE CLAIMS (WC3). In 1996, W3C established the Web Accessibility Initiative (WAI) campaigning for a more accessible Web Conformance Level Website Accessibility Checkpoint for persons with disabilities. For the consortium, Web accessibility was defined as “access to the Web by All priority 1 checkpoints are met. This is the minimum (basic) W3C requirement. Otherwise everyone, regardless of disability” [55]. WAI-A one or more groups of people will find it In 1999, the Web Accessibility Initiative (WAI), a (basic accessibility) impossible to access information from the Project by the World Wide Web Consortium (W3C) website. This is the minimum requirement and must be met. published the Web Content Accessibility Guidelines All priority 1 and 2 checkpoints are satisfied; (WCAG) version 1.0 [56]. These guidelines were widely otherwise one or more groups of people will WAI-AA find it difficult to access information from the accepted in many countries around the world as the (intermediate website. This conformance level status should accessibility) definitive guidelines on creating accessible websites. The be met, as it will remove significant barriers to WAI approach to Web accessibility revolves around accessing Web documents. three interrelated fronts: (i) the content accessibility of All priority 1, 2 and 3 checkpoints are websites for persons with disabilities to perceive, satisfied; otherwise one or more groups of people will find it somehow difficult to access WAI-AAA understand, and use; (ii) making Web browsers and information from the website. This (high accessibility) media players usable for persons with disabilities by conformance level status may be addressed by making them operable through assistive technologies and Web developers to improve access to Website documents. (iii) Web authoring tools and technologies to support production of accessible Web content and sites, so that persons with disabilities can use them effectively. An IV. WEB ACCESSIBILITY EVALUATION TOOLS accessible web site is very similar to an accessible building. An accessible building offers curb cuts, ramps, After Web accessibility evaluation tools are software and elevators to allow a person with disabilities to enter programs or online services that are used to check your and navigate through the building with ease. Hence, an website's accessibility level under web accessibility accessible web site offers similar functionality [12]. guidelines. There is a huge number of accessibility tools for commercial purposes or freely available on the web However, on 11 December 2008, the WAI released the such as Watch Fire Bobby, AChecker, Cynthia Says, WCAG version 2.0 to be up to date while being more EvalAccess, Accessibility Valet Demonstrator technology neutral [57]. (WebThing), AccMonitor Online (HiSoftware), Currently, there are a number of guidelines and tools Torquemada (WebxTutti), Wave 3.5 (WebAIM) and Web designers and webmasters can use to make their Tawdis etc. Some good free web-based website websites accessible to people with disabilities. Such accessibility evaluation tools are linked in [59], [60], [61], guidelines include the Web Content Accessibility [62]. A complete list of accessibility evaluation tools is in guidelines (WCAG) developed by the World Wide Web W3C [63]. These tools are very useful for programmers Consortium (W3C), the US government’s Section 508 and designers to determine whether or not their sites Initiative, Americans with Disabilities Act (ADA), follow WCAG. During the design, implementation, and Australians with Disabilities Act and the National maintenance phases of Web development if these tools Institute on Ageing Guidelines (NIA). Similar guidelines are used carefully, it can help the targeted users in preventing accessibility barriers, repairing encountered exist in Canada, UK and Portugal. In addition to the barriers, and improving the overall quality of Web sites guidelines, automated software tools that help in finding [64]. accessibility flaws in websites before the sites are This study will use Automatic evaluation tools such as publicly posted, are available. Such tools include bobby, AChecker, eXaminator, TAW, Total Validator, WAVE, ramp, infocus and a-prompt. More so new versions of Web AccessibilityAssessment Tool, EvalAccess 2.0, web development tools such as dream weaver and front Cynthia Says, MAGENTA, HERA, Amp and Sort Site page include tools that assist developers with which is considered as the web accessibility test tool accessibility related issues [6]. The most common which able to provide relatively complete analysis of standards Based website Design and development are website accessibility and have been the pioneers and are W3C Web Content Accessibility Guidelines 1.0 or 2.0 the most well-known, due to their usability, ease of use (WCAG 1.0 or WCAG 2.0). WCAG 2.0 was approved as and its quick results. an ISO/IEC 40500 International accessibility standard in V. METHODOLOGY October 2012 [58]. The meaning of that more countries can formally adopt WCAG 2.0 and many countries are In this study, the 25 the official website of the updating their laws to the new version. The W3C Web governments have been analysed. The home page of each Accessibility Initiative (WAI) defines three possible one of the websites has been analysed from three points accessibility conformance levels, as illustrated in Table I. of view: HTML and CSS validity; web accessibility; and,

current use of HTML5 and ARIA. The home page of a problems, those that need to be reviewed by an expert. website is the first contact a user has with the website. If Total Validator is an HTML validator, an accessibility the home page shows problems or is not accessible, it validator, a spell checker, and a broken links checker, all would be very difficult that a disabled user can access included into one tool [71]. This tool is provided in two other pages of the website. Therefore, it is essential to versions: the basic tool for free and the professional tool ensure the accessibility of the home page of a website. that must be purchased. Finally, WAVE is an online All the tests of a web page were conducted during the automatic evaluation tool that helps web developers to same day in order to avoid changes in its content. make their web content more accessible [72]. However, WAVE cannot completely state if a web page is A. HTML and CSS Validity accessible, only a human can determine true accessibility. Two automatic evaluation tools have been used to WAVE detects HTML5 and Accessible Rich Internet evaluate the validity of the HTML and CSS of the Applications (ARIA) features, such as header, footer, websites. The first automatic tool is the Markup ARIA landmarks and roles, and so on. Besides, WAVE Validation Service, a free service by the W3C [65]. This also provides an API that allows automated and remote automatic tool checks the markup validity of web pages accessibility analysis of web pages using the WAVE in HTML, XHTML, SMIL, MathML, etc. According to processing engine. Web Accessibility Assessment Tool is the W3C [66], “Validating web documents is an a Java application developed by the EU FP7 important step which can dramatically help improving ACCESSIBLE project [73]. It evaluates a website and ensuring their quality, and it can save a lot of time according to WCAG 2.0 (level A, level AA, and level and money”. The result of the Markup Validation Service AAA). Providing an option for the users to select among is summarized in the number of errors and warnings in a the success criteria they want to check. Another option is web page. The second tool is the CSS Validator Service, saving a report on the user computer as a PDF file. This another free service by the W3C [67]. Not only, this tool tool can evaluate more than one page; users can define evaluates the style sheets of a web page its conformance the number of pages for evaluation. The results of the with W3C open standards and the CSS specifications. It evaluation are categorized into: errors and warnings. can also detect when CSS poses some risks in terms of EvalAccess is being developed by the Laboratory of HCI usability. It can find errors, typos, or incorrect uses of for Special Needs at the University of the Basque CSS. Country (UPV-EHU). EvalAccess web service checks web pages accessibility, based on the WAI's WCAG 1.0 B. Web Accessibility guidelines. It has been implemented as a web service to Thirteen automatic evaluation tools have been used to allow any other application to use it [74]. Cynthia Says evaluate the accessibility of the websites analysed in this tests your page against predefined checkpoint groups to study: AChecker, eXaminator, TAW, Total Validator, validate it against the US Access Board’s Section 508 or WAVE, Web AccessibilityAssessment Tool, EvalAccess, the W3C’s WCAG 2.0 A-AAA Accessibility Guidelines Cynthia Says, MAGENTA, HERA, Amp and Sort Site [75]. M.A.G.EN.T.A. 2.0 (Multi-Analysis of Guidelines AChecker [68] is an online free service that produces a by an Enhanced Tool for Accessibility) is a system to report of accessibility problems according to different evaluate accessibility of Web sites by checking their guidelines (Section 508, WCAG 1.0 and WCAG 2.0). HTML and CSS code through guidelines, which are to be AChecker classifies accessibility problems into three specified through an XML-compliant specification categories: known problems (problems that have been language called G.A.L. (Guideline Abstract Language) identified with certainty as accessibility barriers), likely that maintains the guidelines separated from the problems (problems that have been identified as probable underlying logic. M.A.G.EN.T.A. 2. 0 is able to validate barriers, but require a human to make a decision) and the accessbility of web pages in relation to the following potential problems (problems that AChecker cannot guidelines: WCAG 2.0 (Level A, AA, AAA), Stanca Act, identify, that require a human decision). AChecker also Visually Impaired [76]. HERA is a tool to check the provides an Application Programming Interface (API) accessibility of Web pages accoridng to the specification that allows remote accessibility analysis. eXaminator is Web Content Accessibility Guidelines (WCAG 1.0). an online free service to check the accessibility of a web HERA performs a preliminary set of tests on the page page developed by Carlos Benav´ıdez [69]. eXaminator and identifies any automatically detectable errors or checks the application of the WCAG 2.0 [57] on the checkpoints met, and which checkpoints need further HTML and CSS contents in a web page and summarizes manual verification [77]. The Accessibility Management the results in an overall score from 1 to 10 that is quite Platform (AMP) provides the infrastructure to facilitate easy to understand by everybody. Of course, the score all aspects of a successful accessibility compliance calculated by eXaminator is a fast check of accessibility, program. AMP’s powerful testing engine and work flow, but automatic evaluation does not cover all of the success reporting support, accessible development best practices, criteria in WCAG 2.0. TAW is a limited online free and extensive training course library allow organizations service to check the web accessibility against WCAG 1.0 to quickly and efficiently incorporate accessibility and 2.0 [70]. TAW classifies accessibility problems into compliance into existing development processes. This automatic problems, those in which the tool is certain ensures that organizations have the infrastructure to that the problem violates the guidelines and manual rapidly conform to Section 508, the Web Content

Accessibility Guidelines (WCAG), the Americans with Policy with 2 errors, Republic of turkey ministry of Disabilities Act (ADA), and other leading accessibility defense with 4 errors. The worst results were obtained requirements [78].SortSite checks sites against the W3C with the website of the Republic of Turkey Ministry of WCAG 1.0 and 2.0 accessibility standards, and Youth and Sports, with 525 errors, and the Republic of compliance with Section 508 of the Rehabilitation Act Turkey Ministry of Economy, with 306 errors. Regarding [79]. the CSS validation, thebest results were the website of C. Current Use of HTML5 and ARIA the Presidency of the Republic of Turkey, Republic of HTML5 is the latest standard and which is an example Turkey Ministry of Justice, Republic of turkey ministry of modern technology. The first draft of HTML5, was of defense, Republic of Turkey Ministry of Interior, published by the W3C in January 2008 [80], HTML5 is Republic of Turkey Ministry of Transport, Maritime expected to be completed and published at the end of Affairs and Communications with 0 errors. On the 2014. Unfortunately, six years later the use of the new opposite side, Republic of Turkey ministry of version of the markup language of the Web is not very development web site presented the highest number of common yet. HTML5 updates the specification to errors with 353. include latest advances and best practices in web B. Web Accessibility development. HTML5 also includes new accessibility features that will improve the accessibility of websites. Due to the lack of space, we cannot include the whole ARIA, another standard of the W3C, addresses the lack results of the web accessibility analysis. Therefore, Fig. 2 of accessibility of many web pages. Complex web summarizes the number of problems detected with applications become inaccessible when assistive automatic evaluation tools and some information has to technologies cannot determine the semantics behind be discarded. Unfortunately, the home pages of all the portions of a document or when the user is unable to websites have accessibility issues. In Fig. 2, column effectively navigate to all parts of it in a usable “AChecker” represents the number of “known problems” way. WAI-ARIA divides the semantics into roles (the that have been detected. According to AChecker, these type defining a user interface element) problems should be fixed. “Likely” and “potential errors” and states and properties supported by the roles. ARIA have not been included in the figure. Column defines ways to make Web content and Web applications “eXaminator” shows the global score provided by this (especially those developed with Ajax and JavaScript) tool, a value from 1 to 10: the higher the value, the better more accessible to people with disabilities. It especially the accessibility of the web page. Column “TAW 1.0 P1” helps with dynamic content and advanced user interface indicates the number of issues to pass the WCAG 1.0 controls developed with Ajax, HTML, JavaScript, and priority 1 (A level) requirement that can be automatically related technologies. ARIA enhances accessibility of detected. The manual errors have been discarded because interactive controls (such as tree menus, drag and drop, they required additional human intervention. Column sliders, sort controls, etc.), provides content roles for “TAW 2.0 Problems” provides the number of problems identifying page structure (navigation, search, main that should be corrected because there is a certainty about content, etc.), areas that can be dynamically updated them. “Warnings” and “Not verified problems” have also (called "live regions" in ARIA), better support for been discarded and they are not showed in the figure. keyboard accessibility and interactivity, and much more. Column “TV” Errors WCAG 2.0 A” shows the number ARIA is a set of special accessibility attributes which can of errors of WCAG 2.0 priority 1 (A level) detected by be added to any markup, but is especially suited to Total Validator. The other errors have been discarded. HTML. The role attribute defines what the general type Column “WAVE Errors” provides the number of errors of object is (such as an article, alert, or slider). ARIA is detected by WAVE. “Alerts” have also been discarded. supported by most up-to-date browsers and screen Column “Web Accessibility Assessment Tool” shows the readers. It is also supported by many scripting libraries. number of errors of WCAG 2.0 priority 1 (A level) Usage of HTML 5 and ARIA together make web content requirement that can be automatically detected. and web applications more accessible to people with “Warnings” have been discarded. Column “EvalAccess disabilities [81], [82]. 2.0” indicates the number of errors to pass the WCAG 1.0 priority 1 requirement that can be automatically detected. “Warnings” have not been included in the

VI. RESULTS figure. Column “Cynthia Says” provides the number of

A. HTML and CSS Validity failures of WCAG 2.0 priority 1 (A level) requirement that can be automatically detected. “Warnings” have been Fig 1 shows the HTML and CSS validity results. discarded. Column “MAGENTA” shows the number of Acolour code is used to clarify the results. An anomalous errors of WCAG 2.0 priority 1 (A level) requirement that situation detected during the analysis: one website could can be automatically detected. Requirement that can be not be analysed, the website of the Prime Ministry of automatically detected. Column “HERA” indicates the Turkey. Only the website of the Republic of Turkey number of errors to pass the WCAG 1.0 priority 1 Ministry of Justice website had 0 validation errors. The requirement that can be automatically detected. Column following websites with the less number of HTML errors “Amp” provides the global percentage provided by this were: Republic of Turkey Ministry of Family and Social tool, a value from %1 to %100: the higher the value, the

better the accessibility of the web page. Finally, Column Ministry of Health 39 0 5 0 “Sort Site” shows the number of issues WCAG 2.0 Transport, 18 0 0 0 Maritime Affairs priority 1 requirement that can be automatically detected. and Comm. In general, the worst results regarding web Ministry of Food, 165 5 64 740 accessibility were obtained with the websites of Republic Agriculture and of Turkey Ministry of Culture and Tourism, Republic of Livestock Turkey Ministry of Economy and Republic of Turkey Ministry of labor 212 72 95 55 and social security Ministry of Youth and Sports. On the other side, the best Ministry of 212 171 10 41 results were obtained with the websites of Turkish Science, Industry Armed Forces/Turkish General Staff, Republic of Turkey and Technology Ministry of Justice and Republic of turkey ministry of Ministry of Energy 283 424 2 4 and Natural defense. Resources Website HTML HTML CSS CSS Ministry of Culture 14 3 54 13 Errors Warns Errors Warns and Tourism Presidency of the 55 49 0 0 Ministry of forest 13 2 16 5 Republic of Turkey and water The Grand 97 26 17 64 Ministry of Family 2 2 4 18 National Assembly and Social Policy of Turkey Ministry for EU 76 112 2 0 Prime Ministry of Affairs Turkey Turkish Armed 97 16 16 0 Ministry of 306 173 123 49 Forces/Turkish Economy General Staff Ministry of Justice 0 0 0 1 Ministry of Youth 525 0 130 89 Ministry of 4 4 0 0 and Sports Defense Ministry of Interior 20 28 0 33 Ministry of Foreign 33 8 79 7 Ministry of 81 2 33 57 Affairs Customs and Trade Ministry of 297 193 57 720 Finance Ministry of 13 1 353 973 Ministry of 134 75 51 58 Development National Education Ministry of 36 12 11 46 Environment and Figure 1 . HTML and CSS validation results Planning

T .

Website

A .

AChecker eXaminato r 1.0 TAW 2.0 TAW TV WAVE Web A EvalAccess 2.0 Cynthia Says MAGENT A HERA Amp Sort Site Presidency of the 27 5.7 18 57 28 9 36 23 3 31 1 %59 18 Republic of Turkey

The Grand National 38 4.6 20 87 59 27 102 20 3 1 2 %58 17 Assembly of Turkey Prime Ministry of Turkey 15 4.6 15 22 10 8 26 3 1 3 %50 13 Turkish Armed 26 2.9 0 26 59 21 22 18 8 46 1 %66 20 Forces/Turkish General Staff Ministry of Justice 2 3.8 6 12 8 1 18 0 27 9 0 %70 12 Ministry of Defense 3 4.7 0 5 2 3 0 0 9 1 0 %68 2 Ministry of Interior 3 4.3 1 24 5 3 6 1 4 19 1 %59 8 Ministry of Foreign 16 5.1 1 28 7 3 27 1 4 29 2 %57 13 Affairs Ministry of Finance 85 4.5 73 8 153 3 73 10 143 12 %52 15 Ministry of National 54 4.7 49 124 89 61 177 49 3 78 1 %59 16 Education Ministry of Environment 13 4.6 0 47 29 8 18 4 4 12 1 %65 10 and Planning Ministry of Health 30 3.4 27 8 66 33 111 27 11 69 1 12 Transport,Maritime 2 6.0 1 18 1 1 24 0 16 2 0 %71 17 Affairs and Comm. Ministry of Food, 40 4.1 33 47 52 34 31 14 54 3 %59 17 Agriculture and Livestock Ministry of labor and 60 3.9 56 164 129 60 125 54 6 111 3 %69 14 social security Ministry of Science, 50 3.8 55 110 77 44 37 10 101 3 %48

Industry and Technology Ministry of Energy and 16 3.1 0 41 54 14 53 10 22 59 3 %53 16 Natural Resources Ministry of Culture and 10 3.9 0 46 29 6 829 0 7 28 0 %63 14 Tourism Ministry of forest and 25 5.1 11 37 36 23 81 11 7 37 1 %59 11 water Ministry of Family and 5 5.8 0 12 26 2 68 0 6 26 1 %58 6 Social Policy Ministry for EU Affairs 25 4.5 6 35 27 16 27 16 6 27 1 %58 17 Ministry of Economy 64 4.1 51 227 95 58 51 9 159 1 17 Ministry of Youth and 63 4.7 50 213 160 60 41 9 169 3 %57 17 Sports Ministry of Customs and 73 3.9 67 113 118 69 156 67 8 210 1 %59 9 Trade Ministry of Development 34 4.9 0 128 115 30 142 28 4 123 3 %53 14

Figure 2. Accessibility results

C. Current Use of HTML5 and ARIA alternative text (null alt text). Vigo et al. [83] tested and compared the capabilities of six automatic current web The DOCTYPE is a declaration that always has to accessibility evaluation tools, by analysing their coverage, appear at the very top of HTML documents. This completeness and correctness with regard to WCAG 2.0 declaration defines the type of document, tells the conformance. The conclusion was that relying on only browser what element to expect as the top-level element, one automatic evaluation tool was an error because none and identifies the version of the type of document. of the analysed tools obtained the best scores in all the According to the results of W3C’s Markup Validation dimensions studied. For example, some tools exhibited Service [65], only 4 web pages (16%) have the HTML5 high completeness scores and low correctness scores at DOCTYPE: Republic of Turkey Ministry of the same time. Therefore, a web accessibility analysis Environment and Planning, Republic of Turkey Ministry based only on automatic evaluation tools should include of Family and Social Policy, Republic of Turkey ministry the results of different tools in order to achieve reliable of customs and trade, Republic of Turkey ministry of results. development. Another limitation is the restriction of our automated Regarding the use of ARIA, WAVE [72] has been accessibility testing on the home page of each tested used to detect ARIA features in the analysed websites. website. In order to achieve a more accurate view of the Only 5 web sites (20%) present some use of ARIA: accessibility of each website, this study is going to be Republic of Turkey Ministry of Family and Social Policy, extended to study hundreds or thousands of web pages in Republic of Turkey Ministry of Food, Agriculture and each website to have a more precise view of the Livestock, Republic of Turkey Ministry of Foreign accessibility. We also note that the accessibility metric, Affairs and Prime Ministry of Turkey. For example, derived from an automatic accessibility evaluation Republic of Turkey Ministry of Foreign Affairs website approach, is a proxy indicator of Website accessibility makes use of: one header (header), one footer (footer), and not a real assessment of accessibility as experienced one navigation sections (nav) by a person with disability. Therefore, our results may not capture all the accessibility issues that disabled VII. LIMITATIONS AND FUTURE WORK individuals might encounter in real-life. However, they Two main limitations have been found in this study. do pinpoint to some major accessibility issues that need The first limitation is related to the exclusive reliance of to be resolved. our accessibility analysis on automated testing results. Throughout the whole investigation to determine the Web accessibility evaluation tools and expert inspections conformance level of accessibility, the researchers cannot substitute user testing, because the difficulties of adopted the various evaluation tools (AChecker, understanding all the interactions between web content eXaminator, TAW, Total Validator, WAVE, etc.), all of and assistive technology. Automatic tools generally them were open source applications. However they are verify the presence of a valid element or attribute, such as widely used and to ensure the scalability of the result we the alt attribute (alternative text) or the label element followed W3C Evaluating Accessibility [64]. Although (description of a form control). However, human the commercial tools (e.g. Bobby) are not freely available judgment is also needed, because some questions are and expensive, we will try to apply both commercial very relevant, such as whether or not the value of the alt evaluation tools and also open source and commercial attribute clearly and effectively conveys the function of assistive Technologies (NVDA, JAWS, etc) them in our the image. For example, there is a big difference between next study. In addition to, in order to obtain more the alternative text that an active or inactive image needs. conclusive results, we plan to compare the results across Indeed, in some cases an image may not need an countries and across different government websites.

Finally, another future work we plan to address is to [5] http://www.kakis.gov.tr/rehberi-indir 24.07.2014 detect the most common problems that recur in the same [6] J. Lazar, A. Dudley-Sponaugle, and K. Greenidge, “Improving web accessibility: A study of webmaster perceptions”, Journal of site and between different sites. Computers in Human Behavior, 20, pp.269–288, 2004. [7] R. Baguma, T. Wanyama, P. V. Bommel, and P. Ogao, “Web VIII. CONCLUSION AND RECOMMENDATIONS Accessibility in Uganda: A study of Webmaster perceptions”, In Proc. 3rd Annual International Conference on Computing & ICT In this paper we addressed Turkish e-government Research (SREC'07), 2007, pp. 183-197. websites accessibility for people with disabilities. 25 [8] United Nations Enable. [Online]. Available: http://www.un.org/disabilities/default.asp?id=18 24.07.2014. websites were tested using automatic tools for checking [9] http://www.bianet.org/bianet/toplum/26969-turkiyede-ozurlu- of websites. Results showed that all Turkish e- orani-yuzde-12-29 24.07.2014. government websites do not address the issue of [10] J. Mankoff, H. Fait, and T. Tran, “Is your web page accessible? A disability-accessibility, and it is clear that the vast comparative study of methods for assessing web page accessibility for the blind,” in Proc. SIGCHI Conference on majority of Turkish government websites do not meet Human Factors in Computing Systems (CHI’05), 2005, pp. 41-50. minimum levels of web accessibility requirements. In our [11] J. Craven and A. Nietzio, “A task-based approach to assessing the study it is difficult to obtain conclusive results because accessibility of web sites,” Performance Measurement and each automatic evaluation tool detects different types of Metrics, vol. 8, no. 2, pp. 98–109, 2007. [12] M. A. Latif and M. N. Masrek, “Accessibility evaluation on errors. Because of this, it is difficult to say which one of Malaysian E-government website,” Journal of e-Government the analysed websites presents the best and the worst Studies and Best Practices, pp. 1-11, 2010. level of web accessibility. [13] P. D. D. Dominic, H. Jati, P. Sellappan, and G. K. Nee, “A The most common detected accessibility issues were comparison of Asian e-government websites quality: Using a non- parametric test,” International Journal of Business Information related to the absence of text equivalents for non-text Systems, vol. 7, no. 2, pp. 220-245, 2011. elements and the failure of the static equivalents for [14] M. K.Baowaly and M. Bhuiyan, “Accessibility analysis and dynamic content to get updated when the dynamic evaluation of Bangladesh government websites,” in Proc. content changes. International Conference on Informatics, Electronics & Vision, 18-19 May 2012, pp. 46–51. As a recommendation, websites designers are [15] M. K. Baowaly, J. Hossain Md., and M. Bhuiyan, “Accessibility encouraged to consider the w3c guidelines because of the analysis and evaluation of government-websites’ in developing increasing number of people with disabilities and in order countries: Case study Bangladesh,” Computer Engineering and to give them their right in accessing websites information Intelligent Systems, vol. 3, no. 4, pp. 1-9, 2012. [16] M. Bakhsh and A. Mehmood, “Web accessibility for disabled: A equally with other. As a future work, different tools case study of government websites in Pakistan,” in Proc. 2012 might be used to check governmental websites to see 10th International Conference on Frontiers of Information whether any differences in the accessibility degree will Technology (FIT), 17-19 Dec. 2012, pp. 342–347. be captured. [17] J. Kuzma, “Global E-government web accessibility: “A case study,” in Proc. British Academy of Management 2010 Based on the work described in this paper, the authors Conference, University of Sheffield, UK., 14-16, September 2010. would like to recommend the following issues as critical [18] M. Goodwin, D. Susar, A. Nietzio, M. Snaprud, and C. S. Jensen, initial steps forwards: Government should either adapt “Global web accessibility analysis of national government portals the existing web accessibility guidelines or develop its and ministry web sites,” Journal of Information Technology and Politics, vol. 8, no. 1, pp. 4167, 2011. own guidelines that are appropriate for their context. [19] W. Isa, M. Suhami, N. Safie, and S. Semsudin, “Assessing the Also, government should set a policy for web usability and accessibility of Malayisa E government website,” accessibility together with an enforcement procedure e.g. American Journal of Economics and Business Administration, vol. making the accessibility of government websites a 3, no. 1, pp. 40-46, 2011. [20] M. S. M. Al-radaideh and A. Wahbeh, “Evaluating accessibility of compulsory requirement. An incentive or reward for Jordanian E-government websites for people with disabilities,” in those who accommodate website accessibility may Proc. International Conference on Information and promote good web accessibility. Communication Systems, Irbid, Jordan. May 22-24 2011. Finally, organizations caring for disabled people have [21] I. Abu-Doush, E. A. Ashraf Bany-Mohammed, and M. A. Al- Betar, “Towards a more accessible e-government in Jordan: An a responsibility to spread the awareness amongst evaluation study of visually impaired users and Web developers,” government organizations for making e-Government Behaviour and Information Technology, vol. 32, no. 3, pp. 273- websites accessible. The successful implementation of e- 293, 2013. Government websiteaccessibility would enable disabled [22] B. Al Mourad and F. Kamoun, “Accessibility evaluation of Dubai e-government websites: Findings and implications,” Journal of E- peoples to get involved directly in the community thus Government Studies and Best Practices, vol. 2013, pp. 1-15, 2013. making it better for all. [23] A. N. Abu Ali, A. Obedidat, and H. Y. Abu-Addose, “Accessibility as an indicator of Jordanian E-government website quality,” in Proc. Fourth International Conference on e-Learning REFERENCES Best Practices in Management, Design and Development of e- [1] http://www.lao.ca.gov/2001/012401_egovernment.html Courses: Standards of Excellence and Creativity, 7-9 May 2013, 24.07.2014 pp. 156-160. [2] M. R. Middleton, “Approaches to evaluation of websites for [24] S. Luj´an-Mora, R. Navarrete, and M. Penafiel, “Egovernment public sector services,” in Kommers, Piet, Eds. in Proc. IADIS and web accessibility in SouthAmerica, ” in Proc. 2014 First Conference on e-Society, Lisbon, Portugal, 2007, pp. 279-284. International Conference on eDemocracy and eGovernment [3] TÜRKİYE BİLİMSEL ve TEKNOLOJİK ARAŞTIRMA (ICEDEG), 24-25 April 2014, pp. 77–82. KURUMU. [Online]. Available: [25] S. Venter and H. Lotriet “Accessibility of South African websites http://www.tubitak.gov.tr/tr/kurumsal/politikalar/icerik-vizyon- to visually disabled users,” South African Journal of Information 2023#bit 24.07.2014 Management, vol. 7, no. 2, June 2005. [4] http://kakis.gov.tr/erisilebilirlik 24.07.2014 [26] J. Choudrie, G. Ghinea, and V. Weerakkody, “Evaluating global e-government sites: A view using web diagnostic tools,”

Electronic Journal of e-government, vol. 2, no. 2, pp. 105-114, [49] C. J. Huang, “Usability of E-government web-sites for people 2004. with disabilities,” in Proc. 36th Annual Hawaii International [27] S. Yuquan, “The accessibility of Chinese local government Conference on System Sciences, 6-9 Jan. 2003. Websites: An exploratory study,” Government Information [50] H. Jati and D. D. Dominic, “Website accessibility performance Quarterly, vol. 24, pp. 377–403, 2007. evaluation in Malaysia,” in Proc. International Symposium [28] A. Potter, “Accessibility of Alabama government websites,” on Information Technology, 2008. IT Sim 2008. (Volume:1 ), 26- Journal of Government Information, pp. 303–317, 2002. 28 Aug. 2008, pp. 1–3. [29] A. M. Abanumy, A. Al-Badi, and P. Mayhew, “E-government [51] E. Loiacono, and S. McCoy, “Web site accessibility: An online website accessibility: In-depth evaluation of Saudi Arabia and sector analysis,” Information Technology and People, vol. 17, pp. Oman,” The Electronic Journal of e-Government, vol. 3, no. 3, pp. 87-101, 2004. 99-106, 2005. [52] M. A. Awan, “E-government: Assessment of GCC (Gulf [30] R. Cullen and C. Houghton, “Democracy online: An assessment Cooperating Council) countries and services provided,” Lecture of New Zealand government Websites,” Government Information Notes in Computer Science, vol. 2739, 2003. Quarterly, vol. 17, no. 3, pp. 243-267, 2000. [53] R. Baguma and J. T. Lubega, “A web design framework for [31] L. Gibson, D. Salon, P. Gregor, and P. Booth, “Auditing improved accessibility for people with disabilities (WDFAD),” Y. accessibility of UK higher education Website,” Interacting with Yesilada & D. Sloan (eds.), W4A (p./pp. 134-140), ACM. Computer, pp. 12313-12325, 2002. [54] R. Baguma, P. Bommel, T. Wanyama, and P. Ogao, “Web [32] J. Kuzma, “Regulatory compliance and web accessibility of UK accessibility in Uganda: A study of webmaster perceptions,” parliament sites,” Journal of Information Law & Technology, vol. Kizza, M. J., Muheirwe, J., Aisbett, J., Gitao, K., Mbarika, V. W., 2, pp. 1-15, 2009. Patel, D. and Rodrigues, A.T., eds. Strengthening the Role of ICT [33] J. Kuzma, G. Weisenborn, T. Philippe, A. Gabel, and R. Dolechek, in Development, vol. III, pp. 183-197. “Analysis of U.S senate web sites for disability accessibility,” [55] X. Zeng, “Evaluation and enhancement of web content International Journal of Business Research, vol. 9, no. 6, pp. 174- accessibility for persons with disabilities,” Ph.D. Thesis. 181, 2009. University of Pittsburgh, 2004. [34] J. Kuzma, Y. Dorothy, and K.Oestreicher, “Global e-government [56] World Wide Web Consortium (W3C). Web Content Accessibility web accessibility: An empirical examination of EU, Asian and Guidelines 1.0. 1999. Internet. [Online]. Available: African sites,” Paper presented at the ICTA’09, Hammamet, http://www.w3.org/TR/WCAG10/ Tunisia, 2009. [57] World Wide Web Consortium (W3C). Web Content Accessibility [35] S. Hong, P. Katerattanakul, and D. Lee, “Evaluating government Guidelines (WCAG) 2.0. 2008. Internet. [Online]. Available: website accessibility: Software tool vs human experts,” http://www.w3.org/TR/WCAG/ Management Research News, vol. 31, no. 1, pp. 27-40, 2007. [58] International Standard Organization. ISO/IEC 40500:2012 [36] S. Hong, P. Katerattanakul, and S. J. Joo, “Evaluating government Information technology–W3C Web Content Accessibility website accessibility: A comparative study,” International Journal Guidelines (WCAG) 2.0. 2012. of Information Technology & Decision Making, vol. 7, no. 3, pp. [59] (2014, August). Free Web-Based Web Site Accessibility 491–515, 2008. Evaluation Tools. [Online]. Available: http://usabilitygeek.com/ 1 [37] I. Basdekis, L. Klironomos, I. Metaxas, and C. Stephanidis, “An 0-free-web-based-web-siteaccessibility-evaluation-tools/ overview of web accessibility in Greece: A comparative study [60] Recommended Web Accessibility Validators. [Online]. Available: 2004–2008,” Universal Access in the Information Society, vol. 9, http://www.indiana.edu/~iuadapts/webaccessibility/access no. 2, pp. 185-190, 2009. validators.html [38] S. H. Kurniawan and P. Zaphiris, “Usability and accessibility [61] 100 Killer Web Accessibility Resources: Blogs, Forums and comparison of governmental, organizational, educational and Tutorials. [Online]. Available: commercial aging/health-related websites,” in Proc. the 9th http://whdb.com/blog/2008/100-killer-web-accessibility- International Conference on Human Computer Interaction, New resources-blogs-forums-and-tutorials/ Orleans, LA: Abridged, 2001, pp. 34-36. [62] Accessibility Testing Tools and Techniques. [Online]. Available: [39] S. Choi, S. Kim, and S. Kim “Korean web site usability for disabled people,” Lecture Notes in Computer Science, vol. http://www.3pillarglobal.com/insights/accessibility- 5068, pp. 405-412, 2008. testing-tools-and-techniques [40] R. Johnson and S. Kent, “Designing universal access: Web- [63] Complete List of Web Accessibility Evaluation Tools. [Online]. applications for the elderly and disabled. Cognition,” Technology Available: http: //www.w3.org/WAI/RC/tools/complete and Work, vol. 9, no. 4, pp. 209–218, 2007. [64] Evaluating Accessibility. [Online]. Available: http://www.w3. [41] J. Evans-Cowley, “The accessibility of municipal government org/WAIIeval/Overview.html websites,” Journal of E-Government, vol. 2, no. 2, pp. 75-90, [65] World Wide Web Consortium (W3C). The W3C Markup 2006. Validation Service. [Online]. Available: http://validator.w3.org/ [42] A. P. Freire, C. M. Russo, and R. P. M. Fortes, “The perception of [66] World Wide Web Consortium (W3C). About The W3C Markup accessibility in web development by academy, industry and Validation Service. [Online]. Available: government: A survey of the Brazilian scenario,” New Review of http://validator.w3.org/about.html Hypermedia and Multimedia, vol. 14, no. 2, pp. 149–175, 2008. [67] World Wide Web Consortium (W3C). The CSS Validation [43] M. Paris, “Website accessibility: A survey of locale-government Service.[Online]. Available: http://jigsaw.w3.org/css-validator/ websites and legislation in Northern Ireland,” Universal Access in [68] Inclusive Design Institute. AChecker. [Online]. Available: the Information Society, vol. 4, pp. 292–299, 2006. Internet: http://achecker.ca/ [44] T. Goette, C. Collier, and J. D. White, “An exploratory study of [69] C. Benav´ıdez. eXaminator. [Online]. Available: the accessibility of state government websites,” Universal Access http://examinator.ws/ in the Information Society, vol. 5, pp. 41–50, 2006. [70] Fundaci´on CTIC. TAW. [Online]. Available: [45] P. T. Jaeger, “Assessing section 508 compliance on federale- http://www.tawdis.net/ government websites: A multi-method, user-centered evaluation [71] Total Validator. Total Validator. [Online]. Available: of accessibility for persons with disabilities,” Government http://www.totalvalidator.com/ Information Quarterly, vol. 23, pp. 169–190, 2006. [72] WebAIM, WAVE. [Online]. Available: http://wave.webaim.org/ [46] P. Jaeger, “User-centered policy evaluations of section 508 of the [73] Chalkia Eleni, Evangelos Bekiaris, “Accessible EU project user rehabilitation act,” Journal of Disability Policy Studies, vol. 19, cases,” in Proc. AEGIS Project Conference, Seville, Spain, 7-8 no. 1, pp. 24-33, June 2008. October 2010. [47] Y. Shi, “E-government web site accessibility in Australia and [74] Eval Access 2.0. [Online]. Available: China: A longitudinal study,” Social Science Computer Review, http://sipt07.si.ehu.es/evalaccess2/howto.html vol. 24, no. 3, pp. 378–385, 2006. [75] HiSoftware Cynthia Says. (2012). [Online]. Available: [48] A. Rabaiah and E. Vandijck, “A strategic framework of e- http://www.cynthiasays.com/ government: generic and best practice,” Electronic Journal of e- http://www.cynthiasays.com/Pages/About.aspx Government, vol. 7, no. 3, pp. 241–258, 2009. [76] http://giove.isti.cnr.it:8080/MagentaWeb2/credits.jsp

[77] What is HERA. [Online]. Available: Yakup Akgül was born in Simav on March 22, http://www.sidar.org/hera/index.php.en 1977 and grew up into Kütahya (Turkey). He [78] AMP. [Online]. Available: studied Department of Information https://amp.ssbbartgroup.com/index.php Management at the university of Hacettepe, [79] SortSite-Accessibility Checker and Validator. [Online]. Available: Ankara (Turkey), from which he graduated in http://www.powermapper.com/products/sortsite/checks/ac 2001. He received Master (2010) and Ph.D. cessibility-checks.htm student in Business Administration at Süleyman [80] World Wide Web Consortium (W3C). HTML 5-A vocabulary and Demirel University, Isparta, Turkey. He works associated APIs for HTML and XHTML-W3C Working Draft. as a lecturer at the Dumlupınar University, (22 January 2008). [Online]. Available: Kütahya, Turkey. His researches focuses on http://www.w3.org/TR/2008/WD-html5-20080122/ knowledge managemet, knowledge managemet strategy, strategy, web [81] WAI-ARIA Overview. [Online]. Available: usability and accessibility. http://www.w3.org/WAI/intro/aria

[82] Accessibility of Rich Internet Applications. [Online]. Available: Kemal Vatansever was born in Burgaz on http://webaim.org/techniques/aria/ August 29, 1979 and grew up into Bursa [83] M. Vigo, J. Brown, and V. Conway, “Benchmarking web (Turkey). He studied Business Administration accessibility evaluation tools: measuring the harm of sole reliance at the university of Osmangazi, Eskişehir on automated tests,” in Proc. the 10th International Cross- (Turkey), from which he graduated in 2002. Disciplinary Conference on Web Accessibility (W4A ’13), 2013, He received Master (2005) and Ph.D. (2010) pp. 1-10. degree in Business Administration at

Dumlupınar University, Kütahya, Turkey. He

works as an Assistant Professor at the

Department of Capital Markets and as a head

of department at the same Department, Pamukkale University, Denizli,

Turkey. His researches focuses on quantitative decision methods, multi

criteria decision making, statistics and operational research.