research paper about web design

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• HHS Author Manuscripts

A Literature Review: Website Design and User Engagement

Renee garett.

1 ElevateU, Los Angeles, CA, USA

Sean D. Young

2 University of California Institute for Prediction Technology, Department of Family Medicine, University of California, Los Angeles, Los Angeles, CA, USA

3 UCLA Center for Digital Behavior, Department of Family Medicine, University of California, Los Angeles, Los Angeles, CA, USA

Proper design has become a critical element needed to engage website and mobile application users. However, little research has been conducted to define the specific elements used in effective website and mobile application design. We attempt to review and consolidate research on effective design and to define a short list of elements frequently used in research. The design elements mentioned most frequently in the reviewed literature were navigation, graphical representation, organization, content utility, purpose, simplicity, and readability. We discuss how previous studies define and evaluate these seven elements. This review and the resulting short list of design elements may be used to help designers and researchers to operationalize best practices for facilitating and predicting user engagement.

1. INTRODUCTION

Internet usage has increased tremendously and rapidly in the past decade ( “Internet Use Over Time,” 2014 ). Websites have become the most important public communication portal for most, if not all, businesses and organizations. As of 2014, 87% of American adults aged 18 or older are Internet users ( “Internet User Demographics,” 2013 ). Because business-to-consumer interactions mainly occur online, website design is critical in engaging users ( Flavián, Guinalíu, & Gurrea, 2006 ; Lee & Kozar, 2012 ; Petre, Minocha, & Roberts, 2006 ). Poorly designed websites may frustrate users and result in a high “bounce rate”, or people visiting the entrance page without exploring other pages within the site ( Google.com, 2015 ). On the other hand, a well-designed website with high usability has been found to positively influence visitor retention (revisit rates) and purchasing behavior ( Avouris, Tselios, Fidas, & Papachristos, 2003 ; Flavián et al., 2006 ; Lee & Kozar, 2012 ).

Little research, however, has been conducted to define the specific elements that constitute effective website design. One of the key design measures is usability ( International Standardization Organization, 1998 ). The International Standardized Organization (ISO) defines usability as the extent to which users can achieve desired tasks (e.g., access desired information or place a purchase) with effectiveness (completeness and accuracy of the task), efficiency (time spent on the task), and satisfaction (user experience) within a system. However, there is currently no consensus on how to properly operationalize and assess website usability ( Lee & Kozar, 2012 ). For example, Nielson associates usability with learnability, efficiency, memorability, errors, and satisfaction ( Nielsen, 2012 ). Yet, Palmer (2002) postulates that usability is determined by download time, navigation, content, interactivity, and responsiveness. Similar to usability, many other key design elements, such as scannability, readability, and visual aesthetics, have not yet been clearly defined ( Bevan, 1997 ; Brady & Phillips, 2003 ; Kim, Lee, Han, & Lee, 2002 ), and there are no clear guidelines that individuals can follow when designing websites to increase engagement.

This review sought to address that question by identifying and consolidating the key website design elements that influence user engagement according to prior research studies. This review aimed to determine the website design elements that are most commonly shown or suggested to increase user engagement. Based on these findings, we listed and defined a short list of website design elements that best facilitate and predict user engagement. The work is thus an exploratory research providing definitions for these elements of website design and a starting point for future research to reference.

2. MATERIALS AND METHODS

2.1. selection criteria and data extraction.

We searched for articles relating to website design on Google Scholar (scholar.google.com) because Google Scholar consolidates papers across research databases (e.g., Pubmed) and research on design is listed in multiple databases. We used the following combination of keywords: design, usability, and websites. Google Scholar yielded 115,000 total hits. However, due to the large list of studies generated, we decided to only review the top 100 listed research studies for this exploratory study. Our inclusion criteria for the studies was: (1) publication in a peer-reviewed academic journal, (2) publication in English, and (3) publication in or after 2000. Year of publication was chosen as a limiting factor so that we would have enough years of research to identify relevant studies but also have results that relate to similar styles of websites after the year 2000. We included studies that were experimental or theoretical (review papers and commentaries) in nature. Resulting studies represented a diverse range of disciplines, including human-computer interaction, marketing, e-commerce, interface design, cognitive science, and library science. Based on these selection criteria, thirty-five unique studies remained and were included in this review.

2.2. Final Search Term

(design) and (usability) and (websites).

The search terms were kept simple to capture the higher level design/usability papers and allow Google scholar’s ranking method to filter out the most popular studies. This method also allowed studies from a large range of fields to be searched.

2.3. Analysis

The literature review uncovered 20 distinct design elements commonly discussed in research that affect user engagement. They were (1) organization – is the website logically organized, (2) content utility – is the information provided useful or interesting, (3) navigation – is the website easy to navigate, (4) graphical representation – does the website utilize icons, contrasting colors, and multimedia content, (5) purpose – does the website clearly state its purpose (i.e. personal, commercial, or educational), (6) memorable elements – does the website facilitate returning users to navigate the site effectively (e.g., through layout or graphics), (7) valid links – does the website provide valid links, (8) simplicity – is the design of the website simple, (9) impartiality – is the information provided fair and objective, (10) credibility – is the information provided credible, (11) consistency/reliability – is the website consistently designed (i.e., no changes in page layout throughout the site), (12) accuracy – is the information accurate, (13) loading speed – does the website take a long time to load, (14) security/privacy – does the website securely transmit, store, and display personal information/data, (15) interactive – can the user interact with the website (e.g., post comments or receive recommendations for similar purchases), (16) strong user control capabilities– does the website allow individuals to customize their experiences (such as the order of information they access and speed at which they browse the website), (17) readability – is the website easy to read and understand (e.g., no grammatical/spelling errors), (18) efficiency – is the information presented in a way that users can find the information they need quickly, (19) scannability – can users pick out relevant information quickly, and (20) learnability – how steep is the learning curve for using the website. For each of the above, we calculated the proportion of studies mentioning the element. In this review, we provide a threshold value of 30%. We identified elements that were used in at least 30% of the studies and include these elements that are above the threshold on a short list of elements used in research on proper website design. The 30% value was an arbitrary threshold picked that would provide researchers and designers with a guideline list of elements described in research on effective web design. To provide further information on how to apply this list, we present specific details on how each of these elements was discussed in research so that it can be defined and operationalized.

3.1. Popular website design elements ( Table 1 )

Frequency of website design elements used in research (2000–2014)

Seven of the website design elements met our threshold requirement for review. Navigation was the most frequently discussed element, mentioned in 22 articles (62.86%). Twenty-one studies (60%) highlighted the importance of graphics. Fifteen studies (42.86%) emphasized good organization. Four other elements also exceeded the threshold level, and they were content utility (n=13, 37.14%), purpose (n=11, 31.43%), simplicity (n=11, 31.43%), and readability (n=11, 31.43%).

Elements below our minimum requirement for review include memorable features (n=5, 14.29%), links (n=10, 28.57%), impartiality (n=1, 2.86%), credibility (n=7, 20%), consistency/reliability (n=8. 22.86%), accuracy (n=5, 14.29%), loading speed (n=10, 28.57%), security/privacy (n=2, 5.71%), interactive features (n=9, 25.71%), strong user control capabilities (n=8, 22.86%), efficiency (n=6, 17.14%), scannability (n=1, 2.86%), and learnability (n=2, 5.71%).

3.2. Defining key design elements for user engagement ( Table 2 )

Definitions of Key Design Elements

In defining and operationalizing each of these elements, the research studies suggested that effective navigation is the presence of salient and consistent menu/navigation bars, aids for navigation (e.g., visible links), search features, and easy access to pages (multiple pathways and limited clicks/backtracking). Engaging graphical presentation entails 1) inclusion of images, 2) proper size and resolution of images, 3) multimedia content, 4) proper color, font, and size of text, 5) use of logos and icons, 6) attractive visual layout, 7) color schemes, and 8) effective use of white space. Optimal organization includes 1) cognitive architecture, 2) logical, understandable, and hierarchical structure, 3) information arrangement and categorization, 4) meaningful labels/headings/titles, and 5) use of keywords. Content utility is determined by 1) sufficient amount of information to attract repeat visitors, 2) arousal/motivation (keeps visitors interested and motivates users to continue exploring the site), 3) content quality, 4) information relevant to the purpose of the site, and 5) perceived utility based on user needs/requirements. The purpose of a website is clear when it 1) establishes a unique and visible brand/identity, 2) addresses visitors’ intended purpose and expectations for visiting the site, and 3) provides information about the organization and/or services. Simplicity is achieved by using 1) simple subject headings, 2) transparency of information (reduce search time), 3) website design optimized for computer screens, 4) uncluttered layout, 5) consistency in design throughout website, 6) ease of using (including first-time users), 7) minimize redundant features, and 8) easily understandable functions. Readability is optimized by content that is 1) easy to read, 2) well-written, 3) grammatically correct, 4) understandable, 5) presented in readable blocks, and 6) reading level appropriate.

4. DISCUSSION

The seven website design elements most often discussed in relation to user engagement in the reviewed studies were navigation (62.86%), graphical representation (60%), organization (42.86%), content utility (37.14%), purpose (31.43%), simplicity (31.43%), and readability (31.43%). These seven elements exceeded our threshold level of 30% representation in the literature and were included into a short list of website design elements to operationalize effective website design. For further analysis, we reviewed how studies defined and evaluated these seven elements. This may allow designers and researchers to determine and follow best practices for facilitating or predicting user engagement.

A remaining challenge is that the definitions of website design elements often overlap. For example, several studies evaluated organization by how well a website incorporates cognitive architecture, logical and hierarchical structure, systematic information arrangement and categorization, meaningful headings and labels, and keywords. However, these features are also crucial in navigation design. Also, the implications of using distinct logos and icons go beyond graphical representation. Logos and icons also establish unique brand/identity for the organization (purpose) and can serve as visual aids for navigation. Future studies are needed to develop distinct and objective measures to assess these elements and how they affect user engagement ( Lee & Kozar, 2012 ).

Given the rapid increase in both mobile technology and social media use, it is surprising that no studies mentioned cross-platform compatibility and social media integration. In 2013, 34% of cellphone owners primarily use their cellphones to access the Internet, and this number continues to grow ( “Mobile Technology Factsheet,” 2013 ). With the rise of different mobile devices, users are also diversifying their web browser use. Internet Explorer (IE) was once the leading web browser. However, in recent years, FireFox, Safari, and Chrome have gained significant traction ( W3schools.com, 2015 ). Website designers and researchers must be mindful of different platforms and browsers to minimize the risk of losing users due to compatibility issues. In addition, roughly 74% of American Internet users use some form of social media ( Duggan, Ellison, Lampe, Lenhart, & Smith, 2015 ), and social media has emerged as an effective platform for organizations to target and interact with users. Integrating social media into website design may increase user engagement by facilitating participation and interactivity.

There are several limitations to the current review. First, due to the large number of studies published in this area and due to this study being exploratory, we selected from the first 100 research publications on Google Scholar search results. Future studies may benefit from defining design to a specific topic, set of years, or other area to limit the number of search results. Second, we did not quantitatively evaluate the effectiveness of these website design elements. Additional research can help to better quantify these elements.

It should also be noted that different disciplines and industries have different objectives in designing websites and should thus prioritize different website design elements. For example, online businesses and marketers seek to design websites that optimize brand loyalty, purchase, and profit ( Petre et al., 2006 ). Others, such as academic researchers or healthcare providers, are more likely to prioritize privacy/confidentiality, and content accuracy in building websites ( Horvath, Ecklund, Hunt, Nelson, & Toomey, 2015 ). Ultimately, we advise website designers and researchers to consider the design elements delineated in this review, along with their unique needs, when developing user engagement strategies.

• Arroyo Ernesto, Selker Ted, Wei Willy. Usability tool for analysis of web designs using mouse tracks. Paper presented at the CHI’06 Extended Abstracts on Human Factors in Computing Systems.2006. [ Google Scholar ]
• Atterer Richard, Wnuk Monika, Schmidt Albrecht. Knowing the user’s every move: user activity tracking for website usability evaluation and implicit interaction. Paper presented at the Proceedings of the 15th international conference on World Wide Web.2006. [ Google Scholar ]
• Auger Pat. The impact of interactivity and design sophistication on the performance of commercial websites for small businesses. Journal of Small Business Management. 2005; 43 (2):119–137. [ Google Scholar ]
• Avouris Nikolaos, Tselios Nikolaos, Fidas Christos, Papachristos Eleftherios. Advances in Informatics. Springer; 2003. Website evaluation: A usability-based perspective; pp. 217–231. [ Google Scholar ]
• Banati Hema, Bedi Punam, Grover PS. Evaluating web usability from the user’s perspective. Journal of Computer Science. 2006; 2 (4):314. [ Google Scholar ]
• Belanche Daniel, Casaló Luis V, Guinalíu Miguel. Website usability, consumer satisfaction and the intention to use a website: The moderating effect of perceived risk. Journal of retailing and consumer services. 2012; 19 (1):124–132. [ Google Scholar ]
• Bevan Nigel. Usability issues in web site design. Paper presented at the HCI; 1997. [ Google Scholar ]
• Blackmon Marilyn Hughes, Kitajima Muneo, Polson Peter G. Repairing usability problems identified by the cognitive walkthrough for the web. Paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems.2003. [ Google Scholar ]
• Blackmon Marilyn Hughes, Polson Peter G, Kitajima Muneo, Lewis Clayton. Cognitive walkthrough for the web. Paper presented at the Proceedings of the SIGCHI conference on human factors in computing systems.2002. [ Google Scholar ]
• Braddy Phillip W, Meade Adam W, Kroustalis Christina M. Online recruiting: The effects of organizational familiarity, website usability, and website attractiveness on viewers’ impressions of organizations. Computers in Human Behavior. 2008; 24 (6):2992–3001. [ Google Scholar ]
• Brady Laurie, Phillips Christine. Aesthetics and usability: A look at color and balance. Usability News. 2003; 5 (1) [ Google Scholar ]
• Cyr Dianne, Head Milena, Larios Hector. Colour appeal in website design within and across cultures: A multi-method evaluation. International journal of human-computer studies. 2010; 68 (1):1–21. [ Google Scholar ]
• Cyr Dianne, Ilsever Joe, Bonanni Carole, Bowes John. Website Design and Culture: An Empirical Investigation. Paper presented at the IWIPS.2004. [ Google Scholar ]
• Dastidar Surajit Ghosh. Impact of the factors influencing website usability on user satisfaction. 2009. [ Google Scholar ]
• De Angeli Antonella, Sutcliffe Alistair, Hartmann Jan. Interaction, usability and aesthetics: what influences users’ preferences?. Paper presented at the Proceedings of the 6th conference on Designing Interactive systems.2006. [ Google Scholar ]
• Djamasbi Soussan, Siegel Marisa, Tullis Tom. Generation Y, web design, and eye tracking. International journal of human-computer studies. 2010; 68 (5):307–323. [ Google Scholar ]
• Djonov Emilia. Website hierarchy and the interaction between content organization, webpage and navigation design: A systemic functional hypermedia discourse analysis perspective. Information Design Journal. 2007; 15 (2):144–162. [ Google Scholar ]
• Duggan M, Ellison N, Lampe C, Lenhart A, Smith A. Social Media update 2014. Washington, D.C: Pew Research Center; 2015. [ Google Scholar ]
• Flavián Carlos, Guinalíu Miguel, Gurrea Raquel. The role played by perceived usability, satisfaction and consumer trust on website loyalty. Information & Management. 2006; 43 (1):1–14. [ Google Scholar ]
• George Carole A. Usability testing and design of a library website: an iterative approach. OCLC Systems & Services: International digital library perspectives. 2005; 21 (3):167–180. [ Google Scholar ]
• Google.com. Bounce Rate. Analyrics Help. 2015 Retrieved 2/11, 2015, from https://support.google.com/analytics/answer/1009409?hl=en .
• Green D, Pearson JM. Development of a web site usability instrument based on ISO 9241-11. Journal of Computer Information Systems. 2006 Fall [ Google Scholar ]
• Horvath Keith J, Ecklund Alexandra M, Hunt Shanda L, Nelson Toben F, Toomey Traci L. Developing Internet-Based Health Interventions: A Guide for Public Health Researchers and Practitioners. J Med Internet Res. 2015; 17 (1):e28. doi: 10.2196/jmir.3770. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• International Standardization Organization. ISO 2941-11:1998 Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 11: Guidance on usability: International Standardization Organization (ISO) 1998. [ Google Scholar ]
• Internet Use Over Time. 2014 Jan 2; Retrieved February 15, 2015, from http://www.pewinternet.org/data-trend/internet-use/internet-use-over-time/
• Internet User Demographics. 2013 Nov 14; Retrieved February 11, 2015, from http://www.pewinternet.org/data-trend/internet-use/latest-stats/
• Kim Jinwoo, Lee Jungwon, Han Kwanghee, Lee Moonkyu. Businesses as Buildings: Metrics for the Architectural Quality of Internet Businesses. Information Systems Research. 2002; 13 (3):239–254. doi: 10.1287/isre.13.3.239.79. [ CrossRef ] [ Google Scholar ]
• Lee Younghwa, Kozar Kenneth A. Understanding of website usability: Specifying and measuring constructs and their relationships. Decision Support Systems. 2012; 52 (2):450–463. [ Google Scholar ]
• Lim Sun. The Self-Confrontation Interview: Towards an Enhanced Understanding of Human Factors in Web-based Interaction for Improved Website Usability. J Electron Commerce Res. 2002; 3 (3):162–173. [ Google Scholar ]
• Lowry Paul Benjamin, Spaulding Trent, Wells Taylor, Moody Greg, Moffit Kevin, Madariaga Sebastian. A theoretical model and empirical results linking website interactivity and usability satisfaction. Paper presented at the System Sciences, 2006. HICSS’06. Proceedings of the 39th Annual Hawaii International Conference on.2006. [ Google Scholar ]
• Maurer Steven D, Liu Yuping. Developing effective e-recruiting websites: Insights for managers from marketers. Business Horizons. 2007; 50 (4):305–314. [ Google Scholar ]
• Mobile Technology Fact Sheet. 2013 Dec 27; Retrieved August 5, 2015, from http://www.pewinternet.org/fact-sheets/mobile-technology-fact-sheet/
• Nielsen Jakob. Usability 101: introduction to Usability. 2012 Retrieved 2/11, 2015, from http://www.nngroup.com/articles/usability-101-introduction-to-usability/
• Palmer Jonathan W. Web Site Usability, Design, and Performance Metrics. Information Systems Research. 2002; 13 (2):151–167. doi: 10.1287/isre.13.2.151.88. [ CrossRef ] [ Google Scholar ]
• Petre Marian, Minocha Shailey, Roberts Dave. Usability beyond the website: an empirically-grounded e-commerce evaluation instrument for the total customer experience. Behaviour & Information Technology. 2006; 25 (2):189–203. [ Google Scholar ]
• Petrie Helen, Hamilton Fraser, King Neil. Tension, what tension?: Website accessibility and visual design. Paper presented at the Proceedings of the 2004 international cross-disciplinary workshop on Web accessibility (W4A).2004. [ Google Scholar ]
• Raward Roslyn. Academic library website design principles: development of a checklist. Australian Academic & Research Libraries. 2001; 32 (2):123–136. [ Google Scholar ]
• Rosen Deborah E, Purinton Elizabeth. Website design: Viewing the web as a cognitive landscape. Journal of Business Research. 2004; 57 (7):787–794. [ Google Scholar ]
• Shneiderman Ben, Hochheiser Harry. Universal usability as a stimulus to advanced interface design. Behaviour & Information Technology. 2001; 20 (5):367–376. [ Google Scholar ]
• Song Jaeki, Zahedi Fatemeh “Mariam”. A theoretical approach to web design in e-commerce: a belief reinforcement model. Management Science. 2005; 51 (8):1219–1235. [ Google Scholar ]
• Sutcliffe Alistair. Interactive systems: design, specification, and verification. Springer; 2001. Heuristic evaluation of website attractiveness and usability; pp. 183–198. [ Google Scholar ]
• Tan Gek Woo, Wei Kwok Kee. An empirical study of Web browsing behaviour: Towards an effective Website design. Electronic Commerce Research and Applications. 2007; 5 (4):261–271. [ Google Scholar ]
• Tarafdar Monideepa, Zhang Jie. Determinants of reach and loyalty-a study of Website performance and implications for Website design. Journal of Computer Information Systems. 2008; 48 (2):16. [ Google Scholar ]
• Thompson Lori Foster, Braddy Phillip W, Wuensch Karl L. E-recruitment and the benefits of organizational web appeal. Computers in Human Behavior. 2008; 24 (5):2384–2398. [ Google Scholar ]
• W3schools.com. Browser Statistics and Trends. Retrieved 1/15, 2015, from http://www.w3schools.com/browsers/browsers_stats.asp .
• Williamson Ian O, Lepak David P, King James. The effect of company recruitment web site orientation on individuals’ perceptions of organizational attractiveness. Journal of Vocational Behavior. 2003; 63 (2):242–263. [ Google Scholar ]
• Zhang Ping, Small Ruth V, Von Dran Gisela M, Barcellos Silvia. A two factor theory for website design. Paper presented at the System Sciences, 2000. Proceedings of the 33rd Annual Hawaii International Conference on.2000. [ Google Scholar ]
• Zhang Ping, Von Dran Gisela M. Satisfiers and dissatisfiers: A two-factor model for website design and evaluation. Journal of the American society for information science. 2000; 51 (14):1253–1268. [ Google Scholar ]

Web design — Past, present and future

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

• Reviews / Why join our community?
• For companies
• Frequently asked questions

What is Web Design?

Web design refers to the design of websites. It usually refers to the user experience aspects of website development rather than software development. Web design used to be focused on designing websites for desktop browsers; however, since the mid-2010s, design for mobile and tablet browsers has become ever-increasingly important.

• Transcript loading…

A web designer works on a website's appearance, layout, and, in some cases, content .

Appearance relates to the colors, typography, and images used.

Layout refers to how information is structured and categorized. A good web design is easy to use, aesthetically pleasing, and suits the user group and brand of the website.

A well-designed website is simple and communicates clearly to avoid confusing users. It wins and fosters the target audience's trust, removing as many potential points of user frustration as possible.

Responsive and adaptive design are two common ways to design websites that work well on both desktop and mobile.

What is Responsive Web Design?

© Interaction Design Foundation, CC BY-SA 4.0

Responsive Web Design (a.k.a. "Responsive" or "Responsive Design") is an approach to designing web content that appears regardless of the resolution governed by the device. It’s typically accomplished with viewport breakpoints (resolution cut-offs for when content scales to that view). The viewports should adjust logically on tablets, phones, and desktops of any resolution.

In responsive design, you can define rules for how the content flows and how the layout changes based on the size range of the screen.

Responsive designs respond to changes in browser width by adjusting the placement of design elements to fit in the available space. If you open a responsive site on the desktop and change the browser window's size, the content will dynamically rearrange itself to fit the browser window. The site checks for the available space on mobile phones and then presents itself in the ideal arrangement.

Best Practices and Considerations for Responsive Design

With responsive design, you design for flexibility in every aspect—images, text and layouts. So, you should:

Take the mobile-first approach —start the product design process for mobile devices first instead of desktop devices.

Create fluid grids and images .

Prioritize the use of Scalable Vector Graphics (SVGs). These are an XML-based file format for 2D graphics, which supports interactivity and animations.

Include three or more breakpoints (layouts for three or more devices).

Prioritize and hide content to suit users’ contexts . Check your visual hierarchy and use progressive disclosure and navigation drawers to give users needed items first. Keep nonessential items (nice-to-haves) secondary.

Aim for minimalism .

Apply design patterns to maximize ease of use for users in their contexts and quicken their familiarity: e.g., the column drop pattern fits content to many screen types.

Aim for accessibility .

What is Adaptive Web Design?

Adaptive design is similar to responsive design—both are approaches for designing across a diverse range of devices; the difference lies in how the tailoring of the content takes place.

In the case of responsive design, all content and functionality are the same for every device. Therefore, a large-screen desktop and smartphone browser displays the same content. The only difference is in the layout of the content. 

In this video, CEO of Experience Dynamics, Frank Spillers, explains the advantages of adaptive design through a real-life scenario.

Adaptive design takes responsiveness up a notch. While responsive design focuses on just the device, adaptive design considers both the device and the user’s context. This means that you can design context-aware experiences —a web application's content and functionality can look and behave very differently from the version served on the desktop.

For example, if an adaptive design detects low bandwidth or the user is on a mobile device instead of a desktop device, it might not load a large image (e.g., an infographic). Instead, it might show a smaller summary version of the infographic.

Another example could be to detect if the device is an older phone with a smaller screen. The website can show larger call-to-action buttons than usual.

Accessibility for Web Design

“The power of the Web is in its universality. Access by everyone regardless of disability is an essential aspect.” —Tim Berners-Lee, W3C Director and inventor of the World Wide Web

Web accessibility means making websites and technology usable for people with varying abilities and disabilities. An accessible website ensures that all users, regardless of their abilities, can perceive, understand, navigate, and interact with the web.

In this video, William Hudson, CEO of Syntagm, discusses the importance of accessibility and provides tips on how to make websites more accessible.

The World Wide Web Consortium (W3C) lists a few basic considerations for web accessibility:

Provide sufficient contrast between foreground and background . For example, black or dark gray text on white is easier to read than gray text on a lighter shade of gray. Use color contrast checkers to test the contrast ratio between your text and background colors to ensure people can easily see your content.

Don’t use color alone to convey information . For example, use underlines for hyperlinked text in addition to color so that people with colorblindness can still recognize a link, even if they can’t differentiate between the hyperlink and regular text.

Ensure that interactive elements are easy to identify . For example, show different styles for links when the user hovers over them or focuses using the keyboard.

Provide clear and consistent navigation options . Use consistent layouts and naming conventions for menu items to prevent confusion. For example, if you use breadcrumbs, ensure they are consistently in the same position across different web pages.

Ensure that form elements include clearly associated labels . For example, place form labels to the left of a form field (for left-to-right languages) instead of above or inside the input field to reduce errors.

Provide easily identifiable feedback . If feedback (such as error messages) is in fine print or a specific color, people with lower vision or colorblindness will find it harder to use the website. Make sure such feedback is clear and easy to identify. For example, you can offer options to navigate to different errors.

Use headings and spacing to group related content. Good visual hierarchy (through typography, whitespace and grid layouts) makes it easy to scan content.

Create designs for different viewport sizes . Ensure your content scales up (to larger devices) and down (to fit smaller screens). Design responsive websites and test them thoroughly. 

Include image and media alternatives in your design . Provide transcripts for audio and video content and text alternatives for images. Ensure the alternative text on images conveys meaning and doesn’t simply describe the image. If you use PDFs, make sure they, too, are accessible.

Provide controls for content that starts automatically . Allow users to pause animations or video content that plays automatically.

These practices not only make a website easier to access for people with disabilities but also for usability in general for everyone.

Learn More about Web Design

Learn how to apply the principles of user-centered design in the course Web Design for Usability . 

For more on adaptive and responsive design, take the Mobile UX Design: The Beginner's Guide course. 

See W3C’s Designing for Web Accessibility for practical tips on implementing accessibility.

Questions related to Web Design

Designing a web page involves creating a visual layout and aesthetic.

Start by defining the purpose and target audience of your page.

Understand the type of content and what actions the user will perform on the web page.

Sketch ideas and create wireframes or mockups of the layout.

Select a color scheme, typography, and imagery that align with your brand identity.

Use design software like Figma or Sketch to create the design.

Finally, gather feedback and make necessary revisions before handing off the development design.

In each step, remember to keep the user experience and accessibility considerations foremost. Here’s why Accessibility Matters: 

The salary of web designers varies widely based on experience, location, and skill set. As of our last update, the average salary for a Web Designer in the United States is reported to be approximately $52,691 per year, according to Glassdoor. However, this figure can range from around $37,000 for entry-level positions to over $73,000 for experienced designers. It is crucial to mention that salaries may differ significantly by region, company size, and individual qualifications. For the most up-to-date and region-specific salary information, visit Glassdoor .

To become a web designer, you should start by understanding design principles, usability best practices, color theory, and typography. Next, learn the essential tools like Adobe Photoshop, Illustrator, and Sketch. Familiarize yourself with web design languages such as HTML, CSS, and JavaScript. It's important to create a portfolio of your top work to impress potential employers. Additionally, consider taking online courses to enhance your knowledge and skills. 

Interaction Design Foundation offers a comprehensive UI Designer Learning Path that can help you become proficient in user interface design, a key component of web design. Lastly, continuously practice web design, seek feedback, and stay up-to-date with the latest trends and technologies.

The role of a web designer entails the task of designing a website's visual design and layout of a website, which includes the site's appearance, structure, navigation, and accessibility. They select color palettes, create graphics, choose fonts, and layout content to create an aesthetically pleasing, user-friendly, and accessible design. Web designers also work closely with web developers to verify that the design is technically feasible and implemented correctly. They may be involved in user experience design, ensuring the website is intuitive, accessible, and easy to use. Additionally, web designers must be aware of designer bias, as discussed in this video. 

Ultimately, a web designer's goal is to create a visually appealing, functional, accessible, and positive user experience.

Web design and coding are closely related, but they are not the same. Web design involves creating the visual elements and layout of a website, while coding involves translating these designs into a functional website using programming languages like HTML, CSS, and JavaScript. Typically, dedicated web developers translate the designs to code. Several design tools can also export code directly.

Although some web designers also have coding skills, it is not a requirement for all web design roles. However, having a basic understanding of coding can be beneficial for a web designer as it helps in creating designs that are both aesthetically pleasing and technically feasible.

Responsive web design guarantees that a website adapts its format to fit any screen size across different devices and screen sizes, from desktops to tablets to mobile phones. It includes the site to the device's resolution, supports device switching and increases accessibility and SEO-friendliness.

As Frank Spillers, CEO of Experience Dynamics mentions in this video, responsive design is a default, and not an optional feature because everyone expects mobile optimization. This approach is vital for Google's algorithm, which prioritizes responsive sites.

To learn web design, start by understanding its fundamental principles, such as color theory, typography, and layout. Practice designing websites, get feedback, and iterate on your designs. Enhance your skills by taking online courses, attending workshops, and reading articles. 

Consider the Interaction Design Foundation's comprehensive UI Designer learning path for essential skills and knowledge. If you're interested in expanding your skill set, consider exploring UX design as an alternative. The article " How to Change Your Career from Web Design to UX Design " on the IxDF Blog offers insightful guidance. Start your journey today!

Absolutely, web design is a rewarding career choice. It offers creative freedom, a chance to solve real-world problems, and a growing demand for skilled professionals. With the digital world expanding, businesses seek qualified web designers to create user-friendly and visually appealing websites. Additionally, web design offers diverse job opportunities, competitive salaries, and the option to work freelance or in-house. Continuously evolving technology ensures that web design remains a dynamic and future-proof career.

Web design and front-end development are related but distinct disciplines. Web design involves creating the visual layout and aesthetics of a website, focusing on user experience, graphics, and overall look. Front-end development, on the other hand, involves implementing the design into a functional website using coding languages like HTML, CSS, and JavaScript. While there is overlap, and many professionals have skills in both areas, web design is more creative, and front-end development is more technical.

In this Master Class webinar, Szymon Adamiak of Hype4 shares his top tips for smooth designer-developer relationships, based on years of working as a front-end developer with teams of designers on various projects.

Yes and no! A web page is a type of user interface—it is the touchpoint between a business and the user. People interact with web pages. They may fill out a form, or simply navigate from one page to another. A web designer must also be familiar with UI design best practices to ensure the website is usable.

That said, in practice, the term UI is most often associated with applications. Unlike web pages, which tend to be more static and are closely related to branding and communication, applications (on both web and mobile) allow users to manipulate data and perform tasks.. 

UI design, as explained in this video above, involves visualizing and creating the interface of an application, focusing on aesthetics, user experience, and overall look. To learn more, check our UI Design Learning Path .

A modal in web design is a secondary window that appears above the primary webpage, focusing on specific content and pausing interaction with the main page. It's a common user interface design pattern used to solve interface problems by showing contextual information when they matter. 

The video above explains the importance of designing good UI patterns to enhance user experience and reduce usability issues. Modals are crucial for successful user-centered design and product development like other UI patterns.

In web design, CMS refers to a Content Management System. It is software used to create and manage digital content. 

The video above implies that the content, including those managed by a CMS, is crucial in every stage of the user experience, from setup to engagement. The top 10 CMS in 2023 are the following:

Magento (more focused on e-commerce)

Squarespace

Shopify (more focused on e-commerce)

The popularity and usage of CMS platforms can vary over time, and there may be new players in the market since our last update. 

Literature on Web Design

Here’s the entire UX literature on Web Design by the Interaction Design Foundation, collated in one place:

Learn more about Web Design

Take a deep dive into Web Design with our course Mobile UX Design: The Beginner's Guide .

In the “ Build Your Portfolio” project, you’ll find a series of practical exercises that will give you first-hand experience with the methods we cover. You will build on your project in each lesson so once you have completed the course you will have a thorough case study for your portfolio.

Mobile User Experience Design: Introduction , has been built on evidence-based research and practice. It is taught by the CEO of ExperienceDynamics.com, Frank Spillers, author, speaker and internationally respected Senior Usability practitioner.

All open-source articles on Web Design

Repetition, pattern, and rhythm.

• 1.2k shares

Adaptive vs. Responsive Design

• 3 years ago

How to Change Your Career from Web Design to UX Design

• 1.1k shares

Emphasis: Setting up the focal point of your design

• 8 years ago

Accessibility: Usability for all

• 2 years ago

How to Design Great 404 Error Pages

• 4 years ago

Emotion and website design

Parallax Web Design - The Earth May Not Move for Us But the Web Can

Fitts’ Law: Tracking users’ clicks

Video and Web Design

• 7 years ago

The Best UX Portfolio Website Builders in 2024

10 of Our Favorite Login Screen Examples

Web Fonts: Definition and 10 Recommendations

What is Eye Tracking in UX?

Open Access—Link to us!

We believe in Open Access and the  democratization of knowledge . Unfortunately, world-class educational materials such as this page are normally hidden behind paywalls or in expensive textbooks.

If you want this to change , cite this page , link to us, or join us to help us democratize design knowledge !

Privacy Settings

Our digital services use necessary tracking technologies, including third-party cookies, for security, functionality, and to uphold user rights. Optional cookies offer enhanced features, and analytics.

Experience the full potential of our site that remembers your preferences and supports secure sign-in.

Governs the storage of data necessary for maintaining website security, user authentication, and fraud prevention mechanisms.

Enhanced Functionality

Saves your settings and preferences, like your location, for a more personalized experience.

Referral Program

We use cookies to enable our referral program, giving you and your friends discounts.

Error Reporting

We share user ID with Bugsnag and NewRelic to help us track errors and fix issues.

Optimize your experience by allowing us to monitor site usage. You’ll enjoy a smoother, more personalized journey without compromising your privacy.

Analytics Storage

Collects anonymous data on how you navigate and interact, helping us make informed improvements.

Differentiates real visitors from automated bots, ensuring accurate usage data and improving your website experience.

Lets us tailor your digital ads to match your interests, making them more relevant and useful to you.

Advertising Storage

Stores information for better-targeted advertising, enhancing your online ad experience.

Personalization Storage

Permits storing data to personalize content and ads across Google services based on user behavior, enhancing overall user experience.

Advertising Personalization

Allows for content and ad personalization across Google services based on user behavior. This consent enhances user experiences.

Enables personalizing ads based on user data and interactions, allowing for more relevant advertising experiences across Google services.

Receive more relevant advertisements by sharing your interests and behavior with our trusted advertising partners.

Enables better ad targeting and measurement on Meta platforms, making ads you see more relevant.

Allows for improved ad effectiveness and measurement through Meta’s Conversions API, ensuring privacy-compliant data sharing.

LinkedIn Insights

Tracks conversions, retargeting, and web analytics for LinkedIn ad campaigns, enhancing ad relevance and performance.

LinkedIn CAPI

Enhances LinkedIn advertising through server-side event tracking, offering more accurate measurement and personalization.

Google Ads Tag

Tracks ad performance and user engagement, helping deliver ads that are most useful to you.

Share Knowledge, Get Respect!

or copy link

Cite according to academic standards

Simply copy and paste the text below into your bibliographic reference list, onto your blog, or anywhere else. You can also just hyperlink to this page.

New to UX Design? We’re Giving You a Free ebook!

Download our free ebook The Basics of User Experience Design to learn about core concepts of UX design.

In 9 chapters, we’ll cover: conducting user interviews, design thinking, interaction design, mobile UX design, usability, UX research, and many more!

web development Recently Published Documents

Total documents.

• Latest Documents
• Most Cited Documents
• Contributed Authors
• Related Sources
• Related Keywords

Website Developmemt Technologies: A Review

Abstract: Service Science is that the basis of knowledge system and net services that judge to the provider/client model. This paper developments a technique which will be utilized in the event of net services like websites, net applications and eCommerce. The goal is to development a technique that may add structure to a extremely unstructured drawback to help within the development and success of net services. The new methodology projected are going to be referred to as {the net|the online|the net} Development Life Cycle (WDLC) and tailored from existing methodologies and applied to the context of web development. This paper can define well the projected phases of the WDLC. Keywords: Web Development, Application Development, Technologies, eCommerce.

Analysis of Russian Segment of the Web Development Market Operating Online on Upwork

The Russian segment of the web services market in the online environment, on the platform of the Upwork freelance exchange, is considered, its key characteristics, the composition of participants, development trends are highlighted, and the market structure is identified. It is found that despite the low barriers to entry, the web development market is very stable, since the composition of entrenched firms that have been operating for more than six years remains. The pricing policy of most Russian companies indicates that they work in the middle price segment and have low budgets, which is due to the specifics of the foreign market and high competition.

Farming Assistant Web Services: Agricultor

Abstract: Our farming assistant web services provides assistance to new as well as establish farmers to get the solutions to dayto-day problems faced in the field. A farmer gets to connect with other farmers throughout India to get more information about a particular crop which is popular in other states. Keywords: Farmers, Assistance, Web Development

Tradução de ementas e histórico escolar para o inglês: contribuição para participação de discentes do curso técnico em informática para internet integrado ao ensino médio em programas de mobilidade acadêmica / Translation of summary and school records into english: contribution to the participation of high school with associate technical degree on web development students in academic mobility programs

Coded websites vs wordpress websites.

This document gives multiple instructions related to web developers using older as well as newer technology. Websites are being created using newer technologies like wordpress whereas on the other hand many people prefer making websites using the traditional way. This document will clear the doubt whether an individual should use wordpress websites or coded websites according to the users convenience. The Responsiveness of the websites, the use of CMS nowadays, more and more up gradation of technologies with SEO, themes, templates, etc. make things like web development much much easier. The aesthetics, the culture, the expressions, the features all together add up in order make the designing and development a lot more efficient and effective. Digital Marketing has a tremendous growth over the last two years and yet shows no signs of stopping, is closely related with the web development environment. Nowadays all businesses are going online due to which the impact of web development has become such that it has become an integral part of any online business.

Cognitive disabilities and web accessibility: a survey into the Brazilian web development community

Cognitive disabilities include a diversity of conditions related to cognitive functions, such as reading, understanding, learning, solving problems, memorization and speaking. They differ largely from each other, making them a heterogeneous complex set of disabilities. Although the awareness about cognitive disabilities has been increasing in the last few years, it is still less than necessary compared to other disabilities. The need for an investigation about this issue is part of the agenda of the Challenge 2 (Accessibility and Digital Inclusion) from GranDIHC-Br. This paper describes the results of an online exploratory survey conducted with 105 web development professionals from different sectors to understand their knowledge and barriers regarding accessibility for people with cognitive disabilities. The results evidenced three biases that potentially prevent those professionals from approaching cogni-tive disabilities: strong organizational barriers; difficulty to understand user needs related to cognitive disabilities; a knowledge gap about web accessibility principles and guidelines. Our results confirmed that web development professionals are unaware about cognitive disabilities mostly by a lack of knowledge about them, even if they understand web accessibility in a technical level. Therefore, we suggest that applied research studies focus on how to fill this knowledge gap before providing tools, artifacts or frameworks.

PERANCANGAN WEB RESPONSIVE UNTUK SISTEM INFORMASI OBAT-OBATAN

A good information system must not only be neat, effective, and resilient, but also must be user friendly and up to date. In a sense, it is able to be applied to various types of electronic devices, easily accessible at any whereand time (real time), and can be modified according to user needs in a relatively easy and simple way. Information systems are now needed by various parties, especially in the field of administration and sale of medicines for Cut Nyak Dhien Hospital. During this time, recording in books has been very ineffective and caused many problems, such as difficulty in accessing old data, asa well as the information obtained was not real time. To solve it, this research raises the theme of the appropriate information system design for the hospital concerned, by utilizing CSS Bootstrap framework and research methodology for web development, namely Web Development Life Cycle. This research resulted in a responsive system by providing easy access through desktop computers, tablets, and smartphones so that it would help the hospital in the data processing process in real time.

Web Development and performance comparison of Web Development Technologies in Node.js and Python

“tom had us all doing front-end web development”: a nostalgic (re)imagining of myspace, assessment of site classifications according to layout type in web development, export citation format, share document.

Numbers, Facts and Trends Shaping Your World

Read our research on:

Full Topic List

Regions & Countries

• Publications
• Our Methods
• Short Reads
• Tools & Resources

Read Our Research On:

Writing Survey Questions

Perhaps the most important part of the survey process is the creation of questions that accurately measure the opinions, experiences and behaviors of the public. Accurate random sampling will be wasted if the information gathered is built on a shaky foundation of ambiguous or biased questions. Creating good measures involves both writing good questions and organizing them to form the questionnaire.

Questionnaire design is a multistage process that requires attention to many details at once. Designing the questionnaire is complicated because surveys can ask about topics in varying degrees of detail, questions can be asked in different ways, and questions asked earlier in a survey may influence how people respond to later questions. Researchers are also often interested in measuring change over time and therefore must be attentive to how opinions or behaviors have been measured in prior surveys.

Surveyors may conduct pilot tests or focus groups in the early stages of questionnaire development in order to better understand how people think about an issue or comprehend a question. Pretesting a survey is an essential step in the questionnaire design process to evaluate how people respond to the overall questionnaire and specific questions, especially when questions are being introduced for the first time.

For many years, surveyors approached questionnaire design as an art, but substantial research over the past forty years has demonstrated that there is a lot of science involved in crafting a good survey questionnaire. Here, we discuss the pitfalls and best practices of designing questionnaires.

Question development

There are several steps involved in developing a survey questionnaire. The first is identifying what topics will be covered in the survey. For Pew Research Center surveys, this involves thinking about what is happening in our nation and the world and what will be relevant to the public, policymakers and the media. We also track opinion on a variety of issues over time so we often ensure that we update these trends on a regular basis to better understand whether people’s opinions are changing.

At Pew Research Center, questionnaire development is a collaborative and iterative process where staff meet to discuss drafts of the questionnaire several times over the course of its development. We frequently test new survey questions ahead of time through qualitative research methods such as  focus groups , cognitive interviews, pretesting (often using an  online, opt-in sample ), or a combination of these approaches. Researchers use insights from this testing to refine questions before they are asked in a production survey, such as on the ATP.

Measuring change over time

Many surveyors want to track changes over time in people’s attitudes, opinions and behaviors. To measure change, questions are asked at two or more points in time. A cross-sectional design surveys different people in the same population at multiple points in time. A panel, such as the ATP, surveys the same people over time. However, it is common for the set of people in survey panels to change over time as new panelists are added and some prior panelists drop out. Many of the questions in Pew Research Center surveys have been asked in prior polls. Asking the same questions at different points in time allows us to report on changes in the overall views of the general public (or a subset of the public, such as registered voters, men or Black Americans), or what we call “trending the data”.

When measuring change over time, it is important to use the same question wording and to be sensitive to where the question is asked in the questionnaire to maintain a similar context as when the question was asked previously (see  question wording  and  question order  for further information). All of our survey reports include a topline questionnaire that provides the exact question wording and sequencing, along with results from the current survey and previous surveys in which we asked the question.

The Center’s transition from conducting U.S. surveys by live telephone interviewing to an online panel (around 2014 to 2020) complicated some opinion trends, but not others. Opinion trends that ask about sensitive topics (e.g., personal finances or attending religious services ) or that elicited volunteered answers (e.g., “neither” or “don’t know”) over the phone tended to show larger differences than other trends when shifting from phone polls to the online ATP. The Center adopted several strategies for coping with changes to data trends that may be related to this change in methodology. If there is evidence suggesting that a change in a trend stems from switching from phone to online measurement, Center reports flag that possibility for readers to try to head off confusion or erroneous conclusions.

Open- and closed-ended questions

One of the most significant decisions that can affect how people answer questions is whether the question is posed as an open-ended question, where respondents provide a response in their own words, or a closed-ended question, where they are asked to choose from a list of answer choices.

For example, in a poll conducted after the 2008 presidential election, people responded very differently to two versions of the question: “What one issue mattered most to you in deciding how you voted for president?” One was closed-ended and the other open-ended. In the closed-ended version, respondents were provided five options and could volunteer an option not on the list.

When explicitly offered the economy as a response, more than half of respondents (58%) chose this answer; only 35% of those who responded to the open-ended version volunteered the economy. Moreover, among those asked the closed-ended version, fewer than one-in-ten (8%) provided a response other than the five they were read. By contrast, fully 43% of those asked the open-ended version provided a response not listed in the closed-ended version of the question. All of the other issues were chosen at least slightly more often when explicitly offered in the closed-ended version than in the open-ended version. (Also see  “High Marks for the Campaign, a High Bar for Obama”  for more information.)

Researchers will sometimes conduct a pilot study using open-ended questions to discover which answers are most common. They will then develop closed-ended questions based off that pilot study that include the most common responses as answer choices. In this way, the questions may better reflect what the public is thinking, how they view a particular issue, or bring certain issues to light that the researchers may not have been aware of.

When asking closed-ended questions, the choice of options provided, how each option is described, the number of response options offered, and the order in which options are read can all influence how people respond. One example of the impact of how categories are defined can be found in a Pew Research Center poll conducted in January 2002. When half of the sample was asked whether it was “more important for President Bush to focus on domestic policy or foreign policy,” 52% chose domestic policy while only 34% said foreign policy. When the category “foreign policy” was narrowed to a specific aspect – “the war on terrorism” – far more people chose it; only 33% chose domestic policy while 52% chose the war on terrorism.

In most circumstances, the number of answer choices should be kept to a relatively small number – just four or perhaps five at most – especially in telephone surveys. Psychological research indicates that people have a hard time keeping more than this number of choices in mind at one time. When the question is asking about an objective fact and/or demographics, such as the religious affiliation of the respondent, more categories can be used. In fact, they are encouraged to ensure inclusivity. For example, Pew Research Center’s standard religion questions include more than 12 different categories, beginning with the most common affiliations (Protestant and Catholic). Most respondents have no trouble with this question because they can expect to see their religious group within that list in a self-administered survey.

In addition to the number and choice of response options offered, the order of answer categories can influence how people respond to closed-ended questions. Research suggests that in telephone surveys respondents more frequently choose items heard later in a list (a “recency effect”), and in self-administered surveys, they tend to choose items at the top of the list (a “primacy” effect).

Because of concerns about the effects of category order on responses to closed-ended questions, many sets of response options in Pew Research Center’s surveys are programmed to be randomized to ensure that the options are not asked in the same order for each respondent. Rotating or randomizing means that questions or items in a list are not asked in the same order to each respondent. Answers to questions are sometimes affected by questions that precede them. By presenting questions in a different order to each respondent, we ensure that each question gets asked in the same context as every other question the same number of times (e.g., first, last or any position in between). This does not eliminate the potential impact of previous questions on the current question, but it does ensure that this bias is spread randomly across all of the questions or items in the list. For instance, in the example discussed above about what issue mattered most in people’s vote, the order of the five issues in the closed-ended version of the question was randomized so that no one issue appeared early or late in the list for all respondents. Randomization of response items does not eliminate order effects, but it does ensure that this type of bias is spread randomly.

Questions with ordinal response categories – those with an underlying order (e.g., excellent, good, only fair, poor OR very favorable, mostly favorable, mostly unfavorable, very unfavorable) – are generally not randomized because the order of the categories conveys important information to help respondents answer the question. Generally, these types of scales should be presented in order so respondents can easily place their responses along the continuum, but the order can be reversed for some respondents. For example, in one of Pew Research Center’s questions about abortion, half of the sample is asked whether abortion should be “legal in all cases, legal in most cases, illegal in most cases, illegal in all cases,” while the other half of the sample is asked the same question with the response categories read in reverse order, starting with “illegal in all cases.” Again, reversing the order does not eliminate the recency effect but distributes it randomly across the population.

Question wording

The choice of words and phrases in a question is critical in expressing the meaning and intent of the question to the respondent and ensuring that all respondents interpret the question the same way. Even small wording differences can substantially affect the answers people provide.

[View more Methods 101 Videos ]

An example of a wording difference that had a significant impact on responses comes from a January 2003 Pew Research Center survey. When people were asked whether they would “favor or oppose taking military action in Iraq to end Saddam Hussein’s rule,” 68% said they favored military action while 25% said they opposed military action. However, when asked whether they would “favor or oppose taking military action in Iraq to end Saddam Hussein’s rule  even if it meant that U.S. forces might suffer thousands of casualties, ” responses were dramatically different; only 43% said they favored military action, while 48% said they opposed it. The introduction of U.S. casualties altered the context of the question and influenced whether people favored or opposed military action in Iraq.

There has been a substantial amount of research to gauge the impact of different ways of asking questions and how to minimize differences in the way respondents interpret what is being asked. The issues related to question wording are more numerous than can be treated adequately in this short space, but below are a few of the important things to consider:

First, it is important to ask questions that are clear and specific and that each respondent will be able to answer. If a question is open-ended, it should be evident to respondents that they can answer in their own words and what type of response they should provide (an issue or problem, a month, number of days, etc.). Closed-ended questions should include all reasonable responses (i.e., the list of options is exhaustive) and the response categories should not overlap (i.e., response options should be mutually exclusive). Further, it is important to discern when it is best to use forced-choice close-ended questions (often denoted with a radio button in online surveys) versus “select-all-that-apply” lists (or check-all boxes). A 2019 Center study found that forced-choice questions tend to yield more accurate responses, especially for sensitive questions.  Based on that research, the Center generally avoids using select-all-that-apply questions.

It is also important to ask only one question at a time. Questions that ask respondents to evaluate more than one concept (known as double-barreled questions) – such as “How much confidence do you have in President Obama to handle domestic and foreign policy?” – are difficult for respondents to answer and often lead to responses that are difficult to interpret. In this example, it would be more effective to ask two separate questions, one about domestic policy and another about foreign policy.

In general, questions that use simple and concrete language are more easily understood by respondents. It is especially important to consider the education level of the survey population when thinking about how easy it will be for respondents to interpret and answer a question. Double negatives (e.g., do you favor or oppose  not  allowing gays and lesbians to legally marry) or unfamiliar abbreviations or jargon (e.g., ANWR instead of Arctic National Wildlife Refuge) can result in respondent confusion and should be avoided.

Similarly, it is important to consider whether certain words may be viewed as biased or potentially offensive to some respondents, as well as the emotional reaction that some words may provoke. For example, in a 2005 Pew Research Center survey, 51% of respondents said they favored “making it legal for doctors to give terminally ill patients the means to end their lives,” but only 44% said they favored “making it legal for doctors to assist terminally ill patients in committing suicide.” Although both versions of the question are asking about the same thing, the reaction of respondents was different. In another example, respondents have reacted differently to questions using the word “welfare” as opposed to the more generic “assistance to the poor.” Several experiments have shown that there is much greater public support for expanding “assistance to the poor” than for expanding “welfare.”

We often write two versions of a question and ask half of the survey sample one version of the question and the other half the second version. Thus, we say we have two  forms  of the questionnaire. Respondents are assigned randomly to receive either form, so we can assume that the two groups of respondents are essentially identical. On questions where two versions are used, significant differences in the answers between the two forms tell us that the difference is a result of the way we worded the two versions.

One of the most common formats used in survey questions is the “agree-disagree” format. In this type of question, respondents are asked whether they agree or disagree with a particular statement. Research has shown that, compared with the better educated and better informed, less educated and less informed respondents have a greater tendency to agree with such statements. This is sometimes called an “acquiescence bias” (since some kinds of respondents are more likely to acquiesce to the assertion than are others). This behavior is even more pronounced when there’s an interviewer present, rather than when the survey is self-administered. A better practice is to offer respondents a choice between alternative statements. A Pew Research Center experiment with one of its routinely asked values questions illustrates the difference that question format can make. Not only does the forced choice format yield a very different result overall from the agree-disagree format, but the pattern of answers between respondents with more or less formal education also tends to be very different.

One other challenge in developing questionnaires is what is called “social desirability bias.” People have a natural tendency to want to be accepted and liked, and this may lead people to provide inaccurate answers to questions that deal with sensitive subjects. Research has shown that respondents understate alcohol and drug use, tax evasion and racial bias. They also may overstate church attendance, charitable contributions and the likelihood that they will vote in an election. Researchers attempt to account for this potential bias in crafting questions about these topics. For instance, when Pew Research Center surveys ask about past voting behavior, it is important to note that circumstances may have prevented the respondent from voting: “In the 2012 presidential election between Barack Obama and Mitt Romney, did things come up that kept you from voting, or did you happen to vote?” The choice of response options can also make it easier for people to be honest. For example, a question about church attendance might include three of six response options that indicate infrequent attendance. Research has also shown that social desirability bias can be greater when an interviewer is present (e.g., telephone and face-to-face surveys) than when respondents complete the survey themselves (e.g., paper and web surveys).

Lastly, because slight modifications in question wording can affect responses, identical question wording should be used when the intention is to compare results to those from earlier surveys. Similarly, because question wording and responses can vary based on the mode used to survey respondents, researchers should carefully evaluate the likely effects on trend measurements if a different survey mode will be used to assess change in opinion over time.

Question order

Once the survey questions are developed, particular attention should be paid to how they are ordered in the questionnaire. Surveyors must be attentive to how questions early in a questionnaire may have unintended effects on how respondents answer subsequent questions. Researchers have demonstrated that the order in which questions are asked can influence how people respond; earlier questions can unintentionally provide context for the questions that follow (these effects are called “order effects”).

One kind of order effect can be seen in responses to open-ended questions. Pew Research Center surveys generally ask open-ended questions about national problems, opinions about leaders and similar topics near the beginning of the questionnaire. If closed-ended questions that relate to the topic are placed before the open-ended question, respondents are much more likely to mention concepts or considerations raised in those earlier questions when responding to the open-ended question.

For closed-ended opinion questions, there are two main types of order effects: contrast effects ( where the order results in greater differences in responses), and assimilation effects (where responses are more similar as a result of their order).

An example of a contrast effect can be seen in a Pew Research Center poll conducted in October 2003, a dozen years before same-sex marriage was legalized in the U.S. That poll found that people were more likely to favor allowing gays and lesbians to enter into legal agreements that give them the same rights as married couples when this question was asked after one about whether they favored or opposed allowing gays and lesbians to marry (45% favored legal agreements when asked after the marriage question, but 37% favored legal agreements without the immediate preceding context of a question about same-sex marriage). Responses to the question about same-sex marriage, meanwhile, were not significantly affected by its placement before or after the legal agreements question.

Another experiment embedded in a December 2008 Pew Research Center poll also resulted in a contrast effect. When people were asked “All in all, are you satisfied or dissatisfied with the way things are going in this country today?” immediately after having been asked “Do you approve or disapprove of the way George W. Bush is handling his job as president?”; 88% said they were dissatisfied, compared with only 78% without the context of the prior question.

Responses to presidential approval remained relatively unchanged whether national satisfaction was asked before or after it. A similar finding occurred in December 2004 when both satisfaction and presidential approval were much higher (57% were dissatisfied when Bush approval was asked first vs. 51% when general satisfaction was asked first).

Several studies also have shown that asking a more specific question before a more general question (e.g., asking about happiness with one’s marriage before asking about one’s overall happiness) can result in a contrast effect. Although some exceptions have been found, people tend to avoid redundancy by excluding the more specific question from the general rating.

Assimilation effects occur when responses to two questions are more consistent or closer together because of their placement in the questionnaire. We found an example of an assimilation effect in a Pew Research Center poll conducted in November 2008 when we asked whether Republican leaders should work with Obama or stand up to him on important issues and whether Democratic leaders should work with Republican leaders or stand up to them on important issues. People were more likely to say that Republican leaders should work with Obama when the question was preceded by the one asking what Democratic leaders should do in working with Republican leaders (81% vs. 66%). However, when people were first asked about Republican leaders working with Obama, fewer said that Democratic leaders should work with Republican leaders (71% vs. 82%).

The order questions are asked is of particular importance when tracking trends over time. As a result, care should be taken to ensure that the context is similar each time a question is asked. Modifying the context of the question could call into question any observed changes over time (see  measuring change over time  for more information).

A questionnaire, like a conversation, should be grouped by topic and unfold in a logical order. It is often helpful to begin the survey with simple questions that respondents will find interesting and engaging. Throughout the survey, an effort should be made to keep the survey interesting and not overburden respondents with several difficult questions right after one another. Demographic questions such as income, education or age should not be asked near the beginning of a survey unless they are needed to determine eligibility for the survey or for routing respondents through particular sections of the questionnaire. Even then, it is best to precede such items with more interesting and engaging questions. One virtue of survey panels like the ATP is that demographic questions usually only need to be asked once a year, not in each survey.

U.S. Surveys

Other research methods, sign up for our weekly newsletter.

Fresh data delivered Saturday mornings

1615 L St. NW, Suite 800 Washington, DC 20036 USA (+1) 202-419-4300 | Main (+1) 202-857-8562 | Fax (+1) 202-419-4372 |  Media Inquiries

Research Topics

• Age & Generations
• Coronavirus (COVID-19)
• Economy & Work
• Family & Relationships
• Gender & LGBTQ
• Immigration & Migration
• International Affairs
• Internet & Technology
• Methodological Research
• News Habits & Media
• Non-U.S. Governments
• Other Topics
• Politics & Policy
• Race & Ethnicity
• Email Newsletters

ABOUT PEW RESEARCH CENTER  Pew Research Center is a nonpartisan fact tank that informs the public about the issues, attitudes and trends shaping the world. It conducts public opinion polling, demographic research, media content analysis and other empirical social science research. Pew Research Center does not take policy positions. It is a subsidiary of  The Pew Charitable Trusts .

Copyright 2024 Pew Research Center

Terms & Conditions

Privacy Policy

Cookie Settings

Reprints, Permissions & Use Policy

Accessibility engineering in web evaluation process: a systematic literature review

• Review Paper
• Open access
• Published: 27 January 2023

Cite this article

You have full access to this open access article

• Jinat Ara 1 ,
• Cecilia Sik-Lanyi 1 &
• Arpad Kelemen 2  

3995 Accesses

3 Citations

3 Altmetric

Explore all metrics

Several works of literature contributed to the web evaluation process in recent years to promote digital inclusion by addressing several accessibility guidelines, methods, processes, and techniques. Researchers have investigated how the web evaluation process could be facilitated by including accessibility issues to obtain an inclusive and accessible solution to improve the user experience and increase user satisfaction. Three systematic literature reviews (SLRs) have been conducted in the context of past research, considering such research focuses. This paper presents a new SLR approach concerning accessibility in the web evaluation process, considering the period from 2010 to 2021. The review of 92 primary studies showed the contribution of publications on different phases of the web evaluation process mainly by highlighting the significant studies in the framework design and testing process. To the best of our knowledge, this is the first study focused on the web accessibility literature reporting the engineering assets for evaluation of new accessible and inclusive web-based solutions (e.g., websites). Besides, in this study, we aim to provide a new direction to the web designers and developers with an updated view of process, methods, techniques, tools, and other crucial aspects to contribute to the accessible process enrichment, as well as depict the gaps and challenges that may be worthy to be investigated in the future. The findings of this SLR introduce a new dimension in web accessibility research on determining and mitigating the research gap of web accessibility issues for web designers, developers, and other practitioners.

Similar content being viewed by others

Toward a Combined Method for Evaluation of Web Accessibility

Automated Evaluation Tools for Web and Mobile Accessibility: A Systematic Literature Review

Comparison of Accessibility Tools for the Evaluation of E-Commerce Websites

Avoid common mistakes on your manuscript.

1 Introduction

In recent years, various aspects have motivated researchers to conduct studies about digital accessibility. The extension and increased availability of the web for multiple purposes (e.g., information search), the representation of the content (e.g., video, audio), and the emergence of new platforms (e.g., Internet of Things) and technologies (e.g., mobile, computer, tablets) are significant aspects to reinforce the investigation of the digital information platform. In particular, from the very beginning of the digital revolution, digital resources become the fundamental source for citizens to access information such as education, health care, government, news, and other information such as entertainment and sports [ 1 , 2 ].

According to the World Wide Web Consortium (W3C) and the Web Accessibility Initiative (WAI) report, accessibility is a broad and extensible term associated with people who have disabilities, incompetent skills, or situational-induced impairment [ 3 ]. This initiative's objective is to ensure accessibility which means people with special needs should be able to access, navigate, interact, and contribute to the information that is available on the Web/Internet, electronic resources/materials, and computer. The current mission of the WAI initiative is to coordinate international, technical, and human efforts to improve web accessibility [ 4 ]. With this mission in mind, WAI launched a set of accessibility guidelines called Web Content Accessibility Guidelines (WCAGs) [ 5 , 6 ]. A detailed description of WCAG is given in Sect.  2 .

The scientific research community has recognized that web design and development must inspect the assorted number of requirements of citizens across the population, including special needs users and elderly citizens. Earlier researchers considered accessibility checking as a supplementary requirement in the evaluation phase of any application development. However, in recent years, researchers suggested that accessibility requirements should be followed from the very beginning of the application design and development. Lack of consideration of accessibility issues during the design and development might introduce violations of accessibility guidelines and consequently basic rights of people with disabilities. A great volume of literature exists addressing accessibility guidelines in the design and development of web platforms [ 7 , 8 ]. More recently, a few studies highlighted the importance and emerging need of considering accessibility throughout the web development life cycle [ 9 , 10 ].

Few studies discussed the importance of systematic literature review (SLR) approaches to present the true insights of a particular topic for highlighting future improvement directions [ 11 , 12 , 13 ]. Campoverde-Molina et al. [ 14 ] mentioned that SLR is a synthesis process of past studies that have been published in different scientific databases focusing on a particular issue. SLR aims to review past literature on a specific domain to determine its effectiveness and find the research gap and new research areas. It helps to identify the way of knowledge improvement, promotes new theories for development, and reveals the new investigated area that needs to focus. Therefore, an SLR focusing on web accessibility engineering assets is essential to determine a way to promote an accessible web platform according to WCAG standards.

Emphasizing the necessity of the SLR approach in the web accessibility context, Akram and Sulaiman [ 14 ] and Campoverde‑Molina et al. [ 15 ] have conducted SLRs to analyze the accessibility of educational institute websites within a specific period. The first SLR performed the analysis regarding the period between 2009 and 2017. The second one conducted the investigation considering the period of 2009 to 2020. In 2021, Campoverde‑Molina et al. [ 16 ] extended their previous work intending to update the result of the past SLR and extended the period from 2002 to 2020. In general, SLR refers to the aggregation of knowledge about a particular domain of research with a set of research questions and solutions. Thus, the SLR process should be as unbiased as possible [ 17 ]. The selected SLRs are auditable and have significant effects. However, the focus on engineering assets such as processes, development techniques, and technologies is limited, which is a drawback of SLRs.

This paper presents an extensive SLR in the context of accessibility in the web evaluation process to identify several engineering processes to improve the accessibility of web platforms. This study will help a wide array of people (developers, designers, inventors, leaders, researchers, and users) and facilitate the accessible web design and evaluation process. The paper is organized as follows: In Sect.  2 , accessibility concepts, importance, and related works are presented. Section 3  describes the details of conducting the SLR. Section  4 represents the result of conducted SLR and discusses the main findings through a broad discussion. In Sect.  5 , we conclude the paper.

2 Background and related work

Digital accessibility is a process to ensure the availability of online tools or content to the users [ 13 ]. The prime objective of digital accessibility is to make an accessible, operable, and interactable online platform to provide equal information accessing opportunities for people with disabilities [ 18 , 19 ]. Several aspects might initiate barriers to implementing and ensuring digital accessible platforms or tools or content, such as limited accessibility knowledge and its guidelines. Sometimes organizational barriers and parameters such as organization size, capital, and cost influence accessibility issues. Addressing these issues, the governments and organizations of several countries declared various guidelines, standards, and conformance levels for the stakeholders [ 20 ]. Following these guidelines, associate authorities might overcome critical issues and ensure digital accessibility.

2.1 Accessibility standards

To develop an accessible solution (e.g., application, websites, software, etc.), several accessibility guidelines have been introduced by the government of several countries and various public and private institutes such as WCAG, Section 508, EN 301 549, YD/T 1761–2012, WAI-ARIA, BITV, ISO 9241 and ATAG are prominent. Web Content Accessibility Guideline (WCAG) was introduced by the Web Accessibility Initiative of the World Wide Web Consortium with several success criteria under 13 guidelines. Section 508 is accessibility requirements rules published by the US Government for digital resources to make the resources accessible. EN 301 549 is a European accessibility requirement that is suitable for public procurement of ICT products and services in Europe. YD/T 1761–2012 refers to the Chinese Technical requirements standards for web accessibility that primarily focus on ensuring accessibility in the digital platform. Besides, the WAI-ARIA standard was published by W3C to define a set of guidelines for HTML attributes to improve semantic accessibility. BITV is a German standard that is issued focusing on WCAG 2.0 to make the website and application accessible for people with disabilities by ensuring perceivable, operable, understandable, and robust guidelines. Similarly, ISO 9241 provides requirements for accessible developments throughout the application development life cycle. It concerns both hardware and software components for interactive design and development. Authoring Tool Accessibility Guidelines (ATAG) is WCAG and User Agent Accessibility guidelines-based instruction for accessible web content design and development.

Among these guidelines, WCAG is the most widely used accessibility standard. WCAG is a documented guideline that explains all the accessibility criteria and step-by-step recommendations about implementation, improvement, and measurement of accessibility to provide a better user experience, especially for people with disabilities. W3C-based WAI first developed the WCAG standards to make the web accessible [ 3 ]. As of July 2022, WAI has published five versions of the WCAG standard, including WCAG 1.0, WCAG 2.0, WCAG 2.1, WCAG 2.2, and WCAG 3.0 (draft version). The WCAG 3.0 is the most sophisticated standard, currently available as a working draft for web developers (front and back end) and designers to develop accessible and usable web content [ 21 ].

In 1999, the first version of WCAG 1.0 was released by W3C with three priorities, 14 guidelines, and 65 checkpoints [ 22 ]. In 2008, W3C released the second version of standards/guidelines, including 61 success criteria and 12 guidelines under four principles: perceivable, operable, understandable, and robust, concerning three conformance levels: Level A, Level AA, and Level AAA [ 23 ]. Furthermore, in 2018, the W3C published an updated version of WCAG 2.0 principles, namely the WCAG 2.1 standard [ 6 ]. It has all the principles, guidelines, success criteria, and conformance levels similar to WCAG 2.0 but they added one new guideline and 17 new success criteria. Therefore, completion of the WCAG 2.1 standard ensures the fulfillment of WCAG 2.0 and is followed with more accessibility concerns. The significant update in WCAG 2.1 is the ‘Operable’ principle. In this principle, a new guideline with six success criteria has been added.

In 2021, W3C extended the WCAG 2.1 guideline and released the WCAG 2.2, an updated version [ 24 ]. In this version, in the Operable principle under guideline (2.4), three new success criteria have been added. In December 2021, the last modified version of WCAG (3.0, working draft) was published, now in progress, waiting for the final draft of guidelines [ 21 ]. Figure  1 shows the WCAG standard with its principles, success criteria, and conformance levels. For the details about success criteria and conformance level, the author refers the reader to [ 24 ]. In addition, all the versions of WCAG followed three conformance levels of A, AA, and AAA to classify web content. By following the WCAG standard, developers and designers can make digital content accessible for a wide range of people with disabilities, including blindness, low vision or vision impairments, deafness and hearing loss, limited movement, dexterity, speech disabilities, sensory disorders, cognitive and learning disabilities, photo-sensitivity and combinations of these [ 25 ]. Nowadays, ensuring an accessible web and improving user experience is crucial for web engineers, researchers, and developers. According to the researchers' opinions, more research needs to be carried out in the next years to improve the accessibility of digital platforms [ 26 ]. Therefore, to understand web accessibility in-depth, a detailed and updated SLR approach is important.

Overview of web content accessibility guidelines (WCAG) version 2.0, 2.1 and 2.2

Our investigation found seven SLRs between 2010 and 2021 related to the area of digital accessibility (two), web accessibility (three), and web-based image and games accessibility (two). The main focus of these seven SLRs is to make digital content accessible for people with disabilities, which is also a prime objective of the digital accessibility consortium. A detailed discussion of the three SLRs concerning web accessibility has been described in the following subsection (2.2) and a comparison of our SLR with the seven earlier SLR studies is conducted in the discussion section.

2.2 Related SLR studies

In the web accessibility context, the first selected SLR was carried out by Akram et al. [ 14 ] to identify the issues with web accessibility of the Saudi Arabian university webpages from the web engineering point of view. To conduct this SLR, they followed three research questions: (1) what are the main principles of Web Content Accessibility Guideline 2.0 (WCAG-2.0) proposed by the W3C to improve web accessibility, (2) what is the compliance level of university and government websites with WCAG-2.0 globally, and (3) what is the compliance level of Saudi Arabian university and government websites with WCAG-2.0. To search past literature, they considered ten scientific databases: Google Scholar, Google search engine, EBSCO host, IEEE Explorer, Science Direct, The Elsevier, Springer Link, ACM Digital Library, Wiley, and Emerald, and found 15 pieces of literature from 2009 to 2017. Their systemic literature review concluded that 87% of the past research employed automatic accessibility testing tools to evaluate university websites. Their SLR also revealed that the most experimented automatic accessibility tools are Bobby, AChecker, eXaminator, TAW, Total Validator, EvalAccess, Cynthia Says, Magenta, Site Analyzer, MAUVE, FAE, WAVE, Valet, and W3C validator service. In addition, they incorporated the manual evaluation process (e.g., interview, questionnaire-based assessment). The manual investigation illustrated that in past research the majority of the work emphasized the improvement of a few accessibility issues such as navigation errors, orientation issues, timing errors, text equivalent to graphics, content, the validity of hypertext markup language (HTML), and cascading style sheets (CSS), use of HTML5, interface design, content, and scripting. However, they conclude that in Saudi Arabia, most universities do not follow World Wide Web Consortium guidelines.

The work proposed by Akram et al. is important in representing the insights of accessibility considering several aspects. However, to validate their represented statistics of implemented automatic accessibility testing with the experimented tools and to identify other possible techniques to validate the accessibility, Campoverde‑Molina et al. [ 15 ] carried out the second SLR and present the empirical results of the accessibility evaluation of educational websites. They have considered 25 past studies from 2009 to 2019 to answer ten research questions. This SLR investigated the selected papers focusing on the bibliometric analysis context and literature review. The SLR determined that 80% of past studies focused on automatic analysis through automatic accessibility evaluation tools, 8% through user incorporation, and 12% through hybrid approaches such as expert invitation, user involvement, and automated tools consideration. This SLR concluded that selected websites did not satisfy any version of the WCAG standard and their conformance levels that introduce the necessity of correction of errors by adopting automated tools and manual observation during website construction.

Following their first SLR, Campoverde‑Molina et al. [ 16 ] extended their previous SLR considering the period from 2002 to 2020 to investigate more research works to represent the accessibility insights in depth. This recent SLR aimed to analyze past literature that focused on the accessibility analysis of university websites. They performed an investigation of 42 selected papers obtained from three scientific databases (Web of Science, Scopus, and IEEE Xplore), focusing on the accessibility standards and accessibility evaluation methodologies. In 42 papers, they found that 38,416 university webpages have been experimented with in the past years. Their SLR result illustrates that all the experimented websites were from Asia. Most of the existing research has experimented with university homepages. All the past literature followed two standards: ISO/IEC 40,500:2012, and Sect. 508, to analyze the accessibility of web pages. They also concluded that past studies considered automatic evaluation tools to validate university web pages, which is around 90.47%. The most frequently used accessibility testing tools are AChecker, WAVE, Bobby, and TAW. However, the inspection result of this SLR is that most of the past investigated university websites showed violations of accessibility guidelines, most commonly adaptability, compatible, distinguishable, input assistance, keyboard accessible, navigable, predictable, readable, and text alternatives that show important accessibility issues.

The selected three systematic literature reviews represent the current insights of the web in detail, considering the term of accessibility context. Despite the importance of these SLR approaches, they have a poor concern about past research domains and lack consideration of engineering approaches, methods, etc. The lack of engineering methods shows the shortcoming of the past SLR that initiate the importance of a detailed future of SLR. In this paper, our presented SLR is unlike the other three systematic literature reviews. We consider a wide range of existing literature intending to determine the engineering approach to initiate future research to mitigate the current research gap.

3 Research methodology

This study aims to conduct a systematic literature review by following the SLR process guidelines and Kitchenham’s guidelines from Kitchenham and Charters [ 27 ]. This research considers three steps to facilitate the SLR approach: (i) planning the SLR process, (ii) conducting the SLR approach, and (iii) reporting the review findings. Figure  2 represents the flowchart of our SLR process.

Flowchart of the proposed systematic literature review (SLR)

3.1 Planning the SLR process

The main sub-activities related to planning the SLR are (i) research question specification, (ii) search string formulation, and (iii) database selection. All these sub-activities are described below.

3.1.1 Research questions

The first step of a literature review is to develop the research questions. Therefore, we developed the research questions according to our research focus. The two research questions are the following:

Research Question-1: What are the available methods, techniques, processes, and approaches to support the evaluation of accessible web?

Research Question-2: What are the current engineering assets (tools, technologies, etc.) to support the evaluation of accessible web?

3.1.2 Search string

To select the appropriate search strings, we defined a set of keywords according to our research questions concerning the accessibility and website domain. We tested the developed set of keywords in different scientific databases by searching manually and refined it based on the relevancy of the output with the research objective. The finally selected set of keywords represented using Boolean operation is the following:

{(Web engineering) or (Website accessibility) or (Web page accessibility) or (Universal accessibility design) or (Accessibility evaluation) or (Accessibility framework) or (Web accessibility methods and algorithms) or (Accessibility measuring software) or (Current accessibility violations)}.

3.1.3 Database selection

For the most relevant and updated literature identification, database selection is crucial. Several scientific databases are available, so appropriate database selection is critical. Herein considering the opinions of other researchers, we selected seven popular databases that provide quality literature and scientific publications. These databases used advanced search algorithms to extract the most related literature according to the user's interest. Seven databases used in this SLR are Scopus, Web of Science, Science Direct, ACM digital library, Google Scholar, IEEE Xplore, and PubMed.

3.2 Conducting the systematic literature review

This phase aims to describe review activities through the specification of (i) database searching and literature extraction, (ii) inclusion and exclusion implication, and (iii) data extraction and quality assessment. These sub-activities are described in detail in the following subsections. Figure  3 shows the flowchart of the review overview.

Flowchart of the review overview

3.2.1 Database searching and literature extraction

We tested the search strings in seven databases to extract past literature. These databases are accepted by scientific committees for scientific publishing. Most of the literature is open access. These databases have advanced search algorithms and semantic technology to retrieve the appropriate literature according to the search strings.

In total, 152 papers were found in the period from 2010 to November 2021 (Scopus: 30, Web of Science: 28, IEEE Xplore: 8, PubMed: 5, Science directory: 20, ACM digital library: 16 and Google Scholar: 45). Five studies were found from other source and were included in the preliminary screening process. These five papers were found in Research Gate (platform of scientific work) based on the suggestion of digital accessibility expertise (3 papers) and other colleagues’ recommendations (2 papers). These works were not available in the seven databases that we have used in this work. The considered five papers have potential contributions to web accessibility and significant observation that addressed the importance of consideration in this systematic literature review. Figure  4 shows the search result considering the number of papers selected in each database through the search query. However, Scopus, Web of Science, and Google scholar have a wider array of literature than other databases. Among 157 papers, we have selected the most related papers required for this review through inclusion and exclusion criteria (described in the next section).

The number of selected literature per database

3.2.2 Inclusion and exclusion criteria

The extracted literature has been evaluated to include the most relevant studies in this research. We excluded the literature that did not meet the inclusion criteria for the review. The inclusion criteria were the following: written in English, papers published in peer-reviewed journals or conferences (i.e., not books), publication period between 2010 and 2021, and describe accessibility improvement, development, or related to accessibility assessment.

The exclusion process was performed to eliminate papers from this review. The exclusion criteria were the following: duplicate papers, non-English papers, not directly related or irrelevant papers, papers that are not freely accessible, and those that are not research papers such as posters, letters, thesis, and editorials. After applying inclusion and exclusion criteria to 157 papers, the following observation was made: 12 papers were duplicates, 11 papers were not in English, 29 papers were not directly related to our research focus, and 7 papers were not research papers. In total, we excluded 59 papers by primary screening. After eliminating these papers, we conducted the proposed SLR process considering the selected 98 papers (including 6 past literature reviews). The entire literature selection process has been performed through the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) technique. The PRISMA flow diagram of the study selection is shown in Fig.  5 .

Study selection through PRISMA approach

3.2.3 Data extraction and quality analysis

In this study, our research was conducted based on the search results during 10–15 January 2022, returning 157 papers. To identify a high-quality paper, data extraction and quality assessment are essential. Several earlier literature reviews followed this technique for the primary evaluation of the selected studies. Therefore, we followed assessment guidelines to identify quality papers, complete paper reading, and answer our research questions. Table 1 shows the assessment criteria for the evaluation of selected studies.

For each question, we set the score to 0 or 1. For each positive answer, a paper gets a score of 1. If not relevant to the assessment questions, the score is 0. For the Q1 indexed journal, the additional points are + 0.50. Similarly, the extra points for the Q2, Q3, and Q4 indexed journal is + 0.40, + 0.30, and + 0.20, respectively. We incorporated Equation-1 and Equation-2 to calculate the final and normalization score to estimate the quality of each selected paper. After conducting the quality analysis, we consider only those studies that passed at least four quality assessment questions with α  ≥ 0.4 normalized scores. However, among 98 selected studies, six were excluded from this SLR (as shown in Fig.  5 , PRISMA diagram) based on the result of the quality assessment criteria. Table 2 shows the quality assessment result of the qualified 92 papers for this review.

3.3 Reporting the findings

In general, selected papers were related to web development, web accessibility, and information and communications technology (ICT) tools. The statistics of past research showed that existing SLRs focused on a few criteria, but other aspects also need to be considered. However, this study focused on previous SLR results and added new findings from our investigation results that were not highlighted in the earlier SLRs. Earlier SLRs considered accessibility requirements, standards, frequent violations, and improvement suggestions. However, accessible development criteria, evaluation tools development and their engineering methods, and updated validation and testing procedure need to highlight to identify the new research area. According to Durdu and Yerlikaya [ 28 ], before ensuring accessible web, web developers and designers should consider the standard guidelines and the requirements of people with disabilities. Also, Bradbard and Peters [ 29 ] shared the same observation. They highlighted that the majority of developers and designers have no adequate knowledge about accessibility requirements for people with disability and also lack knowledge about accessible web application development. Thus, in recent days, accessibility specialists have suggested checking accessibility criteria during the development and testing process through automatic accessibility testing tools and user and expert testing. Past works introduced various aspects of developing an effective webpage, but recent studies revealed that accessibility issues completely align with user satisfaction or usability. Therefore, the government of different countries and public and private organizations initiated a few guidelines concerning accessibility and usability criteria [ 30 ] that directed a new research area to make the development easier and barrier-free. In the Following, we would like to describe our findings and analysis results of the selected literature in the context of two research questions.

RQ-1: What are the available methods, techniques, processes, and approaches to support the evaluation of accessible web?

To answer the first research question, we analyzed 92 selected studies. The selected papers were classified into seven groups/processes: (i) accessibility requirements (AR), (ii) challenges (C), (iii) improvement directions (ID), (iv) framework design (FD), (v) framework implementations (FI), (vi) testing (T), and (vii) evaluation (E). All these phases are described in detail in the following subsections. Figure  6 presents the seven processes with an accounted number of papers for each process. Furthermore, nineteen studies emphasized two activities as presented in the Venn diagram of Fig.  7 , which is: {2 (AR & E) + 1 (AR & T) + 2 (AR & FI) + 1 (C & ID) + 1 (C & FD) + 4 (ID & T) + 1 (ID & E) + 1 (ID & FD) + 2 (FD & E) + 4 (T & E)}. The Venn diagram represents the number of papers that have multiple focuses instead of a particular focus or objective. In total 19 unique papers have been found that have multiple focuses. Figure  7 shows the number of papers with their associated activities through the blue arrow. For example, considering ‘accessibility requirements,’ 2 papers focused on accessibility requirements and evaluation process, 2 papers focused on accessibility requirements and implementation, and 1 paper focused on accessibility requirements and testing. Figure  7 shows the complete view of the number of papers with their multi-focused area. Moreover, results depict that past research mostly emphasized the technical processes, especially improvement direction, testing, and evaluation.

Percentage of studies considering each process related to web evaluation and accessible web applications

Venn diagram representing the number of studies for certain activities and multiple activities

3.3.1 Accessibility requirements (AR)

This section describes the accessibility and usability requirements with new methods for imposing the accessibility and usability requirements on the current web. Among 92 papers, nine (9) were related to accessibility requirements (representing 9.7% of the total literature) that emphasized ensuring the accessibility guidelines. These studies could be grouped into three main topics of interest, as presented in Table 3 .

In the context of accessibility requirements, Bai [ 31 ] and Henry et al. [ 32 ] described the importance of accessibility and usability criteria in web and mobile software applications. They added that improving web accessibility is essential for users with disabilities and non-disabled users. They indicate a significant gap between the needed strategies and the developed solutions for people with disabilities, including auditory, cognitive, neurological, physical, speech, and visual impairments. Therefore, the requirements of people with disabilities should be acknowledged during development as accessible technology is essential for equal access and interaction in today's digital world. Also, Riley-Huff [ 34 ] pointed out that the first step to developing an accessible website is following the web accessibility guidelines/standards. To ensure higher accessibility standards, a possible way is to improve accessibility and usability [ 35 ]. Thus, automatic accessibility testing tools are essential. To ensure usability, they mentioned a few existing models that are prominent to analyze. Another study by Wu et al. [ 33 ] investigated data visuality (chart type, chart embellishment, and data continuity) for people with intellectual and developmental disabilities. They emphasized that people with intellectual and developmental disabilities perform information processing differently. But the actual scenario is quite challenging that complicates the process of data visuality for these people. Thereby they suggested considering all the potential requirements with disabilities during development to improve data visuality and accessibility.

Sauer et al. [ 36 ] identified three criteria: accessibility, usability, and user experience. These are essential for making the internet platform accessible and convenient for people with and without disabilities. They suggested several methods to ensure accessibility (checklists, cognitive barrier walkthroughs, automatic checking), usability (user testing, observation, questionnaires, interviews, focus groups, heuristic evaluation, cognitive walkthrough, and data logging), and user experience. They suggested that accessibility and usability could be imposed during development to improve the user experience. Vu et al. [ 37 ] addressed that low-quality web designs often lead to user frustration that might cause abandonment of undesirable sites. They highlighted several potential and usable web design and evaluation components/methods to improve website usability, which results in a better user experience.

Furthermore, Almeida and Baranauskas [ 38 ] pointed out that web accessibility requirements for people with disabilities are crucial. The difficulty of understanding accessibility guidelines is the prime cause of inaccessible design and development. They also added that developers and designers are not experts and have limited knowledge of accessibility requirements. Therefore, they proposed an inclusive web-based collaborative tool to evaluate and modify the guidelines according to the universal and accessible design and development guidelines. It helps to represent the accessibility guidelines more skillfully. Furthermore, Gaggi and Pederiva [ 39 ] developed a tool for designers and developers concerning the same issue. They assisted the importance of accessibility measurement with a complete direction about guidelines that need attention during the web design and development phase.

3.3.2 Challenges (C)

This section describes the accessibility challenges that are generally liable for the current inaccessible web platform. Among 92 papers, four (4) studies were related to accessibility challenges (representing 4.3% of the total literature). These investigated studies could be grouped into three main topics of interest, as presented in Table 4 .

Researchers are trying to ensure an accessible web for more than a decade, including digital content, websites, user-machine interface, software, etc. Acosta-Vargas et al. [ 40 ] pointed out that to implement an accessible web, web researchers have found several challenges. They specified that accessible web page development required adequate knowledge that demands financial investment such as manufacturing and maintaining costs, testing costs, and quality assurance costs. These deliberations are crucial to improving the accessibility of the developed system. However, these deliberations rely on the organization's size, capital, opportunities, etc. Thus, ensuring these necessities is comparatively challenging. Inal et al. [ 41 ] showed their effort by conducting a user survey about digital accessibility practices to identify the challenges of creating an accessible system. They invited user experience (UX) professionals to find the most common challenges. The challenges were associated with time constraints, lack of training cost constraints, work overload, not being a requirement for the organization, not being a customer requirement, and people with disabilities or special needs not included as target users. Inal et al. highlighted that such challenges act as barriers to considering accessibility requirements seriously, which is responsible for the current inaccessible web.

Another study by Brajnik and Vigo [ 42 ] addressed some crucial challenges that need to consider for introducing an accessible web. The most pressing ones are validity, reliability, sensitivity, and adequacy of user-tailored metrics. Challenges with validity are associated with different validation systems of metrics. For example, there are no specific/gold standards to produce the output for the validation process. The reliance on tools and their limited coverage, completeness, and correctness are heterogonous issues that arise as challenges during metrics result in validation. The reliability of several evaluation performance metrics (human judgment, automatic evaluation, etc.) depends on the evaluation metric transparency and their reproducible and comparable results. Brajnik and Vigo depict that the actual cause for low reliability is the adopted sampling method to evaluate the pages, such as accessibility violation criteria, identified data, formulae, or methods to compute the final score. Sensitivity and adequacy are related to the meaningfulness and suitability of the generated scores through metrics. User-tailored metrics depend on the user's ability as all users have different needs. Accessibility barriers affect different ability users in various manners. Thus, such aspects addressed by Brajnik and Vigo need to be considered in future research.

Furthermore, Palaskar et al. [ 43 ] revealed that existing automated accessibility testing tools consider around 50% of Web Content Accessibility Guidelines. Though most of the rules are easy to understand, sometimes it is pretty challenging to implement all the natural language rules in an automatic system. They also claimed that some rules are unacceptable for ensuring accessibility, and others are inappropriate, for example, rules for color schemes and image captions accessibility checking. To develop an accessible website, consideration of some specific aspects is insufficient. Sometimes, accessibility checking requires more than the considered rules. Thus, validating the appropriate rules and incorporating all the guidelines is the major challenge for current web-based accessibility research.

3.3.3 Improvement directions (ID)

This section describes directions for future improvement of accessible development. Among 92 papers, ten (10) studies were related to improvement directions (representing 10.8% of the total literature). These investigated studies could be grouped into two main topics of interest, as presented in Table 5 .

In the context of improvement direction, few studies focused on the technological aspects of accessible development. Edelberg and Verhulsdonck [ 44 ] addressed that web developers and associated authorities choose the colors and font based on the choice of organization identity. However, through this process, it is not always possible to address accessibility issues such as inaccessible color and contrast, and fonts, which makes a difference in design and development for people with disabilities such as a person with low vision. They suggested that colors, fonts, and supporting elements should be perceived correctly during the development phase. Development should be encoded according to the content management system (CMS) to enhance the user experience of a wider audience. Brajnik and Vigo [ 42 ] pointed out the significant progress for accessibility metrics in the last decades. However, immaturity is still present in modern development. Thus, future research for further improvements is indicated. Based on their observation, they added a few improvement directions. For example, the implementation should follow Agile, an iterative development model to keep track of accessibility issues. In addition, following the hybrid approach like human judgments through different levels of expertise and users, such as disability type or user, might improve the accessibility of the development. Miesenberger et al. [ 45 ] presented some accessibility challenges related to cognitive disability with associated improvement direction. For instance, individual user-centered and personal services-based design and development should ensure. Accessibility requirements should be tested in development cycles with several testing tools (keyboard/mouse logging, eye tracking, etc.). To better usability design, an advanced development framework or platform for R&D should incorporate, and the development should follow the process model (e.g., Waterfall, Iterative, Spiral, Agile, etc.). In addition, Alismail and Chipidza [ 46 ] recommended following the WCAG 2.0 and 2.1 guidelines to develop accessible websites by addressing potential accessibility issues. Also, they emphasized user testing by involving people with disabilities, integrating assistive technologies during web accessibility evaluation, incorporating accessibility requirements during design, development, and maintenance phases, and arranging training for web developers and designers to spread accessibility awareness.

In the accessible prototype design context, Bhagat and Joshi [ 47 ] presented a few technical recommendations to overcome accessibility challenges. They also mentioned that all the accessibility requirements should be checked and validated by the website's quality assurance (QA) team. Ojha et al. [ 48 ] provided a few guidelines based on their detailed study on improving website accessibility with readability. Readability improvement suggestions are related to website structural components such as hyperlinks and image alt-text. These functions should ensure by incorporating the variable weight-based approach for different elements of web pages. Also, website dynamism should be considered in readability score computation to improve the readability in terms of the accessibility of the website. Furthermore, Morris et al. [ 52 ] emphasized ensuring alt text of visual content for screen reader users. They have articulated design guidelines for the representation of visual content with prototype design requirements, especially for people with vision impairments, to facilitate and improve visual content accessibility.

3.3.4 Framework design (FD)

This section describes several frameworks designed to contribute to the web evaluation process to facilitate web platform accessibility. Among 92 papers, seventeen (17) were related to framework design (representing 18.4% of the total literature). These investigated studies could be grouped into three main topics of interest, as presented in Table 6 .

To contribute to accessible user-centric design, Alahmadi [ 53 ] proposed a state-of-the-art framework for web accessibility evaluation to facilitate accessibility measurement and identify accessibility standards errors. The proposed model ensures user-centered design (UCD) based on usability and accessibility guidelines for deaf, visually impaired, and deaf-blindness people. Kaur and Gupta [ 54 ] proposed a quality index evaluation framework to evaluate website design to ensure the quality of web design and development. Hassouna et al. [ 55 ] addressed some significant issues for users with visual impairment. Concerning the accessibility requirements for users with vision impairment, they designed an accessible web page prototype. Few studies focused on ontology design. For example, Sapna and Mohanty [ 56 ] proposed a large-scale test scenario management process using ontology modeling with the help of Web Ontology Language (OWL) to facilitate the software and web development, and testing process by providing faster and more reliable services. Kourtiche et al. [ 57 ] designed an ontology of user-profiles considering user disability context to understand various user requirements during accessible web development. Another study proposed by Fayzrahmanov et al. [ 58 ] developed a user interface to improve web navigability considering the user requirements with visual impairment.

In the context of web accessibility evaluation, Li et al. [ 59 ] designed an interactive web accessibility evaluation system based on the Chinese government guidelines. This framework incorporates automatic tools and human inspection to make evaluation feasible for large web pages. Alsaeedi [ 60 ] proposed a novel framework for evaluating the performance of two accessibility testing protocols in webpage evaluation. Song et al. [ 61 ] designed a crowdsourcing-based web accessibility evaluation framework to validate against WCAG. It generates the automatic accessibility score of each evaluated webpage according to the weight of each checkpoint. Giovanna et al. [ 62 ] developed an open support accessibility evaluation tool to improve automatic accessible support following accessibility conformance testing (ACT) rules. Sanchez-Gordon and Luján-Mora [ 64 ] proposed an agile environment-based accessibility evaluation framework to improve evaluation results based on automated tools, simulators, and expert and user-based testing. In further evaluation, Song et al. [ 65 ] addressed the complexity of accessibility evaluation methods and the shortage of experts in this field. These aspects make the accessibility evaluation process difficult and reduce their significance. Thus, they proposed a crowdsourcing-based web accessibility evaluation system that uses decision strategies such as the golden set strategy and time-based golden set strategy. Palaskar et al. [ 43 ] claimed that most existing Americans with Disabilities (ADA) tools detect only 50% to 60% of accessibility violations because the rules are not understandable. They developed an API to test websites according to the WCAG 2.0 guidelines and A, AA, and AAA conformance level. Additionally, Acosta-Vargas et al. [ 66 ] designed a heuristic method to enable accessibility measurement of websites to ensure an accessible and inclusive web platform.

Additionally, Won [ 67 ] developed a color tool to understand website color meaning for accessible design practice. The proposed approach can evaluate webpage HTML design prototypes and provide a clear understanding of product-specific colors, cross-cultural color meanings, and color preference. It assists designers in making better color decisions during the design and development phase.

3.3.5 Framework implementations (FI)

This section describes several studies that implemented different approaches to contribute to evaluating an accessible web platform. Among 92 papers, seventeen (17) were related to implementation purposes (representing 18.4% of the total literature). These investigated studies could be grouped into three main topics of interest, as presented in Table 7 .

Concerning web accessibility evaluation, many research studies proposed decision support systems, evaluation tools, algorithms, frameworks, models, and interfaces. Mohamad et al. [ 68 ] developed a decision support system for large-scale compliance assessment against web accessibility recommendations and legislation. This architecture aims to provide scalable, interoperable, and integrated web accessibility assessment in the context of user-centric design to develop accessible web and mobile applications. Li et al. [ 69 ] proposed an EDBA decision support system for website accessibility evaluation at a lower cost. Among other scientific studies, Žuliček et al. [ 70 ] developed an accessibility evaluation tool to evaluate the whole webpage, including subpages, to provide a detailed analysis and simplified code refinement. Oliveira et al. [ 71 ] developed an accessibility assessment tool to analyze the strength and weaknesses of the website following the Web Content Accessibility Guidelines. Rashida et al. [ 72 ] developed an automated web-based tool to identify the quality of academic websites by considering websites' content of information, loading time, and overall performance metrics. Lim et al. [ 73 ] proposed an open-source customized automated accessibility testing tool based on the existing Axe accessibility testing engine to scale up the accessibility testing process. Gaggi and Pederiva [ 39 ] developed an automatic tool to assist designers and developers in understanding development aspects that should be considered during the development process to introduce an accessible website. In addition, Duarte et al. [ 74 ] developed an algorithm to automatically identify the semantic similarity between web content and its textual description in the context of web accessibility evaluation guidelines or rules. Wu et al. [ 75 ] developed a semi-supervised regression algorithm involving manual evaluation (webpage sampling) and automatic accessibility testing to generate the overall evaluation result of the website. Almeida and Baranauskas [ 38 ] developed a framework following universal design (UD) accessibility guidelines to help designers overcome accessibility barriers in a web-based system. Morato et al. [ 76 ] proposed a framework for automatic website accessibility checking in the context of readability through a linguistic characteristics analyzer to identify the best linguistic feature to detect text readability.

In the context of accessibility evaluation for visually impaired people, Michailidou et al. [ 78 ] implemented an open-source web accessibility prediction model to predict and visualize the complexity of web pages in the form of a pixelated heat map. Another work proposed by Bonacin et al. [ 79 ] developed an adaptive interface focusing on the requirements of Color Vision Deficiency (CVD) people considering automatic recoloring facilities to facilitate the interaction of CVD people with the web.

Additionally, for accessible prototype development, Matošević et al. [ 82 ] developed a machine learning algorithms-based expert knowledge system to classify web pages or parts of web pages to improve search engine optimization (SEO) guidelines.

3.3.6 Testing (T)

This section describes the studies associated with the testing purpose for accessibility validation of web platforms. Among 92 papers, thirty (30) were related to accessibility testing (32.6% of the total literature). These investigated studies could be grouped into five main topics of interest, as presented in Table 8 .

In the context of testing, many research studies focused on accessibility testing tools to validate website accessibility considering several disabilities. Few studies tested accessibility issues by incorporating automated accessibility testing tools. Martins et al. [ 83 ] tested eHealth websites using a single accessibility testing tool to identify the accessibility issues. Addressing the effectiveness of multiple automatic testing tools, Padure and Pribeanu [ 84 ] applied six accessibility evaluation tools to evaluate their selected websites. They suggested that a single testing tool is not enough to identify all the accessibility issues of a website. In other studies, Marino and Alfonzo [ 35 ] claimed that automatic tools are inadequate to clarify all the accessibility issues of websites. Thus, further manual observation is required. Therefore, Hassouna et al. [ 85 ] initiated a semi-automated evaluation process utilizing an automatic tool and human observation to evaluate design prototypes of websites. The considered evaluation tool is effective as it identifies problems in the design stage. For example, if it detects any error, it redirects to the design stage to show the problem and repair the design problems without modifying the original code. Also, Bhagat and Joshi [ 47 ] observed that a lack of awareness regarding assistive technologies and global accessibility standards is responsible for less inclusive and less accessible website design and development. Thus, they conducted the experimental procedure following automatic and user testing to help service providers, government divisions, and ministries ensure maximum accessibility of online platforms. Rysavy and Michalak [ 92 ] evaluated the library tools and services in terms of accessibility and usability with open-source tools that emphasized the involvement of blind student workers to validate the resulting transparency.

In contrast, few studies evaluated websites considering several tools and techniques to measure the performance of accessibility, usability, readability, and quality. Akgül [ 93 ] evaluated website accessibility, usability, quality, and readability using several tools and techniques. The author employed online open-source tools for accessibility testing and visual and manual inspection for usability testing considering several design standards and Google search results. For quality performance, Akgül incorporated webpage monitoring software considering download time, page size, and objects per website. Finally, evaluated readability, considering text alignment, webpage language, and all-caps text. Grant et al. [ 97 ] examined web accessibility and user experience using the hidden code optimization technique. They aim to motivate better web development practices and improve the overall holistic user experience. Ajuji et al. [ 96 ] and Kumar et al. [ 98 ] proposed two scientific studies. They depict that though the increasing web interactivity is significantly visible, still people with disabilities are finding it difficult to access. They highlighted that website has non-compliant issues against the W3C guidelines. Thus, to evaluate the websites' conformance to the WCAG, Ajuji et al. implemented an automatic accessibility testing tool to evaluate the websites in terms of Perceivable, Operable, Understandable, and Robust. Kumar et al. considered a simulator to visualize the accessibility issues for different types of disabilities. Burkard et al. [ 99 ] counteracted that the importance and awareness of digital accessibility are often not recognized during web development. Due to the complexity of the guidelines, people are not motivated to follow them. Therefore, automatic accessibility barrier checking, identifying, and fixing is an important issue. They considered several accessibility monitoring systems to validate the websites and compare tools in the context of completeness and correctness. Also, Alshamari [ 51 ] evaluated the accessibility of E-commerce websites through multiple accessibility evaluation tools to generate evaluation reports, locate potential errors, and direct warnings to help in accessible website design and development. Furthermore, Król et al. [ 105 ] evaluated the quality of the websites through automatic testing tools considering website performance, SEO quality, website availability, and mobile friendliness.

In the context of better user experience, Bai [ 31 ] emphasized accessibility and usability observation as website accessibility and usability are highly correlated. Bai choose the most frequently used automatic conformance testing tools and several usability testing models. Another study proposed by Yi [ 106 ] claimed that most websites are not accessible to people with visual impairment, even not readable by the screen reader. This problem happens as websites have too many menus, multiple frames, and a lack of alternative text. Thus, Yi proposed the web accessibility evaluation process using questionnaire-based user testing incorporating people with visual impairment. All the users tested websites using assistive technologies such as screen readers to share their opinions by answering questions about the websites’ accessibility.

3.3.7 Evaluation

This section describes several accessibility evaluation methods and techniques. Among 92 papers, nineteen (19) were related to accessibility evaluation (representing 20.6% of the total literature). These investigated studies could be grouped into three main topics of interest, as presented in Table 9 .

Here, we focus on the studies performed in the context of evaluation purposes. For accessibility evaluation, few studies focused on the questionnaire and expert-based evaluation. Hassouna et al. [ 55 ] and Moreno et al. [ 107 ] argued that the web is less accessible for people with vision impairments. They utilized questionnaire-based evaluation for accessibility prototypes with the participation of people with visual impairment. For descriptive analysis of the questionnaire result, they used statistical techniques to observe the relationship between the questionnaire items and the dependent variables. Another study by Hadadi [ 110 ] stated that designers are not careful about considering the requirements of disabilities such as color blindness. Thus, they overlooked the accessibility criteria to integrate into the design tools. This work evaluated the accessibility of widely used design tools through user feedback. The aim was to increase accessibility awareness and encourage product designers to design and develop an accessible solution. Alcaraz Martínez et al. [ 114 ] addressed that several statistical charts on websites are valuable for representing the information. Unfortunately, charts on websites are not accessible for people with low vision and CVD. Thus, they performed a heuristic accessibility evaluation of statistical charts focusing on the needs of people with low vision and CVD to find the usability problems in user interface design. In another study, Giovanna et al. [ 62 ] conducted a quantitative and qualitative analysis of user feedback regarding task completion time and computing success rate metric. In addition, some existing literature focused on automatic testing validator performance assessment and effeteness. Krawiec and Dudycz [ 108 ] evaluated the performance of automatic accessibility testing validator considering standards, the number of page validation ability, user interface interactivity, software update, free/commercial, etc. This assessment system helps to understand the most effective tool according to the specific requirements. Kous et al. [ 102 ] reinforced that several statistical methods using the quantitative data analysis concept are valuable for validating automatic web accessibility testing results. Grantham et al. [ 109 ] claimed that low literacy and numeracy skills sometimes affect user access and understanding of the website's content. Following accessibility guidelines and incorporating advanced assessment criteria against international legal accessibility requirements should be considered to ensure an accessible web.

Considering readability, Kimmons [ 111 ] claimed that most websites have accessibility issues with content understanding (readability) and structural elements. These issues introduce serious accessibility problems and act as a leading cause of reducing accessibility. Another work is conducted by Sun et al. [ 113 ] to assess e-textbooks’ accessibility. They investigated accessibility considering reading time and accuracy to content-related questions. They evaluated experiment results through composite, average, and weighted average scores to examine user experience and performance. However, Ojha et al. [ 48 ] addressed a wide array of accessibility and readability evaluation metrics for online content based on machine learning and statistical language modeling techniques.

In the context of usability evaluation, Radcliffe et al. [ 112 ] conducted m-Health app evaluation considering accessibility and usability concerns. The evaluation was performed through rapid user-testing and quantifying usability feedback. The user testing result and usability feedback were validated through several standardized evaluation methods for inclusive design requirements specification. Wu et al. [ 33 ] addressed that web designers and developers should focus on usability criteria instead of user experience as ensuring usability improves accessibility. Thus, their paper presents several methods and techniques for usability and accessibility evaluation of web design, such as naturalistic observation, participatory evaluation, web-based methods, prototyping, usability inspections, and usability laboratory testing. Giraud et al. [ 116 ] indicated that filtering redundant and irrelevant information is crucial for people with visual impairments similar to sighted users to improve the accessibility of the web. Therefore, to improve website usability, some specific needs of users with visual impairment are emerging to consider. They conducted experiments with users with vision impairment to determine the accessibility of web content in terms of filtered or not irrelevant and redundant information. Also, cognitive load, performance, and participants' satisfaction were investigated through the dual-task paradigm.

Figure  8 represents the number of papers on each topic of interest according to the seven processes. This figure depicts that the number of proposed approaches for accessibility testing to identify accessibility issues is more frequent than other approaches such as development, implementation, and evaluation. The observation result of research question 1 concludes that the number of the proposed approach for the development and implementation of the accessible web evaluation approach was relatively lower, which addresses a further concern of the web researcher.

Number of studies on each topic of research interest according to the seven processes/phase

RQ-2: What are the current engineering assets (tools, technologies, etc.) to support the evaluation of accessible web?

We analyzed the selected papers and identified several groups of interest considering our seven processes. Table 10 summarizes the 22 groups of topics of interest related to the seven processes, including verities of methods, tools, and techniques to answer our second research question.

Asset description

From the 7 process groups and 22 topics of interest (Table 10 ), we aimed to highlight the main assets related to the engineering aspects to support the technical process we have found in our SLR offered by past researchers. These findings will help developers, web engineers, accessibility researchers, and associated authorities to support the accessible design and development process. The addressed assets are listed and described below.

Assets of accessibility requirements (AR)

(AR 1 .) Assets for the importance of accessibility and usability guidelines: (1)  explanation  of higher accessibility standards in website evaluation [ 31 ]; (2)  explanation  of the importance of accessibility guidelines and user requirements for people with disabilities [ 32 ]; (3) sets of usability requirements  for conventional visualization elements design for cognitive barriers people [ 33 ]; (4)  explanation  of web usability and accessibility requirements of WCAG 2.0 and ISO 9241 standards [ 34 ].

Therefore, in this group, we identified two subgroups of assets: s-group-1: 3 studies for the explanation, and s-group-2: 1 study for requirements.

(AR 2 .) Assets for accessibility, usability, and user experience improvement methods: (1)  methods  to improve accessibility, usability, and user experience [ 36 ]; (2)  methods  to understand user perception to improve usability and user experience [ 37 ].

Therefore, in this group, we identified one subgroup of assets: s-group-1: 2 studies for methods.

(AR 3 .) Assets for accessibility requirements specification: (1) Faware is a framework for accessibility requirements representation and implementation in visualization elements design and development [ 38 ]; (2) WCAG4All is a tool for understanding accessibility requirements following standards guidelines [ 39 ];

Therefore, in this group, we identified two subgroups of assets: s-group-1: 1 study for framework and s-group-2: 1 study for tools.

Assets of challenges (C)

(C 1 .) Assets of limited resource adequacy: (1)  cost  for maintaining, testing, and quality assurance is challenging that depends on organization size, capital, and opportunities [ 40 ]; (2)  opportunities for the training program, learning materials, etc. are not enough for accessibility knowledge improvement [ 56 ]; 3) practical experience and advanced knowledge  of UX professionals from different countries are limited [ 41 ].

Therefore, in this group, we identified three subgroups of assets: s-group-1: 1 study for cost, s-group-2: 1 study for opportunities, and s-group-3: 1 study for experience and knowledge.

(C 2 .) Assets of success criteria validation: (1)  metrics  for accessibility evaluation concerning validity, reliability, sensitivity, and adequacy are challenging to ensure [ 42 ].

Therefore, in this group, one subgroup of assets was found: s-group-1: 1 study for metrics.

(C 3 .) Assets of rules optimization : (1)  guidelines  are not enough or appropriate, even difficult to incorporate in automated systems or web development processes [ 43 ].

Therefore, we found one subgroup of assets in this group: s-group-1: 1 study for guidelines.

Assets of improvement directions (ID)

(ID 1 .) Assets for technological aspects: (1) guidelines for accessible and functional prototype design and development [ 44 ]; (2) directions for accessible development [ 42 ]; (3) directions for cognitive disabilities and their particular accessibility barriers in recent development [ 45 ]; (4) suggestions for development that would be facilitated and tested during the design and development phase [ 46 ].

Therefore, in this group, we identified three subgroups of assets: s-group-1: 1 study for guidelines, s-group-2: 2 studies for directions, and s-group-3: 1 study for suggestions.

(ID 2 .) Assets for accessible prototype design: (1)  directions  for accessible prototype design and development [ 47 ]; (2)  guidelines  for improving website readability by ensuring proper structural components and website dynamism [ 48 ]; (3)  suggestions  for spreading awareness, organizing training, and focusing on the accessible prototype design to make the websites accessible to all, including people with special needs [ 49 ]; (4)  suggestions  for accessible prototype design to ensure advanced multimedia components [ 50 ]; (5)  suggestions  for some potential features that should be taken into consideration during feature development [ 51 ]; (6)  guidelines  for visual content representation and accessible prototype design for screen reader users [ 52 ].

Therefore, in this group, we identified three subgroups of assets: s-group-1: 2 studies for guidelines, s-group-2: 1 study for directions, and s-group-3: 3 studies for suggestions.

Assets of framework design (FD)

(FD 1 .) Assets for accessible user-centric design practice: (1) state-of-the-art  framework for university website accessibility evaluation for students with hearing and visual impairment [ 53 ]; (2)  evaluation  of web page prototype design considering blind user requirement [ 55 ]; (3) OUPIP is a user profile-based ontological  model  for designers and developers to develop applications, and devices considering user’s needs, disability type and dynamic context [ 57 ]; (4) multi-axial serialization  framework  for the users with visual impairment to understand and find the required information in the webpage [ 58 ]; (5)  Ontology  for test management process to provide detailed knowledge about the specific domain and captured requirements for testing [ 56 ]; (6) tool for quantitative measurement by evaluating website HTML code to identify the quality of the website design [ 54 ].

Therefore, in this group, we identified five subgroups of assets: s-group-1: 2 studies for framework, s-group-2: 1 study for evaluation, s-group-3: 1 study for models, s-group-4: 1 study for tool, and s-group-5: 1 study for ontology.

(FD 2 .) Assets for web accessibility evaluation: (1) design a cost-effective crowdsourcing  framework  for web accessibility evaluation considering 25 checkpoints and 5 conformance levels [ 59 ]; (2) proposed a  framework  in order to evaluate the well-known automatic accessibility tools in terms of webpage accessibility through their proposed measurement metrics [ 60 ]; (3) a crowdsourcing  framework  for web accessibility evaluation against web accessibility content guidelines checkpoints [ 61 ]; (4) an open and flexible accessibility testing  tool  to support single and multi-page validation [ 62 ]; (5) WUAM is a framework  for websites usability and accessibility evaluation to improve website performance [ 63 ]; (6) proposed a framework  for web accessibility improvement following ISTQB in agile contexts [ 64 ]; (7) proposed a crowdsourcing  framework  for website accessibility evaluation to identify the accessibility barriers and determine the overall accessibility level [ 65 ]; (8) proposed an API  based website accessibility testing tool following ADA guidelines to identify the potential errors and violations, even without prior knowledge [ 43 ]; (9) proposed a framework  for website accessibility barrier measurement according to several variable magnitude techniques [ 50 ]; 10) proposed a heuristic  method  to determine the level of accessibility of high ranked websites [ 66 ].

Therefore, in this group, we identified four subgroups of assets: s-group-1: 7 studies for framework, s-group-2: 1 study for method, s-group-3: 1 study for tool, and s-group-4:1 study for API.

(FD 3 .) Assets for accessible color design: (1) an accessible color suggestions  tool  for designers to improve their color judgment ability and increase their inspiration for accessible design practice [ 67 ].

Therefore, in this group, we identified one subgroup of assets: s-group-1: 1 study for tools.

Assets of framework implementation (FI)

(FI 1 .) Assets for web accessibility evaluation system: (1) user-centric holistic decision support environment  system  for web and mobile application accessibility evaluation [ 68 ]; (2) a cost-effective task assignment-based decision support  system  for web accessibility evaluation [ 69 ]; (3) a module  for automatically analyzing, identifying and solving the accessibility issues [ 70 ]; (4) an automated website readability assessment  model to improve the accessibility and readability of the website [ 76 ]; (5) ShoppingForAll is a tool  for evaluating and identifying the strength and weaknesses of the website in terms of user satisfaction and accessibility criteria [ 71 ]; (6) an  algorithm  for semantic similarity improvement of website content from the web accessibility perspective [ 74 ]; (7) a  tool  for quality assessment of the university websites by assessing website source code [ 72 ]; (8) FAware is a  tool  to provide accessibility issues and available suggestions [ 38 ]; (9) a semi-supervised  model  to evaluate and predict website accessibility [ 75 ]; (10) an open-source, industry-standard  tool  to addresses the shortcomings of current accessibility testing tools for the local government context [ 73 ]; (11) WCAG4All is a  tool  for consulting web designers and developers about accessibility guidelines [ 39 ]; (12) WAccess is a browser extension open-source accessibility testing  tool  to evaluate websites against WCAG guidelines [ 77 ].

Therefore, in this group, we identified five subgroups of assets: s-group-1: 2 studies for systems, s-group-2: 1 study for modules, s-group-3: 2 studies for models, s-group-4: 6 studies for tools, and s-group-5: 1 study for algorithms.

(FI 2 .) Assets for web accessibility evaluation for visually impaired users: (1) ViCRAM is a tool  to predict the visual complexity of the web pages associated with accessibility issues for people with visually impaired or low vision people [ 78 ]; (2) FAIBOUD is a framework  to facilitate the interaction of CVD people with the web [ 79 ]; (3) proposed an automatic  system  for identifying website drop-down menu widgets [ 80 ].

Therefore, in this group, we identified three subgroups of assets: s-group-1: 1 study for tool, s-group-2: 1 study for framework, and s-group-3: 1 study for the system.

(FI 3 .) Assets for accessible prototype improvements : (1) proposed a method  to improve accessibility issues by modifying faulty code into correct code to make content management system-based websites more accessible [ 81 ]; (2) An expert knowledge  system  to detect web page SEO quality [ 82 ].

Therefore, in this group, we identified two subgroups of assets: s-group-1: 1 study for method and s-group-2: 1 study for the expert system.

Assets of testing (T)

(T 1 .) Assets for automatic detection of accessibility issues : (1) ACCESSWEB is an automated validator  for accessibility evaluation considering different accessibility guidelines [ 83 ]; (2) TAW is an automated validator  for web pages evaluation against the web content standards [ 35 ]; (3) Total Validator is an automated validator  to validate accessibility against standards guidelines [ 86 ]; (4) A semi-automated  process  is to evaluate website design prototypes and repair without modifying the original page code [ 85 ]; (5) AChecker and TAW  automated validators are to validate the accessibility of the website and identify the associated issues that violated accessibility guidelines [ 87 ]; (6) automatic testing by AChecker, Total Validator, WAVE, and HTML/CSS/ARIA  automated validators  for evaluation of higher educational institute websites [ 88 ]; (7) hybrid accessibility testing  process  with AChecker, WAVE, and aXe automatic accessibility testing tools and JAWS and Non-Visual Desktop Access, two open-source screen reader applications [ 47 ]; (8) WAVE is an automated validator  to indicate accessibility issues and related accessibility features [ 89 ]; (9) AChecker, Cynthia Says, Mauve, TAW, Total Validator, and Wave are automated validators to identify the accessibility issues and compare their result to understand the effectiveness of the system [ 84 ]; (10) AChecker, WAVE, and SortSite are automated validators  to identify the shortcoming of websites [ 46 ]; (11) AChecker, Cynthia Says, EIII Checker, MAUVE, SortSite, TAW, Tenon, and WAVE are automated validators  to identify the effectiveness of result considering coverage completeness, correctness, specificity, inter-reliability and intra-reliability, validity, efficiency, and capacity [ 90 ]; (12) multi-tool accessibility assessment through  automated validators  such as AChecker, Cynthia Says, Tenon, WAVE, Mauve, and Hera to perform a comparative analysis of websites to identify the effective testing tool [ 49 ].

Therefore, in this group, we identified two subgroups of assets: s-group-1: 10 studies for the automated validator, and s-group-2: 2 studies for the process.

(T 2 .) Assets for content evaluation for osteoarthritis: (1) SMOG and FOG are two  automated validators  to determine webpage content readability considering informative images and relevant video [ 91 ].

Therefore, we identified one subgroup of assets in this group: s-group-1: 1 study for the automated validator.

(T 3 .) Assets for accessibility evaluation for blind users: (1) WAVE is an online  automated validator  for accessibility issues identification of library tools and services for blind users [ 92 ].

Therefore, this group identified one subgroup of assets: s-group-1: 1 study for the automated validator.

(T 4 .) Assets for accessibility evaluation: (1) propose a hybrid  evaluation  approach for improving user experience [ 97 ]; (2 ) A hybrid  evaluation  process for accessibility, usability, quality and readability testing [ 93 ]; (3) A semi-automated evaluation process incorporating AChecker, Total Validator, WAVE and expert opinion to examine the webpage code [ 94 ]; (4) AChecker is an automated validator  to analyze education cooperative websites to determine its accessibility considering disabilities [ 95 ]; (5) TAW is an automated validator  to validate websites against the conformance of WCAG 2.0 [ 96 ]; (6) A simulator  for visual, hearing and mobility impairment to visualize the accessibility issue associated with the particular disability [ 98 ]; (7) A semi-automated process considering axe Monitor, Pope Tech, Siteimprove, ARC with user feedback to validate websites accessibility [ 99 ]; (8) WAVE is an automated validator  to validate the accessibility of COVID-19 vaccine registration portals [ 100 ]; (9) accessibility evaluation through comparative  analysis  using automatic accessibility testing protocols and statistical observation [ 101 ]; (10) AChecker is an automated validator  to evaluate website accessibility [ 102 ]; (11) AChecker, Cynthia Says, and TAW are automated validators  to validate website e-accessibility [ 103 ]; (12) A comparative analysis using Webaccessibility automated accessibility validator and statistical technique to validate the websites against WCAG 2.1 conformance guidelines [ 104 ]; (13) Google PageSpeed Insights, Blink Audit Tool, Backlink Checker, WAVE and Bulk are automated validator  to assess and evaluate website quality [ 105 ]; (14) Achecker, TAW, Eval Access, MAUVE and FAE are automated validators  to identify the accessibility issues of the selected websites [ 51 ].

Therefore, in this group, we identified four subgroups of assets: s-group-1: 2 studies for evaluation, s-group-2: 7 studies for the automated validator, s-group-3: 1 study for the simulator, s-group-4: 2 studies for analysis, and s-group-5: 2 studies for the process.

(T 5 .) Assets for better user experience: (1) FAE, Nielsen’s10-item metric, and Baker’s six-dimension are automated validators  for accessibility and usability testing for better user experience [ 31 ]; (2) questionnaire-based user  assessment  to identify the accessibility incompatibility with screen reader application [ 106 ].

Therefore, in this group, we identified two subgroups of assets: s-group-1: 1 study for automated validator and s-group-2: 1 study for assessment.

Assets of evaluation (E)

(E 1 .) Assets for accessibility evaluation methods: (1) manual  assessment  through assistive technology with users and experts in this field [ 83 ]; (2) questionnaire-based  assessment  for people with visual impairment through several data analysis techniques [ 85 ]; (3) questionnaire-based  evaluation  for discovering the navigation strategies of low vision people that cause to experience accessibility barriers [ 107 ]; (4) automatic  assessment  system to identify the most effective validator for accessibility testing [ 108 ]; (5) statistical data  analysis  to validate the reliability of the questionnaire result [ 55 ]; (6) quantitative data  analysis  using statistical analysis methods [ 102 ]; (7) manual  assessment  criteria for accessibility assessment of Australian private and governmental websites against DDA standards [ 109 ]; (8) user  evaluation  of Adobe online design platforms tool with the help of mix panel data analysis [ 110 ]; (9) statistical  evaluation  for quality analysis of the websites [ 105 ].

Therefore, in this group, we identified three subgroups of assets: s-group-1: 4 studies for assessment, s-group-2: 3 studies for evaluation, and s-group-3: 2 studies for analysis.

(E 2 .) Assets for readability evaluation tools and techniques : (1)  metrics / tools  for website content readability measurement to make website content universally accessible [ 48 ]; (2) descriptive  evaluation  of university homepage to validate the readability [ 111 ].

Therefore, in this group, we identified two subgroups of assets: s-group-1: 1 study for metrics/tool and s-group-2: 1 study for evaluation.

(E 3 .) Assets for usability evaluation methods: (1) statistical  techniques  for usability testing of m-Health application [ 112 ]; (2) questionnaire-based  evaluation  for user experience testing [ 113 ]; (3) quantitative and qualitative  analysis  considering the user performance, computing task completion time, and correct task completion ratio [ 62 ]; (4) quantitative and qualitative  analysis  with statistical measurements to evaluate user perceptions [ 115 ]; (5) statistical  analysis  to determine the relationship between web accessibility and usability [ 31 ]; (6) user  evaluation  to improve website accessibility and interface usability by reducing the cognitive load of people with blindness [ 116 ]; (7) hybrid  evaluation  process to identify the effectiveness of usability and interface design [ 33 ]; (8) quantitative and qualitative  analysis  to evaluate user perceptions for interactive user interface design [ 114 ].

Therefore, in this group, we identified three subgroups of assets: s-group-1: 1 study for technique, s-group-2: 3 studies for evaluation, and s-group-3: 4 studies for analysis.

Figure  9 shows the graphical representation of assets obtained in this SLR. The observation result of research question 2 (as shown in Fig.  9 ) illustrates that automated validators, tools, and frameworks are the main research assets in the investigated area. It demonstrated that most past researchers and the scientific community contributed to accessibility research using the existing automated validators. Recently researchers focused on developing accessibility testing tools and designing frameworks to contribute to accessibility practice, though the number of developed tools and frameworks is limited. In addition, a small group of researchers has conducted studies on other aspects in the accessibility context.

Identified assets of the research outcome

4 Discussion

4.1 research context’s investigation results.

This section highlights the context we focused on in our investigation to reveal in this SLR. The first context of the discussion is the invested domain of past studies. Figure  10 shows the number of papers in each research area found in this SLR. Most studies focused on education, such as government and higher education institute websites. However, few studies focused on other areas such as libraries, health care, electronic materials (e.g., eBooks, visual charts, etc.), tourism, and E-commerce. Accessibility research has a significant contribution to national and international legislation to develop accessible software or web in different domains. However, more investigation for accessibility measurement should be carried out considering other areas to present accessible systems within a broad scope of future research. Besides, during the COVID-19 pandemic, accessible healthcare websites were significantly valuable and were a crucial requirement for the world community [ 117 ]. However, the observation result depicts that the number of proposed studies focusing on the healthcare domain is not adequate, which is the present research gap in this particular domain. This finding exposes the necessity of devoting continued effort to investigating the healthcare domain in future research.

Number of studies of each area of research considering accessibility domain

Figure  11 shows that according to the investigated platforms, most of the selected studies focused on web systems (75 studies), four (4) studies focused on tools and applications, and six (6) studies presented platform-independent approaches.

Number of investigated studies of each platform

Regarding guidelines, most of the selected studies followed WCAG standards to evaluate and develop the web or software application. Figure  12 depicts that WCAG is the dominant and accepted standard for referencing primary accessibility guidelines for the accessible solution and prototype design or user-centric design issues. WCAG is also extensively used as a referencing guideline in accessibility assessment or testing tool development. However, as WCAG is incorporated widely; a few deliberations are laborious to solve by imposing this standard alone. Thus, a wide variety of supporting resources and other guidelines or standards is crucial help for web developers and designers to improve accessibility issues and overcome the current accessibility limitation.

Number of studies according to the focused guideline

Regarding programming language, the frequently used programming language to implement the proposed methods, tools, and frameworks were JavaScript (object-oriented), Python (high-level programming language), HTML (markup language), CSS/SCSS (style description language), PHP (scripting language), C+ + (case sensitive language), OWL (knowledge representation language) and SWRL (logical inference engine). The most frequently marked engineering tools were Apache and MySQL webserver, Oracle database, JavaScript (React), FontAwesome, Axe, Chrome, and HTML Code Sniffer accessibility evaluation libraries. Frequently applied Application Programming Interfaces (APIs) are Clarifai (for image and video), Indico (for semantic matching), Swoogle, AATT, and REST API for Windows and Linux Operating systems. Most tested websites followed content management systems such as WordPress, Joomla, and Drupal. The tested report represents in extensible markup language (XML), enhanced address recognition logic (EARL), and portable document format (PDF). However, Selenium Web Browser Automation and ChromeDriver Tools with Webdriver and MutationObserver API are effective among other web engineering tools.

Generally, the effectiveness and performance of the web concerning accessibility issues have been assessed through automatic testing (accessibility and usability) and human observation. Figure  13 shows that frequently implemented testing tools are WAVE, AChecker, and TAW. However, earlier studies also addressed other accessibility and usability testing tools such as Mauve, Cynthia Says, Total Validator, aXe Monitor, Tenon, Siteimprove, SortSite, etc. Among several automatic testing tools, some specific tools have been implemented frequently in the past literature. Despite the availability of a wide array of accessibility testing tools (approximately 75 according to W3C), most tools are underrated, and even web designers and developers have no idea about these tools and their effectiveness [ 118 ]. In the investigated works of literature, only three pieces of literature compared multiple automatic accessibility testing tools to evaluate their effectiveness. This limited number of comparative analyses is not sufficient to show the usefulness of the existing automated tools. Thus, it is crucial to devote continued effort to perform further comparative analysis considering the benefits of automatic testing tools in future accessibility research.

Number of studies considering implemented testing tools

Concerning the accessibility and usability evaluation and validation results, SPSS, Microsoft Excel, and STATISTICA were the most used statistical analysis tools. Frequently used statistical standards are standard deviation (SD), Pearson’s correlation analysis, one-way ANOVA, System Usability Scale (SUS), Tierney’s 7-min accessibility assessment and app rating system, z-score calculation, Kolmogorov–Smirnov test, Shapiro–Wilk test, Wilcoxon signed-rank test, arithmetic mean, median, coefficient of variation, minimum and maximum value computation. According to past literature, these statistical techniques are effective in accessibility evaluation and validation practice.

Concerning the publication frequency, the observation result shows that between 2010 and 2021, seven (7) studies were published per year on average. Figure  14 displays that the observed number of published studies was low until 2017. Since then, the number of published works has grown. Between 2020 and 2021, the number of publications has shown tremendous growth. This significant growing number of publications depicts that nowadays, web researchers are concerned about the importance of accessible web and ensuring accessibility of the digital platform.

Number of publications per study year for the SLR

Considering our seven processes, we classified the selected papers into three periods: 2010–2013, 2014–2017, and 2018–2021. As shown in Fig.  15 , the number of publications between 2018 and 2021 was much higher in each of the 7 processes compared to the earlier periods. This increase was greatest in testing. The rise of articles in the implementation, evaluation, and design areas is also remarkable. These statistics indicate that concern about digital accessibility has increased in recent years. Compared with other processes, accessibility requirements, challenges, and improvement directions are underrated topics in accessibility research. In addition, the number of papers for development methods (development and implementation) is also limited. This observation directs the importance of devoting continued efforts to conducting future research concerning accessibility requirements, challenges, improvement directions, and development methods.

Number of publications in seven processes of the SLR considering three time periods

As the prime objective of accessibility research is to ensure online platforms are accessible to people with disabilities, thus, in this SLR study, we classified the past studies according to their focused disability type. Almost one-third of the selected studies did not focus on any group of disabilities (see Fig.  16 ). A prominent number of studies focused on issues with every disability. The number of studies focused on visual impairment is also noticeable. However, compared to these three criteria (AI (area independent), AD (all types of disabilities), and VD (visual impairment)), a few studies considered the cognitive, sensory impairment, and physical disabilities issues. Apart from the invested disability types, it is crucial to show the continued effort for other exceptional cases, such as hearing disabilities, moving disabilities, special children, and autism.

Publications with focused disabilities group

Despite the importance of applications to support during the web development process to ensure accessible application development, studies related to application development for accessibility direction are still limited compared to studies on web accessibility evaluation. This result shows the importance of putting effort into methods, tools, and assets to support the development of accessible web and web applications, considering the engineering feature of this platform.

4.2 Web accessibility in past studies

In our search for past studies, we found seven SLRs addressing web accessibility. Najadat et al. [ 119 ] indicated that research on web accessibility has grown since 2007. However, the development of accessibility evaluation tools, metrics, and standards was addressed poorly by past literature. They showed the most common web metrics regarding design, speed, size, diagnosis tools, and metrics for better provision of services. Following this, an SLR carried out by Muniandy and Sulaiman [ 120 ] depicts that for years, accessible computer application design, including mobile applications, computer applications, and online web applications for visually impaired people, has gained immense popularity. Research conducted by Baldwin and Ching [ 121 ] identified that user-centric web prototype design would be helpful to improve accessibility in upcoming development for people with disabilities.

Addressing these issues, an SLR carried out by Akram and Sulaiman [ 14 ] indicated that many studies published between 2009 to 2017 devoted to automated tools development to validate the technical aspects against the accessibility conformance or guidelines. Despite the importance of automatic accessibility testing tools, the lack of advanced techniques to develop these tools required human observation to interact with people with disabilities with interactive systems. With the same focus, an SLR carried out by Campoverde-Molina et al. [ 15 ] stated that a synthesis study is crucial to determine the web accessibility standards and the evaluation methods. They also indicated that the testing process remains the main focus of the current web research. In another SLR, Campoverde-Molina et al. [ 16 ] added that the majority of the experimented websites have potential accessibility issues that address further investigation and more research in this field.

In our findings, we identified a few studies related to the accessible design pattern of rich internet application (RIA), accessibility guidelines visualization, and user interface designs. Compared with the previous SLR studies proposed by Akram and Sulaiman and Campoverde-Molina et al., our proposed study also identifies the importance and growth of accessibility requirements elicitation. They added that research on accessible development and evaluation techniques, user-centric design, and user requirements with disabilities should consider.

Further, an SLR conducted by Oh et al. [ 122 ] indicated that web accessibility research in the area of web image analysis and web-based gamification or game development has increased. They added that understanding visual information (e.g., images) is a critical challenge for people with low vision. Another SLR proposed by Salvador-Ullauri et al. [ 123 ] depicted that web-based games are helpful for teaching and learning for people with disabilities. Web and game developers and designers are fascinated by implementing accessible features as accessibility guidelines are not limited to a particular domain of people. However, from the comparative analysis of previous SLRs, we can observe that (Table 11 ) most of the past SLR studies have lacked consideration of development and implementation approaches for web evaluation that are necessary to include in our SLR process.

4.3 Observation of research

In the investigated studies of this research, among the considered seven processes, challenges, and accessibility requirements experienced with less literature. The primary reason might be aligned with the current research focus. The majority of the research focused on the development of evaluation and testing methods, though addressing accessibility challenges during web development and enhancing the importance of ensuring accessibility guidelines is also important [ 124 ]. Without demonstrating the challenges that might be raised during the development process and their associated solutions, it is barely possible to ensure accessibility for digital sources (e.g., websites, software, etc.). To improve these issues, more attention should be given to the current research focus to identify the major challenges associated with the development of the accessible solution and demonstrate the accessibility guidelines with its advancements. Besides, the literature for framework design and development/implementation is not significant compared to the other processes (e.g., testing). Also, there was limited investigation for evaluation metrics to evaluate the correlation between experimented results and user (e.g., people with disabilities) perceptions, which introduces an urgent need to investigate accessibility result validation systems. In addition, our SLR result illustrates that most of the research focused on automatic accessibility testing tools to investigate the accessibility of the web platform. The articles found considered automatic accessibility testing tools while largely neglecting engineering asset development. Therefore, our proposed SLR depicts the importance of future research for updated methods, techniques, processes, and approaches to support the ensurement of an accessible web.

However, a positive finding observed in this SLR was the rapid growth of the number of studies in the accessibility context. Improving accessibility means developing accessible applications and solutions to help users with various disabilities. This perspective emphasized that developed systems should focus on user requirements (especially for special needs users) to ensure user-centric design, considering user involvement and global accessibility design guidelines for digital inclusion. To enable accessible development tendencies in companies and governmental organizations, several governments have proposed rules to improve the accessibility of digital services; for instance, the United Kingdom, the European Union, the Chinese government, and other public and private organizations. Despite several new digital content accessibility guidelines, investigating new processes, tools and techniques is a significant challenge that directs the importance of future investigations or state-of-the-art research.

5 Conclusion

A systematic literature review is presented in this paper, considering accessibility in the context of web evaluation processes. In this paper, we attempted to take a small step toward contributing to this research by pointing to a new direction for future goals and considerations.

This study showed automatic accessibility testing and evaluation of the focused area of research in the last decade for ensuring the inclusion of accessible web content. There was a great increase in the number of published works after 2017 compared to the previous years.

In the past, most of the literature focused on visual impairment, and very few papers discussed other disabilities, such as hearing, physical, and cognitive disabilities. In this SLR, we found requirements, challenges, engineering techniques, ontology, frameworks, API, algorithms, and testing tools for different levels of satisfaction associated with disabilities, but especially for visual impairment. Therefore, we identified and reported a research gap regarding other disabilities.

Unfortunately, there are few reference architectures for referring to accessible web design, development, and evaluation processes. For example, a framework for accessibility improvement of people with color vision deficiency [ 79 ], an approach for automatically identifying widgets [ 80 ], and an accessibility testing and refinement tool for the early design phase [ 110 ]. It would be beneficial to develop other reference architecture focusing on other contributing areas to solving three problems: (i) framework for the developer to identify and implement accessibility features to improve the accessibility issues, (ii) easy methods to understand and ensure accessibility requirements concerning every type of disabilities during the development phase, and (iii) updated automatic accessibility testing protocols incorporating the latest WCAG standards rules. To overcome these problems, we can note that developing new methods and tools could be a research topic in the upcoming years.

Considering the accessibility of current web platforms, in general, currently available web resources (websites, web-based games, web/mobile applications, etc.) are not accessible. Recently, the governments of many countries-imposed accessibility-related laws (i.e., WCAG) to ensure accessibility requirements. Furthermore, the methods and tools to solve the accessibility problems have limitations that direct future research concerning the development of engineering approaches.

For current accessibility research, there are many challenges to incorporating updated WCAG. Regarding automatic accessibility testing protocol, several studies focused on the limited number of guidelines and disability requirements. Studies for the design and development of accessibility testing protocols are limited. Thus, automatic accessibility testing protocol development concerning different disabilities and elderly user requirements could be a research area in the upcoming years.

Finally, consideration of several methodologies and open-source developments for ensuring accessibility is significantly important. Recently, several researchers and companies have been developing web-based solutions by adopting accessibility requirements. They develop open-source software that has an essential role for end-users and corporations. Accessibility is a crucial technological aspect of developing a new solution for any domain.

Abbreviations

Accessibility conformance testing

Americans with disabilities

Application programming interface

Cascading style sheets

Color vision deficiency

Disabilities discrimination act

Enhanced address recognition logic

Evaluator-decision-based assignment

Hypertext markup language

Information and communications technology

International software testing qualifications board

Web ontology language

Portable document format

Preferred reporting items for systematic reviews and meta-analyses

Rich internet application

Syntactically awesome style sheet

Search engine optimization

• Systematic literature review

Semantic web rule language

User-centric design

Universal design

Web accessibility initiative

Web content accessibility guidelines

Extensible markup language

Nagano, Y., Suginome, K., Yoshimoto, K., Tsuchiya, Y.: Activities for improving web accessibility. Fujitsu Sci. Tech. J. 45 (2), 239–246 (2009)

Google Scholar  

Basdekis, I., Klironomos, I., Metaxas, I., Stephanidis, C.: An overview of web accessibility in Greece: a comparative study 2004–2008. Univ. Access Inf. Soc. 9 (2), 185–190 (2010)

Article   Google Scholar  

World Wide Web Consortium.: 2022a: W3C—Web Accessibility Initiative (WAI), Making the Web Accessible. http://www.w3.org/WAI/

World Wide Web Consortium.: 2022b: About w3C, Groups. https://www.w3.org/Consortium/activities

World Wide Web Consortium.: 2022c. Web content accessibility guidelines (WCAG). https://www.w3.org/standards/

World Wide Web Consortium.: 2018. Web content accessibility guidelines (WCAG) 2.1. https://www.w3.org/TR/WCAG22/

Koga, T.: Policy issues regarding electronic government and Web accessibility in Japan (2006)

Wentzel, J., Velleman, E., Geest, T.V.D.: Developing accessibility design guidelines for wearables: Accessibility standards for multimodal wearable devices. In: International Conference on Universal Access in Human-Computer Interaction, pp. 109–119. Springer, Cham (2016)

Reichling, M., Cherfi, S.S.S.: Integrating accessibility as a quality property in web developments. In: IEEE 7th International Conference on Research Challenges in Information Science (RCIS), pp. 1–12). IEEE (2013)

Horton, S., Sloan, D.: Accessibility in practice: a process-driven approach to accessibility. In: Inclusive Designing, pp. 105–115. Springer, Cham (2014)

Moreno, L., Martínez, P.: A review of accessibility requirements in elderly users’ interactions with web applications. In: Proceedings of the 13th International Conference on Interacción Persona-Ordenador, pp. 1–2 (2012)

Estrada-Martínez, F.J., Hilera, J.R., Otón, S., Aguado-Delgado, J.: Semantic web technologies applied to software accessibility evaluation: a systematic literature review. Universal Access in the Information Society, 1–25 (2020)

Chadli, F.E., Gretete, D., Moumen, A.: Digital accessibility: a systematic Literature Review. In: SHS Web of Conferences, Vol. 119. EDP Sciences (2021)

Akram, M., Sulaiman, R.B.: A systematic literature review to determine the web accessibility issues in Saudi Arabian university and government websites for disable people. Int. J. Adv. Comput. Sci. Appl. 8 (6) (2017)

Campoverde-Molina, M., Lujan-Mora, S., Garcia, L.V.: Empirical studies on web accessibility of educational websites: a systematic literature review. IEEE Access 8 , 91676–91700 (2020)

Campoverde-Molina, M., Luján-Mora, S., Valverde, L.: Accessibility of university websites worldwide: a systematic literature review. Universal Access in the Information Society, 1–36 (2021)

Stapic, Z., López, E.G., Cabot, A.G., de Marcos Ortega, L., Strahonja, V.: Performing systematic literature review in software engineering. In: Central European Conference on Information and Intelligent Systems, p. 441. Faculty of Organization and Informatics Varazdin (2012)

Bai, A., Fuglerud, K.S., Skjerve, R.A., Halbach, T.: Categorization and comparison of accessibility testing methods for software development (2018)

Chiang, M.F., Cole, R.G., Gupta, S., Kaiser, G.E., Starren, J.B.: Computer and world wide web accessibility by visually disabled patients: problems and solutions. Surv. Ophthalmol. 50 (4), 394–405 (2005)

Caldwell, R.: Web-accessibility, e-learning, and academic libraries. Int. J. Public Inf. Syst. 2 (1) (2006)

World Wide Web Consortium.: 2022d. Web content accessibility guidelines (WCAG) 3.0. https://www.w3.org/TR/wcag-3.0/

World Wide Web Consortium.: 1999. Web content accessibility guidelines (WCAG) 1.0. https://www.w3.org/TR/WAI-WEBCONTENT/

World Wide Web Consortium.: 2008. Web content accessibility guidelines (WCAG) 1.0. https://www.w3.org/TR/WCAG20/

World Wide Web Consortium.: 2021. Web content accessibility guidelines (WCAG) 2.2. https://www.w3.org/TR/WCAG22/

Dror, A.A., Morozov, N.G., Layous, E., Mizrachi, M., Daoud, A., Eisenbach, N., Sela, E.: United by hope, divided by access: country mapping of COVID-19 information accessibility and its consequences on pandemic eradication. Front. Med. 7 , 1133 (2021)

Kim, N.W., Joyner, S.C., Riegelhuth, A., Kim, Y.: Accessible visualization: design space, opportunities, and challenges. In: Computer Graphics Forum, Vol. 40, No. 3, pp. 173–188 (2021)

Kitchenham, B., Charters, S.: Guidelines for performing systematic literature reviews in software engineering (2007)

Durdu, P.O., Yerlikaya, Z.: The perception of website accessibility: A survey of Turkish software professionals. AJIT-e: Bilişim Teknolojileri Online Dergisi 11 (41), 42–71 (2020)

Bradbard, D.A., Peters, C.: Web accessibility theory and practice: an introduction for university faculty (2010)

Yu, S.Y.: A review of the accessibility of ACT COVID-19 information portals. Technol. Soc. 64 , 101467 (2021)

Bai, Y.: The relationship between Website accessibility and usability: an examination of US county government online portals. Electron. J. e-Gov. 17 (1), 47–62 (2019)

Henry, S.L., Abou-Zahra, S., Brewer, J.: The role of accessibility in a universal web. In: Proceedings of the 11th Web for all Conference, pp. 1–4 (2014)

Wu, K., Petersen, E., Ahmad, T., Burlinson, D., Tanis, S., Szafir, D.A.: Understanding data accessibility for people with intellectual and developmental disabilities. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–16 (2021)

Riley-Huff, D.A.: Web accessibility and universal design. Libr. Technol. Rep. 48 (7), 29–35 (2012)

Marino, S.I., Alfonzo, P.L.: A web accessibility empirical analysis, a case study (2021)

Sauer, J., Sonderegger, A., Schmutz, S.: Usability, user experience and accessibility: towards an integrative model. Ergonomics 63 (10), 1207–1220 (2020)

Vu, K.P.L., Proctor, R.W., Hung, Y.H.: Website Design and Evaluation. Handbook of human factors and ergonomics, pp. 1016–1036 (2021)

Almeida, L.D.A., Baranauskas, M.C.C.: Universal design principles combined with web accessibility guidelines: a case study. In: Proceedings of the IX Symposium on Human Factors in Computing Systems, pp. 169–178 (2010)

Gaggi, O., Pederiva, V.: WCAG4All, a tool for making web accessibility rules accessible. In: 2021 IEEE 18th Annual Consumer Communications & Networking Conference (CCNC), pp. 1–6. IEEE (2021)

Acosta-Vargas, P., Acosta, T., Lujan-Mora, S.: Challenges to assess accessibility in higher education websites: a comparative study of Latin America universities. IEEE Access 6 , 36500–36508 (2018)

Inal, Y., Guribye, F., Rajanen, D., Rajanen, M., Rost, M.: Perspectives and practices of digital accessibility: a survey of user experience professionals in uwait countries. In: Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society, pp. 1–11 (2020)

Brajnik, G., Vigo, M.: Automatic Web Accessibility Metrics. In: Web Accessibility, pp. 505–521. Springer, London (2019)

Palaskar, A., Gondkar, S.R., Gondkar, R.R.: Web Accessibility test approach on developed websites. Available at SSRN 3835104 (2021)

Edelberg, J., Verhulsdonck, G.: Addressing accessibility as advocacy: special needs users in industry web development processes. In: The 39th ACM International Conference on Design of Communication, pp. 343–346 (2021)

Miesenberger, K., Edler, C., Heumader, P., Petz, A.: Tools and applications for cognitive accessibility. In: Web Accessibility, pp. 523–546. Springer, London (2019)

Alismail, S., Chipidza, W.: Accessibility evaluation of COVID-19 vaccine registration websites across the United States. J. Am. Med. Inform. Assoc. 28 (9), 1990–1995 (2021)

Bhagat, S., Joshi, P.: Evaluation of accessibility and accessibility audit methods for e-governance portals. In: Proceedings of the 12th International Conference on Theory and Practice of Electronic Governance, pp. 220–226 (2019)

Ojha, P.K., Ismail, A., Srinivasan, K.K.: Perusal of readability with focus on web content understandability. J. King Saud Univ. Comput. Inf. Sci. 33 (1), 1–10 (2021)

Ismail, A., Kuppusamy, K.S., Nengroo, A.S.: Multi-tool accessibility assessment of government department websites: a case-study with JKGAD. Disabil. Rehabil. Assist. Technol. 13 (6), 504–516 (2018)

Kuppusamy, K.S., Balaji, V.: Evaluating web accessibility of educational institutions websites using a variable magnitude approach. Universal Access in the Information Society, pp. 1–10 (2021)

Alshamari, M.: Accessibility evaluation of Arabic e-commerce web sites using automated tools. J. Softw. Eng. Appl. 9 (9), 439–451 (2016)

Morris, M.R., Johnson, J., Bennett, C.L., Cutrell, E.: Rich representations of visual content for screen reader users. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 1–11 (2018)

Alahmadi, T.: A multi-method evaluation of university website accessibility: foregrounding user-centred design, mining source code and using a quantitative metric. In: Proceedings of the 14th International Web for All Conference, pp. 1–2 (2017)

Kaur, S., Gupta, S.K.: A fuzzy-based framework for evaluation of website design quality index. Int. J. Digit. Libr. 22 (1), 15–47 (2021)

Hassouna, M.S., Sahari, N., Ismail, A.: University website accessibility for totally blind users. J. Inf. Commun. Technol. 16 (1), 63–80 (2017)

Sapna, P.G., Mohanty, H.: An ontology based approach for test scenario management. In: International Conference on Information Intelligence, Systems, Technology and Management, pp. 91–100. Springer, Berlin, Heidelberg (2011)

Kourtiche, A., Mohamed Benslimane, S., Hacene, S.B.: OUPIP: ontology based user profile for impairment person in dynamic situation aware social networks. Int. J. Knowl.-Based Org. (IJKBO) 10 (2), 12–34 (2020)

Fayzrahmanov, R.R., Göbel, M.C., Holzinger, W., Krüpl, B., Baumgartner, R.: A unified ontology-based web page model for improving accessibility. In: Proceedings of the 19th International Conference on World Wide Web, pp. 1087–1088 (2010)

Li, L.C., Bu, J.J., Zhi, Y., Wei, W.A., Can, W.A.: An overview of web accessibility evaluation of government websites in China. In: International Conference on Social Sciences and Development (ICSSD), pp. 2–5 (2016)

Alsaeedi, A.: Comparing web accessibility evaluation tools and evaluating the accessibility of webpages: proposed frameworks. Information 11 (1), 40 (2020)

Song, S., Bu, J., Artmeier, A., Shi, K., Wang, Y., Yu, Z., Wang, C.: Crowdsourcing-based Web accessibility evaluation with golden maximum likelihood inference. In: Proceedings of the ACM on Human-Computer Interaction, 2(CSCW), pp. 1–21 (2018)

Giovanna, B., Manca, M., Paternó, F., Pulina, F.: Flexible automatic support for web accessibility validation. In: Proceedings of the ACM on Human–Computer Interaction, p. 4 (2020)

Zeleke, Y.: Usability and accessibility model for e-government websites in Ethiopia (2020)

Sanchez-Gordon, S., Luján-Mora, S.: A method for accessibility testing of web applications in agile environments. In: Proceedings of the 7th World Congress for Software Quality (WCSQ). En proceso de publicación.(citado en la página 13, 15, 85) (2017)

Song, S., Bu, J., Wang, Y., Yu, Z., Artmeier, A., Dai, L., Wang, C.: Web accessibility evaluation in a crowdsourcing-based system with expertise-based decision strategy. In: Proceedings of the 15th International Web for All Conference, pp. 1–4 (2018)

Acosta-Vargas, P., Salvador-Ullauri, L.A., Luján-Mora, S.: A heuristic method to evaluate web accessibility for users with low vision. IEEE Access 7 , 125634–125648 (2019)

Won, S.: Conveying colour research to design practice: design and evaluation of a web-based colour tool. Color. Res. Appl. 46 (2), 350–361 (2021)

Mohamad, Y., Velasco, C.A., Kaklanis, N., Tzovaras, D., Paternò, F.: A holistic decision support environment for web accessibility. In: International Conference on Computers Helping People with Special Needs, pp. 3–7. Springer, Cham (2018)

Li, L., Wang, C., Song, S., Yu, Z., Zhou, F., Bu, J.: A task assignment strategy for crowdsourcing-based web accessibility evaluation system. In: Proceedings of the 14th International Web for All Conference, pp. 1–4 (2017)

Žuliček, L., Tomić, S., Bosnić, I.: Adapting modularized web applications to web accessibility standards. In: 2021 44th International Convention on Information, Communication and Electronic Technology (MIPRO), pp. 470–475. IEEE (2017)

Oliveira, R.P., Afonso, A.P., Pinto, A.S.: E-commerce website accessibility evaluation: case study in Portugal. Şubat 10 , 2021 (2020)

Rashida, M., Islam, K., Kayes, A.S.M., Hammoudeh, M., Arefin, M.S., Habib, M.A.: Towards developing a framework to analyze the qualities of the university websites. Computers 10 (5), 57 (2021)

Lim, Z.Y., Chua, J.M., Yang, K., Tan, W.S., Chai, Y.: Web accessibility testing for uwaitie government e-services. In: Proceedings of the 17th International Web for All Conference, pp. 1–5 (2020)

Duarte, C., Matos, I., Carriço, L.: Semantic content analysis supporting web accessibility evaluation. In: Proceedings of the 15th International Web for All Conference, pp. 1–4 (2018)

Wu, Y., Yu, Z., Li, L., Wang, W., Bu, J., Zhang, Y., Dai, L.: Using semi-supervised group sparse regression to improve web accessibility evaluation. In: International Conference on Computers Helping People with Special Needs, pp. 52–56. Springer, Cham (2018)

Morato, J., Iglesias, A., Campillo, A., Sanchez-Cuadrado, S.: Automated readability assessment for uwaiti e-government information. J. Inf. Syst. Eng. Manag. 6 (2), em0137 (2021)

Boyalakuntla, K., Sri Manasa Venigalla, A., Chimalakonda, S.: Waccess—a web accessibility tool based on the latest WCAG 2.2 guidelines. arXiv e-prints, arXiv-2107 (2021)

Michailidou, E., Eraslan, S., Yesilada, Y., Harper, S.: Automated prediction of visual complexity of web pages: tools and evaluations. Int. J. Hum. Comput. Stud. 145 , 102523 (2021)

Bonacin, R., Reis, J.C.D., de Araujo, R.J.: An ontology-based framework for improving color vision deficiency accessibility. Universal Access in the Information Society, pp. 1–26 (2021)

Antonelli, H.L., Igawa, R.A., Fortes, R.P.D.M., Rizo, E.H., Watanabe, W.M.: Drop-down menu widget identification using HTML structure changes classification. ACM Trans. Access. Comput. (TACCESS) 11 (2), 1–23 (2018)

Csontos, B., Heckl, I.: Improving accessibility of CMS-based websites using automated methods. Universal Access in the Information Society, pp. 1–15 (2020)

Matošević, G., Dobša, J., Mladenić, D.: Using machine learning for web page classification in search engine optimization. Future Internet 13 (1), 9 (2021)

Martins, J., Gonçalves, R., Branco, F.: A full scope web accessibility evaluation procedure proposal based on Iberian eHealth accessibility compliance. Comput. Hum. Behav. 73 , 676–684 (2017)

Padure, M., Pribeanu, C.: Comparing six free accessibility evaluation tools. Informatica Economica 24 (1), 15–25 (2020)

Hassouna, M.S., Sahari, N., Ismail, A.: Accessibility factors for totally blind users of Palestinian universities websites (2015)

Pribeanu, C., Fogarassy-Neszly, P.: A review of municipal web sites for accessibility: a computer-aided evaluation approach. Stud. Inf. Control 20 (3), 265–272 (2011)

Verkijika, S.F., De Wet, L.: Accessibility of South African university websites. Univ. Access Inf. Soc. 19 (1), 201–210 (2020)

AlMeraj, Z., Boujarwah, F., Alhuwail, D., Qadri, R.: Evaluating the accessibility of higher education institution websites in the State of Kuwait: empirical evidence. Univ. Access Inf. Soc. 20 (1), 121–138 (2021)

Sharma, A.: Web Accessibility of Indian University library website: an evaluation with WAVE website evaluation tool (2021)

Abduganiev, S.G.: Towards automated web accessibility evaluation: a comparative study. Int. J. Inf. Technol. Comput. Sci. 9 (9), 18–44 (2017)

Chapman, L., Brooks, C., Lawson, J., Russell, C., Adams, J.: Accessibility of online self-management support websites for people with osteoarthritis: a text content analysis. Chronic Illn. 15 (1), 27–40 (2019)

Rysavy, M.D., Michalak, R.: Assessing the accessibility of library tools & services when you aren’t an accessibility expert: Part 1. J. Libr. Adm. 60 (1), 71–79 (2020)

Akgül, Y.: Accessibility, usability, quality performance, and readability evaluation of university websites of Turkey: a comparative study of state and private universities. Univ. Access Inf. Soc. 20 (1), 157–170 (2021)

Doush, I., AlMeraj, Z.: Evaluating the accessibility of uwaiti e-government websites. Jordanian J. Comput. Inf. Technol. (JJCIT) 5 (03) (2019)

Baule, S.M.: Evaluating the accessibility of special education cooperative websites for individuals with disabilities. TechTrends 64 (1), 50–56 (2020)

Ajuji, M., Abubakar, A., Adam, Y.A., Emmanuel, D.U.: Evaluation of accessibility of randomly selected websites. ATBU J. Sci. Technol. Educ. 9 (3), 50–57 (2021)

Grant, J., Sumilang, G., Providenti, M., Cho, S., Reichler, D.: Accountability for the hidden codes toward a better user experience: case study of Hrsimple communication design for web accessibility and SEO. In: The 39th ACM International Conference on Design of Communication, pp. 347–350 (2021)

Kumar, S., Shree, D.V.J., Biswas, P.: Comparing ten WCAG tools for accessibility evaluation of websites. Technology and Disability, (Preprint), pp. 1–23 (2021)

Burkard, A., Zimmermann, G., Schwarzer, B.: Monitoring systems for checking websites on accessibility. Front. Comput. Sci. 2 (2021)

Jo, G., Habib, D., Varadaraj, V., Smith, J., Epstein, S., Zhu, J., Swenor, B.K.: COVID-19 Vaccine Website Accessibility Dashboard. Disabil. Health J. 101325 (2022)

Eusébio, C., Teixeira, L., Teixeira, P., Caneiro, M.J., Lemos, D., Silveiro, A.: The State of Web Accessibility for tourists with disabilities: a comparative study between different tourism supply agents. Disabil. Rehabil. Assistive Technol. 1–13 (2021)

Kous, K., Kuhar, S., Pavlinek, M., Heričko, M., Pušnik, M.: Web accessibility investigation of Slovenian municipalities’ websites before and after the adoption of European Standard EN 301 549. Univ. Access Inf. Soc. 20 (3), 595–615 (2021)

Ali, L.: Accessible websites for everyone—a case of UAE universities websites. Int. J. Inf. Educ. Technol. 11 (4) (2021)

Zare, S., Rahmatizadeh, S., Valizadeh-Haghi, S.: Academic medical libraries and accessibility challenges: the conformance of the websites with the WCAG2. 1. DESIDOC J. Lib. Inf. Technol. 41 (2) (2021)

Król, K., Zdonek, D.: The quality of infectious disease hospital websites in Poland in light of the COVID-19 pandemic. Int. J. Environ. Res. Public Health 18 (2), 642 (2021)

Yi, Y.J.: Web accessibility of healthcare Web sites of Korean government and public agencies: a user test for persons with visual impairment. Univ. Access Inf. Soc. 19 (1), 41–56 (2020)

Moreno, L., Valencia, X., Pérez, J.E., Arrue, M.: An exploratory study of web adaptation techniques for people with low vision. Univ. Access Inf. Soc. 20 (2), 223–237 (2021)

Krawiec, Ł., Dudycz, H.: The approach to the assessment of validators for studying web-accessibility. Inf. Syst. Manag. 6 (2017)

Grantham, J., Grantham, E., Powers, D.: Website accessibility: an Australian view. In: Proceedings of the Thirteenth Australasian User Interface Conference-Volume 126, pp. 21–28 (2012)

Hadadi, S.: Adee: bringing accessibility right inside design tools. In: The 23rd International ACM SIGACCESS Conference on Computers and Accessibility, pp. 1–4 (2021)

Kimmons, R.: Open to all? Nationwide evaluation of high-priority web accessibility considerations among higher education websites. J. Comput. High. Educ. 29 (3), 434–450 (2017)

Radcliffe, E., Lippincott, B., Anderson, R., Jones, M.: A pilot evaluation of mHealth app accessibility for three top-rated weight management apps by people with disabilities. Int. J. Environ. Res. Public Health 18 (7), 3669 (2021)

Sun, Y.T., Vu, K.P.L., Strybel, T.Z.: A validation test of an accessibility evaluation method. In International Conference on Applied Human Factors and Ergonomics, pp. 625–633. Springer, Cham 2017

Alcaraz Martínez, R., Turró, M.R., Granollers Saltiveri, T.: Methodology for heuristic evaluation of the accessibility of statistical charts for people with low vision and color vision deficiency. Universal access in the information society, pp. 1–32 (2021)

Cao, S., Loiacono, E.: Perceptions of web accessibility guidelines by student website and app developers. Behav. Inf. Technol. 1–19 (2021)

Giraud, S., Thérouanne, P., Steiner, D.D.: Web accessibility: Filtering redundant and irrelevant information improves website usability for blind users. Int. J. Hum. Comput. Stud. 111 , 23–35 (2018)

Ara, J., Sik-Lanyi, C.: Investigation of COVID-19 vaccine information websites across Europe and Asia using automated accessibility protocols. Int. J. Environ. Res. Public Health 19 (5), 2867 (2022)

Abascal, J., Arrue, M., Valencia, X.: Tools for web accessibility evaluation. In: Yesilada, Y., Harper, S. (eds) Web Accessibility. Human–Computer Interaction Series. Springer, London (2019). https://doi.org/10.1007/978-1-4471-7440-0_26

Najadat, H., Al-Badarneh, A., Alodibat, S.: A review of website evaluation using web diagnostic tools and data envelopment analysis. Bull. Electr. Eng. Inf. 10 (1), 258–265 (2021)

Muniandy, M., Sulaiman, S.: The role of multimedia design elements in computer accessibility for visually impaired: a complete review. J. Theor. Appl. Inf. Technol. 95 (20) (2017)

Baldwin, S.J., Ching, Y.H.: Accessibility in online courses: a review of national and statewide evaluation instruments. TechTrends 65 (5), 731–742 (2021)

Oh, U., Joh, H., Lee, Y.: Image accessibility for screen reader users: a systematic review and a road map. Electronics 10 (8), 953 (2021)

Salvador-Ullauri, L., Acosta-Vargas, P., Luján-Mora, S.: Web-based serious games and accessibility: a systematic literature review. Appl. Sci. 10 (21), 7859 (2020)

Araa, J., Cecilia, S.L.: Artificial intelligence in web accessibility: potentials and possible challenges. In: Proceedings of IAC 2022 in Vienna, p. 173 (2022)

Download references

Open access funding provided by University of Pannonia. The authors declare that this article is their research and was not financially supported.

Author information

Authors and affiliations.

Department of Electrical Engineering and Information Systems, University of Pannonia, Egyetem u. 10, Veszprem, 8200, Hungary

Jinat Ara & Cecilia Sik-Lanyi

Department of Organizational Systems and Adult Health, University of Maryland Baltimore, 655 W. Lombard St #455B, Baltimore, MD, 21201, USA

Arpad Kelemen

You can also search for this author in PubMed   Google Scholar

Contributions

JA and CS-L conceptualized the study. JA undertook the investigation and statistical analyses and managed the data with support from CS-L and AK and wrote the first draft of the paper. All authors critically inputted into the draft and reviewed it and agreed on the final version. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Jinat Ara .

Ethics declarations

Conflict of interest.

The authors declare that there is no conflict of interest.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Ara, J., Sik-Lanyi, C. & Kelemen, A. Accessibility engineering in web evaluation process: a systematic literature review. Univ Access Inf Soc (2023). https://doi.org/10.1007/s10209-023-00967-2

Download citation

Accepted : 10 January 2023

Published : 27 January 2023

DOI : https://doi.org/10.1007/s10209-023-00967-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Universal inclusion
• Digital accessibility
• Web accessibility
• Engineering assets
• Accessibility insights
• Find a journal
• Publish with us
• Track your research

This paper is in the following e-collection/theme issue:

Published on 24.4.2024 in Vol 26 (2024)

The Impact of Video-Based Microinterventions on Attitudes Toward Mental Health and Help Seeking in Youth: Web-Based Randomized Controlled Trial

Authors of this article:

Original Paper

• Diana Lemmer 1, 2 , MSc   ; 
• Markus Moessner 1 , PD, PhD   ; 
• Nicolas Arnaud 3 , PhD   ; 
• Harald Baumeister 4 , Prof Dr   ; 
• Agnes Mutter 4 , MSc   ; 
• Sarah-Lena Klemm 5 , BSc   ; 
• Elisa König 6 , Dipl-Psych   ; 
• Paul Plener 6, 7 , Prof Dr Med   ; 
• Christine Rummel-Kluge 5 , Prof Dr Med   ; 
• Rainer Thomasius 3 , Prof Dr Med   ; 
• Michael Kaess 8, 9 , Prof Dr Med   ; 
• Stephanie Bauer 1, 2, 10 , Prof Dr  

1 Center for Psychotherapy Research, Center for Psychosocial Medicine, University Hospital Heidelberg, Heidelberg, Germany

2 Ruprecht-Karls University Heidelberg, Heidelberg, Germany

3 German Centre for Addiction Research in Childhood and Adolescence, University Medical Centre Hamburg-Eppendorf, Hamburg, Germany

4 Department of Clinical Psychology and Psychotherapy, Ulm University, Ulm, Germany

5 Department of Psychiatry and Psychotherapy, University of Leipzig Medical Center, Leipzig, Germany

6 Department of Child and Adolescent Psychiatry and Psychotherapy, University Hospital Ulm, Ulm, Germany

7 Department of Child and Adolescent Psychiatry, Medical University of Vienna, Vienna, Austria

8 University Hospital of Child and Adolescent Psychiatry and Psychotherapy, University of Bern, Bern, Switzerland

9 Clinic of Child and Adolescent Psychiatry, Center for Psychosocial Medicine, University Hospital Heidelberg, Heidelberg, Germany

10 German Center for Mental Health (DZPG), Partner site Mannheim/Heidelberg/Ulm, Heidelberg, Germany

Corresponding Author:

Stephanie Bauer, Prof Dr

Center for Psychotherapy Research

Center for Psychosocial Medicine

University Hospital Heidelberg

Bergheimer Str. 54

Heidelberg, 69115

Phone: 49 6221 56 7345

Fax:49 6221 56 7350

Email: [email protected]

Background: Mental health (MH) problems in youth are prevalent, burdening, and frequently persistent. Despite the existence of effective treatment, the uptake of professional help is low, particularly due to attitudinal barriers.

Objective: This study evaluated the effectiveness and acceptability of 2 video-based microinterventions aimed at reducing barriers to MH treatment and increasing the likelihood of seeking professional help in young people.

Methods: This study was entirely web based and open access. The interventions addressed 5 MH problems: generalized anxiety disorder, depression, bulimia, nonsuicidal self-injury, and problematic alcohol use. Intervention 1 aimed to destigmatize and improve MH literacy, whereas intervention 2 aimed to induce positive outcome expectancies regarding professional help seeking. Of the 2435 participants who commenced the study, a final sample of 1394 (57.25%) participants aged 14 to 29 years with complete data and sufficient durations of stay on the video pages were randomized in a fully automated manner to 1 of the 5 MH problems and 1 of 3 conditions (control, intervention 1, and intervention 2) in a permuted block design. After the presentation of a video vignette, no further videos were shown to the control group, whereas a second, short intervention video was presented to the intervention 1 and 2 groups. Intervention effects on self-reported potential professional help seeking (primary outcome), stigma, and attitudes toward help seeking were examined using analyses of covariance across and within the 5 MH problems. Furthermore, we assessed video acceptability.

Results: No significant group effects on potential professional help seeking were found in the total sample ( F 2,1385 =0.99; P =.37). However, the groups differed significantly with regard to stigma outcomes and the likelihood of seeking informal help ( F 2,1385 =3.75; P =.02). Furthermore, separate analyses indicated substantial differences in intervention effects among the 5 MH problems.

Conclusions: Interventions to promote help seeking for MH problems may require disorder-specific approaches. The study results can inform future research and public health campaigns addressing adolescents and young adults.

Trial Registration: German Clinical Trials Register DRKS00023110; https://drks.de/search/de/trial/DRKS00023110

Introduction

Mental health (MH) problems in youth are prevalent and pose severe health-related, social, and financial burdens on individuals [ 1 - 5 ] and societies [ 6 , 7 ]. Approximately half of all mental disorders first manifest before the age of 18 years [ 8 ], and MH problems in youth often persist and aggravate over the life span [ 9 - 14 ]. Therefore, the need for effective prevention and intervention programs targeting young people is an important public health goal. However, while effective MH services exist, most youth with MH problems do not seek professional help. Low uptake has been reported for various conventional [ 15 - 18 ] as well as digital MH services [ 19 - 22 ]. The burden of mental illness can only be alleviated at the population level if a substantial proportion of the population uses the available services [ 23 , 24 ]. Otherwise, the public health impact of MH services remains limited. Thus, increasing the reach of MH services (ie, fostering the uptake and use of professional help) is vital for the improvement of youth MH at the population level.

To facilitate service use, specific barriers to help seeking need to be addressed. Previous research has indicated that attitudinal factors pose larger impediments to help seeking than structural factors (eg, treatment costs and inconvenient scheduling) [ 25 , 26 ]. Specifically, self-reliance, a low perceived need for help [ 25 - 27 ], low treatment expectations [ 28 ], stigma [ 26 , 27 , 29 , 30 ], and poor MH literacy [ 27 , 31 , 32 ] have been identified as major contributors to the lack of professional help seeking.

Different approaches to facilitate help seeking and promote positive attitudes toward MH issues and help seeking in youth have been evaluated in previous research, including face-to-face and digital interventions. In a systematic mapping review, 84% (106/126) of the studies focused on school-based interventions, whereas only 10 (8%) articles covered internet-based approaches to improve MH literacy, MH-related attitudes, stigma, and help-seeking behavior in adolescents [ 33 ]. The internet-based interventions included both minimal, single-session interventions [ 34 , 35 ] and multisession approaches intended to be used over several weeks [ 36 , 37 ], with different outcome measures. A total of 4 studies focused on MH more broadly, whereas 6 studies investigated interventions for specific MH problems (depression: n=5; eating disorders: n=1). Keeping the limited number of studies in this area of research in mind, the results nevertheless point to the potential of internet-based interventions with respect to reduced stigma (2 studies), enhanced help-seeking intentions (2 studies), and improved help-seeking behaviors (1 study).

Clearly, there is a need for more research in this area, particularly with respect to digital brief and microinterventions (ie, highly focused in-the-moment interventions with a narrower scope and time frame than standard interventions [ 38 ]), which allow for a flexible, easily accessible, scalable, and efficient delivery of MH content. Initial research on such brief and microinterventions with psychoeducational and destigmatizing components has shown promising results. For instance, a brief acceptance-facilitating intervention that included a text-based personalized psychoeducation component had a small but significant effect on the intention to use MH services in German university students [ 39 ]. More recently, randomized controlled trials (RCTs) in young adults, university students, and adolescents with short video interventions demonstrated effects with regard to public stigma toward schizophrenia [ 40 , 41 ] and depression [ 42 , 43 ], as well as help-seeking intentions [ 42 ] and attitudes [ 43 ]. Furthermore, an Australian pilot study with international students found that a brief, web-based MH literacy intervention alleviated MH stigma. However, it had no significant effect on help-seeking intentions or MH literacy [ 44 ].

Another component of previous help seeking–facilitating strategies has been storytelling. A pilot study on a video-based intervention indicated that storytelling was well accepted and perceived as engaging [ 45 ]. In addition, an RCT evaluated internet-based storytelling programs with varying interactivity and stigma-related content. Significant reductions in MH stigma and microaggression toward individuals with MH problems were observed [ 46 ].

Concerning the theoretical foundation of interventions, few studies have investigated help seeking–promoting strategies that were explicitly based on the premises of health behavior models. Logsdon et al [ 47 ] evaluated an internet-based depression intervention for adolescent mothers, which was conceptualized according to the theory of planned behavior. The intervention led to significant improvements in help-seeking attitudes, intentions, and behavior. Another well-established and yet more recent health behavior model, which incorporates elements of previously developed approaches, is the Health Action Process Approach (HAPA) [ 48 ]. It encompasses a stage theoretical perspective on health behavior and includes a motivational, intention-forming phase as well as a volitional phase, where planning and behavior maintenance occur. In both the HAPA model and the updated version of the theory of planned behavior, namely, the reasoned action approach, outcome expectancies (or instrumental attitudes) play a crucial role in the formation of intentions, and intentions significantly predict actual behavior [ 49 , 50 ]. The results of previous research on a trauma recovery internet intervention support the use of the HAPA model for the prediction of e-MH engagement. Specifically, outcome expectations significantly predicted the intention to use the intervention (β=.36) [ 51 ]. Skepticism about treatment effectiveness has further been identified as a predictor for not using MH services in another study with university students [ 28 ].

Building on the findings of previous research, this study investigates the short-term effectiveness of 2 brief animated video interventions to promote potential professional help seeking in a general sample of adolescents and young adults aged 14 to 29 years using a web-based RCT approach. Both interventions aimed to improve participants’ willingness to seek professional help (ie, psychotherapists, psychiatrists, and counseling services) for 5 MH problems (generalized anxiety disorder [GAD], depression, bulimia, nonsuicidal self-injury [NSSI], and problematic alcohol use). The inclusion of various MH problems allowed for the investigation of potential differential effects. While one intervention followed a destigmatizing and psychoeducational approach, the other intervention aimed to induce positive outcome expectancies in accordance with the HAPA model through storytelling. The interventions were both compared to each other and to a nonintervention control group (CG) where participants were presented with a stand-alone video vignette without an additional intervention video. This approach was chosen due to both contextual (ie, vignette characters were described as experiencing difficulties in several life domains, and thus, additional control videos referring to the vignettes were unfeasible) and practical (ie, the creation of 10 additional videos was not necessary) considerations.

This study had the following objectives:

• To investigate the short-term effectiveness of the 2 interventions in the promotion of potential MH help seeking (professional and informal), whereby self-reported professional help seeking was defined as the primary outcome.
• To investigate the interventions’ effectiveness in the improvement of self-reported attitudes toward MH problems and MH service use (stigmatization and attitudes toward seeking MH services).
• To evaluate the interventions’ self-reported acceptability.

Within the framework of this study, the videos were evaluated as stand-alone interventions. They were not developed to replace existing interventions. However, in case of favorable outcomes, they have the potential to complement existing health care services. Results and procedures are reported in accordance with the Checklist for Reporting Results of Internet E-Surveys [ 52 ] and the CONSORT-EHEALTH (Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and Online Telehealth) [ 53 ]. The study was preregistered at the German Clinical Trials Register on September 23, 2020 (DRKS00023110).

Study Design

This anonymous, fully automated, web-based, parallel-group exploratory RCT compared the effects of intervention 1 (psychoeducational intervention) and intervention 2 (positive consequences of help seeking) against those of the CG (no further videos after the case vignette) with regard to potential help seeking, attitudes toward help seeking, and stigma. The design comprised 15 conditions in total (5 MH problems × 3 interventional conditions). Randomization was stratified by gender and implemented using a permuted block design (block sizes: 15 and 30). Due to anonymous participation and automated randomization, researchers were unable to assign specific conditions to individuals. However, 2 of the authors were able to view the randomization list. The video material was aligned with the participants’ gender to increase identification with the character (ie, participants who identified as woman, female, or nonbinary viewed videos with a female protagonist [Paula], and participants identifying as man or male viewed videos with a male protagonist [Paul]). The study components and conditions as well as the study procedure are shown in Figure 1 .

Recruitment and Sample

Recruitment started in October 2020 and ended in May 2022. Youth aged between 14 and 29 years with sufficient German language skills were eligible for participation. The age of 14 years is widely accepted as appropriate to provide informed consent for medical decisions and participation in studies [ 54 , 55 ]. The upper age limit of 29 years aligns with the definition of emerging adulthood, a separate life stage between adolescence and adulthood [ 56 , 57 ]. Participants were primarily recruited through the web on social media platforms and via mailing lists, web-based marketplaces, and forums for adolescents and young adults (eg, accounts and emails of youth clubs and student associations). As an incentive to complete the study, participants were offered to take part in an optional gift card lottery at the end of the study (100 gift cards of €20 [US $21.58]). We asked participants for a valid email address if they were interested in the lottery and stored email addresses separately from other study data and user IDs to ensure anonymous participation.

We recorded page change time stamps. Participants whose time stamp data indicated that the video or the videos they were assigned to had not been fully viewed (ie, duration of stay<length of the respective videos) were excluded from statistical analyses. Furthermore, only data from participants who completed all questionnaires were included in the final analysis (n=1394; completion rate: 1394/2435, 57.25%). We also excluded 5 cases with duplicate user IDs, which occurred due to a technical error and indicated repeated participation ( Figure 1 ). HTTP cookies were used to assign individual user IDs to participants. For each session, new cookies were generated and used. Therefore, duplicate participation was possible after the completion of each study session and was not registered by the system. In the 5 aforementioned cases, duplicate IDs were mistakenly generated when participants tried to use the “back” button of their web browser and restarted their participation.

This study was conducted in an open access, voluntary web-based setting. A website was established to provide study information and enable participation. The ASMO software (Center for Psychotherapy Research) [ 58 ] was used to implement the RCT. A randomization list with numbers representing the conditions was generated and embedded in our ASMO database [ 58 ] before recruitment. Data were collected at the Center for Psychotherapy Research, Heidelberg. The study’s technical functionality and usability were extensively tested before recruitment by the authors and their colleagues at their respective institutions. Before their participation, the youth received detailed information about the aims, scope, procedures, data processing, and data storage of the study on the website. Participants were informed that they would be randomly assigned to 1 of 5 MH problems and 1 of 3 video versions. They were not informed about the specific health issues or the conditions’ details before participation. As the aim of the conditions was to provide information about a specific MH problem, blinding of participants after assignment to the interventions was not possible. Only participants who provided informed consent through a web-based checkbox were eligible for participation. After study completion, participants were debriefed in writing about the objectives on the study website. The debriefing form also included contact information for formal help services. Study duration amounted to approximately 30 minutes. Participants were first asked to complete sociodemographic and screening questionnaires; were then randomly assigned to 1 of the 15 experimental conditions; and, finally, were presented with the outcome questionnaires. The whole study (including informed consent and gift card lottery pages) comprised 26 pages with 1 to 12 items on each page. Each segment or measure was presented on 1 or 2 separate pages depending on its respective length. Some items were conditional for adaptive questioning (eg, when lifetime NSSI was denied, no further questions about NSSI were presented). Changes to the item responses could only be made while they had not been confirmed through a click on the “next” button, which brought participants to the next page. There was no “back” button.

Sociodemographics and Screening

All measures were self-reported. The sociodemographic form asked participants about their age, gender, migration background, education, whether they knew someone with MH problems, and participants’ previous or current MH service use (actual help seeking). Thereafter, participants’ subjective psychological distress was assessed using several screening instruments.

Anxiety symptoms were measured using the 7-item Generalized Anxiety Disorder Scale (GAD-7) [ 59 ]. Symptom frequency within the previous 2 weeks was indicated on a 4-point response scale. Total scores (potential range 0-21) were used for further analyses. Scores of ≥5 indicate a mild anxiety symptomatology, scores of ≥10 indicate a moderate anxiety symptomatology, and scores of ≥15 indicate a severe anxiety symptomatology [ 59 ].

The 9-item Patient Health Questionnaire (PHQ-9) [ 60 ] was used for depression symptomatology assessment. Frequencies of depression symptoms within the previous 2 weeks were measured on a 4-point scale. Total scores (potential range 0-27) were calculated for further analyses. Total scores of ≥5 were interpreted as mild, scores of ≥10 were interpreted as moderate, scores of ≥15 were interpreted as moderately severe, and scores of ≥20 were interpreted as severe depression symptomatology [ 60 ].

The Weight Concerns Scale (WCS) [ 61 , 62 ] assessed weight and body shape concerns. It consists of 5 items with varying response scale types (4- to 7-point scales). The response categories of each item represent scores between 0 and 100. The mean across all items was used for further analyses. Scores of ≥57 are indicative of a high risk of eating disorders [ 61 ].

Problematic alcohol use during the previous 12 months was measured using the Alcohol Use Disorders Identification Test for Consumption (AUDIT-C) [ 63 , 64 ]. It comprises 3 items with 5-point response scales. Sum scores range between 0 and 12. A score of 0 indicates abstinence, whereas scores between 1 and 3 are interpreted as moderate alcohol consumption. Scores of ≥4 indicate hazardous alcohol consumption [ 63 , 65 ].

A total of 4 items of the Self-Injurious Thoughts and Behaviors Interview [ 66 ] served to assess NSSI. The first item identified whether participants had ever harmed themselves without suicidal intention. If participants reported lifetime NSSI, the 3 subsequent questions were presented. These items measured the frequency of NSSI within the last year, the age at the first occurrence of NSSI, and the age at the last occurrence of NSSI. Item responses were analyzed separately and descriptively.

Experimental Conditions and Materials

The interventional strategies were applied using short animated videos. The videos were created with the Pro+ version of the web-based animation tool Powtoon (Powtoon Limited) [ 67 ]. Each research group involved in this study prepared materials for 1 of the 5 MH problems based on their respective field of expertise. The materials were structured in a similar fashion across MH problems. The main characters, Paul and Paula, were introduced as students aged 16 years in each condition. In total, 30 videos were created: 5 MH problems × 2 main character genders × 3 video types. Participants in the control condition only viewed a vignette, whereas participants in both intervention groups each viewed 1 additional video (either for intervention 1 or intervention 2). A subset of the videos was pretested between July 2020 and September 2020 with a convenience sample of 9 youths (mean age 18.56, SD 3.74 years; range 14-24 years; 3/9, 33% male), who confirmed comprehensibility and overall acceptability.

All participants viewed a case vignette. Each vignette depicted the respective main character, who was affected by 1 of 5 MH problems (GAD, depression, bulimia, NSSI, or problematic alcohol use). The vignettes introduced the characters to the viewers in a third-person perspective and described their challenges in their everyday lives due to their MH conditions (eg, difficult emotions and cognitions, physical symptoms, and social and school-related issues). The accurate diagnostic labels were not presented in the vignettes [ 68 ]. Vignette duration ranged from 2 minutes, 19 seconds to 2 minutes, 47 seconds (mean 2 min, 29 s; SD 11 s). The bulimia vignettes were developed first. They were inspired by the vignettes by Mond et al [ 69 ] and adapted in accordance with International Classification of Diseases, 10th Revision and Diagnostic and Statistical Manual of Mental Disorders, 5th Edition diagnostic criteria, as well as further literature on the symptomatology and psychological strain of bulimia [ 70 ]. The bulimia vignettes then served as a template for the vignettes of the other 4 MH problems.

Intervention 1

Intervention 1 aimed to improve MH literacy and decrease stigmatization through the presentation of psychoeducational information to encourage help seeking. These intervention videos first presented the correct diagnostic label, prevalence rates, and core symptoms of the condition shown in the vignette. Next, 5 destigmatizing and psychoeducational facts about the respective condition were presented (eg, “Bulimia is a serious illness and not a lifestyle”), which were inspired by the work by Bulik [ 71 ]. The videos then presented treatment options, information about potential challenges in professional help seeking, and encouraging statements about the benefits of professional MH support. Intervention 1 video durations ranged from 4 to 5 minutes (mean 4 min, 27 s; SD 21 s). The information provided in these intervention videos was based on epidemiological, etiological, diagnostic, barrier-related, and interventional findings on the respective MH problems (eg, the studies by Bulik [ 71 ], Keski-Rahkonen and Mustelin [ 72 ], and Nagl et al [ 73 ] for bulimia).

Intervention 2

The second strategy (intervention 2) was based on the premises of the HAPA [ 48 ]. Intervention 2 was designed to induce positive outcome expectancies of professional help seeking through the continuation of Paul and Paula’s stories. The videos showed the main characters 1 year after their initial situation as described in the vignettes. Intervention 2 videos first demonstrated the help-seeking process of the main characters in a retrospective fashion. Encouraged by their teachers, friends, or parents, the main characters sought and received professional support from a psychotherapist. The psychotherapist’s gender matched the gender of the main character. The videos showed how the psychotherapist informed the main character about the correct diagnostic label of their condition and shortly portrayed the therapeutic process. The process included initial difficulties of the main character, such as feelings of insecurity about disclosing their experiences to their therapist, which were resolved over time, and the main characters became invested in their psychotherapy. Then, 5 positive consequences of psychotherapy were presented, such as decreased impairment and an improved quality of life. The videos ended with the notion that the main character was still facing occasional difficulties, but substantial improvements in overall well-being and satisfaction with their decision to seek help were emphasized. Intervention 2 video durations ranged from 4 minutes, 1 second to 4 minutes, 29 seconds (mean 4 min, 15 s; SD 14 s). These interventions were designed in accordance with previous literature on the therapeutic process in MH conditions, including treatment expectations, experiences, and consequences [ 74 ].

Outcome Measures

Primary outcome measure.

Our primary outcome was the potential use of professional MH services (ie, the hypothetical likelihood of seeking formal sources of help if participants experienced Paul’s or Paula’s MH problem), measured using a 12-item version of the General Help Seeking Questionnaire (GHSQ) [ 75 ]. The GHSQ measures the willingness of seeking various formal and informal sources of help within the next 4 weeks for an indicated MH problem on a 7-point rating scale (1=“extremely unlikely”; 7=“extremely likely”). The maximum score among the 3 items, which measured potential help seeking with professional MH services (psychotherapists, psychiatrists, and counseling services), was used as our primary outcome. The GHSQ is the most frequently used instrument for help seeking [ 76 ].

Secondary Outcome Measures

GHSQ data on the potential use of informal sources (romantic partner, friend, parent, or other family member) and no intended help seeking (1 item) were used as secondary outcomes. For informal sources of support, the items’ maximum score was used for the analyses.

Attitudes toward help seeking were measured using the Inventory of Attitudes Toward Seeking Mental Health Services (IASMHS) [ 77 ] on a 5-point rating scale. It comprises 24 items. Its 3 dimensions—“psychological openness,” “help-seeking propensity,” and “indifference to stigma”—are represented with 8 items. Subscale scores range from 0 to 32. Higher scores indicate more positive attitudes.

The Universal Stigma Scale (USS) [ 78 ] was used for stigma measurement. It consists of 11 statements in 2 subscales (“blame/personal responsibility”: 5 items; “impairment/distrust”: 6 items). The extent of agreement with these statements is indicated on a 5-point Likert scale. Means were calculated for each of the 2 subscales. Lower scores indicate higher stigmatization.

Transportation (ie, the immersiveness of the stories presented in the videos) was measured using an adapted version of the Transportation Scale–Short Form [ 79 ]. Adjustments were made to suit the medium of the narratives (ie, video material in contrast to written stories). Our adapted version contained 5 items on a 7-point Likert scale.

Video acceptability was measured using a translated and adapted 4-item version of the acceptability and likability scale used by Gaudiano et al [ 45 ]. In total, 3 items measured overall likability, comprehensibility, and interestingness of the videos on a 5-point rating scale.

Statistical Analysis

Sociodemographic, screening, and outcome data were first analyzed descriptively. Intervention effects on potential professional help seeking (primary outcome) and secondary outcomes in the total sample (ie, across all MH problems and across participants with and without actual help seeking as reported in the screening) were analyzed via analyses of covariance (ANCOVAs) at an α level of P <.05. In addition to the intervention group, the models included age as a covariate, the participants’ actual help seeking (fixed effects), and the 5 MH problems (random effects) as control variables. The results of the main ANCOVA in the total sample are presented in the Results section.

Subgroup ANCOVAs were conducted for each of the 5 MH problems separately. In this case, the respective screening scores (GAD-7, PHQ-9, WCS, number of NSSI events during the last year, and AUDIT-C) were included as additional covariates. Subgroup analyses were further conducted for cases with and without actual help seeking in the total sample and within each of the 5 MH issue groups.

In case of significant ( P <.05) and trend ANCOVA effects, pairwise group comparisons were conducted using 2-tailed t tests. All tests were 2-sided with an α level of 5%. Mean differences (MDs) adjusted for covariates are reported in the Results section.

An a priori power analysis was conducted using G*Power (Heinrich-Heine-Universität Düsseldorf) [ 80 ]. Under the assumption of a medium effect size ( f =0.25), a minimum of 240 participants (80 per condition) were needed to test the expected effect within each of the 5 MH problems via ANCOVAs with a significance criterion of α=.05 and a power of 90%. Statistical analyses were performed using R (version 4.3; R Foundation for Statistical Computing) [ 81 ] and SPSS (version 28; IBM Corp) [ 82 ]. R was also used to generate the random allocation sequence. Authors involved in data analysis and interpretation were not blinded with respect to the assigned experimental conditions.

Deviations From the Protocol

In the beginning of recruitment, the upper age limit was raised from 25 years originally to 29 years due to the aforementioned findings of previous research.

Ethical Considerations

Ethics approval was obtained from Ethics Committee I of the Heidelberg Medical Faculty on July 27, 2020 (protocol S378/2020). The procedures were in accordance with the Helsinki Declaration of 1975, as revised in 2000. All participants received information about the study’s aims, scope, procedures, data processing, and data storage on the study website in written form. Furthermore, all participants received contact information if they wished to clarify questions via telephone or email. Only participants who provided their informed consent through a web-based checkbox were eligible for participation. Participants were able to opt out of the study at any time by closing the study website, which they were informed of before their participation. Participants were offered to take part in an optional gift card lottery at the end of the study (100 gift cards of €20 [US $21.58] each). If they were interested in the lottery, they could enter their email address. Email addresses were stored separately from other study data and user IDs to ensure anonymity. All other data were collected and are reported anonymously. Thus, this study does not contain any individual data of identifiable participants.

Sample Characteristics

Figure 1 shows the flow of participants. Of the 2208 participants who were randomized to 1 of the 15 conditions, 472 (21.38%) were excluded because their time spent on the video pages fell below the durations of the videos they were assigned to, indicating that they did not view the entire videos. Of the remaining 1736 participants, 342 (19.7%) were excluded due to incomplete data (ie, they did not complete all the relevant scales that the study entailed [beginning with informed consent up to and including the last acceptability item]). Our final sample consisted of 1394 youths aged 14 to 29 years (mean 20.97, SD 3.67 years). Sociodemographic and screening results are shown in Table 1 .

a MH: mental health.

b GAD-7: 7-item Generalized Anxiety Disorder Scale.

c PHQ-9: 9-item Patient Health Questionnaire.

d WCS: Weight Concerns Scale.

e SITBI-G: German version of the Self-Injurious Thoughts and Behaviors Interview.

f NSSI: nonsuicidal self-injury.

g We excluded 4 cases in “Age of last NSSI – age of first NSSI” due to invalid values (age of first NSSI>age of last NSSI).

h AUDIT-C: Alcohol Use Disorders Identification Test for Consumption.

A total of 79.56% (1109/1394) of the sample identified as woman or girl, and 44.76% (624/1394) were help seekers (ie, they used professional MH services at the time of or before data collection). On average, the youth were moderately anxious (mean GAD-7 score 8.38, SD 5.00) and depressed (mean PHQ-9 score 9.56, SD 6.07). While 22.88% (319/1394) reported abstinence in the AUDIT-C, 30.85% (430/1394) indicated hazardous alcohol consumption. A total of 19.58% (273/1394) were at high risk of developing an eating disorder according to the WCS. One-third (479/1394, 34.36%) of the sample reported a lifetime history of NSSI according to the Self-Injurious Thoughts and Behaviors Interview, with a 12-month prevalence rate of 19.01% (265/1394).

Intervention Effects

The main results are presented in Table 2 .

a Results controlled for help seeking (fixed factor), mental health issue (random factor), and age (covariate).

b CG: control group.

c GHSQ: General Help Seeking Questionnaire.

d Higher scores represent a greater level of agreement.

e Pairwise comparisons were conducted in case of significant or trend analysis of covariance effects. Empty cells indicate that pairwise comparisons were not conducted due to the analysis of covariance results.

f USS: Universal Stigma Scale.

g Higher scores represent more positive attitudes toward mental health issues and help seeking.

h IASMHS: Inventory of Attitudes Toward Seeking Mental Health Services.

i TS-SF: Transportation Scale–Short Form.

Figure 2 summarizes the results of the overall efficacy and the MH issue–specific subgroup analyses graphically. Specific results of the subgroup analyses can be found in Multimedia Appendices 1 , 2 , and 3 .

Primary Outcome: Potential Professional Help Seeking (GHSQ)

On the 7-point scale of the GHSQ, most participants (1046/1394, 75.04%) selected a score of ≥4 (CG: 409/554, 73.8%; intervention 1: 316/410, 77.1%; intervention 2: 321/430, 74.7%). In total, 19.23% (268/1394; CG: 105/554, 19%; intervention 1: 74/410, 18%; intervention 2: 89/430, 20.7%) of participants reported a score of 7 (“extremely likely”), whereas 6.74% (94/1394; CG: 47/554, 8.5%; intervention 1: 24/410, 5.9%; intervention 2: 23/430, 5.3%) responded with a score of 1 (“extremely unlikely”). Across all MH problems, no statistically significant group main effect was found on potential professional help seeking ( F 2,1385 =0.99; P =.37; Table 2 ).

Secondary Outcomes

Potential informal help seeking (ghsq).

For informal sources of support, most participants (1190/1394, 85.37%) selected a score of ≥5 on the 7-point scale (CG: 478/554, 86.3%; intervention 1: 338/410, 82.4%; intervention 2: 374/430, 87%). For 43.69% (609/1394) of the participants, informal help seeking was “extremely likely,” with a selected score of 7 (CG: 244/554, 44%; intervention 1: 161/410, 39.3%; intervention 2: 204/430, 47.4%), whereas a minority of 1% (14/1394; CG: 4/554, 0.7%; intervention 1: 7/410, 1.7%; intervention 2: 3/430, 0.7%) responded with a score of 1 (“extremely unlikely”). In the total sample, significant group differences were found regarding informal help seeking ( F 2,1385 =3.75; P =.02), with intervention 2 showing a significantly higher mean score than intervention 1 (adjusted MD=0.25; P =.007; Table 2 ). In the subsample of help seekers across MH problems, the same pattern was observed ( F 2,616 =3.21; P =.04; adjusted MD=0.37; P =.01; Multimedia Appendix 1 ). A significant group effect was also found for the total sample in the problematic alcohol use conditions ( F 2,273 =3.51; P =.03; Multimedia Appendix 2 ). Both the CG (adjusted MD=0.42; P =.02) and intervention 2 (adjusted MD=0.41; P =.03) had greater mean scores than intervention 1.

No Potential Help Seeking (GHSQ)

With regard to no intention of seeking help with any of the potential sources listed in the GHSQ (“I would not seek help from anyone” item), almost half (674/1394, 48.35%) of participants selected a score of 1 or 2 (1=“extremely unlikely”; CG: 264/554, 47.7%; intervention 1: 185/410, 45.1%; intervention 2: 225/430, 52.3%), whereas 15.42% (215/1394) responded with a score of 6 or 7 (7=“extremely likely”; CG: 89/554, 16.1%; intervention 1: 68/410, 16.6%; intervention 2: 58/430, 13.5%). There were no statistically significant group differences in the total sample ( P =.07; Table 2 ). However, there were trends for group differences in some of the MH issue subgroups ( Multimedia Appendices 2 and 3 and Figure 2 ).

Public Stigma: Blame and Personal Responsibility (USS)

With regard to the USS blame and personal responsibility subscale, statistically significant group differences were found in the total sample ( F 2,1385 =3.25; P =.04; Table 2 ) and in non–help seekers across MH problems ( F 2,762 =3.21; P =.04; Multimedia Appendix 1 ). In the total sample, both intervention 1 and intervention 2 had significantly greater means compared to the CG (intervention 1>CG: adjusted MD=0.084 and P =.03; intervention 2>CG: adjusted MD=0.085 and P =.03). In the subgroup of non–help seekers, there was a significant difference between intervention 2 and the CG (adjusted MD=0.13; P =.02). Further subgroup analyses revealed no additional differences between experimental conditions. It should be noted that blame and personal responsibility data distributions were heavily skewed to the left (total sample: skew=−1.58). As logarithmic, natural logarithm, square root, and reciprocal transformations did not normalize the distributions, we decided to perform ANCOVAs using the untransformed blame data. Therefore, results should be interpreted with caution.

Public Stigma: Impairment and Distrust (USS)

For the USS distrust subscale, ANCOVAs revealed statistically significant group differences in the total sample ( F 2,1385 =8.01; P <.001; Table 2 ) in both help seekers ( F 2,616 =4.39; P =.01) and non–help-seekers across MH problems ( F 2,762 =3.74; P =.02; Multimedia Appendix 1 ). Moreover, statistically significant group differences were found in the total problematic alcohol use subsample ( F 2,273 =4.49; P =.01; Multimedia Appendix 2 ) and its subgroup of non–help seekers ( F 2,144 =4.00; P =.02; Multimedia Appendix 3 ). In the NSSI subgroup of non–help seekers, a significant group main effect was observed ( F 2,160 =4.50; P =.01; Multimedia Appendix 3 ). Across MH problems, both in the total sample (intervention 1>CG: adjusted MD=0.13 and P =.005; intervention 2>CG: adjusted MD=0.17 and P <.001) and the subsample of help seekers (intervention 1>CG: adjusted MD=0.16 and P =.02; intervention 2>CG: adjusted MD=0.17 and P =.01), significantly larger means in both interventions as compared to the CG were observed. Among participants without previous help seeking across MH problems, post hoc comparisons only revealed a statistically significant difference between intervention 2 and the CG (adjusted MD=0.16; P =.007). In the NSSI subgroup of non–help seekers, intervention 2 differed significantly from both the CG (adjusted MD=0.36; P =.005) and intervention 1 (adjusted MD=0.31; P =.02). For problematic alcohol use, in both the total sample and the subsample of non–help seekers, significant post hoc differences between intervention 2 and the CG (MD for the total=0.32 and P =.003; MD for those without previous help seeking=0.42 and P =.006) were found.

Psychological Openness (IASMHS)

No statistically significant group main effect on the IASMHS psychological openness subscale was found in the total sample ( Table 2 ). Significant effects were found in the total depression sample ( F 2,255 =4.59; P =.01; Multimedia Appendix 2 ) and its subgroup of non–help seekers ( F 2,138 =4.20; P =.02; Multimedia Appendix 3 ). In the total depression sample, intervention 1 showed a greater mean in comparison to the CG (adjusted MD=1.38; P =.046) and intervention 2 (adjusted MD=2.24; P =.003). In the subsample of non–help seekers in the depression conditions, intervention 1 was found to have a greater mean than intervention 2 (adjusted MD=2.75; P =.004), but no significant difference was found with the CG (adjusted MD=1.55; P =.09). No significant group main effects were observed in the other subsamples.

Help Seeking Propensity (IASMHS)

In the total sample, no significant group main effect was found for the IASMHS help seeking propensity subscale ( Table 2 ). Subgroup analyses revealed significant differences in the total bulimia sample ( F 2,271 =3.27; P =.04), where both intervention 1 (adjusted MD=1.51; P =.03) and intervention 2 (adjusted MD=1.40; P =.04) showed larger means than the CG ( Multimedia Appendix 2 ). No further group differences were found in the other subsamples.

Indifference to Stigma (IASMHS)

For the IASMHS indifference to stigma subscale, differential group main effects were found in the total sample ( F 2,1385 =3.18; P =.04; Table 2 ), in the subsample of non–help seekers ( F 2,762 =3.74; P =.02; Multimedia Appendix 1 ), in the total ( F 2,288 =3.22; P =.04; Multimedia Appendix 2 ) and non–help-seeking ( F 2,176 =4.48; P =.01; Multimedia Appendix 3 ) GAD samples, in the total bulimia sample ( F 2,271 =3.45; P =.03; Multimedia Appendix 2 ), and in the NSSI subsample of non–help seekers ( F 2,160 =3.23; P =.04; Multimedia Appendix 3 ). Across MH problems, the CG showed a larger mean than intervention 1 in the total sample (adjusted MD=0.97; P =.02), whereas a greater mean score in the CG compared to those of both intervention 1 (adjusted MD=1.20; P= .02) and intervention 2 (adjusted MD=1.15; P =.02) was found in the subsample without previous help seeking. A similar pattern emerged in the total GAD sample and its subsample of non–help seekers, where the CG’s means were significantly larger in comparison to those of intervention 1 (adjusted MD=2.09; P =.02) in the total sample and of both intervention 1 (adjusted MD=2.57; P =.02) and intervention 2 (adjusted MD=2.91; P =.008) among non–help seekers. In the total bulimia sample, intervention 1 had a significantly higher mean than intervention 2 (adjusted MD=2.26; P =.009), whereas both the CG (adjusted MD=2.10; P =.04) and intervention 2 had greater means than intervention 1 (adjusted MD=2.59; P= .02) in the NSSI subsample of non–help seekers.

Video Acceptability and Transportation

In the total sample, most participants (1041/1394, 74.68%) rated the videos with a score of “4” (705/1394, 50.57%) or “5” (336/1394, 24.1%) on the overall likability item. Regarding comprehensibility, 83.93% (1170/1394) rated the videos as “very comprehensible” (“5” on the 5-point scale), whereas 14.13% (197/1394) assigned them a score of “4.” With respect to the videos’ interestingness, the responses were distributed across the 5-point scale as follows: 27.4% (382/1394) of participants gave a rating of “5,” a total of 41.61% (580/1394) gave the videos a rating of “4,” a total of 22.02% (307/1394) assigned them a score of “3,” and 7.32% (102/1394) gave them a rating of “2.” A minority of participants (23/1394, 1.65%) rated the videos with a score of “1” on the interestingness scale.

In the total sample ( Table 2 ), the intervention 1 videos were rated as generally more likable ( F 2,1385 =12.20; P <.001; intervention 1>CG: adjusted MD=0.25 and P <.001; intervention 1>intervention 2: adjusted MD=0.20 and P <.001) and interesting ( F 2,1385 =6.39; P =.002; intervention 1>CG: adjusted MD=0.06 and P =.02; intervention 1>intervention 2: adjusted MD=0.07 and P <.001) in comparison to those of the CG and intervention 2. The groups did not differ significantly in video comprehensibility ( F 2,1385 =2.01; P =.13). Participants felt more “transported” into the videos’ narratives in the CG and intervention 1 as compared to participants in intervention 2 ( F 2,1385 =4.23; P =.02; CG>intervention 2: adjusted MD=0.17 and P =.03; intervention 1>intervention 2: adjusted MD=0.23 and P =.006; Table 2 ). Most subgroup analyses revealed either similar patterns with regard to general likability and interestingness (eg, total help seekers, total GAD sample, and GAD non–help seekers) or no significant differences (eg, GAD help seekers, bulimia help seekers, all depression samples, and all alcohol use samples; Multimedia Appendices 1 - 3 ). In the cases of bulimia (total and non–help-seeking subsamples; Multimedia Appendices 2 and 3 ) and NSSI ( Multimedia Appendix 2 ), different patterns emerged. In the total bulimia sample, the videos of both the CG and intervention 1 scored significantly higher on the interestingness scale than those of intervention 2 ( F 2,271 =4.49; P =.01; CG>intervention 2: adjusted MD=0.33 and P =.02; intervention 1>intervention 2: adjusted MD=0.44 and P =.005). In the total NSSI sample, the videos of both intervention 1 and intervention 2 were rated as significantly more likable than those of the CG ( F 2,277 =10.31; P <.001; intervention 1>CG: adjusted MD=0.51 and P <.001; intervention 2>CG: adjusted MD=0.29 and P =.008).

Principal Findings

This study developed and tested the short-term effectiveness of 2 brief video-based strategies targeted at adolescents and young adults (aged 14 to 29 years) aiming to foster potential professional help seeking (main outcome) and related attitudes for 5 MH problems. In the total sample, we did not find effects of either intervention 1 (psychoeducation) or intervention 2 (positive outcome expectancies) on our primary outcome. However, significant group effects were found with respect to potential informal help seeking, stigma toward others, and indifference to stigma in the total sample. While both intervention groups showed more favorable attitudes than the CG with regard to public stigma, this did not translate to participants’ own indifference to stigma. In this case, the CG showed significantly more positive attitudes in comparison to intervention 1. However, this finding was not apparent in the MH issue–specific subgroup analysis with the exception of GAD. Unintended adverse effects of MH interventions have been reported in previous research [ 83 - 85 ], which underlines the need for thorough evaluations of such interventions before their public dissemination. Accordingly, we would advise against the implementation of our GAD interventions at the current stage and would recommend the development and evaluation of other tailored strategies for this MH problem.

With regard to informal help, participants in intervention 2 showed a greater willingness to approach friends, family members, or romantic partners for help than participants in intervention 1. This might have been due to the interventions’ design as intervention 2 explicitly depicted improvements in social relationships after the main characters in the videos had sought professional support. Overall, the videos were well accepted and rated as quite interesting, with some room for improvement and with the videos of intervention 1 outperforming those of the other 2 conditions. All videos were, on average, rated as very comprehensible, and no significant group differences were observed in this regard. Interestingly, participants felt more transported into the narratives in the CG and intervention 1 as compared to those in intervention 2. As intervention 2 followed a narrative approach, continuing Paul and Paula’s vignette stories, this was surprising. However, as previously stated, intervention 1 was generally more liked and viewed as more interesting in comparison to intervention 2. The animated and fictional third-person approach of intervention 2 seemed to not have sparked as much interest in participants as the facts presented in intervention 1. While we aimed to increase identification with our main characters through the alignment of their genders with those of the participants, intervention 2 might have been insufficient with regard to the perceived “realness” of the story and the characters, which has been identified as crucial for the formation of narrative transportation and, in turn, attitudes and intentions [ 86 ]. The rather optimistic portrayal of the help-seeking process might have contributed to a lack of perceived authenticity in this sample as well. Furthermore, implicit MH statements in the videos’ scenarios could have been more fruitful. For instance, the viewer could have watched directly how Paul and Paula discussed their issues with a psychotherapist rather than having a narrator describe the situation to them. These types of videos have been associated with improved health literacy and more beneficial attitudes toward cervical cancer [ 87 ], and their application to the field of MH would be interesting.

We further observed differential outcomes with respect to the assigned MH problems and participants’ actual help-seeking status. While the CG outperformed either intervention 1 (total) or both interventions (subgroup of non–help seekers) for GAD with regard to stigma indifference and no further outcomes were found for GAD, different patterns emerged in the other MH issue groups. Results were mixed, where both intervention 1 and intervention 2 outperformed the other conditions in some of the outcomes but not in others (eg, in depression), or a clearer tendency toward the superiority of one of the interventions emerged (eg, in NSSI). In summary, our results point toward the usefulness of tailored interventions with regard to MH issue type and previous help-seeking experiences of potential target groups. Our finding that different strategies might work differently for each of the 5 MH problems included in our study is in accordance with those of previous research. For example, Ebneter and Latner [ 78 ] found varying stigmatizing attitudes among different MH problems. The participants in their study blamed a vignette character with an eating disorder more for their condition than a character with depression, whereas the latter was regarded as more impaired. Our finding that a destigmatizing and psychoeducational intervention such as intervention 1 might work better for bulimia fits their recommendation to target stigmatizing attitudes toward specific MH problems [ 78 ]. Similarly, alcohol dependency and self-endangering behaviors were perceived as particularly dangerous in a Swiss vignette study [ 88 ]. The humanizing depiction of our NSSI and problematic alcohol use characters within a framework of close supportive relationships, which improved in quality through psychotherapy, might have been a relevant factor for reduced distrust scores in the intervention 2 condition as compared to the CG and in the case of non–help seekers in the NSSI condition as compared to intervention 1 as well. This approach might be advantageous to reduce public stigma regarding MH problems that are viewed as particularly dangerous. Decisions for one or another interventional strategy may also depend on the specific goal and targeted outcome. While more research is needed, our study provides preliminary evidence for the tailored strategies suggested in Table 3 .

a Check marks (✓) represent recommended use of an intervention; crosses (X) represent advice against the use of an intervention; and question marks (?) represent inconclusive results and, therefore, no clear recommendation. Recommendations are solely based on the results of this study.

b GAD: generalized anxiety disorder.

c No recommended use.

d NSSI: nonsuicidal self-injury.

Limitations

One limitation of this study lies within the sole investigation of effects on hypothetical intentions and attitudes instead of actual help-seeking behavior. While intentions are substantially associated with behavior and provide valuable insights, they do not translate directly to behavior change [ 49 ]. Moreover, only short-term effects were investigated. Previous studies on short video-based interventions have demonstrated sustaining destigmatizing effects for 1 [ 40 ] and 5 months after their delivery [ 43 ]. While potential long-term effects of our interventions in particular remain unknown and might be investigated in the future, the current state of research points toward potentially impactful long-term effects of low-threshold microinterventions. Related to this, we did not investigate dose-response effects. Research on optimal doses (ie, durations, frequencies, and amounts of intervention components) needed for sustainable change through microinterventions and interventions in general is crucial for well-founded recommendations for or against specific interventions [ 89 ].

Just as in other microintervention studies among youth [ 39 ], group differences in our study were small. However, the potential high reach of easily accessible, low-threshold interventions such as the ones evaluated in this study is apparent in the final sample size of 1394. As it depends on both effectiveness and reach, this allows for a comparably high public health impact [ 23 , 24 ]. We also included trend effects in our overview ( Figure 2 ), which should be interpreted with caution. However, these findings might be useful to inform the planning of subsequent research in this field.

Future research should focus on improvements in the effectiveness of microinterventions. One approach could be the investigation of interventional framing. In a Japanese study on depression, loss-framed messages (ie, emphasizing negative consequences of refraining from help seeking) had a greater impact on help-seeking intentions than gain-framed (positive consequences of help seeking) or neutral (eg, prevalence rates) messages as well as unformatted, plain-text messages in middle-aged adults [ 90 ]. Thus, it might be interesting to conduct future studies on the effects of video-based microinterventions with differently framed messaging as the videos in this study emphasized potential gains of help seeking rather than potential losses of help seeking restraint.

Furthermore, our sample showed, on average, a high level of education, very little public stigma, a pronounced willingness to seek help, and a high rate of actual professional help seeking (624/1394, 44.76%), which limits the generalizability of our findings. More than 90% (1285/1394, 92.18%) of our sample knew someone with MH problems. While we were careful not to recruit MH experts, such as university students of medicine and psychology, youth with a personal interest in MH-related topics seemed to have been more inclined to participate. Related to this, we aimed for a community youth sample rather than a clinical sample. The scenarios that our items referred to were hypothetical and did not necessarily reflect participants’ own experiences due to the random assignment to 1 of the 5 MH problems. A similar approach with targeted interventions according to youth’s actual MH status and more individualized elements with regard to gender-related [ 91 ] and cultural [ 92 , 93 ] aspects could be promising in future research. Thus, upcoming studies should strive to align their research objectives more closely with the characteristics and needs of the selected target groups.

Finally, we did not include a comprehension check to assess participants’ understanding of and engagement with the content presented in the videos. While we accounted for the time participants spent on the video pages and only included participants with sufficient durations of stay in the final analyses, they may not have fully comprehended or attended to the video material. The substantial number of excluded participants who completed the study without meeting the time threshold (472/2208, 21.38%) underlines this potential issue. Thus, future studies should address this limitation by including comprehension checks to improve the robustness of the findings.

Conclusions

The low uptake of professional MH services in youth reflects the need for appropriate strategies to facilitate professional help seeking. This study investigated the effectiveness of 2 short video-based strategies targeted at youth (aged 14 to 29 years) on potential professional help seeking and related attitudes for 5 MH problems. While we did not find intervention effects on potential professional help seeking (with the exception of previous help seekers in the depression conditions), differential intervention effects depending on each MH problem and participants’ actual help-seeking status were found in our secondary outcomes, such as public stigma. Our study results can be used to inform the development of new antistigma interventions, which, based on our results, we would recommend tailoring to specific MH problems, target groups, and outcomes. While group differences were small, such low-threshold interventions can be easily disseminated and, therefore, hold potential for a high reach and, thus, a meaningful impact at the population level. More research is needed for more robust and generalizable recommendations.

Acknowledgments

This study was funded by the German Federal Ministry of Education and Research (funding identifier: 01GL1904). The Federal Ministry of Education and Research had no influence on the design of the study and was not involved in data collection, analysis, and interpretation or the writing of the manuscript. For the publication fee, the authors acknowledge financial support from Heidelberg University. The authors thank Sabrina Baldofski, Elisabeth Kohls, Felicitas Mayr, and Maria I. Austermann for their support in the creation of video materials for this study (depression and alcohol abuse videos). The authors thank Lutfi Arikan (University Hospital Heidelberg) for enabling the technical implementation of this study.

Data Availability

Individual participant data underlying the results presented in this publication (text, tables, figures, and appendices) and statistical code will be shared during the period of 3 months to 5 years following publication of the paper with researchers who provide a methodologically sound proposal to the corresponding author.

Authors' Contributions

SB, MM, and MK conceptualized the study. SB led the grant application. SB, MM, MK, and DL contributed to the study design and selection of screening and outcome measures. DL prepared the technical implementation of this study. DL, AM, EK, and SK prepared the video interventions under the supervision of MM, NA, HB, PP, CRK, RT, and SB, who also provided feedback and information with regard to their fields of expertise. MM generated the random allocation sequence. DL was responsible for study recruitment, which was supported by student assistants. DL analyzed the data, wrote the first draft of this manuscript, and created its tables and figures, with SB and MM providing further feedback, guidance, and supervision during each step. All authors provided feedback on the manuscript. All authors read and approved the final manuscript.

Conflicts of Interest

PP is an advisor for Boehringer Ingelheim and has received speaker’s honoraria from InfectoPharm, GSK, Janssen, and Oral B.

Separate analysis of covariance results and pairwise comparisons for participants with and without previous help-seeking experience.

Separate analysis of covariance results and pairwise comparisons for outcomes per mental health problem.

Separate analysis of covariance results and pairwise comparisons for participants with and without previous help-seeking experience per mental health problem.

CONSORT-eHEALTH Checklist V 1.6.1.

• Gibb SJ, Fergusson DM, Horwood LJ. Burden of psychiatric disorder in young adulthood and life outcomes at age 30. Br J Psychiatry. Aug 2010;197(2):122-127. [ CrossRef ] [ Medline ]
• Weitkamp K, Daniels JK, Romer G, Wiegand-Grefe S. Health-related quality of life of children and adolescents with mental disorders. Health Qual Life Outcomes. Jul 31, 2013;11(1):129. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sawyer MG, Whaites L, Rey JM, Hazell PL, Graetz BW, Baghurst P. Health-related quality of life of children and adolescents with mental disorders. J Am Acad Child Adolesc Psychiatry. May 2002;41(5):530-537. [ CrossRef ] [ Medline ]
• Romanowska S, MacQueen G, Goldstein BI, Wang J, Kennedy SH, Bray S, et al. Social and role functioning in youth at risk of serious mental illness. Early Interv Psychiatry. Aug 10, 2020;14(4):463-469. [ CrossRef ] [ Medline ]
• Gardner A, Filia K, Killackey E, Cotton S. The social inclusion of young people with serious mental illness: a narrative review of the literature and suggested future directions. Aust N Z J Psychiatry. Jan 2019;53(1):15-26. [ CrossRef ] [ Medline ]
• Erskine HE, Moffitt TE, Copeland WE, Costello EJ, Ferrari AJ, Patton G, et al. A heavy burden on young minds: the global burden of mental and substance use disorders in children and youth. Psychol Med. May 2015;45(7):1551-1563. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Suhrcke M, Pillas D, Selai C. Economic aspects of mental health in children and adolescents. Social Cohesion for Mental Wellbeing among adolescents. 2008. URL: https:/​/pure.​york.ac.uk/​portal/​en/​publications/​economic-aspects-of-mental-health-in-children-and-adolescents [accessed 2024-04-08]
• Solmi M, Radua J, Olivola M, Croce E, Soardo L, Salazar de Pablo G, et al. Age at onset of mental disorders worldwide: large-scale meta-analysis of 192 epidemiological studies. Mol Psychiatry. Jan 02, 2022;27(1):281-295. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kessler RC, Avenevoli S, Costello EJ, Georgiades K, Green JG, Gruber MJ, et al. Prevalence, persistence, and sociodemographic correlates of DSM-IV disorders in the National Comorbidity Survey Replication Adolescent Supplement. Arch Gen Psychiatry. Apr 01, 2012;69(4):372-380. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hofstra MB, Van Der Ende J, Verhulst FC. Adolescents' self-reported problems as predictors of psychopathology in adulthood: 10-year follow-up study. Br J Psychiatry. Sep 02, 2001;179(3):203-209. [ CrossRef ] [ Medline ]
• Pardini D, White HR, Stouthamer-Loeber M. Early adolescent psychopathology as a predictor of alcohol use disorders by young adulthood. Drug Alcohol Depend. Apr 2007;88 Suppl 1(Suppl 1):S38-S49. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Heijmens Visser JH, van der Ende J, Koot HM, Verhulst FC. Predictors of psychopathology in young adults referred to mental health services in childhood or adolescence. Br J Psychiatry. Jul 02, 2000;177(1):59-65. [ CrossRef ] [ Medline ]
• Reef J, Diamantopoulou S, van Meurs I, Verhulst F, van der Ende J. Child to adult continuities of psychopathology: a 24-year follow-up. Acta Psychiatr Scand. Sep 03, 2009;120(3):230-238. [ CrossRef ] [ Medline ]
• Pearson CM, Miller J, Ackard DM, Loth KA, Wall MM, Haynos AF, et al. Stability and change in patterns of eating disorder symptoms from adolescence to young adulthood. Int J Eat Disord. Jul 15, 2017;50(7):748-757. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hintzpeter B, Metzner F, Pawils S, Bichmann H, Kamtsiuris P, Ravens-Sieberer U, et al. Inanspruchnahme von ärztlichen und psychotherapeutischen Leistungen durch Kinder und Jugendliche mit psychischen Auffälligkeiten. Kindh Entwickl. Oct 2014;23(4):229-238. [ CrossRef ]
• Wang PS, Aguilar-Gaxiola S, Alonso J, Angermeyer MC, Borges G, Bromet EJ, et al. Use of mental health services for anxiety, mood, and substance disorders in 17 countries in the WHO world mental health surveys. Lancet. Sep 08, 2007;370(9590):841-850. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Cotter P, Kaess M, Corcoran P, Parzer P, Brunner R, Keeley H, et al. Help-seeking behaviour following school-based screening for current suicidality among European adolescents. Soc Psychiatry Psychiatr Epidemiol. Jun 6, 2015;50(6):973-982. [ CrossRef ] [ Medline ]
• Vanheusden K, Mulder CL, van der Ende J, van Lenthe FJ, Mackenbach JP, Verhulst FC. Young adults face major barriers to seeking help from mental health services. Patient Educ Couns. Oct 2008;73(1):97-104. [ CrossRef ] [ Medline ]
• Kaltenthaler E, Sutcliffe P, Parry G, Beverley C, Rees A, Ferriter M. The acceptability to patients of computerized cognitive behaviour therapy for depression: a systematic review. Psychol Med. Nov 2008;38(11):1521-1530. [ CrossRef ] [ Medline ]
• Lawrence D, Johnson S, Hafekost J, Boterhoven de Haan K, Sawyer M, Ainley J, et al. The mental health of children and adolescents: report on the second Australian child and adolescent survey of mental health and wellbeing. Government of Australia. 2015. URL: https:/​/www.​health.gov.au/​sites/​default/​files/​documents/​2020/​11/​the-mental-health-of-children-and-adolescents_0.​pdf [accessed 2024-04-08]
• Lillevoll KR, Vangberg HC, Griffiths KM, Waterloo K, Eisemann MR. Uptake and adherence of a self-directed internet-based mental health intervention with tailored e-mail reminders in senior high schools in Norway. BMC Psychiatry. Jan 21, 2014;14(1):14. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Woodford J, Farrand P, Bessant M, Williams C. Recruitment into a guided internet based CBT (iCBT) intervention for depression: lesson learnt from the failure of a prevalence recruitment strategy. Contemp Clin Trials. Sep 2011;32(5):641-648. [ CrossRef ] [ Medline ]
• Glasgow RE, Estabrooks PE. Pragmatic applications of RE-AIM for health care initiatives in community and clinical settings. Prev Chronic Dis. Jan 04, 2018;15:E02. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. Sep 1999;89(9):1322-1327. [ CrossRef ] [ Medline ]
• Andrade LH, Alonso J, Mneimneh Z, Wells JE, Al-Hamzawi A, Borges G, et al. Barriers to mental health treatment: results from the WHO World Mental Health surveys. Psychol Med. Aug 09, 2013;44(6):1303-1317. [ CrossRef ]
• Mojtabai R, Olfson M, Sampson NA, Jin R, Druss B, Wang PS, et al. Barriers to mental health treatment: results from the National Comorbidity Survey replication. Psychol Med. Aug 2011;41(8):1751-1761. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gulliver A, Griffiths KM, Christensen H. Perceived barriers and facilitators to mental health help-seeking in young people: a systematic review. BMC Psychiatry. Dec 30, 2010;10:113. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Eisenberg D, Golberstein E, Gollust SE. Help-seeking and access to mental health care in a university student population. Med Care. Jul 2007;45(7):594-601. [ CrossRef ] [ Medline ]
• Clement S, Schauman O, Graham T, Maggioni F, Evans-Lacko S, Bezborodovs N, et al. What is the impact of mental health-related stigma on help-seeking? a systematic review of quantitative and qualitative studies. Psychol Med. Jan 2015;45(1):11-27. [ CrossRef ] [ Medline ]
• Schnyder N, Panczak R, Groth N, Schultze-Lutter F. Association between mental health-related stigma and active help-seeking: systematic review and meta-analysis. Br J Psychiatry. Apr 2017;210(4):261-268. [ CrossRef ] [ Medline ]
• Furnham A, Swami V. Mental health literacy: a review of what it is and why it matters. Int Perspect Psychol. Oct 2018;7(4):240-257. [ CrossRef ]
• Jorm AF. Mental health literacy: empowering the community to take action for better mental health. Am Psychol. Apr 2012;67(3):231-243. [ CrossRef ] [ Medline ]
• Patafio B, Miller P, Baldwin R, Taylor N, Hyder S. A systematic mapping review of interventions to improve adolescent mental health literacy, attitudes and behaviours. Early Interv Psychiatry. Dec 03, 2021;15(6):1470-1501. [ CrossRef ] [ Medline ]
• Howard KA, Griffiths KM, McKetin R, Ma J. Can a brief biologically-based psychoeducational intervention reduce stigma and increase help-seeking intentions for depression in young people? a randomised controlled trial. J Child Adolesc Ment Health. May 15, 2018;30(1):27-39. [ CrossRef ] [ Medline ]
• Jaber R, Farroukh M, Ismail M, Najda J, Sobh H, Hammad A, et al. Measuring depression and stigma towards depression and mental health treatment among adolescents in an Arab-American community. Int J Cult Ment Health. Sep 24, 2015;8(3):247-254. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bruning Brown J, Winzelberg AJ, Abascal LB, Taylor CB. An evaluation of an internet-delivered eating disorder prevention program for adolescents and their parents. J Adolesc Health. Oct 2004;35(4):290-296. [ CrossRef ] [ Medline ]
• O'Kearney R, Kang K, Christensen H, Griffiths K. A controlled trial of a school-based internet program for reducing depressive symptoms in adolescent girls. Depress Anxiety. 2009;26(1):65-72. [ CrossRef ] [ Medline ]
• Baumel A, Fleming T, Schueller SM. Digital micro interventions for behavioral and mental health gains: core components and conceptualization of digital micro intervention care. J Med Internet Res. Oct 29, 2020;22(10):e20631. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ebert DD, Franke M, Kählke F, Küchler AM, Bruffaerts R, Mortier P, et al. WHO World Mental Health - International College Student collaborators. Increasing intentions to use mental health services among university students. Results of a pilot randomized controlled trial within the World Health Organization's World Mental Health International college student initiative. Int J Methods Psychiatr Res. Jun 2019;28(2):e1754. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Amsalem D, Markowitz JC, Jankowski SE, Yang LH, Valeri L, Lieff SA, et al. Sustained effect of a brief video in reducing public stigma toward individuals with psychosis: a randomized controlled trial of young adults. Am J Psychiatry. Jul 26, 2021;178(7):635-642. [ CrossRef ] [ Medline ]
• Amsalem D, Jankowski SE, Pagdon S, Valeri L, Smith S, Yang LH, et al. "It is hard to be a woman with schizophrenia": randomized controlled trial of a brief video intervention to reduce public stigma in young adults. J Clin Psychiatry. Dec 19, 2022;84(1):22m14534. [ CrossRef ] [ Medline ]
• Amsalem D, Martin A. Reducing depression-related stigma and increasing treatment seeking among adolescents: randomized controlled trial of a brief video intervention. J Child Psychol Psychiatry. Feb 06, 2022;63(2):210-217. [ CrossRef ] [ Medline ]
• Conceição V, Rothes I, Gusmão R. The effects of a video-based randomized controlled trial intervention on depression stigma and help-seeking attitudes in university students. Psychiatry Res. Feb 2022;308:114356. [ CrossRef ] [ Medline ]
• Clough BA, Nazareth SM, Casey LM. Making the grade: a pilot investigation of an e-intervention to increase mental health literacy and help-seeking intentions among international university students. Br J Guid Couns. Oct 03, 2019;48(3):347-359. [ CrossRef ]
• Gaudiano BA, Davis CH, Miller IW, Uebelacker L. Pilot randomized controlled trial of a video self-help intervention for depression based on acceptance and commitment therapy: feasibility and acceptability. Clin Psychol Psychother. May 28, 2020;27(3):396-407. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fong TH, Mak WW. The effects of internet-based storytelling programs (amazing adventure against stigma) in reducing mental illness stigma with mediation by interactivity and stigma content: randomized controlled trial. J Med Internet Res. Aug 12, 2022;24(8):e37973. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Logsdon CM, Myers J, Rushton J, Gregg JL, Josephson AM, Davis DW, et al. Efficacy of an internet-based depression intervention to improve rates of treatment in adolescent mothers. Arch Womens Ment Health. Jun 2018;21(3):273-285. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Schwarzer R, Luszczynska A. How to overcome health-compromising behaviors. Eur Psychol. Jan 2008;13(2):141-151. [ CrossRef ]
• Zhang CQ, Zhang R, Schwarzer R, Hagger MS. A meta-analysis of the health action process approach. Health Psychol. Jul 2019;38(7):623-637. [ CrossRef ] [ Medline ]
• Fishbein M, Ajzen I. Predicting and Changing Behavior: The Reasoned Action Approach. New York, NY. Psychology Press; 2010.
• Yeager CM, Shoji K, Luszczynska A, Benight CC. Engagement with a trauma recovery internet intervention explained with the Health Action Process Approach (HAPA): longitudinal study. JMIR Ment Health. Apr 10, 2018;5(2):e29. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Eysenbach G. Improving the quality of web surveys: the checklist for reporting results of internet E-surveys (CHERRIES). J Med Internet Res. Sep 29, 2004;6(3):e34. [ CrossRef ] [ Medline ]
• Eysenbach G, CONSORT-EHEALTH Group. CONSORT-EHEALTH: improving and standardizing evaluation reports of web-based and mobile health interventions. J Med Internet Res. Dec 31, 2011;13(4):e126. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Santelli JS, Rosenfeld WD, DuRant RH, Dubler N, Morreale M, English A, et al. Guidelines for adolescent health research: a position paper of the society for adolescent medicine. J Adolesc Health. Nov 1995;17(5):270-276. [ CrossRef ] [ Medline ]
• Schachter D, Kleinman I, Harvey W. Informed consent and adolescents. Can J Psychiatry. Aug 01, 2005;50(9):534-540. [ CrossRef ] [ Medline ]
• Arnett JJ. Emerging adulthood: a theory of development from the late teens through the twenties. Am Psychol. 2000;55(5):469-480. [ CrossRef ]
• Mehta CM, Arnett JJ, Palmer CG, Nelson LJ. Established adulthood: a new conception of ages 30 to 45. Am Psychol. May 2020;75(4):431-444. [ CrossRef ] [ Medline ]
• Wilhelm M, Feldhege J, Bauer S, Moessner M. Einsatz internetbasierter verlaufsmessung in der psychotherapieforschung. Psychotherapeut. Oct 12, 2020;65(6):505-511. [ CrossRef ]
• Spitzer RL, Kroenke K, Williams JB, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med. May 22, 2006;166(10):1092-1097. [ CrossRef ] [ Medline ]
• Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. Sep 2001;16(9):606-613. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Killen JD, Taylor CB, Hammer LD, Litt I, Wilson DM, Rich T, et al. An attempt to modify unhealthful eating attitudes and weight regulation practices of young adolescent girls. Int J Eat Disord. May 1993;13(4):369-384. [ CrossRef ] [ Medline ]
• Killen JD, Taylor CB, Hayward C, Wilson DM, Haydel KF, Hammer LD, et al. Pursuit of thinness and onset of eating disorder symptoms in a community sample of adolescent girls: a three-year prospective analysis. Int J Eat Disord. Nov 1994;16(3):227-238. [ CrossRef ] [ Medline ]
• Bush K, Kivlahan DR, McDonell MB, Fihn SD, Bradley KA. The AUDIT alcohol consumption questions (AUDIT-C): an effective brief screening test for problem drinking. Ambulatory care quality improvement project (ACQUIP). Alcohol use disorders identification test. Arch Intern Med. Sep 14, 1998;158(16):1789-1795. [ CrossRef ] [ Medline ]
• Saunders JB, Aasland OG, Babor TF, de la Fuente JR, Grant M. Development of the alcohol use disorders identification test (AUDIT): WHO collaborative project on early detection of persons with harmful alcohol consumption--II. Addiction. Jun 1993;88(6):791-804. [ CrossRef ] [ Medline ]
• Kuitunen-Paul S, Roerecke M. Alcohol Use isorders Identification Test (AUDIT) and mortality risk: a systematic review and meta-analysis. J Epidemiol Community Health. Sep 2018;72(9):856-863. [ CrossRef ] [ Medline ]
• Fischer G, Ameis N, Parzer P, Plener PL, Groschwitz R, Vonderlin E, et al. The German version of the self-injurious thoughts and behaviors interview (SITBI-G): a tool to assess non-suicidal self-injury and suicidal behavior disorder. BMC Psychiatry. Sep 18, 2014;14(1):265. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Home page. Powtoon. URL: https://www.powtoon.com/ [accessed 2024-04-08]
• Wright A, Jorm AF, Mackinnon AJ. Labels used by young people to describe mental disorders: which ones predict effective help-seeking choices? Soc Psychiatry Psychiatr Epidemiol. Jun 2012;47(6):917-926. [ CrossRef ] [ Medline ]
• Mond JM, Hay PJ, Paxton SJ, Rodgers B, Darby A, Nillson J, et al. Eating disorders "mental health literacy" in low risk, high risk and symptomatic women: implications for health promotion programs. Eat Disord. 2010;18(4):267-285. [ CrossRef ] [ Medline ]
• DeJong H, Hillcoat J, Perkins S, Grover M, Schmidt U. Illness perception in bulimia nervosa. J Health Psychol. Apr 2012;17(3):399-408. [ CrossRef ] [ Medline ]
• Bulik CM. Towards a science of eating disorders: replacing myths with realities: the fourth Birgit Olsson lecture. Nord J Psychiatry. Sep 10, 2016;70(3):224-230. [ CrossRef ] [ Medline ]
• Keski-Rahkonen A, Mustelin L. Epidemiology of eating disorders in Europe: prevalence, incidence, comorbidity, course, consequences, and risk factors. Curr Opin Psychiatry. Nov 2016;29(6):340-345. [ CrossRef ] [ Medline ]
• Nagl M, Jacobi C, Paul M, Beesdo-Baum K, Höfler M, Lieb R, et al. Prevalence, incidence, and natural course of anorexia and bulimia nervosa among adolescents and young adults. Eur Child Adolesc Psychiatry. Aug 2016;25(8):903-918. [ CrossRef ] [ Medline ]
• Laberg S, Törnkvist Å, Andersson G. Experiences of patients in cognitive behavioural group therapy: a qualitative study of eating disorders. Cogn Behav Ther. Jan 2001;30(4):161-178. [ CrossRef ]
• Wilson CJ, Deane FP, Ciarrochi J, Rickwood D. Measuring help-seeking intentions: properties of the general help-seeking questionnaire. Can J Couns. 2005;39(1):15-28. [ FREE Full text ]
• Divin N, Harper P, Curran E, Corry D, Leavey G. Help-seeking measures and their use in adolescents: a systematic review. Adolescent Res Rev. Jan 17, 2018;3(1):113-122. [ CrossRef ]
• Mackenzie CS, Knox VJ, Gekoski WL, Macaulay HL. An adaptation and extension of the attitudes toward seeking professional psychological help scale. J Applied Social Pyschol. Jul 31, 2006;34(11):2410-2433. [ CrossRef ]
• Ebneter DS, Latner JD. Stigmatizing attitudes differ across mental health disorders: a comparison of stigma across eating disorders, obesity, and major depressive disorder. J Nerv Ment Dis. Apr 2013;201(4):281-285. [ CrossRef ] [ Medline ]
• Appel M, Gnambs T, Richter T, Green MC. The Transportation Scale–Short Form (TS–SF). Media Psychol. Feb 03, 2015;18(2):243-266. [ CrossRef ]
• Faul F, Erdfelder E, Lang A, Buchner A. G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav Res Methods. May 2007;39(2):175-191. [ CrossRef ]
• R Core Team. R: a language and environment for statistical computing. R Foundation for Statistical Computing. URL: https://www.r-project.org/ [accessed 2024-04-08]
• IBM SPSS Statistics for Windows, version 28.0. IBM Corp. URL: https://www.ibm.com/support/pages/downloading-ibm-spss-statistics-28010 [accessed 2024-04-08]
• Lienemann BA, Siegel JT, Crano WD. Persuading people with depression to seek help: respect the boomerang. Health Commun. Oct 2013;28(7):718-728. [ CrossRef ] [ Medline ]
• Lienemann BA, Siegel JT. State psychological reactance to depression public service announcements among people with varying levels of depressive symptomatology. Health Commun. Jun 18, 2016;31(1):102-116. [ CrossRef ] [ Medline ]
• Siegel JT, Lienemann BA, Rosenberg BD. Resistance, reactance, and misinterpretation: highlighting the challenge of persuading people with depression to seek help. Soc Pers Psychol Compass. Jun 05, 2017;11(6):e12322. [ CrossRef ]
• van Laer T, de Ruyter K, Visconti LM, Wetzels M. The extended transportation-imagery model: a meta-analysis of the antecedents and consequences of consumers' narrative transportation. J Consum Res. Feb 01, 2014;40(5):797-817. [ FREE Full text ] [ CrossRef ]
• Murphy ST, Frank LB, Chatterjee JS, Baezconde-Garbanati L. Narrative versus non-narrative: the role of identification, transportation and emotion in reducing health disparities. J Commun. Feb 07, 2013;63(1):116-137. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sowislo JF, Gonet-Wirz F, Borgwardt S, Lang UE, Huber CG. Perceived dangerousness as related to psychiatric symptoms and psychiatric service use - a vignette based representative population survey. Sci Rep. Apr 03, 2017;8(1):45716. [ FREE Full text ] [ CrossRef ] [ Medline ]
• McVay MA, Bennett GG, Steinberg D, Voils CI. Dose-response research in digital health interventions: concepts, considerations, and challenges. Health Psychol. Dec 2019;38(12):1168-1174. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Suka M, Yamauchi T, Yanagisawa H. Comparing responses to differently framed and formatted persuasive messages to encourage help-seeking for depression in Japanese adults: a cross-sectional study with 2-month follow-up. BMJ Open. Nov 12, 2018;8(11):e020823. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kaushik A, Kostaki E, Kyriakopoulos M. The stigma of mental illness in children and adolescents: a systematic review. Psychiatry Res. Sep 30, 2016;243:469-494. [ CrossRef ] [ Medline ]
• Misra S, Jackson VW, Chong J, Choe K, Tay C, Wong J, et al. Systematic review of cultural aspects of stigma and mental illness among racial and ethnic minority groups in the United States: implications for interventions. Am J Community Psychol. Dec 2021;68(3-4):486-512. [ CrossRef ] [ Medline ]
• Papadopoulos C, Foster J, Caldwell K. 'Individualism-collectivism' as an explanatory device for mental illness stigma. Community Ment Health J. Jun 27, 2013;49(3):270-280. [ CrossRef ] [ Medline ]

Abbreviations

Edited by G Eysenbach, T de Azevedo Cardoso; submitted 13.11.23; peer-reviewed by A Morgan; comments to author 15.12.23; revised version received 22.12.23; accepted 08.03.24; published 24.04.24.

©Diana Lemmer, Markus Moessner, Nicolas Arnaud, Harald Baumeister, Agnes Mutter, Sarah-Lena Klemm, Elisa König, Paul Plener, Christine Rummel-Kluge, Rainer Thomasius, Michael Kaess, Stephanie Bauer. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 24.04.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.