literature review about technology

• Skip to main content
• Accessibility help

Information

We use cookies to collect anonymous data to help us improve your site browsing experience.

Click 'Accept all cookies' to agree to all cookies that collect anonymous data. To only allow the cookies that make the site work, click 'Use essential cookies only.' Visit 'Set cookie preferences' to control specific cookies.

Your cookie preferences have been saved. You can change your cookie settings at any time.

Literature Review on the Impact of Digital Technology on Learning and Teaching

This literature review was commissioned by the Scottish Government to explore how the use of digital technology for learning and teaching can support teachers, parents, children and young people in improving outcomes and achieving our ambitions for education in Scotland

Digital learning and raising attainment

Key findings

There is conclusive evidence that digital equipment, tools and resources can, where effectively used, raise the speed and depth of learning in science and mathematics for primary and secondary age learners. There is indicative evidence that the same can be said for some aspects of literacy, especially writing and comprehension. Digital technologies appear to be appropriate means to improve basic literacy and numeracy skills, especially in primary settings.

The effect sizes are generally similar to other educational interventions that are effective in raising attainment, though the use of digital learning has other benefits. Also, the extent of the effect may be dampened by the level of capability of teachers to use digital learning tools and resources effectively to achieve learning outcomes. More effective use of digital teaching to raise attainment includes the ability of teachers to identify how digital tools and resources can be used to achieve learning outcomes and adapting their approach, as well as having knowledge and understanding of the technology. This applies in all schools.

Where learners use digital learning at home as well as school for formal and non-formal learning activities these have positive effects on their attainment, because they have extended their learning time. This is particularly important for secondary age learners.

The assessment framework, set out in Annex 2 , identifies a number of educational benefits that digital learning and teaching has the potential to help learners aged 5 to 18 to realise, through the opportunity to learn in different ways, access more sources of information, and be tested and get feedback differently. In terms of raising attainment, these benefits include short term outcomes, such as having a greater feeling of control over learning and more confidence to practise a skill, through to medium term outcomes such as faster acquisition of knowledge and skills, and improved impacts in terms of learners achieving higher exam or test results where digital technology has been used.

In this section, the impact of digital technology on children's attainment in a range of areas is discussed, followed by the impact on aspects of numeracy, literacy and science learning.

Raising children's attainment

There is a substantial body of research that has examined the impact of digital tools and resources on children's attainment in a range of areas.

Higgins et al (2012) provide a summary of research findings from studies with experimental and quasi-experimental designs, which have been combined in meta-analyses to assess the impact of digital learning in schools. Their search identified 48 studies which synthesised empirical research of the impact of digital tools and resources on the attainment of school age learners (5-18 year olds).

They found consistent but small positive associations between digital learning and educational outcomes. For example, Harrison et al (2004) identified statistically significant findings, positively associating higher levels of ICT use with school achievement at each Key Stage in England, and in English, maths, science, modern foreign languages and design technology. Somekh et al (2007) identified a link between high levels of ICT use and improved school performance. They found that the rate of improvement in tests in English at the end of primary education was faster in ICT Test Bed education authorities in England than in equivalent comparator areas. However, Higgins et al note that while these associations show, on average, schools with higher than average levels of ICT provision also have learners who perform slightly higher than average, it may be the case that high performing schools are more likely to be better equipped or more prepared to invest in technology or more motivated to bring about improvement.

Higgins et al report that in general analyses of the impact of digital technology on learning, the typical overall effect size is between 0.3 and 0.4 - just slightly below the overall average for researched interventions in education (Sipe & Curlette, 1997; Hattie, 2008) and no greater than other researched changes to teaching to raise attainment, such as peer tutoring or more focused feedback to learners. The range of effect sizes is also very wide (-0.03 to 1.05),which suggests that it is essential to take into account the differences between technologies and how they are used.

Table 4: Summary of meta-analyses published between 2000 and 2012 (in Higgins et al 2012)

In an earlier meta-analysis, Liao et al (2007), considered the effects of digital tools and resources on elementary school learners' achievement in Taiwan. Synthesizing research comparing the effects of digital learning (equipment, tools and resources) with traditional instruction on elementary school learners' achievement, they considered quantitative and qualitative information from 48 studies including over 5,000 learners. Of the 48 studies, 44 (92%) showed positive effects in favour of a computer assisted intervention, while four (8%) were negative and favoured a traditional instruction method. Nearly 60% of the studies examined the effects of computer aided instruction for teaching mathematics or science. Another 11% of the studies concentrated on the teaching of reading and language. They found an overall positive effect size across all the studies of 0.45 (study-weighted grand mean), which is considered to be a moderate effect, with a wide range of effect sizes (from 0.25 to 2.67).

No significant differences were found between subject areas, and the authors suggest that digital learning has the potential to be implemented in many different subject areas. They found that the two subjects that showed the highest effects were reading and languages, which had a high positive effect size of 0.7. Studies using computer simulations also had higher effects. The authors suggest this may be because simulations can provide learners with the opportunity to engage in a learning activity which could not be replicated in a classroom.

More qualitative studies have identified how improvements in attainment are achieved. From a wide study of primary and secondary schools in England that were early adopters in using digital learning and teaching, Jewitt et al (2011) concluded that:

• Using digital resources provided learners with more time for active learning in the classroom;
• Digital tools and resources provided more opportunity for active learning outside the classroom, as well as providing self-directed spaces, such as blogs and forums, and access to games with a learning benefit;
• Digital resources provided learners with opportunities to choose the learning resources;
• The resources provided safer spaces for formative assessment and feedback.

The sections below focus on specific key areas of attainment: literacy, numeracy, and science learning.

There is a large body of research that has examined the impact of digital equipment, tools and resources on children's literacy. The effects are generally positive, though not as large as the effects found where digital learning is used to improve numeracy, and consistent in finding that ICT helps improve reading and writing skills, as well as developing speaking and listening skills.

Effect of context

Archer and Savage (2014) undertook a meta-analysis to reassess the outcomes presented in three previous meta-analyses considering the impact of digital learning on language and literacy learning: Slavin et al (2008 and 2009) and Torgenson and Zhu (2003). Overall they found a relatively small average positive effect size of 0.18, with a few of the studies having a negative effect and three studies showing moderate to large effect sizes. The authors found that programmes with a small number of participants tended to show larger effect sizes than larger programmes but that not all were statistically significant.

Archer and Savage sought to understand whether the context within which the digital tool or resource was used has an impact on outcomes. In particular, they examined whether training and support given to the teachers or other staff delivering the programme had an impact. The authors found that training and support could be identified in around half of the studies and that it did appear to have a positive impact on the effectiveness of the literacy intervention, with the average effect size rising to 0.57. The authors conclude that this indicates the importance of including implementation factors, such as training and support, when considering the relative effectiveness of digital learning and teaching.

Effect on specific literacy skills

In their meta-analysis, Higgins et al (2012) found that digital learning has a greater impact on writing than on reading or spelling. For example, Torgenson and Zhu (2003) reviewed the impact of using digital technology on the literacy competences of 5-16 year-olds in English and found effect sizes on spelling (0.2) and reading (0.28) much lower than the high effect size for writing (0.89).

In their meta-analysis of studies investigating the effects of digital technology on primary schools in Taiwan, Laio et al (2007) considered studies over a range of curriculum areas; 11 of which addressed the effects of using digital learning in one or more literacy competence. They found no significant differences in effect size between the different subject areas, suggesting the potential for digital technology to raise outcomes is equal across different subjects. However, they did note that the two areas that showed the highest effect sizes (over 0.7) were reading and comprehension.

Effect of specific digital tools and resources

Somekh et al (2007) evaluated the Primary School Whiteboard Expansion ( PSWB ) project in England. They found that the length of time learners were taught with interactive whiteboards ( IWB s) was a major factor in learner attainment at the end of primary schooling, and that there were positive impacts on literacy (and numeracy) once teachers had experienced sustained use and the technology had become embedded in pedagogical practice. This equated to improvements at Key Stage 2 writing (age 11), where boys with low prior attainment made 2.5 months of additional progress.

Hess (2014) investigated the impact of using e-readers and e-books in the classroom, among 9-10 year olds in the USA . The e-books were used in daily teacher-led guided reading groups, replacing traditional print books in these sessions. Teachers also regularly used the e-readers in sessions where the class read aloud, and e-readers were available to learners during the school day for silent reading. The study found a significant difference in reading assessment scores for the group using the e-readers. Scores improved for both male and female learners and the gap between males and females decreased.

The use of digital tools and resources also appears to affect levels of literacy. Lysenko and Abrami (2014) investigated the use of two digital tools on reading comprehension for elementary school children (aged 6-8) in Quebec, Canada. The first was a multimedia tool which linked learning activities to interactive digital stories. The tool included games to engage learners in reading and writing activities, and instructions were provided orally to promote listening comprehension. The second tool was a web-based electronic portfolio in which learners could create a personalised portfolio of their reading and share work with peers, teachers and parents to get feedback. The authors found that in classes where both tools were used together during the whole school year learners performed significantly better both in vocabulary and reading comprehension (with medium-level effect sizes) than learners in classes where the tools were not part of English language instruction.

Rosen and Beck-Hill (2012) reported on a study programme that incorporated an interactive core curriculum and a digital teaching platform. At the time of their report it was available for 9-11 year old learners in English language, arts and mathematics classes in Dallas, Texas. The online platform contained teaching and learning tools. Learners were assessed using standardised tests administered before the programme and after a year's participation. The results of increased achievement scores demonstrated that in each of the two school year groups covered, the experimental learners significantly outperformed the control learners in reading and maths scores. In observations in classrooms that used the programme, the researchers observed higher teacher-learner interaction, a greater number and type of teaching methods per class, more frequent and complex examples of differentiation processes and skills, more frequent opportunities for learner collaboration, and significantly higher learner engagement. The authors report that the teaching pedagogy observed in the classrooms differed significantly from that observed in more traditional classrooms. The teachers following the programme commented that the digital resources made planning and implementing 'differentiation' more feasible. This is differentiation of teaching in terms of content, process, and product, to reflect learners' readiness, interests, and learning profile, through varied instructional and management strategies.

Effect of the amount and quality of digital technology use

The uses of digital technology and access to it appear to be critical factors. Lee et al (2009) analysed how in the US 15-16 year-old learners' school behaviour and standardised test scores in literacy are related to computer use. Learners were asked how many hours a day they typically used a computer for school work and for other activities. The results indicated that the learners who used the computer for one hour a day for both school work and other activities had significantly better reading test scores and more positive teacher evaluations for their classroom behaviours than any other groups [5] . This was found while controlling for socio-economic status, which has been shown to be a predictor of test scores in other research. The analysis used data from a national 2002 longitudinal study, and it is likely that learners' usage of computers has increased and changed since that time.

Biagi and Loi (2013), using data from the 2009 Programme for International Student Assessment ( PISA ) and information on how learners used digital technology at school and at home (both for school work and for entertainment), assessed the relationship between the intensity with which learners used digital tools and resources and literacy scores. They examined uses for: gaming activities (playing individual or collective online games), collaboration and communication activities (such as linking with others in on-line chat or discussion forums), information management and technical operations (such as searching for and downloading information) and creating content, knowledge and problem solving activities (such as using computers to do homework or running simulations at school). These were then compared to country specific test scores in reading. The authors found a positive and significant relationship between gaming activity and language attainment in 11 of the 23 countries studied. For the other measures, where relationships existed and were significant, they tended to be negative.

The more recent PISA data study ( OECD , 2015, using 2012 results) also found a positive relationship between the use of computers and better results in literacy where it is evident that digital technology is being used by learners to increase study time and practice [6] . In addition, it found that the effective use of digital tools is related to proficiency in reading.

There is a large body of research which has examined the impact of digital equipment, tools and resources on children's numeracy skills and mathematical competences throughout schooling. Higgins et al (2012) found from their meta-analysis that effect sizes of tested gains in knowledge and understanding tend to be greater in mathematics and science than in literacy. The key benefits found relate to problem solving skills, practising number skills and exploring patterns and relationships (Condie and Monroe, 2007), in addition to increased learner motivation and interest in mathematics.

Effect on specific numeracy skills

Li and Ma's (2010) meta-analysis of the impact of digital learning on school learners' mathematics learning found a generally positive effect. The authors considered 46 primary studies involving a total of over 36,000 learners in primary and secondary schools. About half of the mathematics achievement outcomes were measured by locally-developed or teacher-made instruments, and the other half by standardized tests. Almost all studies were well controlled, employing random assignment of learners to experimental or control conditions.

Overall, the authors found that, on average, there was a high, significantly positive effect of digital technology on mathematics achievement (mean effect size of 0.71), indicating that, in general, learners learning mathematics with the use of digital technology had higher mathematics achievement than those learning without digital technology. The authors found that:

• Although the difference was small, younger school learners (under 13 years old) had higher attainment gains than older secondary school learners;
• Gains were more positive where teaching was more learner-centred than teacher-centred. In this regard, the authors differentiate between traditional models, where the teacher tends to teach to the whole class, and a learner-centred teaching model which is discovery-based (inquiry-oriented) or problem-based (application-oriented) learning;
• Shorter interventions (six months or less) were found to be more effective in promoting mathematics achievement than longer interventions (between six and 12 months). It is suggested that such gains in mathematics achievement are a result of the novelty effects of technology, as suggested in other research, and as learners get familiar with the technology the novelty effects tend to decrease;
• The authors found no significant effects from different types of computer technology on mathematics achievement. Whether it was used as communication media, a tutorial device, or exploratory environment, learners displayed similar results in their mathematics achievement;
• Equally, the authors found no significant relationship between the effect of using digital technology and the characteristics of learners included in the samples for studies, such as gender, ethnicity, or socio-economic characteristics.

The studies by Lee et al (2009) and Biagi and Loi (2013) found similar results for mathematics as they did for reading and literacy in relation to the use of digital equipment. Learners who used a computer at least one hour a day for both school work and other activities had significantly better mathematics test scores and more positive teacher evaluations for their classroom behaviour in mathematics classes than those who did not use the computer. Biagi and Loi (2013) found a significant positive relationship between intensity of gaming activity and maths test scores in 15 countries out of the 23 studied. As with language, the authors found that learners' total use of digital technologies was positively and significantly associated with PISA test scores for maths in 18 of the 23 countries studied.

Studies have found that using digital equipment for formal learning is also associated with increases in learners' motivation for learning mathematics. House and Telese (2011 and 2012) found that:

• For learners aged 13 and 14 in South Korea, for example, those who expressed high levels of enjoyment at learning mathematics, more frequently used computers in their mathematics homework. However, learners who more frequently played computer games and used the internet outside of school tended to report that they did not enjoy learning mathematics;
• Learners in the USA and Japan aged 13 and 14 who showed higher levels of algebra achievement also used computers more at home and at school for school work. Those who used computers most for other activities had lower test scores. In each of the USA and Japan they found that overall computer usage which included use for school work was significantly related to improvements in test scores.

Somekh et al (2007) found that, once the use of IWB s was embedded, in Key Stage 1 mathematics (age 7) in England, high attaining girls made gains of 4.75 months, enabling them to catch up with high attaining boys. In Key Stage 2 mathematics (age 11), average and high attaining boys and girls who had been taught extensively with the IWB made the equivalent of an extra 2.5 to 5 months' progress over the course of two years.

Digital tools and resources can also increase some learners' confidence in mathematics as well as their engagement in new approaches to learning and their mathematical competences. Overcoming learners' anxieties about mathematics and their competence in specific aspects of the subject are common concerns in teaching mathematics which hampers their ability to learn (reported in Huang et al 2014).

Huang et al (2014) researched the outcomes, in Taiwan, from a computer game simulating the purchase of commodities, from which 7 and 8 year-old primary school learners can learn addition and subtraction, and apply mathematical concepts. The model combined games-based learning with a diagnosis system. When the learner made a mistake, the system could detect the type of mistake and present corresponding instructions to help the learner improve their mathematical comprehension and application. The authors compared two learning groups: both used the game-based model but one without the diagnostic, feedback element. They found that the learning achievement post-test showed a significant difference and also that the mathematics anxiety level of the two learner groups was decreased by about 3.5%.

Passey (2011) found that among over 300 schools in England using Espresso digital resources, those that had been using them over a longer period made significantly greater increases in end of primary school numeracy test results than schools which were recent users.

Science learning

Effects on science knowledge and skills

In their meta-analysis, Laio et al (2007) considered 11 studies looking at the impact of digital technology on science learning. These had a moderate average effect size of 0.38 and generally had positive effects. Condie and Monroe (2007) identified that digital learning made science more interesting, authentic and relevant for learners and provided more time for post-experiment analysis and discussion.

In their study of the PISA data, Biagi and Loi (2013) found a significant positive relationship between learners' total use of digital equipment and science test scores in 21 of the 23 countries they studied. They also found evidence of a significant positive relationship between the intensity of using gaming activity and science scores in 13 of the 23 countries they studied. Somekh et al (2007) found that in primary school science all learners, except high attaining girls, made greater progress when given more exposure to IWB s, with low attaining boys making as much as 7.5 months' additional progress.

Effects of specific digital tools and resources

Digital tools and resources generally have a positive effect on learners' science learning. This can be seen from a number of studies assessing outcomes for learners in different stages of education.

Hung et al (2012) explored the effect of using multi-media tools in science learning in an elementary school's science course in Taiwan. Learners were asked to complete a digital storytelling project by taking pictures with digital cameras, developing the story based on the pictures taken, producing a film based on the pictures by adding subtitles and a background, and presenting the story. From the experimental results, the authors found that this approach improved the learners' motivation to learn science, their attitude, problem-solving capability and learning achievements. In addition, interviews found that the learners in the experimental group enjoyed the project-based learning activity and thought it helpful because of the digital storytelling aspect.

Hsu et al (2012) investigated the effects of incorporating self-explanation principles into a digital tool facilitating learners' conceptual learning about light and shadow with 8-9 year old learners in Taiwan. While they found no difference in the overall test scores of the experimental and control groups, they found a statistically significant difference in retention test scores. Those learners who had paid more attention to the self-explanation prompts tended to outperform those in the control group.

Anderson and Barnett's (2013) study, in the US , examined how a digital game used by learners aged 12-13 increased their understanding of electromagnetic concepts, compared to learners who conducted a more traditional inquiry-based investigation of the same concepts. There was a significant difference between the control and experimental groups in gains in knowledge and understanding of physics concepts. Additionally, learners in the experimental group were able to give more nuanced responses about the descriptions of electric fields and the influence of distance on the forces that change experience because of what they learnt during the game.

Güven and Sülün (2012) considered the effects of computer-enhanced teaching in science and technology courses on the structure and properties of matter, such as the periodical table, chemical bonding, and chemical reactions, for 13-14 year olds in Turkey. Their proposition was that computer-enhanced teaching can instil a greater sense of interest in scientific and technological developments, make abstract concepts concrete through simulation and modelling, and help to carry out some dangerous experiments in the classroom setting. They found a significant difference in achievement tests between the mean scores of the group of learners who were taught with the computer-enhanced teaching method and the control group who were taught with traditional teaching methods.

Belland (2009) investigated the extent to which a digital tool improved US middle school children's ability to form scientific arguments. Taking the premise that being able to construct and test an evidence-based argument is critical to learning science, he studied the impact of using a digital problem based learning tool on 12-14 year olds. Learners worked in small groups and were asked to develop and present proposals for spending a grant to investigate an issue relating to the human genome project. Those in the experimental group used an online system which structured the project into stages of scientific enquiry. The system prompted the learners to structure and organise their thinking in particular ways: by prompting the learners individually, sharing group members' ideas, tasking the group to form a consensus view, and prompting the group to assign specific tasks among themselves.

Using pre- and post- test scores to assess the impact on learners' abilities to evaluate arguments, Belland found a high positive effect size of 0.62 for average-achieving learners compared to their peers in the control group. No significant impacts were found for higher or lower-achieving learners. Belland suggests that for high-achieving learners, this may be because they already have good argument making skills and are already able to successfully structure how they approach an issue and gather evidence. The study also used qualitative information to consider how the learners used the digital tool and compared this to how learners in the control group worked. The author found that in the experimental group they made more progress and were more able to divide tasks up between them, which saved time. They also used the tool more and the teacher less to provide support.

Kucukozer et al (2009) examined the impact of digital tools on teaching basic concepts of astronomy to 11-13 year old school children in Turkey. Learners were asked to make predictions about an astronomical phenomenon such as what causes the seasons or the phases of the moon. A digital tool was used to model the predictions and display their results. The learners were then asked to explain the differences and the similarities between their predictions and their observations. In the prediction and explanation phase the learners worked in groups to discuss their ideas and come to a conclusion. In the observation phase they watched the 3D models presented by their teacher. Thereafter, they were asked to discuss and make conclusions about what they had watched. The authors found that instruction supported by observations and the computer modelling was significantly effective in bringing about better conceptual understanding and learning on the subject.

Ingredients of success

Where studies examine the process that brings about positive results from digital learning and teaching compared to traditional approaches, it is evident that these are more likely to be achieved where digital equipment, tools and resources are used for specific learning outcomes and built into a teaching model from the outset. This broadly supports Higgins et al's (2012) conclusions that:

• Digital technology is best used as a supplement to normal teaching rather than as a replacement for it;
• It is not whether technology is used (or not) which makes the difference, but how well the technology is applied to support teaching and learning by teachers;
• More effective schools and teachers are more likely to use digital technologies effectively than other schools.

Differences in effect sizes and the extent that learners achieve positive gains in attainment are ascribed by most authors of the studies above to:

• The quality of teaching and the ability of teachers to use the digital equipment and tools effectively for lessons;
• The preparation and training teachers are given to use equipment and tools;
• The opportunities teachers have to see how digital resources can be used and pedagogies adapted (Rosen and Beck-Hill, 2012; Belland, 2009).

Teachers have to adapt to learner-centred approaches to learning if they are to use digital tools and resources (Li and Ma, 2010).

As well as ensuring digital tools and resources are supporting learning goals, success appears to also be linked to some other factors:

• The availability of equipment and tools within schools (and at home);
• How learners use digital equipment. Higgins et al (2012) found that collaborative use of technology (in pairs or small groups) is usually more effective than individual use, though some learners - especially younger children - may need guidance in how to collaborate effectively and responsibly;
• The extent that teaching continues to innovate using digital tools and resources (Higgins et al, 2012).

Fullan (2013) suggested four criteria that schools should meet if their use of digital technology to support increased attainment is to be successful. These were that systems should be engaging for learners and teachers; easy to adapt and use; ubiquitous - with access to the technology 24/7; and steeped in real life problem solving.

Fullan and Donnelly (2013) developed these themes further, proposing an evaluation tool to enable educators to systematically evaluate new companies, products and school models, using the context of what they have seen as necessary for success. Questions focus on the three key criteria of pedagogy (clarity and quality of intended outcome, quality of pedagogy and the relationship between teacher and learner, and quality of assessment platform and functioning); system change (implementation support, value for money, and whole system change potential) and technology (quality of user experience/model design, ease of adaptation, and comprehensiveness and integration).

Email: Catriona Rooke

There is a problem

Thanks for your feedback

Your feedback helps us to improve this website. Do not give any personal information because we cannot reply to you directly.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• How to Write a Literature Review | Guide, Examples, & Templates

How to Write a Literature Review | Guide, Examples, & Templates

Published on January 2, 2023 by Shona McCombes . Revised on September 11, 2023.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research that you can later apply to your paper, thesis, or dissertation topic .

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates, and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarize sources—it analyzes, synthesizes , and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

Table of contents

What is the purpose of a literature review, examples of literature reviews, step 1 – search for relevant literature, step 2 – evaluate and select sources, step 3 – identify themes, debates, and gaps, step 4 – outline your literature review’s structure, step 5 – write your literature review, free lecture slides, other interesting articles, frequently asked questions, introduction.

• Quick Run-through
• Step 1 & 2

When you write a thesis , dissertation , or research paper , you will likely have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and its scholarly context
• Develop a theoretical framework and methodology for your research
• Position your work in relation to other researchers and theorists
• Show how your research addresses a gap or contributes to a debate
• Evaluate the current state of research and demonstrate your knowledge of the scholarly debates around your topic.

Writing literature reviews is a particularly important skill if you want to apply for graduate school or pursue a career in research. We’ve written a step-by-step guide that you can follow below.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research problem and questions .

Make a list of keywords

Start by creating a list of keywords related to your research question. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list as you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some useful databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can also use boolean operators to help narrow down your search.

Make sure to read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

You likely won’t be able to read absolutely everything that has been written on your topic, so it will be necessary to evaluate which sources are most relevant to your research question.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models, and methods?
• Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible , and make sure you read any landmark studies and major theories in your field of research.

You can use our template to summarize and evaluate sources you’re thinking about using. Click on either button below to download.

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It is important to keep track of your sources with citations to avoid plagiarism . It can be helpful to make an annotated bibliography , where you compile full citation information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

The only proofreading tool specialized in correcting academic writing - try for free!

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

Try for free

To begin organizing your literature review’s argument and structure, be sure you understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly visual platforms like Instagram and Snapchat—this is a gap that you could address in your own research.

There are various approaches to organizing the body of a literature review. Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarizing sources in order.

Try to analyze patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organize your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text , your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, you can follow these tips:

• Summarize and synthesize: give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: don’t just paraphrase other researchers — add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically evaluate: mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: use transition words and topic sentences to draw connections, comparisons and contrasts

In the conclusion, you should summarize the key findings you have taken from the literature and emphasize their significance.

When you’ve finished writing and revising your literature review, don’t forget to proofread thoroughly before submitting. Not a language expert? Check out Scribbr’s professional proofreading services !

This article has been adapted into lecture slides that you can use to teach your students about writing a literature review.

Scribbr slides are free to use, customize, and distribute for educational purposes.

Open Google Slides Download PowerPoint

If you want to know more about the research process , methodology , research bias , or statistics , make sure to check out some of our other articles with explanations and examples.

• Sampling methods
• Simple random sampling
• Stratified sampling
• Cluster sampling
• Likert scales
• Reproducibility

 Statistics

• Null hypothesis
• Statistical power
• Probability distribution
• Effect size
• Poisson distribution

Research bias

• Optimism bias
• Cognitive bias
• Implicit bias
• Hawthorne effect
• Anchoring bias
• Explicit bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarize yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your thesis or dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, September 11). How to Write a Literature Review | Guide, Examples, & Templates. Scribbr. Retrieved April 9, 2024, from https://www.scribbr.com/dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, what is a theoretical framework | guide to organizing, what is a research methodology | steps & tips, how to write a research proposal | examples & templates, unlimited academic ai-proofreading.

✔ Document error-free in 5minutes ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

To read this content please select one of the options below:

Please note you do not have access to teaching notes, a literature review exploring the role of technology in business survival during the covid-19 lockdowns.

International Journal of Organizational Analysis

ISSN : 1934-8835

Article publication date: 18 May 2021

Issue publication date: 23 November 2022

The Covid-19 pandemic has affected every aspect of human life. Even though the pandemic length was not too long, a huge volume of research relating to Covid-19 has been published in different contexts. This paper aims to review the literature investigating the impact of Covid −19 on businesses generally and explore studies examining the technology role of business survival during the Covid-19 lockdowns specifically.

Design/methodology/approach

This study implemented the concept of a systematic review approach to review the literature that has been conducted in the business field during the Covid-19 crisis in general. Additionally, it looks into the research examining the role of technology in business survival in the Covid-19 crisis specifically. All studies were conducted in 2020. A total of 53 studies were identified and categorised into different themes. The research methods, theories and locations have also been analysed.

It was found that Covid-19 pandemic has affected all business sectors in several ways. Technology adoption has a critical role for business survival during the Covid-19 crises especially with small businesses. Very limited research has been conducted on the adoption of different technologies during the Covid-19 lockdowns.

Originality/value

This study presents the most frequent themes and topics that have been explored in the literature during the Covid-19 crisis in the business field. It highlights the methods used in addition to the theories and research locations present in this literature. Finally, it proposes the possible implications of this literature review.

• Literature review

Abed, S.S. (2022), "A literature review exploring the role of technology in business survival during the Covid-19 lockdowns", International Journal of Organizational Analysis , Vol. 30 No. 5, pp. 1045-1062. https://doi.org/10.1108/IJOA-11-2020-2501

Emerald Publishing Limited

Copyright © 2021, Emerald Publishing Limited

Related articles

We’re listening — tell us what you think, something didn’t work….

Report bugs here

All feedback is valuable

Please share your general feedback

Join us on our journey

Platform update page.

Visit emeraldpublishing.com/platformupdate to discover the latest news and updates

Questions & More Information

Answers to the most commonly asked questions here

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Center for Substance Abuse Treatment (US). Using Technology-Based Therapeutic Tools in Behavioral Health Services. Rockville (MD): Substance Abuse and Mental Health Services Administration (US); 2015. (Treatment Improvement Protocol (TIP) Series, No. 60.)

Using Technology-Based Therapeutic Tools in Behavioral Health Services.

• Order print copy from SAMHSA

Section 1 A Review of the Literature

The use of communication technologies (e.g., the Internet, email, video conferencing, telephone) to prevent and/or treat mental and substance use disorders has been recognized by the Center for Substance Abuse Treatment (CSAT) as important in helping meet unaddressed treatment needs ( CSAT, 2009a ). This review covers the therapeutic use of such technologies, whether they are delivered via telephones or computers, as well as their use in supervising and training program staff members. It is not concerned with most other uses of new technologies (e.g., electronic record keeping, computer modeling, biotechnology, social media). Although technology-assisted care (TAC) provides a number of opportunities to enhance behavioral health services ( Eonta et al., 2011 ), this review focuses on interventions that use technologies as a primary means of delivering services.

As with the rest of this Treatment Improvement Protocol (TIP), the literature review focuses on research involving adults. Because a good deal of the research in this area has been conducted outside the United States, studies involving foreign populations are identified as such; those that are not so identified should be assumed to have taken place in the United States. This review focuses on the past 10 years of research, with occasional references to older, seminal literature. When possible, it uses other reviews to summarize earlier studies. The review generally does not draw conclusions, but instead tries to present several points of view so that readers who are interested in the issue may seek out the appropriate literature and draw their own conclusions. Thus, readers should not accept the presentation of one article's findings as an endorsement of one position over another.

The first two sections after “Overview” provide some of the basic information about the technologies included in this TIP and give some idea about how they are currently being used in behavioral health as well as more general claims about their effectiveness. The larger sections that follow discuss the use of such technologies to address prevention and treatment specifically of mental and substance use disorders and are organized by the disorder addressed. Those sections may include research on one or multiple types of technology, depending on what recent literature is available.

• Understanding Technologies

Introduction

The “Understanding Technologies section covers basic technologies that are being used in the treatment and prevention of mental and substance use disorders:

Telephone/Audio Counseling

Video/web conferencing, self-directed, web-based, and computer-based therapeutic tools, web-based text communication, mobile (handheld) technologies.

These categories are not exclusive. One intervention may involve components that use any number of these technologies, and there is almost always some overlap with other categories (e.g., mobile technologies typically use phone and/or text communication).

Although the system of categorizing interventions by the technology used is common in the literature, it is not the only way to categorize them, and there are other features of these interventions that can be used to distinguish one from another. For example, interventions can be categorized as either synchronous (involving communications occurring in real time) or asynchronous (occurring outside real time, with some delay between the sending and receiving of the communication; Suler, 2004 ; Yellowlees et al., 2010 ). The larger portion of this review discusses interventions according to the disorder or problem targeted by the intervention.

This section introduces these technologies, presents basic findings about their use and effectiveness (drawing on other reviews when available), and also highlights interventions that can be used to address multiple substance use and mental disorders (as opposed to interventions directed at a single disorder or group of disorders, such as anxiety disorders).

Counseling has been conducted via telephone for quite some time, and many counselors report positive results using that technology ( Maheu, Pulier, Wilhelm, McMenamin, & Brown-Connolly, 2004 ). Potential benefits for clients of telephone-based services, relative to in-person services, include lower expense, greater convenience, greater anonymity, and a greater sense of control ( Reese, Conoley, & Brossart, 2002 ). Telephones, either using live interviewers or automated systems, have been successfully used to screen and assess mental and substance use disorders and cognitive impairment ( Kobak, Williams, & Engelhardt, 2008 ; Marks et al., 1998 ; Martin-Khan, Wootton, & Gray, 2010 ; Rohde, Lewinsohn, & Seeley, 1997 ; Simon, Revicki, & VonKorff, 1993 ; Tunstall, Prince, & Mann, 1997 ; Xu et al., 2012 ). However, some disorders (e.g., adjustment disorder with depressed mood) may be more difficult to assess by phone than in person ( Rohde et al., 1997 ).

Telephones have also been used to improve treatment/medication compliance ( Maust et al., 2012 ), monitor recovery from mental and substance use disorders ( Godleski, Cervone, Vogel, & Rooney, 2012 ), and motivate potential clients to enter treatment ( Seal et al., 2012 ). Adding phone calls to a Web-based intervention may also improve treatment compliance and outcomes ( Graham et al., 2011 ; Titov, Andrews, Choi, Schwenke, & Johnston, 2009 ). Leach and Christensen (2006) , in a literature review on telephone-based interventions for mental and substance use disorders, located 14 studies involving interventions for depression (6 studies), anxiety (3), eating disorders (3), substance use disorders (1), and schizophrenia (1). They concluded that such interventions could reduce symptoms of anxiety and depression as well as disordered eating behaviors. They also found limited and somewhat flawed evidence that such interventions could reduce alcohol use for individuals with alcohol use disorders and hospitalization rates for people with schizophrenia. However, most of the studies they reviewed had methodological problems, such as small sample sizes, high dropout rates, and a lack of randomization, which limited their ability to draw firm conclusions about effectiveness. They also noted that effective telephone-based interventions were highly structured and made use of homework assignments for clients. Another review by Mohr, Vella, Hart, Heckman, and Simon (2008) , which included 12 trials of phone-based interventions for depression, also found that such interventions were associated with significantly greater reductions in depressive symptoms than were control conditions; these interventions were also associated with reductions in symptoms from baseline to posttreatment follow-up that were comparable with those observed in many in-person interventions.

Other studies have found telephone-based interventions to be more effective than no-treatment controls and/or about as effective as some standard treatments for smoking cessation ( Cummins, Bailey, Campbell, Koon-Kirby, & Zhu, 2007 ; Rabius, McAlister, Geiger, Huang, & Todd, 2004 ; Regan, Reyen, Lockhart, Richards, & Rigotti, 2011 ), continuing care for substance use disorders ( Farabee et al., 2012 ; McKay, Lynch, Shepard, & Pettinati, 2005 ; McKay et al., 2011 ; Stout, Rubin, Zwick, Zywiak, & Bellino, 1999 ), depression ( Mohr, Carmody, Erickson, Jin, & Leader, 2011 ; Mohr et al., 2012 ; Mohr et al., 2008 ; Piette et al., 2011 ), obsessive–compulsive disorder (OCD; Kenwright, Marks, Graham, Franses, & Mataix-Cols, 2005 ; Lovell, Fullalove, Garvey, & Brooker, 2000 ), problem gambling ( Rodda & Lubman, 2012 ), posttraumatic stress disorder (PTSD) symptoms ( DuHamel et al., 2010 ), and the promotion of positive behavior change related to healthy eating and exercise ( Eakin, Lawler, Vandelanotte, & Owen, 2007 ). Self-guided treatment, using phone calls from counselors, has also been found to be effective for anxiety disorders ( Cuijpers, Donker, van Straten, Li, & Andersson, 2010 ). Also, telephone-based cognitive–behavioral therapy (CBT) can improve health outcomes for people with physical disorders ( Muller & Yardley, 2011 ).

Dorstyn, Mathias, and Denson (2011) conducted a meta-analytic review of telephone-based counseling interventions for people with acquired physical disabilities (e.g., spinal cord injuries, severe burns) but not, for the most part, people with mental or substance use disorders; they found that such interventions were associated with significant improvements in the use of coping skills, in community integration, and in symptoms of depression immediately following telephone counseling as well as more modest, but lasting, improvements in quality of life.

Telephone helplines, or hotlines set up so that individuals in need of services can call into a centralized location and speak with a counselor, have also been effective in suicide prevention ( Gould, Kalafat, Harris Munfakh, & Kleinman, 2007 ), tobacco cessation ( Cummins et al., 2007 ; Stead, Perera, & Lancaster, 2007 ), and addressing general mental health concerns, including panic attacks ( Burgess, Christensen, Leach, Farrer, & Griffiths, 2008 ). In their interviews with therapists, Day and Schneider (2000) found that some counselors felt that treatment using audio only caused them to miss important information (e.g., body language, client's physical state), but other counselors observed that a lack of the visual element increased the ease of communication between client and therapist. Clients who had tried telephone-based services generally expressed satisfaction with them and found those services helpful for a variety of behavioral health problems ( Reese et al., 2002 ; Reese, Conoley, & Brossart, 2006 ). Many clients also expressed a preference for telephone counseling. In one survey of clients who had received both telephone-based and in-person counseling, 96 percent stated they would be willing to seek telephone-based services again; by comparison, only 63 percent said they would be willing to use in-person services again ( Reese et al., 2006 ).

Video services for behavioral health are typically provided through video conferencing using computers connected to the Internet ( Zack, 2004 ), but they can also be transmitted using videophones connected to phone lines, although that is a lower-quality option ( Godleski, Nieves, Darkins, & Lehmann, 2008 ). Video conferencing, which provides both audio and video, has been used in a variety of behavioral health settings, usually to provide what would otherwise be an in-person service to clients who are not able to reach the provider's location. A comprehensive review of these services (entitled Evidence-Based Practice for Telemental Health) is available from the American Telemedicine Association (ATA; 2009) . The review is focused on interactive video conferencing because reviewers found that this technology had the largest research base in support of its use of any of the technologies they considered.

Backhaus et al. (2012) reviewed 65 studies involving the use of video conferencing specifically for the provision of psychotherapy. They concluded that:

• This was a feasible approach to providing therapy.
• Therapists were able to develop a therapeutic alliance using this technology (although that might be limited to one-on-one therapy, as studies involving group and family therapy found some problems in this area).
• Most users were satisfied with this method of delivery and reported a level of satisfaction comparable with that reported by clients receiving in-person therapy, and the major sources of dissatisfaction were technical difficulties.
• Clients using video conferencing had similar levels of retention and showed similar levels of clinical improvement to those receiving in-person treatments, with some differences depending on the specific disorder being treated (e.g., adolescents being treated for depression had faster improvements when treated via video conferencing).

García-Lizana and Muñoz-Mayorga (2010b) conducted a review of randomized controlled trials of video conferencing interventions for mental illness, of which they found 10. Although they found the research insufficient to draw a strong conclusion, the data that were available indicated that this approach was about as effective as in-person services and was an appropriate option, especially with clients who had difficulties accessing in-person services.

Richardson, Frueh, Grubaugh, Egede, and Elhai (2009) also reviewed literature on the use of video conferencing for behavioral health. They summarized earlier literature, as presented in older literature reviews, which consisted mainly of case studies, program descriptions, and anecdotal support for the use of video conferencing technology. Taken together, the literature does provide strong support for the acceptability to clients of such services and the reliability of assessments conducted using such technology. In the literature published since 2003, they found further support for those claims, and some studies that indicated that video conferencing interventions were superior to no treatment or a reduced level of in-person services. They also reviewed three studies that compared video conferencing interventions with in-person treatment and did not find any significant differences in outcomes for participants in the two groups, with both groups experiencing improvements. These three studies involved relatively brief interventions, and the authors of the largest of them ( O'Reilly et al., 2007 ) questioned the extent to which their results were generalizable to more complex treatments that may rely more on a therapeutic alliance. The authors also observed a number of methodological problems with many of the studies of video conferencing interventions and noted that many counselors remain wary of trying such interventions in spite of research suggesting their effectiveness. These reviewers also found that research generally indicated that video conferencing was cost-effective relative to in-person treatment, especially when costs related to travel and expenses for counselors were taken into account.

Norman (2006) reviewed 72 articles concerning video conferencing, with a focus on how well this technology might be adapted in the United Kingdom, and concluded that it appeared to be an effective way to deliver counseling services and was promising for clients living in rural areas. The author also concluded that although some early research did not find video conferencing to be cost-effective, most studies did find it to be so, and current developments in technology were likely to increase its cost-effectiveness. More recent data from the U.S. Department of Veterans Affairs (VA) confirmed that video conferencing interventions can aid healthcare systems in cutting overall costs. An evaluation of telemental health services (i.e., behavioral health services conducted using video conferencing) provided by VA to 98,609 clients between 2006 and 2010 found that hospital admissions for those clients decreased on average by 24.2 percent, and days hospitalized decreased by 26.6 percent ( Godleski, Darkins, & Peters, 2012 ). VA emphasizes that the value derived from implementing telehealth technologies is based on the enhancements such technologies bring to disease management, care/case management, health informatics, and the ability to offer the correct care in the correct place at the correct time. Over the past 5 years, the veteran's home has grown in importance as the “correct place” for the delivery of VA telemental health services, and home telehealth is now a major component of VA telemental healthcare and an ongoing topic of research ( Godleski et al., 2008 ).

How VA Defines Telehealth

“The wider application of care and case management principles to the delivery of healthcare services using health informatics, disease management and telehealth technologies to facilitate access to care and improve the health of designated individuals and populations with the intent of providing the right care in the right place at the right time.”

Source: http://www.telehealth.va.gov/about/index.asp

However, some reviewers have drawn less promising conclusions about video conferencing approaches to behavioral health and have found methodological problems with the literature supporting its use ( Hailey, Roine, & Ohinmaa, 2008 ; Simpson, 2009 ). For example, Hailey et al. (2008) found that although there is evidence supporting the use of video conferencing for a number of mental and substance use disorders, the quality of evidence is stronger for the use of Web- and phone-based interventions. They also observed methodological problems with much of the research involving video conferencing. A review by Simpson (2009) focused on the use of video conferencing to conduct psychotherapy and also observed a lack of rigor in most of the research. More recently, Kramer et al. (2012) discussed some of the methodological problems with the current research involving video conferencing and discussed ways that research could be improved by using a standard evaluation model.

Factors such as bandwidth, image resolution, and display size may affect clients' and counselors' experiences with video conferencing ( ATA, 2009 ). There is some research indicating that at least one of these factors (i.e., bandwidth) can affect outcomes ( Hyler, Gangure, & Batchelder, 2005 ). Other research indicates that certain aspects of assessment (e.g., assessing negative symptoms of psychosis) may be more accurate when done with higher-bandwidth connections, and clients are more likely to accept and express satisfaction with video conferencing when they use higher-bandwidth connections ( Sharp, Kobak, & Osman, 2011 ). Maheu et al. (2004) discussed in greater detail these and other important technical aspects of video conferencing in relation to behavioral health services.

Some counselors are using publicly available video conferencing software such as Skype, rather than professional video conferencing software, to communicate with clients. However, a recent review of the use of Skype for behavioral health services was able to find only small, poorly designed studies and was thus unable to draw conclusions about its use ( Armfield, Gray, & Smith, 2012 ). In addition, Skype may not be compliant with the Health Insurance Portability and Accountability Act, although some have argued that it can be if counselors set up the service appropriately (see the National Association of Social Workers' review papers on the subject by Morgan & Polowy, 2011 see the National Association of Social Workers' review papers on the subject by Morgan & Polowy, 2012 ). Morgan and Polowy (2011) concluded that it can be difficult to protect and ensure the confidentiality of clients' communications over Skype. A variety of professional programs are available for counselors to conduct this type of therapy, and such programs may be more appropriate. The independent Web site http://www.telementalhealthcomparisons.com provides information on a variety of these programs and allows for comparisons among them. Although it is used primarily for individual therapy ( Simpson, 2009 ), video conferencing technology has been used to provide other types of treatment, including group and marriage/family therapy ( ATA, 2009 ). As noted, video conferencing has been shown in a number of studies to be an effective technology for conducting clinical interviews and other assessments ( ATA, 2009 ; Richardson et al., 2009 ).

Video conferencing may be particularly valuable for clients living in rural or remote areas ( Grady & Singleton, 2011 ; LaMendola, 2000 ; Norman, 2006 ) and other clients who would not otherwise be able to find counselors with the appropriate skill sets in their areas, such as refugees or members of cultural groups who do not have a strong local presence ( Mucic, 2010 ). Specific populations that have had success using this type of technology include rural residents of nursing homes ( Rabinowitz et al., 2010 ), American Indian veterans ( Shore et al., 2012 ), and people who are incarcerated ( Magaletta, Fagan, & Peyrot, 2000 ). Video conferencing has also been used to assess psychiatric emergency patients in Finland ( Sorvaniemi, Ojanen, & Santamäki, 2005 ) and for involuntary commitment hearings ( Price & Sapci, 2007 ). As noted, video conferencing interventions are typically well-received by clients, and there is also some evidence that clients may participate more in counseling sessions if video conferencing is offered as a treatment option or in addition to in-person sessions ( Day & Schneider, 2002 ). Some research also indicates that some clients may feel more comfortable revealing information via video conferencing than in person ( Yuen, Goetter, Herbert, & Forman, 2012 ).

Despite the fact that some therapists have been reluctant to use video conferencing for treatment because of concerns about building a therapeutic relationship with clients ( Day & Schneider, 2000 ), those who regularly use it have stated that they find it possible to develop a strong therapeutic relationship. This has been confirmed in the few studies that have been conducted in this area ( Simpson & Morrow, 2010 ). For example, in a Canadian study, Germain, Marchand, Bouchard, Guay, and Drouin (2010) found that ratings of the strength of therapeutic alliances for clients with PTSD did not differ significantly between clients treated via video conferencing and those treated in person.

In discussions with therapists who were asked to compare in-person, video, and audio sessions, some therapists complained about being more distant from their clients when using video conferencing, but others observed that this technology made it easier for some clients to self-disclose ( Day & Schneider, 2000 ). Most therapists did believe that a strong therapeutic relationship could be developed in this medium, and some observed that the alliances they developed were even stronger than they might have been if the therapy had been conducted in person. Another potential concern with video conferencing is whether or not interventions developed for delivery in person can be adapted to this medium. Although research is limited, Morland et al. (2011) found that compliance with a manualized behavioral health services intervention did not differ significantly whether the intervention was delivered using video conferencing or in person.

Patient-rated alliance is also a significant factor to consider, and although some studies have found patient-rated alliance to be a positive predictor of favorable outcomes, the complex combinations of patient ratings, therapist ratings, and particular conditions can yield results that are challenging to interpret ( Huppert et al., 2014 ).

Computerized interventions for mental and substance use disorders have been in use for decades ( Carr, Ghosh, & Marks, 1988 ; Selmi, Klein, Greist, Sorrell, & Erdman, 1990 ), as has Web-based counseling using computers (see historical review by Grohol, 2004 ). Currently, such interventions are typically delivered via the Internet, and a number of them are entirely or largely mutual-help/self-directed interventions with little or no counselor involvement. Web-based interventions may be purely text-based or may make use of audio and/or video content. However, text-based interventions are discussed in a separate section of this literature review. Barak, Klein, and Proudfoot (2009) separated interventions/programs into four basic categories:

• Web-based intervention
• Online counseling and therapy
• Internet-operated therapeutic software
• Other online activities

Although these categories may be helpful in better understanding the differences in types of services offered, they are still not widely accepted, and many interventions make use of components from more than one of these categories. Most Internet-based interventions serve individuals, but group therapy ( Bellafiore, Colon, & Rosenberg, 2004 ; Golkaramnay, Bauer, Haug, Wolf, & Kordy, 2007 ; Roth, 2005 ) and family or family-oriented therapy ( Alemi, Haack, Dill, and Harge, 2005 ; Bischoff, 2004 ; Dausch, Miklowitz, Nagamoto, Adler, & Shore, 2009 ; Jencius & Sager, 2001 ) may also be conducted over the Internet. Behavioral health programs can also make use of social media sites ( Chou, Hunt, Beckjord, Moser, & Hesse, 2009 ), virtual communities ( Enos, 2008 ; Gorini, Fasano, Gaggioli, Vigna, & Riva, 2008 ), and even computer games ( Wilkinson, Ang, & Goh, 2008 ) to aid in treatment and recovery.

Computers, whether they make use of the Internet or not, can be used effectively for psychoeducation, and they appear to be at least as effective as printed materials according to studies conducted in the United States, Finland, and Australia ( Finkelstein, Lapshin, & Cha, 2008 ; Pitkänen et al., 2012 ; Proudfoot et al., 2007 ; Välimäki, Hätönen, Lahti, Kuosmanen, & Adams, 2012 ). Psychoeducation can be delivered by computer in such a way that clients can be exposed to material at their own pace or in the formats that are most appropriate for their individual learning styles ( Newman, Koif, Przeworski, & Llera, 2010 ). Online interventions can also be used to improve specific areas of functioning for people with mental illness. For example, van der Zanden, Speetjens, Arntz, and Onrust (2010) reported on an online course in the Netherlands to teach parenting skills to parents with mental illness. Although the dropout rate in the pilot study was high (only 58 percent completed the posttreatment assessment), those who did complete the intervention had significant improvements in parenting skills and parental competence.

A number of research reviews have found that computerized interventions can be effective at treating a variety of mental and substance use disorders with a greater level of effectiveness than no-treatment controls and, in a number of cases, a level of effectiveness comparable with in-person treatments ( Andrews, Cuijpers, Craske, McEvoy, & Titov, 2010 ; Barak, Hen, Boniel-Nissim, & Shapira, 2008 ; Chen et al., 2012 ; Cuijpers et al., 2009 ; Green & Iverson, 2009 ; Kaltenthaler et al., 2006 ; Kiluk et al., 2011 ; Moore, Fazzino, Garnet, Cutter, & Barry, 2011 ). Web-based interventions have also been found to be effective at changing such behaviors as those related to diet, exercise, and risky sexual activity ( Wantland, Portillo, Holzemer, Slaughter, & McGhee, 2004 ).

Barak et al. (2008) conducted a meta-analytic review of 64 studies involving interventions that addressed a variety of behavioral health concerns and found Web-based interventions to be about as effective as those delivered in person. Of the interventions included in that meta-analysis, those using CBT appeared to be the most effective compared with those using psychoeducational or purely behavioral approaches. The authors also found that there was a significantly greater effect size for interventions that were delivered individually compared with a group therapy format and that interventions appeared to be more effective for clients ages 19 to 39 compared with those who were 40 or older. Web sites that were interactive also appeared to be more effective than those where users passively received information and/or instructions. However, a later review by Hanley and Reynolds (2009) , which focused on text-based online therapy only (see discussion later in this section), cautioned that Barak and colleagues' (2008) conclusions on the greater effectiveness of CBT interventions delivered online reflected a more general bias in research toward more technical and less relational interventions (as the former are easier to research).

Similarly, a review of computerized CBT interventions by Green and Iverson (2009) found good evidence to support the use of such interventions for anxiety disorders, depressive disorders, eating disorders, smoking cessation, and problem drinking. The authors also noted that although data are limited, the available research indicates that these interventions will perform as well in community settings as they do in research trials. Another review by Kiluk et al. (2011) of 75 randomized controlled trials that focused on the methodological soundness of computer-assisted interventions for mental and substance use disorders found some evidence that interventions delivered via computer can be effective. The authors found that computer-assisted interventions were more effective than waitlist controls in 88 percent of the studies, more effective than placebo conditions in 65 percent of the studies, and more effective than active control conditions in 48 percent of the studies. The authors did not find any significant differences in effectiveness across four different categories of target problems (depression, anxiety, nicotine dependence, and substance use disorders). Studies that used worse methodologies were significantly more likely to find computerized interventions more effective than control conditions than studies that used better methodologies.

A major problem involved in comparing Web-based interventions with one another and with other types of interventions is that research quality varies considerably among studies, and most studies have methodological problems ( Cunningham & Van Mierlo, 2009 ; Kiluk et al., 2011 ). In Kiluk and colleagues' (2011) review, each of the 75 studies was rated according to how well it met 14 different criteria of methodological soundness. None of the studies met the minimum standard for all of the criteria, only three met 13 of the criteria, and the mean quality score was 13.6 (out of a possible 28).

Most recent research does indicate that Web-based interventions have significantly lower costs than do traditional treatments, thus making them more likely to be cost-effective ( Hedman et al., 2011 ; Mitchell, Stanimirovic, Klein, & Vella-Brodrick, 2009 ). Tate, Finkelstein, Khavjou, and Gustafson (2009) reviewed eight studies of Web-based interventions that provided data on cost savings and concluded that such interventions are more cost-effective than traditional services. These authors also discussed some of the specific cost considerations involved in the development and use of Web-based behavioral health services. An earlier review by Palmqvist, Carlbring, and Andersson (2007) , based on fewer studies, also found that such interventions promised to be more cost-effective. Both reviews, however, included a number of studies from outside the United States, where different approaches to healthcare might affect costs, as well as studies that did not include all startup costs.

For most mental and substance use disorders, some therapist contact is optimal, but Web- and computer-based treatments can reduce the amount of time a therapist needs to provide care and reach clients who might not otherwise seek or be able to access care ( Andersson, 2009 ; Andrews, Davies, & Titov, 2011 ; Kiropoulos et al., 2008 ). Research has consistently found that Web-based treatment is less labor intensive for staff members, and many studies have found relatively low time requirements for the staff members involved in service delivery ( Andrews et al., 2011 ; Kiropoulos et al., 2008 ; Marks, Kenwright, McDonough, Whittaker, & Mataix-Cols, 2004 ).

Web-based interventions can be either guided or unguided. In guided interventions, a therapist or other staff member communicates with the client to assist him or her in using the online intervention (e.g., by explaining homework assignments, giving feedback about progress, reminding clients to complete certain tasks), whereas in an unguided intervention, the client only interacts with the software or other self-guided materials (e.g., automated emails, published literature; Furmark et al., 2009 ; Watkins, Smith, Kerber, Kuebler, & Himle, 2011 ). Some interventions have found better results when even a minimal amount of contact in the form of simple reminders was added to an online treatment (e.g., Clarke et al., 2005 ; Moritz, Schilling, Hauschildt, Schröder, & Treszl, 2012 ). Interventions vary as to the type and amount of guidance provided by staff members (as well as the type of staff member providing that guidance), and those factors are likely to be important in determining whether a guided intervention will be more effective than an unguided one. In their review of Web-based therapeutic interventions, Palmqvist and colleagues (2007) concluded that greater therapist involvement is associated with larger effect sizes. However, other studies involving treatments for social anxiety disorder (SAD; Furmark et al., 2009 ; Berger, Caspar et al., 2011 ) and depression ( Berger, Hämmerli, Gubser, Andersson, & Caspar, 2011 ; Farrer, Christensen, Griffiths, & Mackinnon, 2011 ) have not found any significant difference in outcomes between interventions that were guided and those that were unguided.

The effect of therapist contact may also vary according to the type and frequency of contact. Klein and colleagues (2009) did not find any significant differences in outcomes for people receiving a Web-based treatment for panic disorder when they received three emails a week from their therapist instead of one email. However, an evaluation of an online program to facilitate recovery from bipolar disorder found that adding email communications from a peer coach did significantly improve initial and long-term use of the program ( Simon, Ludman, et al., 2011 ). In a large ( N =2,005) trial of a Web-based smoking cessation intervention, the addition of proactive phone calls from counselors to an interactive intervention was associated with significantly better long-term abstinence rates than were found with users of either a static or an interactive intervention alone ( Graham et al., 2011 ). Also, European research on individual differences among therapists relating to outcomes in Web-delivered CBT interventions for people with major depressive disorder failed to find any relationship between such factors and changes in depressive or anxiety symptoms ( Almlöv, Carlbring, Berger, Cuijpers, & Andersson, 2009 ). From this small study, the authors concluded that these therapist factors likely play less of a role in Web-delivered interventions than they do in ones delivered in person.

Internet technology has been used to deliver prevention, screening and assessment, early intervention, acute care, and recovery support services for a wide variety of behavioral health problems. Although most studies focus on one particular type of behavioral health intervention, Webb, Joseph, Yardley, and Michie (2010) conducted a meta-analytic review of 85 Web-based behavioral health interventions that sought to effect a specific behavioral change (e.g., increase physical activity, decrease alcohol use, promote smoking cessation). They evaluated three different sets of intervention characteristics: the use of theory in designing the intervention, the use of specific behavioral change techniques, and the mode of delivery.

Greater use of theory in developing interventions was associated with a greater effect size for interventions; the biggest effect size was for the use of theory or target constructs as predictors of behavior to select recipients for the intervention. Only three theories were used by enough studies to be evaluated; those theories were, in order of greatest to smallest effect size: theory of reasoned action/planned behavior, the transtheoretical model of readiness for change, and social–cognitive theory.

Brouwer et al. (2011) reviewed literature on effective program characteristics of Web-based lifestyle promotion interventions, which included smoking cessation and drinking reduction, as well as interventions targeting diet and exercise. They found that having email and/or phone contact with users and providing regular updates concerning Web site content were both associated with a greater number of logins to the site, whereas providing peer and/or counselor support was associated with increased time spent using the site. Research regarding the preferred methods for Web site design for behavioral health programs/interventions is limited, but Danaher, McKay, and Seeley (2005) discussed different possibilities for design (i.e., the matrix, tunnel, hierarchical, and hybrid models) and how each model may be best used for certain behavioral change goals. In a European survey of experts on different aspects of Web-based healthcare and related topics (e.g., e-commerce, Web site development, Web design), the majority of respondents believed that potential clients would initially be most motivated to use such Web sites if they saw them as personally relevant; would be most likely to extend the time spent using a site if they received tailored feedback, found the information provided to be reliable and relevant, and were able to navigate the site without much difficulty; and would be more likely to return to the site if they expected to see new content and were given the opportunity to monitor their own progress toward behavioral health goals (Brouwer et al., 2008).

Rotondi et al. (2012) evaluated the effectiveness of different Web site design elements for clients with co-occurring substance use disorders and serious mental illness. They varied 12 different design factors and found large differences in users' abilities to complete tasks on the site according to the elements used. Users were more likely to complete online tasks successfully on sites that had a shallow hierarchy (i.e., fewer pages to navigate), fewer hyperlinks per page, fewer topic areas, fewer words per page, no graphics, and no tool bars.

Web-based and computerized interventions appear to be a better option for clients with less severe mental illness than for those with more severe mental illness. In an Australian study, Sunderland, Wong, Hilvert-Bruce, and Andrews (2012) analyzed data from clients who completed online CBT treatments for either depression ( n =302) or generalized anxiety disorder (GAD; n =361) to evaluate factors associated with treatment response. They found that these treatments were effective for the majority (75 to 80 percent) of clients, but individuals who did not respond to treatment had significantly higher levels of symptom severity and psychological distress prior to treatment than did those who did respond. However, in spite of the promise Web-delivered programs and interventions hold (especially for certain populations), there are a number of difficulties involved in developing and implementing such interventions ( Cunningham & Van Mierlo, 2009 ; Kiluk et al., 2011 ). There can be other problems involved in the development of such interventions ( Cunningham & Van Mierlo, 2009 ; Danaher et al., 2005 ) as well:

• There can be difficulties in adapting existing interventions to this method of delivery.
• People may respond differently in front of a computer than they do to another person.
• Greater distractions may exist for someone using the Internet than in a one-on-one setting.

Some counselors have observed that developing a therapeutic alliance may be difficult with online interventions, given the impersonal nature of computer-based interactions ( Callan & Wright, 2010 ). However, Hanley and Reynolds (2009) reviewed five studies that provided data on 161 clients who received online therapy, four of the five of which made comparisons between in-person and Web-delivered therapies. Overall, participants in those studies receiving Web-based services perceived their relationship with the counselor delivering those services to be moderate or high in strength (in most cases, measured with the Working Alliance Inventory, and in one case, with the Agnew Relationship Measure). The authors concluded, based on these data and clinical evidence from other sources, that a good-quality relationship can be developed between counselors and clients working online and that such relationships have the strength necessary to produce therapeutic change. A more recent review by Sucala et al. (2012) , which included 11 articles relating to a variety of different types of online therapies, also found that evidence appeared to indicate that clients and counselors could develop a strong therapeutic alliance online, although the evidence was not conclusive.

Kang and Gratch (2011) found that clients working with virtual counselors online expressed a preference for counselors who self-disclosed a high level of intimate information about themselves (as opposed to those who disclosed a medium or low level of personal information). Some participants (i.e., those who did not consistently engage in either a high or low level of self-disclosure) also revealed more information about themselves to virtual counselors who self-disclosed high levels of personal information.

One of the potential benefits of Web-based interventions over self-guided interventions that relay information on paper is that the former can tailor the information to the specific needs of the client, and many of the Web-based interventions discussed later in this section provide some sort of tailored content. To describe different mechanisms and types of tailoring, Lustria, Cortese, Noar, and Glueckauf (2009) reviewed 30 Web-based interventions that provide tailored content. Web-based interventions can also make use of multiple media, potentially increasing the impact of interventions and improving clients' ability to learn from presented materials ( Villani & Riva, 2012 ).

Ritterband, Thorndike, Cox, Kovatchev, and Gonder-Frederick (2009) proposed a behavioral change model for Web-delivered interventions that may be of help in understanding how Web-delivered services effect client change. Their model involves a nine-step process in which (1) the user of a Web-based intervention is (2) influenced by environmental factors (e.g., ease of access to the Internet, opinions of family/friends) that then affect (3) the user's use of the Web site and compliance with treatment, while that use is also affected by (4) support (e.g., email reminders from staff people) and (5) Web site characteristics, such as methods of engaging users and presenting content. The use of the Web site subsequently leads to (6) behavior change and then (7) symptom improvement occurring through (8) mechanisms of change (e.g., motivation, attitudes, beliefs, self-efficacy, self-monitoring). These improvements in symptoms are sustained through (9) treatment maintenance activities.

One development in computer technology that has received a good deal of attention in behavioral health is the use of virtual reality (VR) software. Different reviews have found that VR exposure therapy (VRET) can effectively treat PTSD ( Gerardi, Cukor, Difede, Rizzo, & Rothbaum, 2010 ; McLean, Steenkamp, Levy, & Litz, 2010 ) and at least some specific phobias ( Gerardi et al., 2010 ; Newman, Szkodny, Llera, & Przeworski, 2011a ; see also the “ Use in Treatment of Anxiety Disorders ” section). Meyerbröker and Emmelkamp (2010) reviewed only controlled studies of VRET and concluded that, for fear of heights or flying, there was good evidence of its effectiveness, although they also observed that only limited research indicated that VRET may also be effective for panic disorder, seasonal affective disorder, and PTSD. See Part 1, Chapter 1 of this TIP for more on VR/VRET.

In addition to its use in exposure therapy, VR technology has potential uses in clinical role-playing, efficacy-building exercises, and skills training and practice (Botella et al., 2004). This technology has also been used to treat a range of other behavioral health problems, including eating disorders ( Ferrer-García & Gutiérrez-Maldonado, 2012 ), male sexual dysfunction ( Optale et al., 2004 ), nicotine dependence ( Moon & Lee, 2009 ), alcohol use disorders ( Lee, Kwon, Choi, Yang, 2007 ), and other health problems that have a behavioral component, such as obesity and diabetes ( Morie & Chance, 2011 ). See Part 1, Chapter 1 of this TIP for more on other uses of VR.

Alcañiz, Lozano, and Rey (2004) explained some of the technical aspects of VR as it is used in medical and behavioral health settings, with particular attention to the hardware required to set up a VR environment. The cost and potential difficulties involved in developing virtual environments (i.e., three-dimensional simulations of real or imagined scenarios) have been factors that limit the use of VR technology. Riva and colleagues (2007) reported on the development of an open-source VR platform that enables providers to design and deliver new VR environments according to their clients' specific needs. Virtual worlds can also be used to assist in behavioral health. Morie (2009) described a U.S. Army program that uses the Second Life virtual world to help personnel returning from deployment socialize and learn about available behavioral health resources. The U.S. Department of Defense (DoD) has also created a virtual clinic to provide treatment for PTSD within the Second Life environment ( Yellowlees, Holloway, & Parish, 2012 ).

As an alternative to the time and expense of VR, Bledsoe and Simmerok (2014) offered what they called “augmented reality.” Although it was designed to be used via the Internet for distance learning or other educational programs, rather than for therapeutic purposes, their augmented reality amounts to a rich multimedia platform constructed with low- or no-cost, readily accessible ways to make any online experience more engaging. For example, the authors took a picture of their college counseling center, made their Web site look like the picture, and added elements to the picture, along with videos and audio clips that were all designed to be related to the information and to be used as materials in the delivery of their educational course. Such augmented reality could also make Web sites for therapeutic interventions more engaging and effective without incurring the effort and expenses required to produce a VR capability.

Text-based communications include a variety of technologies (e.g., text messaging, email, Internet chat rooms) that allow for simple written communication between providers and clients or, in the case of mutual-help groups and activities, among clients. For the most part, these communications support activities occurring elsewhere in person, by computer, or using video/audio communication technology ( Maheu et al., 2004 ). Counselors can use email to conduct therapy or as an adjunct to in-person therapy ( Recupero & Harms, 2010 ), or they can use an online chat program or instant messaging for the same purposes ( Derrig-Palumbo, 2010 ). Although research on evidence-based behavioral health interventions using social media is not available yet, online social networks also show promise as platforms for text-based behavioral health interventions ( Levine et al., 2011 ).

Not much research has evaluated Web-based text communications in behavioral health services, especially as stand-alone interventions. Atherton, Sawmynaden, Sheikh, Majeed, and Car (2012) reviewed the literature on the use of email for clinical communication in a variety of healthcare settings and found nine controlled trials, but most were related to communication in nonbehavioral health settings. They were also unable to draw conclusions about the effectiveness of such communications because of the poor quality of the research. A single study that compared telephone and email contact did find the former to be more effective, but it also had significant methodological problems. An earlier review of text-based online therapy by Hanley and Reynolds (2009) concluded that despite very limited evidence, such interventions showed a great deal of promise.

There is some doubt about the strength of therapeutic alliances built through text-based therapy, as not much research on the subject has been conducted. However, a study by Reynolds, Stiles, and Grohol (2006) found that clients receiving email therapy rated the impact of their sessions and the strength of their therapeutic alliances about as highly as clients receiving in-person counseling. A review by Hanley and Reynolds (2009) , which included five studies of therapeutic alliances in text-based therapies, also found that therapeutic alliances developed in this medium were about as strong as those developed through in-person therapy. However, one of the studies cited in those reviews did find that therapeutic alliances tended to be rated stronger when Internet chat was used than when email was used for text communication ( Cook & Doyle, 2002 ).

Specific studies have found that text-based interventions can be effective in the treatment of eating disorders ( Robinson & Serfaty, 2008 ), depression ( Vernmark et al., 2010 ), smoking cessation ( Polosa et al., 2009 ; Te Poel, Bolman, Reubsaet, & de Vries, 2009 ), and alcohol use disorders ( Blankers, Koeter, & Schippers, 2011 ); they can likewise be effective for people with schizophrenia and their families/support systems ( Rotondi et al., 2010 ). Text-based interventions that use email and/or chat room discussions have also been effective in promoting weight loss, and Tate (2011) discussed how this research can inform similar interventions in substance use disorder treatment. A German study involving 114 individuals who had completed inpatient treatment for a mental disorder and participated in text-based continuing care groups and 114 matched controls who did not use text-based continuing care found that participants in the text-based groups had a significantly lower risk for negative outcomes (according to a composite measure of behavioral and physical health) than did those in the control group ( Golkaramnay et al., 2007 ). Participants in the chat groups were also more likely (77 percent), but not significantly so, to maintain improvements made during treatment than were those in the control group (65.2 percent). The intervention also had a relatively low dropout rate and a high level of attendance.

Text components are often part of larger interventions. For example, researchers evaluating an online recovery support intervention for people with bipolar disorder found that the addition of an email communication component significantly increased the odds that a participant would return to the Web site and would use it for a longer period of time ( Simon, Ralston, et al., 2011 ). However, an evaluation of a Web-based smoking cessation intervention found that although the addition of an online discussion group increased use of the site, increased use did not translate into significantly better outcomes ( Stoddard, Augustson, & Moser, 2008 ). Text and numerical data, transmitted via computer or over telephone lines, can also be used for symptom monitoring and/or ongoing assessment ( Godleski et al., 2012 ).

Although research evaluating the effectiveness of online text-based peer discussion groups is limited, these groups also appear to be valuable as sources of information and support for individuals with a variety of mental and substance use disorders, including eating disorders ( Eichhorn, 2008 ; Fernández-Aranda et al., 2009 ), depression ( Griffiths, Calear, Banfield, & Tam, 2009 ; Houston, Cooper, & Ford, 2002 , Melling & Houguet-Pincham, 2011 ), psychosomatic disorders ( Haug, Sedway, & Kordy, 2008 ), and substance use disorders ( Hall & Tidwell, 2003 ). In addition, a number of the interventions discussed in the “ Promise of Technology for Specific Populations ” section also include a group chat or bulletin board for more informal group discussions and peer support ( An, Klatt, et al., 2008 ; Stoddard et al., 2008 ; Titov, Andrews, Schwencke, et al., 2009 ). For more information on Web-based peer support, see the “ Peer Support/Mutual-Help Groups ” section.

Support groups with a professional facilitator may benefit clients more than those with peer moderators ( Barak, Boneh, & Doley-Cohen, 2010 ). Also, research suggests that, for some clients (e.g., those with serious mental illness), unmoderated support discussion groups can actually have a detrimental effect ( Kaplan, Salzer, Solomon, Brusilovskiy, & Cousounis, 2011 ). Suler (2004) and Anthony (2004) discussed the psychology of text-based interactions, with particular attention to how they affect therapeutic relationships. Alemi et al. (2007) described their own practice using email communication to support substance use disorder treatment services and used their experiences to write some guidelines for others who wish to incorporate email into existing treatment programs.

Mobile technologies include a variety of handheld and mobile devices for communicating information. Currently, the term is most often used to refer to mobile phones—both smartphones (handheld computers that can run software like a computer) and feature phones (which are used only to communicate via audio and sometimes text). The use of mobile devices is now very widespread; many people can access Internet and/or phone service only through such devices ( International Telecommunication Union, 2012 ). For certain populations (e.g., people who are homeless), mobile devices may be the only reliable method clients have for receiving phone and/or Internet communications ( Eyrich-Garg, 2010 ; Rice, Lee, & Taitt, 2011 ). Such devices can be used to receive phone calls, access Web-based interventions, or send/receive text and/or numerical data.

Mobile technology has been used successfully to assess cravings in individuals with substance use disorders ( Ferguson & Shiffman, 2011 ; Freedman, Lester, McNamara, Milby, & Schumacher, 2006 ), aid in continuing care/relapse prevention for people with substance use disorders ( McTavish, Chih, Shah, & Gustafson, 2012 ), aid in smoking cessation ( Whittaker et al., 2012 ), assist in continuing care for people with eating disorders ( Bauer, Percevic, Okon, Meermann, & Kordy, 2003 ; Robinson et al., 2006 ), supplement treatment for borderline personality disorder (BPD; Rizvi, Dimeff, Skutch, Carroll, & Linehan, 2011 ), monitor suicide risk for veterans ( Rimoldi, Lewis, & Jampala, 2012 ), monitor PTSD symptoms for veterans ( Smith, Harms, et al., 2012 ), monitor mood for people with BPD ( Bopp et al., 2010 ), and monitor symptoms and manage medication for people with schizophrenia ( Granholm, Ben-Zeev, Link, Bradshaw, & Holden, 2012 ; Sablier et al., 2012 ).

Shiffman (2009) reviewed research on ecological momentary assessment (i.e., the real-time assessment of mood, behavior, symptoms, and so forth, using portable devices) for clients in substance use disorder treatment and smoking cessation programs. He observed that research trials have demonstrated good compliance but that little has been done in the way of external validation to evaluate the accuracy of this method of reporting for these populations. Heron and Smyth (2010) reviewed a number of studies involving the use of mobile technology to treat clients in real time and in real-world settings (also known as ecological momentary interventions). They concluded that, taken together, research studies indicate that this is an effective mode of treatment for a variety of behavioral health problems (including substance-related cravings, eating disorder symptoms, and anxiety disorder symptoms) as evaluated with a diverse group of participants.

Boschen and Casey (2008) reviewed pre-2008 literature on the use of mobile devices in psychotherapy. At that time, they were only able to locate seven (mostly small) studies, but they did conclude that this technology was promising, particularly for CBT interventions. A more recent review of the research involving handheld devices, including mobile phones, in behavioral healthcare observed that such devices have been found to be effective in the treatment of anxiety and nicotine dependence ( Ehrenreich, Righter, Rocke, Dixon, & Himelhoch, 2011 ). Mobile devices have also been used effectively to promote behavior change in relation to general health concerns, such as aiding in weight loss for individuals who were overweight ( Gerber, Stolley, Thompson, Sharp, & Fitzgibbon, 2009 ; Patrick et al., 2009 ) and for HIV prevention with young men who are sexually active ( Juzang, Fortune, Black, Wright, & Bull, 2011 ). Fjeldsoe, Marshall, and Miller (2009) reviewed studies involving the use of mobile phones to deliver text messages for a variety of behavioral changes relating to health problems, such as asthma, hypertension, and diabetes, and Nundy et al. (2014) successfully used mobile phone text messaging to improve glycemic control in employees with diabetes.

• Promise of Technology for Specific Populations

Web- and phone-based interventions may be able to reach potential clients who are not currently being served by the behavioral health system because of cost, availability of services, accessibility, or other reasons ( Alleman, 2002 ; Andersson, 2009 ; Callan & Wright, 2010 ; Postel, De Haan, ter Huurne, Becker, & de Jong, 2011 ). This section details some of the populations who may especially benefit from such interventions.

Many potential clients also express a definite interest in, if not a preference for, such interventions. Mohr et al. (2010) surveyed 658 primary care patients regarding treatment preferences. Of those respondents who expressed an interest in behavioral health services ( n =492), 18.7 percent were definitely interested in telephone-based treatment, and 43.7 percent would consider it, whereas 11.6 percent were definitely interested in Web-based treatment, and 36.4 percent would consider it. Individuals who cited time constraints as a potential barrier to treatment seeking were significantly more likely to be interested in telephone- or Web-based interventions. Computerized interventions are also cost-effective, and they may be indicated when a client cannot afford many in-person sessions but still requires some form of continued contact or treatment ( Newman et al., 2010 ).

Rural Populations

Given the difficulty of accessing trained professionals in their communities, people living in rural, frontier, or remote areas may benefit from services provided via telephone or the Web ( McGinty, Saeed, Simmons, & Yildirim, 2006 ). According to a survey of VA patients and providers living and/or working in rural areas, distance was most often cited as the greatest barrier to treatment ( Buzza et al., 2011 ); Internet and phone technologies can help overcome this barrier. The VA has successfully instituted a number of services for veterans living in rural areas. For example, a review of VA's American Indian Telemental Health Clinics indicated that telemental health services (mental health services delivered using various telecommunication technologies) provided by these clinics were well-received by clients, generally showed diagnostic reliability, and were less expensive than the same services provided in person ( Shore et al., 2012 ).

A report on focus groups conducted with users of telemental health services and their therapists in frontier areas found that both therapists and consumers expressed a high level of satisfaction with such services ( LaMendola, 2000 ). Consumers, however, had concerns that these services may not be paid for by insurers. Studies of Web-based interventions for people in rural areas have also generally found these interventions to be acceptable to this group of clients ( Finfgeld-Connett, 2009 ; Griffiths & Christensen, 2007 ; Stoops et al., 2009 ).

People With Disabilities

Various phone- and Web-based interventions can help extend care to people with disabilities, who may otherwise have problems accessing appropriate care because of physical or cultural factors. For example, in many parts of the country, people who are Deaf may not have access to programs with staff members who are fluent in American Sign Language; Web-based interventions may help improve their access to such staff ( Titus & Guthmann, 2010 ). There have been trials of self-directed, Web-based smoking cessation ( Jones, Goldsmith, Effken, Button, & Crago, 2010 ) and substance use disorder treatment programs ( Moore, Guthmann, Rogers, Fraker, & Embree, 2009 ) for people who are Deaf; both are discussed in more detail in the “ Use in the Treatment of Smoking/Smokeless Tobacco Use ” section. Pollard, Dean, O'Hearn, and Haynes (2009) observed that health-related materials for people who are Deaf can be improved using video, as English is a second language for many. See Vignette 4 , “Incorporating TAC Into Behavioral Health Services for Clients Who Are Hearing Impaired,” in Part 1, Chapter 2 of this TIP.

Web-based behavioral health interventions can also be targeted to people with specific physical illnesses. For example, van Bastelaar, Pouwer, Cuijpers, Riper, and Snoek (2011) reported on a Dutch Web site designed to reduce depression among people with type 1 or 2 diabetes. In a randomized controlled study, use of the site was associated with significantly greater reductions in symptoms of depression and diabetes-specific emotional distress than were found in a waitlist control group. Another Web-based intervention is also being tried in the Netherlands to reduce depressive symptoms among people with multiple sclerosis (MS; Boeschoten et al., 2012 ). In this pilot study, participants who used the Web site experienced significant decreases in symptoms of depression (measured with the Beck Depression Inventory, second edition [BDI-II]).

American Academy of Neurology (AAN) Guideline on Telephone-Administered CBT for MS

The Guideline Development Subcommittee of the AAN convened a panel of experts to make recommendations for evidence-based interventions for MS. The panel concluded that a 16-week telephone-administered CBT program (which AAN endorsed) “is possibly effective and may be considered in treating depressive symptoms” in people with MS; AAN endorsed this conclusion. (Minden et al., 2014). The weekly 50-minute telephone calls provide CBT to help clients with MS change thought processes and behaviors that reinforce depressive symptoms, manage stress, and deal with interpersonal and other problems and situations ( Dolan, 2014 ).

Web-based interventions may also improve services for clients with cognitive deficits, as one study conducted with 160 participants in a methadone maintenance program found better abstinence outcomes among individuals with greater impairment in cognitive functioning when they used a Web-based intervention than when they received services delivered in person ( Acosta, Marsch, Xie, Guarino, & Aponte-Melendez, 2012 ). Research from outside the behavioral health field indicates that phone and Internet messaging can help both patients and their caregivers engage in tasks that can improve behavioral as well as physical health, such as practicing relaxation training and using compensatory strategies relating to cognitive deficits ( Forducey, Glueckauf, Bergquist, Maheu, & Yutsis, 2012 ). Although it is targeted at online educators, an October 2013 special issue of the Journal of Asynchronous Learning Networks (Volume 17, Issue 3) focuses on considerations related to accessibility and disabilities. These same concerns are relevant to the provision of behavioral health services online.

Students/Young Adults

College students and other young adults may be more interested in Web-delivered interventions, as their comfort level is fairly high with the technology, and they often seek health information and social support through the Internet ( Wyn, Cuervo, Woodman, & Stokes, 2005 ). As of 2014, 97 percent of adults ages 18 to 29 used the Internet either through a computer or a mobile device, and 98 percent owned a cell phone ( Pew Research Center, 2014 ). Use of the Internet is even more common among college students ( Smith, Rainie, & Zickuhr, 2011 ). Other research confirms that college students often look to the Internet as a source of health information ( Stellefson et al., 2011 ).

In one study of a screening and brief intervention for problem drinking delivered to first-year college students, 41 percent of participants who were screened as potentially having a drinking problem expressed a preference for getting further information about drinking over the Internet, and 6 percent expressed a preference for getting information by phone ( Saitz et al., 2007 ). A number of prevention and early intervention programs targeting college students have been developed and are discussed in the “ Technology To Aid in Substance Use Disorder Prevention ” section. It should also be noted that although younger people may be more technically savvy and interested in these technologies, behavioral health programs using such technologies have been successfully implemented for people of all ages, including older adults ( Ramos-Ríos, Mateos, Lojo, Conn, & Patterson, 2012 ; Westphal, Dingjan, & Attoe, 2010 ).

There are some indications that women may have a greater preference for, and be more likely than men to engage with, computer-delivered interventions. In an evaluation of the acceptability of a computer-delivered CBT intervention for depression, women were significantly more likely to have a favorable response than were men; no relationship between age and treatment acceptability was found ( Cavanagh et al., 2009 ). In that study, a positive expectation prior to entering treatment was associated with treatment completion, but not with treatment outcomes.

Studies on smoking cessation programs (for more detail, see the “ Use in the Treatment of Smoking/Smokeless Tobacco Use ” section) have also found that, in a multifaceted program, women were significantly more likely than men to use both phone and Web components of the program ( Zbikowski, Hapgood, Barnwell, & McAfee, 2008 ). In an intervention with only a Web component, women engaged more extensively than men with the site ( Strecher et al., 2008 ).

Tsan and Day (2007) evaluated gender differences in attitudes toward different modes of behavioral health services (e.g., in person, video conferencing) in a group of 176 college students. They found that women held significantly more positive attitudes than men toward in-person and email counseling but not toward counseling via instant messaging, video conferencing, or voice-only communication via the Internet. Women may also benefit more than men do from some Web-based interventions. For example, female college students who used the MyStudentBody-Alcohol program drank significantly less on special occasions and had fewer negative consequences as a result of drinking after participation in the program, but that was not the case for men. In another computerized screening and brief intervention program for problem drinking among college students, women whose drinking was problematic experienced greater reductions in alcohol use than their male counterparts ( Saitz et al., 2007 ).

Researchers in the Netherlands also found that women were substantially more likely to have a positive response when using a Web-based intervention for drinking than were men ( Riper et al., 2008 ). Another Dutch study of a Web-based intervention for subclinical depression found that women had significantly better outcomes than did men ( Spek, Nyklíček, Cuijpers, & Pop, 2008 ). Particular groups of women have also been successfully targeted with computer-based interventions, such as women at risk for an alcohol-exposed pregnancy ( Tenkku et al., 2011 ) and postpartum women ( Ondersma, Svikis, & Schuster, 2007 ). Lipman, Kenny, and Marziali (2011) reported on a pilot study that used video conferencing to conduct support groups for single mothers with low incomes; they observed that the intervention was well-received, but the study was not large enough to detect significant changes in measures of behavioral health.

People Who Are Homeless

Communication technology can help providers reach people who are homeless. For example, a pilot study was conducted with 30 individuals who were homeless and in outpatient treatment for cocaine use disorders. Each individual received a cell phone to provide real-time information concerning cravings and substance use ( Freedman et al., 2006 ). The authors found that people who are homeless could reliably use cell phones for this purpose.

Eyrich-Garg (2010) reported on the feasibility of using cell phones for prevention and treatment with people who are homeless and not using the shelter system (hence, not easily reachable through the usual channels of service provision to people who are homeless). In a sample of 100 such individuals in Philadelphia, 44 percent already had cell phones, and 20 percent used their cell phones to access the Internet. There were no significant differences in cell phone ownership between those who had prior substance use disorder treatment and those who had none, and there were minimal differences in regard to the prevalence of mental illness. A survey conducted in Los Angeles of 169 adolescents and young adults who were homeless found that 62 percent owned cell phones, and 40 percent had phones that were currently operational ( Rice et al., 2011 ).

Members of Specific Cultural Groups

Individuals who belong to cultural groups that have difficulties finding culturally and linguistically appropriate services from local providers may also benefit from telephone- and Web-based services that can connect them to culturally competent providers ( Mucic, 2010 ; Skinner & Latchford, 2011 ).

Skinner and Latchford (2011) and Shore, Savin, Novins, and Manson (2006) discussed the provision of culturally responsive behavioral health services using phone and Internet technology. Data evaluating the use of these technologies to provide culturally responsive services at a distance is scarce. However, Mucic (2010) reported on a Danish pilot study that used video conferencing technology to connect behavioral health clients (largely asylum seekers/refugees) to therapists who spoke their language and understood their cultural background. Participants in the pilot program ( N =61) were largely comfortable with the procedure (75 percent reported no discomfort with the program), and most (85 percent) preferred it to local psychiatric services using an interpreter. Choi et al. (2012) described an 8-week Web-delivered CBT depression treatment for Chinese Australians and reported that its use was associated with significant decreases in depressive symptoms that persisted 3 months after treatment ended. They also observed that participants found the treatment acceptable and that 68 percent completed all online lessons.

McDonnell, Kazinets, Lee, and Moskowitz (2011) evaluated a Web-based smoking cessation intervention for Korean Americans, a population with a high rate of smoking; a Korean American community partner believed that members of this population were often reluctant to participate in in-person cessation programs. Participants either used the online intervention ( n =562) or received similar information via printed materials ( n =550). Although 30-day cessation rates did not differ significantly between the two groups, 26 percent of those who completed the online intervention had quit for at least 30 days at the follow-up assessment conducted 50 weeks after enrollment, whereas just 10 percent of those who did not complete the intervention had done so.

Groups Less Suitable for Web- and/or Phone-Based Interventions

Some technologies may be unsuitable or less suitable for certain clients. Providers of services to clients in frontier areas have observed that individuals with paranoid delusions, for example, may find some of the technology used in these interventions disturbing ( LaMendola, 2000 ). Individuals with poor reality testing, strong transference reactions, and problems with impulsivity/aggressiveness may also have difficulties engaging in online therapy ( Suler, 2001 ).

People with certain personality disorders may also fare better with in-person treatment than with treatment delivered via communication technologies. A reanalysis of data from a study that compared Web-based and in-person interventions for panic disorder found that symptoms of a personality disorder (in the avoidant anxious category) were associated with significantly worse outcomes for Web-based treatment than in-person treatment ( Andersson, Carlbring, & Grimlund, 2008 ). The authors suggested that this is because it may be easier for such clients to “repair misunderstandings” when in the counselor's office. Many research studies of these interventions exclude participants who lack reading skills or have problems with written comprehension, but as Andersson (2009) noted, multimedia technology may make this limitation irrelevant.

Although people with lower socioeconomic status (SES) are less likely to have access to computer technology, and hence are less able to benefit from TAC, access is expanding relatively quickly ( McNeill, Puleo, Bennett, & Emmons, 2007 ). Programs are in development to create public computer centers for underserved populations ( National Telecommunications and Information Administration, 2010 ). Due to the recent increase in use of smartphone technology, many people with lower SES now have regular access to the Internet, and rates of access using that technology are likely to continue to increase ( Fox & Duggan, 2012 ).

Suler (2001) summarized recommendations from the International Society for Mental Health Online clinical case study group regarding individuals' potential suitability for online therapy. Among other factors, he suggested that counselors consider client preferences regarding communication methods, the client's knowledge about the use of the technologies involved, the client's comfort with and knowledge about online communication and relationships, the client's skills and comfort with the medium being used, the client's personality type and specific disorders, and the presence of significant impairments or chronic medical conditions.

• Technology To Aid in Substance Use Disorder Prevention

Many substance use disorder prevention programs use computer and phone technologies (e.g., Web sites) as program components. In fact, such use is so common that it is beyond the scope of this TIP to address it comprehensively, so this section focuses on prevention programs that exclusively use those technologies (typically, Web-based interventions). In addition to these specific interventions, this technology has some more general applications relating to prevention. For example, the Internet can be used to train prevention providers ( McPherson, Cook, Back, Hersch, & Hendrickson, 2006 ) or to improve fidelity of implementation and increase accessibility for existing prevention programs ( Bishop, Bryant, Giles, Hansen, & Dusenbury, 2006 ).

A number of projects to increase capacity for community prevention programs have made use of such technology. Chinman, Tremain, Imm, and Wandersman (2009) examined 18 prevention coalitions in Missouri that were using the Getting to Outcomes (GTO) prevention program with a Web-based component to help programs complete GTO tasks. The authors compared these coalitions with eight coalitions in the same state that were using GTO without the additional component. They found that those that used the Web-based component did significantly better at performing key planning, implementation, and evaluation activities related to GTO.

One area of computer-delivered prevention that has been reasonably well-evaluated is the use of these technologies to help reduce drinking/binge drinking among college students. Carey, Scott-Sheldon, Elliott, Bolles, and Carey (2009) conducted a meta-analysis of 43 separate interventions (from 35 publications) that were intended to reduce alcohol use among college students. They found such interventions more effective at reducing the amount of alcohol consumed on specific occasions in the short term and total alcohol consumption (over periods of weeks or months) in the long term than no-treatment or assessment-only controls. They concluded that such interventions are cost-effective and can reduce drinking in this population, being more effective than no-treatment controls and about as effective as active controls.

MyStudentBody

MyStudentBody.com includes a series of prevention modules aimed at college students, including three specifically oriented toward preventing the use/misuse of alcohol, drugs, and tobacco. Each component has been or is in the process of being evaluated, and other modules (notably MyStudentBody-Stress [MSB-S]) may have an effect in reducing substance use disorders.

For the evaluation of MyStudentBody-Alcohol (MSB-A), 265 students were randomly assigned to use the MSB-A interactive Web site or a Web site that provided research-based articles on drinking ( Chiauzzi, Green, Lord, Thum, & Goldstein, 2005 ). All were assessed using the Daily Drinking Questionnaire, the Rutgers Alcohol Problem Index, and the Readiness to Change Questionnaire at baseline, at the conclusion of the intervention, and 3 months later. Participants in both intervention and control groups reduced alcohol consumption over the course of the study, and there were few differences between the groups as a whole. However, for women only, the use of MSB-A compared with use of the control Web site was associated with significantly less drinking during special occasions and fewer negative consequences as a result of drinking. Also, for persistent, heavy binge drinkers and for participants who had low motivation to change their drinking behavior, use of MSB-A was associated with a more rapid decrease in average and peak alcohol consumption compared with use of the control site.

MyStudentBody-Parent (MSB-P) is a newer intervention aimed at parents of college students to help them communicate with their children about hazardous drinking ( Donovan, Wood, Frayjo, Black, & Surette, 2012 ). In an evaluation involving 558 parents randomly assigned to the intervention or to a control group who received general information about college student drinking and drug use via email, those who participated in MSB-P were significantly more likely to discuss strategies to reduce or avoid alcohol use with their children, and their children were significantly more likely to state that they used such strategies.

College Alc

The College Alc program is a multimedia educational intervention that provides information about drinking norms, the effects of alcohol, and safe drinking practices. College Alc was evaluated (Paschall, Bersamin, Fearnow-Kenney, Wyrick, & Currey, 2006) with first-year college students who were randomly assigned to receive the intervention ( n =173) or to a control group ( n =197). Although there were no significant differences between the two groups in terms of drinking outcomes, those who received the intervention had significantly more knowledge about alcohol, fewer positive attitudes toward alcohol use, and greater expressed intentions to limit alcohol-related harm following the intervention than did those in the control group. In a secondary analysis of these data, Bersamin, Paschall, Fearnow-Kenney, and Wyrick (2007) found that participants who reported drinking in the month prior to the baseline assessment and who received the intervention had decreases in incidents of heavy drinking, drinking to intoxication, and alcohol-related consequences; those who reported past-month drinking and were in the control group had increases in those areas.

Another college-oriented, Web-delivered alcohol prevention program, AlcoholEdu, was designed for use with all students attending a school (instead of a targeted, high-risk group). It has been evaluated at 225 different campuses with 24,877 students who were randomly assigned to the intervention or to no-intervention control groups ( Wall, 2007 ). In this initial assessment of the program, which looked only at those participants who completed a follow-up assessment (49.5 percent of the total), students in the intervention group reported significantly fewer negative consequences of drinking, fewer days of heavy drinking, and less intentionally risky drinking behavior. However, those results need to be considered in light of high participant dropout.

Two more recent evaluations of AlcoholEdu had mixed results. In a randomized trial conducted with 1,891 first-year students, participants who received the intervention had better alcohol knowledge but failed to show better alcohol-related outcomes (with the exception of less frequent participation in drinking games) than did those who were in an assessment-only control group (Croom et al., 2009). In the other study, conducted with 1,620 first-year students randomly assigned to the intervention or to an assessment-only control group, 91.5 percent of those who received the intervention and 67.9 percent of those in the control group completed the 1-month follow-up ( Lovecchio, Wyatt, & DeJong, 2010 ). In this trial, the authors did find that, at the 1-month postintervention assessment, those who used the AlcoholEdu Web site had, compared with those in the control group, significantly fewer negative consequences from drinking and reported significantly less alcohol use. The authors noted, however, that participants who did not complete the study were lighter drinkers in the baseline assessment, and thus larger dropout rates in the control group could have skewed the results.

Yet another evaluation of AlcoholEdu was conducted by Paschall, Antin, Ringwalt, and Saltz (2011) with students from 30 different colleges and universities across the United States (15 of which used AlcoholEdu and 15 of which provided a control group). Compared with students in the control group schools, students at schools providing AlcoholEdu had significantly greater reductions in past-month alcohol use and binge drinking episodes, but the differences did not persist after the semester in which the intervention was delivered. However, in a post hoc analysis, the authors did find a significantly greater effect in schools that had a higher rate of completion for the intervention (often because a school mandated completion).

Hustad, Barnett, Borsari, and Jackson (2010) randomly assigned 150 first-year college students to receive AlcoholEdu or the electronic program Check-Up to Go (e-CHUG; see the next section) or to an assessment-only control group. At a 1-month follow-up assessment, participants in both interventions reported significantly less alcohol use than did those in the control group. Those who used the AlcoholEdu program had significantly fewer negative consequences from drinking (evaluated with the Young Adult Alcohol Consequences Questionnaire) than did those in the control group, but the difference in that outcome measure was not significant for those who used e-CHUG (although the trend was in the same direction).

e-CHUG and Drinker's Check-Up Programs

A third commercially available program developed to reduce alcohol use by college students is the e-CHUG program, which also provides assessment and individualized feedback; it appears to be more promising as a form of indicated prevention than as a universal prevention program ( Hustad et al., 2010 ).

The program was evaluated with a group of 106 first-year students who had previously indicated that they engaged in heavy episodic drinking as well as 245 students who were abstainers or light drinkers ( Walters, Vader, & Harris, 2007 ). Participants were randomly assigned to receive the e-CHUG intervention or to receive periodic drinking assessments alone. Among those who were heavy episodic drinkers prior to the study, use of e-CHUG was associated with significantly greater reductions in drinks per week and estimated blood alcohol content on the day of heaviest drinking at the 8-week assessment compared with assessment alone. However, by the 16-week assessment, there were no longer significant differences between the groups due to drinking reductions among the control group. There were no significant effects on drinking for individuals who were abstainers or light drinkers prior to the study.

In another, smaller evaluation of e-CHUG, 80 first-year students (52 of whom were available for a 3-month assessment) were randomly assigned to receive e-CHUG or assessment alone ( Doumas & Andersen, 2009 ). Among participants who engaged in binge drinking in the 2 weeks prior to the study (and were thus considered high-risk drinkers), those who used the e-CHUG site reported significantly less drinking and had significantly fewer problems resulting from drinking (according to the Rutgers Alcohol Problem Index), but there were no significant differences for participants who were not high-risk drinkers. However, the significance of this finding was equivocal and varied according to the method used to determine significance. Also, individuals who did not complete the study had significantly more alcohol-related problems at baseline than did those who remained in the study, which may have affected findings. Another evaluation of e-CHUG for first-year university students ( N =350) found that those who used the program, compared with those assigned to an assessment-only control group, had significantly greater reductions in days of heavy drinking and in drinking-related negative consequences ( Doumas, Kane, Navarro, & Roman, 2011 ). Also, students who were assessed as being at high risk for alcohol use disorders and who used e-CHUG reduced peak drinking by 58 percent and drinking to intoxication by 65 percent, compared with increases of 11 percent and 15 percent, respectively, for high-risk participants in the control group.

In another study of e-CHUG, 103 students were randomly assigned to the intervention, to behavioral skills training in self-management, or to a control group ( Lane, Lindemann, & Schmidt, 2012 ). The authors found that drinking increased significantly for heavy drinkers if they were assigned to the self-management or control groups, but not if they used e-CHUG. On the other hand, lighter drinkers decreased their drinking significantly more if they received the self-management training rather than if they were in the eCHUG or control groups. The authors concluded that schools may get a better response if a given student's drinking behavior is used to match that student with a prevention program.

The e-CHUG program is based on a computerized brief motivational intervention known as Drinker's Check-Up. In an initial study of Drinker's Check-Up conducted with 61 individuals who were considered problem drinkers (i.e., had scores of 8 or higher on the Alcohol Use Disorders Identification Test [AUDIT]), those who received the intervention and completed follow-up assessments had significant reductions in drinking quantity, frequency of drinking, and alcohol-related problems that were greater in the short term than those of the delayed treatment control group and that persisted for 1 year after treatment ( Hester, Squires, & Delaney, 2005 ). In a more recent evaluation involving two trials conducted with college students ( N =226), those who used the Drinker's Check-Up had significantly greater improvements that were found at both 1-month and 12-month posttreatment assessments on one or more measures of alcohol consumption (depending on the specific trial) than did individuals in delayed assessment-only control groups ( Hester, Delaney, & Campbell, 2012 ).

Other Substance Use Disorder Prevention Programs

Lewis, Neighbors, Oster-Aaland, Kirkeby, and Larimer (2007) researched an alcohol-related prevention program for first-year students who had indicated in a survey conducted in class that they had engaged in at least one heavy drinking episode in the prior month. Participants ( N =316) were assigned to a gender-neutral or a gender-specific version of the intervention or to an assessment-only control group. For the interventions, participants provided information on drinking behavior as well as beliefs about drinking. Feedback was then generated that addressed the participant's drinking behavior, his or her perception of typical student drinking, and information about actual student drinking norms that was either generalized to all students or specific to students of the participant's gender. At a 5-month postintervention assessment, students who received either version of the intervention were reporting significantly fewer drinking occasions and fewer drinks per week than were those in the control group. Although results were not significantly different between the two interventions, those who received the gender-specific feedback did report less drinking than those who received the gender-neutral feedback. At follow-up, those who received the intervention also reported significantly lower estimates of their fellow students' drinking frequency and drinks per week.

Neighbors, Lee, Lewis, Fossos, and Walter (2009) evaluated a Web-based prevention intervention specifically designed to reduce drinking on college students' 21st birthdays. Students who had expressed an intention to consume two or more drinks on their 21st birthday were randomly assigned to use either a control Web site that simply assessed their alcohol use or the intervention Web site, which, in addition to assessment, provided personalized feedback about drinking. Participants were reassessed 1 week after their birthdays. Those who used the intervention site had significantly lower estimated blood alcohol content on their birthdays compared with those who were in the control group. This effect primarily occurred for individuals who had intended to drink large quantities on their birthdays. Doumas, McKinley, and Book (2009) compared a normative feedback Web site ( http://www.CheckYourDrinking.net ) and an alcohol education Web site ( http://www.judicialeducator.com/main.asp ) with 76 students who were referred for mandated services as a result of violating the school's alcohol and drug policy. At a 30-day postintervention follow-up assessment, students who used the CheckYourDrinking intervention reported significantly less weekly drinking, lower alcohol consumption on the day of greatest drinking, less frequent drinking to intoxication, and lower estimates of peer drinking compared with students who used the alcohol education site.

Technology-Based Products To Prevent High-Risk Drinking Among College Students

The Substance Abuse and Mental Health Services Administration (SAMHSA) sponsored a challenge for the top three products to prevent college high-risk drinking, and in 2013 the winners were Syracuse University's BeWise interactive Web site, the University of Central Florida's Expectancy Challenge Alcohol Literacy Curriculum mobile app (designed to present information in a nonjudgmental manner), and the University of Tennessee's Alcohol and You (an online module to educate all incoming first-year students about the choices regarding and consequences of alcohol use).

Source: Lucey, 2005.

Although almost all substance use disorder/illicit use prevention programs for adults are aimed at college students, some have been developed for women of childbearing age. Tenkku et al. (2011) evaluated a Web-based intervention to reduce alcohol use among women ( N =458, of whom 319 were available for follow-up) who were considered to be at risk for an alcohol-exposed pregnancy because they had consumed alcohol in the past 30 days and did not use reliable contraception. Participants were randomly assigned to complete the intervention or to receive printed materials through the mail that conveyed the same information. Four months after the intervention, 58 percent of all participants were no longer considered at risk for an alcohol-exposed pregnancy. Women who used the Web-based intervention were 34 percent less likely to be at risk for alcohol-exposed pregnancy at the follow-up assessment than were those who received the printed materials, but the difference was not significant.

In a study of 150 women who were receiving Women, Infants, and Children services in San Diego County and reported drinking at least at a moderately risky level (i.e., who scored two or higher on the Tolerance, Annoyed, Cut-Down, Eye-Opener screening instrument) were randomly assigned either to use a Web-based assessment and brief intervention, which was adapted from the e-CHUG intervention described in the “ e-CHUG and Drinker's Check-Up Programs ” section, or to receive printed information about normal drinking patterns and the health effects of alcohol on women and unborn children ( Delrahim-Howlett et al., 2011 ). Although there were no significant differences in outcomes for participants in the two groups, 70 percent of all participants did reduce their number of risky drinking episodes.

• Technology To Aid in Mental Health Promotion

Programs are making use of new technologies in a number of ways to further mental health promotion/mental disorder prevention. A meta-analytic review of 75 randomized controlled trials involving computer/Internet interventions for reducing behavioral health risks and/or promoting behavioral health found that such interventions can result in significant improvements in a number of areas relating to behavioral health, especially in the short term (i.e., when assessed at the conclusion of the intervention; Portnoy, Scott-Sheldon, Johnson, & Carey, 2008 ). The authors cautioned, however, that many of the studies they reviewed have methodological problems, and more long-term research is needed.

Mitchell and colleagues (2009) observed that Web applications allow for the tailoring of mental health promotion/mental illness prevention activities to the individual client, thereby potentially increasing the efficacy of such interventions compared with those delivered through traditional media (e.g., television or printed public service announcements). They also noted the ability of Web-based activities to reach a larger audience at lower cost and to be sustainable over a longer period of time. Screening programs for prevention as well as for problem identification are discussed further in the “ Technology in the Treatment of Mental Illness ” section.

Positive Psychology

One area of mental health promotion that has embraced the use of the Internet is the field of positive psychology, an approach to mental health that seeks to increase happiness (perceived as involving positive emotion, engagement, and meaningfulness), which in turn builds mental health resilience and appears to decrease symptoms of some mental illnesses ( Seligman, Rashid, & Parks, 2006 ; Seligman, Steen, Park, & Peterson, 2005 ). The first research trial of a positive psychology intervention using the Internet had a very basic design in which a convenience sample of participants (411 of whom completed all follow-up assessments) was recruited over the Internet and received instructions via email ( Seligman et al., 2005 ). Participants were assigned to complete one of six different exercises (five related to personal happiness and one that was a placebo). At an assessment 6 months after the intervention, completion of two of the exercises was associated with significant increases in happiness (according to the researchers' own measure) and significant decreases in depressive symptoms as measured by the Center for Epidemiological Studies Depression Scale (CES-D).

An Australian study by Mitchell et al. (2009) used a more dynamic, interactive Web-based approach for delivery and compared a strengths-based positive psychology intervention that taught and helped people practice problem-solving skills with an intervention that provided information on problem-solving skills without any interactive features or any attempt to get participants to apply those skills. The authors found that although participation in the positive psychology intervention was, at a 3-month postintervention follow-up, associated with significant improvements in participants' well-being (or happiness) according to two of the instruments used, it was not associated with any changes in depression, anxiety, or stress as measured by the Depression, Anxiety, Stress Scales–21.

Depression and Anxiety Prevention

Another example of a Web-based program that could be considered mental health promotion as well as substance use disorder prevention is the MSB-S intervention (related to the other MyStudentBody sites already described), which teaches and helps students use stress management techniques ( Chiauzzi, Brevard, Thum, Decembrele, & Lord, 2008 ). In a research trial of the intervention, students at six different colleges were randomly assigned to the MSB-S Web site intervention ( n =77), to a control group that used a Web site that provided health information and sought to encourage physical exercise ( n =78), or to a no-treatment control group ( n =80). Those who used the MSB-S site increased their use of specific stress management skills. Also, participants who used the MSB-S site showed an initial significant decline in anxiety (as measured by the College Adjustment Scales anxiety subscale), but at the last follow-up assessment (6 months after the intervention), anxiety scores were about equal for the intervention and control Web site groups.

A Web-based anxiety disorder prevention program was evaluated in Australia with a group of college students ( N =42) who had previously screened as having high anxiety sensitivity ( Kenardy, McCafferty, & Rosa, 2006 ). In a 6-month follow-up assessment, those participants who had been randomly assigned to the intervention had, compared with those in a waitlist control group, significantly lower ( p< 0.01) anxiety-related thoughts (according to scores on the Catastrophic Cognitions Questionnaire-Modified) and fewer depressive symptoms (on the CES-D). Other measures of anxiety also indicated declines in anxiety, but differences from the control group were not significant.

Other interventions are available for mental illness or mental illness symptoms, but for many of them, there are few data to support their use. Patten (2003) reported on a psychoeducational program delivered by computer or telephone that was designed to prevent depressive symptoms, but a randomized controlled evaluation of 786 participants found that the intervention was associated with no better outcomes than an information-only control group. Finkelstein and Lapshin (2007) reported on a Web-based program that, although not related to prevention per se, was designed to reduce the effects of prejudice associated with depression and increase knowledge about the disorder, which in turn might aid in early treatment-seeking for those with depressive symptoms. In an evaluation with 42 college students and faculty, use of the program was associated with significant decreases in depression-related prejudice (assessed using the Bogardus Social Distance Scales).

Suicide Prevention

Suicide prevention activities have a long history of being conducted using communication technologies, and suicide or crisis intervention hotlines are widespread and generally considered an effective way to prevent suicides ( Gould et al., 2007 ). For people in crisis who may not be suicidal, these hotlines also provide referrals and help reduce callers' feelings of being overwhelmed ( Gould et al., 2007 ). Within the United States, many crisis hotlines have joined together as part of the National Suicide Prevention Lifeline network (1-800-273-TALK), which is able to provide nationwide coverage 24 hours a day, 7 days a week. Online resources for suicide prevention have been developed, including the National Suicide Prevention Lifeline Web site ( http://www.suicidepreventionlifeline.org ), which is sponsored by SAMHSA. Since 2011, SAMHSA and the National Suicide Prevention Lifeline have collaborated with Facebook, and in 2015, SAMHSA continues to use Facebook in its prevention efforts (see http://www.samhsa.gov/prevention-week ).

Web-based suicide risk monitoring and referral programs are also available. Haas and colleagues (2008) evaluated such a program with 1,162 students at two different universities. The students completed an online screening instrument that incorporated an existing depression screener (the Patient Health Questionnaire 9-Item Scale [PHQ-9]) as well as other items relating to suicide (e.g., substance use, anxiety). The computer scored responses into three categories of suicide risk; those who were considered to have a moderate or high risk for suicide received personalized email responses from a counselor that addressed issues of potential concern for the student and provided different referral options for further counseling. A large percentage of the respondents (84.4 percent) fell into the moderate-risk or high-risk categories. As a result of the program, 19.4 percent attended an evaluation session with a counselor, and 13.5 percent entered treatment. If students engaged in an online dialog with a counselor after receiving their initial feedback email, they were almost three times more likely to receive an evaluation and to enter treatment.

Computer software that can screen and assess suicide risk has been available for quite some time. In an early pilot evaluation involving 21 individuals with suicidal ideation, 58 percent expressed a preference for talking with a computer instead of a person ( Greist et al., 1974 ). Later research indicated that, in evaluating case histories of 20 individuals receiving mental health services (10 of whom later attempted suicide), the computer was significantly more accurate than a group of psychiatrists and psychiatric residents in predicting suicide attempts. The computer predicted 70 percent of cases compared with counselors, who predicted 38 percent ( Gustafson, Greist, Strauss, Erdman, & Laughren, 1977 ). Rimoldi et al. (2012) presented information from a VA pilot study involving the use of text messages sent via mobile phones to assess and manage suicide risk in a group of 11 veterans considered to be at high risk for suicide. Participants regularly responded to questions concerning their mental status, and when no answer was received or responses indicated an increase in risk, a Suicide Prevention Coordinator responded accordingly. The intervention was well-received, but three participants dropped out because of personal costs.

Another military suicide prevention program, which is run by DoD and thus serves active-duty personnel, uses email or handwritten messages sent at regular intervals to individuals who have been hospitalized for a mental disorder ( Luxton, Kinn, et al., 2012 ). Although program users may select email or paper messages, the majority preferred the email option (72 percent of participants in the pilot study). The pilot study, however, did not yield any significant data.

Video conferencing can also be used to monitor suicide risk and intervene when necessary. Godleski et al. (2008) discussed VA efforts to assess suicidality using high-speed video conferencing technology. They presented VA's best practice guidelines and discussed legal and licensing concerns. Gros, Veronee, Strachan, Ruggiero, and Acierno (2011) presented a case study, also involving a veteran, of the use of video conferencing to manage suicidality.

Eating Disorder Prevention

Another area where there have been a number of efforts to use communication technology is in the prevention of eating disorders. However, not all interventions in this area have been effective. Newton and Ciliska's (2006) review of five published studies did not find any significant effect on reducing eating disorders following implementation of Web-based prevention programs. A review by Yager and O'Dea (2008) of eating disorder prevention programs for college students, which included interventions delivered in person as well as those delivered by computer, found that information-based, CBT, and psychoeducational interventions had only limited success but that interventions that built self-esteem to improve body image, including ones that used computer technology, were more effective.

Two studies of the Student Bodies Web-based program, which includes modules addressing factors related to risk for eating disorders (including body image, weight, and nutrition) have found that the program may be helpful, at least in some cases, in reducing eating disorders among college-age women. In the larger study ( N =480), there were no significant differences in eating disorder risk between intervention and control groups taken as a whole, but for participants who began the study with an elevated body mass index (of 25 or greater) and those who engaged in compensatory behaviors (e.g., self-induced vomiting, laxative use, diet pill use) at baseline, participation in Student Bodies was associated with significantly better outcomes ( Taylor et al., 2006 ).

In another study ( Low et al., 2006 ), 61 female undergraduates were randomly assigned to use Student Bodies by itself ( n =14), the program with the addition of an unmoderated online discussion component ( n =19), the program with moderated discussion ( n =14), or a no-intervention control group ( n =14). The authors found a significant relationship between participation in the program (for all three intervention groups taken together) and lower risk for eating disorders as indicated by scores on the Eating Disorders Inventory Drive for Thinness and Body Dissatisfaction subscales (but not the bulimia subscale) during the 8- to 9-month follow-up period. Eleven participants who did not complete the study were not included in the analysis. Although the authors cautioned that it was difficult to evaluate differences among the three intervention groups because of the small sample size, participants in the group with unmoderated discussion had the greatest reductions in eating disorder risk, suggesting that having a moderated discussion was unnecessary.

Another study of an indicated prevention intervention that used moderated discussion as well as psychoeducation, homework tasks, and a monitoring component (in which the discussion group monitor attempted to identify problematic messages that might require clinical intervention) also found benefits for college-age women who were at risk for developing eating disorders based on scores on the Weight Concerns Scale ( Zabinski, Wilfey, Calfas, Winzelberg, & Taylor, 2004 ). At an assessment 10 weeks after the intervention, participants in the intervention group ( n =30), compared with those in the control group ( n =30), had significantly greater self-esteem and significantly greater improvements in eating disorder pathology as measured by the Eating Disorder Examination Questionnaire and two of its subscales (Eating Concern and Weight Concern).

A Web-based program developed in Germany is designed to provide both prevention and early intervention for eating disorders by including an eating disorder monitoring component that results in referrals to treatment when eating disorder symptoms reach too severe a level ( Bauer, Moessner, Wolf, Haug, & Kordy, 2009 ). A pilot program found that the intervention could be successfully implemented, but outcome data are still lacking. Franko et al. (2005) evaluated a CD-ROM eating disorder prevention program called Food, Mood, and Attitude with 240 female first-year college students who were randomly assigned to the intervention or to an assessment-only control group. At a 3-month postintervention assessment, those who used the intervention had significantly greater improvements than control group members in a number of areas connected to the development of eating disorders (e.g., attitudes relating to thinness, body shape, and weight) and were significantly more likely to decrease the use of purging methods (e.g., vomiting, laxatives, diuretics).

In an Australian trial ( Paxton, McLean, Gollings, Faulkner, & Wertheim, 2007 ), women who had been screened as having a high degree of body dissatisfaction (according to Body Shape Questionnaire or Bulimia Test Revised scores) were randomly assigned to a group receiving an in-person intervention ( n =42), a group receiving a Web-delivered intervention ( n =37), or a waitlist control group ( n =37). Individuals were not excluded if they met criteria for a current eating disorder, but they did not have to meet those criteria, either; therefore, this intervention could be considered as treatment or indicated prevention. Participation in either intervention group was associated with significant improvements in terms of body dissatisfaction, but the effect during the 6-month follow-up period was more lasting for those who received the intervention in person rather than over the Internet.

• Technology in the Treatment of Mental Illness

As noted previously, phone and computer technologies have been successfully used for screening, assessing, and treating a variety of mental disorders. They have been used as stand-alone interventions and as adjuncts to treatment that takes place in person. This section of the literature review discusses general screening and assessment, the treatment of specific mental disorders, and other applications of these technologies (e.g., peer support groups, family support services).

VA Reports 80,000 Veterans Took Part in More Than 200,000 Telehealth Consultations in 2012

The VA's telehealth system is “a service designed to allow thousands of veterans from remote parts of the country to consult with specialists in medical facilities hundreds of miles away.… The virtual appointments included vets receiving care and counseling for chronic medical conditions and for mental health counseling.

Dr. Robert Petzel, under secretary for health at the VA, said that virtual appointments are making a significant difference in the lives and health of patients. He told of one veteran who was driving 45 minutes in city traffic to keep in-person appointments with his mental health counselor. ‘He would fight the traffic, be angry and upset by the time he got to see his therapist,’ Petzel said, ‘and it would not be a particularly productive therapy session. He now participates in a tele-mental health program where he does his therapy from his home.’

‘We've interviewed him. He points out the fact that now he's relaxed, in an environment he's comfortable in, he's not upset about the traffic and he's getting infinitely—and he used the word infinitely—more out of his therapy sessions now that he doesn't have to travel that distance.’”

Source: http://www.military.com/daily-news/2013/07/31/va-touts-telehealth-program-used-by-80k-vets.html

Use in Screening and Assessment

Computer and phone technologies have been used for many years for screening and assessing mental illness (as well as substance use disorders, which are discussed later). Although an in-person assessment is usually preferable, it is not always feasible, and the use of computer technologies, particularly video conferencing technology, can be effective for assessing clients' behavioral health ( Barak & Buchanan, 2004 ; Bickel, Christensen, & Marsch, 2011 ; Butcher, Perry, & Hahn, 2004 ; Maheu et al., 2004 ).

Phone-based screening and assessment

A variety of research has found that telephone interviews are an effective way to screen, assess, and diagnose a number of different mental disorders ( Maheu et al., 2004 ). For example, Kobak et al. (1997) found, in a study that compared diagnoses made using automated telephone screening with those made using the Structured Clinical Interview for the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (SCID-IV) delivered by clinicians over the phone ( N =200), no substantive differences in rates of specific mental disorders were found, with the exception of alcohol abuse, which was more likely to be reported in the automated interview. Findings were similar in a comparison of the Hamilton Depression Rating Scale and the Hamilton Anxiety Rating Scale administered via phone by a computer versus by trained clinicians ( Kobak, Greist, Jefferson, Mundt, & Katzelnick, 1999 ).

Some disorders, however, may be harder to diagnose over the phone. For example, Rohde et al. (1997) found poor agreement between telephone and in-person assessments of adjustment disorder with depressed mood, but not for other disorders assessed in their study. Voice-automated telephone technology also makes it possible to use phones to conduct automated screening and assessment. For example, Xu et al. (2012) described and provided preliminary psychometric data for a voice-automated PTSD screening and monitoring program that is still in development. Phones can also be used to provide ongoing assessment and monitoring for clients with mental illness and to improve self-care and medication compliance. For example, Maust et al. (2012) found that a Commonwealth of Pennsylvania program that used telephones to provide regular clinical assessments and feedback to older adults taking psychotropic medications was associated with significantly greater medication regimen compliance and significantly greater improvements in depressive symptoms and overall mental well-being than found in older adults using standard services alone.

Screening and assessment using video conferencing

Assessments conducted using video conferencing have also been found to be accurate in a number of studies ( Bickel et al., 2011 ; García-Lizana & Muñoz-Mayorga, 2010a ; Norman, 2006 ; Sharp et al., 2011 ). For example, Singh, Arya, and Peters (2007) compared assessments conducted using video conferencing technology with ones conducted in person with the same group of 37 consecutive admissions to a community mental health facility, and they found substantial agreement in almost every area. The one exception to this was in the assessment of an individual's risk to others (but not to self).

A meta-analysis of 14 studies that included data from objective assessments conducted using video conferencing over the Internet and comparison data from in-person interviews found no differences in assessments conducted using the two methods ( Hyler et al., 2005 ). The authors did, however, find that low bandwidth may be less effective. In its review of video conferencing, the ATA (2009) concluded that there is good evidence showing that clients adapt to assessment interviews conducted with video technology and respond with as much information as they would if assessed in person. They also noted anecdotal evidence that people with certain disorders (i.e., PTSD, agoraphobia, eating disorders) may feel safer if assessed using this technology.

Another review by Richardson et al. (2009) found strong evidence that clinical assessments conducted using video conferencing technology were as reliable as assessments conducted in person. Other reviews of video conferencing have also concluded that the technology is reliable for conducting assessments and psychiatric interviews ( Sharp et al., 2011 , Simpson & Morrow, 2010 ). In a study of video conferencing as a means of conducting psychiatric interviews, which involved 43 interviews of 14 different clients, Turner (2001) found that both the psychiatrists who performed assessments and the clients who received them had a favorable attitude toward the technology.

Yellowlees et al. (2012) evaluated the feasibility of using asynchronous mental health evaluations that involved video sent via the Internet. In that study, psychiatric interviews were conducted by a primary care doctor using the Mini-International Neuropsychiatric Interview and recorded for viewing by a psychiatrist at a later time, who in turn provided a psychiatric diagnosis. The authors found that the process worked well and enabled psychiatrists to observe and diagnose clients who would not otherwise be able to see such professionals because of a lack of mental health professionals in their area. However, some research indicates that certain aspects of assessment (e.g., assessing negative symptoms of psychosis) may be more accurate when done with higher-bandwidth connections ( Sharp et al., 2011 ). Maheu and colleagues (2004) discussed in greater detail these and other important technical aspects of video conferencing in relation to behavioral health services. There are also some indications that substance use disorders may not be assessed as accurately using video conferencing as they are when assessed in person ( Shore, Savin, Orton, Beals, & Manson, 2007 ).

Web-based screening and assessment

Computerized assessments have been developed for a large number of instruments, and research generally indicates that computer delivery is effective for screening and assessing most clients. Butcher et al. (2004) reviewed research on the use of computerized assessment in three areas: personality assessment, neuropsychological assessment, and computerized adaptive testing (CAT). They cited research supporting the use of computers in all three areas, concluding that these methods are effective and that their use improves behavioral health services. Research has also indicated that, in general, clients seeking treatment for mental illness respond favorably to automated assessments ( Hile & Adkins, 1997 ).

Many well-known and often-used instruments have been adapted for the computer, but a complete list of all computerized assessments is beyond the scope of this literature review. Butcher, Perry, and Atlis (2000) reviewed earlier literature on the validity and utility of some of the most widely used instruments, including iterations of the Minnesota Multiphasic Personality Inventory, the World Health Organization Composite International Diagnostic Interview, and the Computerized Diagnostic Interview Schedule. For some clients, completing a computerized or Web-delivered assessment is preferable to answering questions delivered by another person, as the latter may be more embarrassing ( Butler, Villapiano, & Malinow, 2009 ; Joinson & Paine, 2007 ; Lessler, Caspar, Penne, & Barker, 2000 ; Newman et al., 2010 ). Computerized assessments can also help ensure standardized delivery and completeness of data ( Newman et al., 2010 ).

Guided assessment

Technology is now available that can guide clients through a self-assessment process using audio instructions, a process known as audio computer-assisted self-interviewing (ACASI). Chinman et al. (2007) conducted a feasibility study of ACASI in mental health clinics and found that it was well-received by providers and by clients with serious mental illness. The same type of system was also evaluated for use with a more extensive interview—that used in a National Household Survey on Drug Abuse ( Lessler et al., 2000 ). The ACASI version of the survey was given to 1,982 respondents and was found to improve reporting of drug use among young people and to be helpful in correcting inconsistencies in reporting, and it was particularly useful for respondents who would have had problems with text-based questions.

Adaptive testing

Another way that new technology can improve assessment is through the application of CAT techniques, which use answers from prior questions to select only the necessary questions for the assessment and thereby decrease the amount of time required to give and score the instrument ( Butcher et al., 2004 ). Butcher and colleagues (2004) reviewed earlier research on the use of adaptive testing in behavioral health services.When Gibbons and colleagues (2008) combined item response theory and CAT to study features of the 626-item Mood and Anxiety Spectrum Scales with 800 clients, they found that 95 percent of participants (765 clients) preferred answering the questions using a computer rather than a paper-and-pencil method.

Web-based screening and assessment instruments

In general terms, simple screening instruments and surveys are as likely to be valid when delivered over the Internet as when delivered by mail ( Ritter, Lorig, Laurent, & Matthews, 2004 ). However, more complex assessment instruments may not adapt equally well to Internet use, as factors such as social desirability, willingness to self-disclose, and computer anxiety may affect the validity of online assessment ( Vallejo, Jordan, Diaz, Comeche, & Ortega, 2007 ). Butcher and colleagues (2004) also discussed some potential problems involved in online psychological assessment; these included ensuring equivalent test-taking attitudes, test norms that are appropriate for Internet delivery, test validity, and test security.

The delivery via the Internet of a number of specific instruments has been evaluated. For example, Vallejo et al. (2007) evaluated whether or not the psychometric properties of the General Health Questionnaire-28 (GHQ-28) and the Symptoms Check-List-90-Revised (SCL-90-R) varied when administered over the Internet versus with paper and pencil. They found that the psychometric properties of the GHQ-28 were similar for both delivery methods and thus recommended the Internet version for use. However, although the SCL-90-R had high internal consistency regardless of the delivery method, the paper version produced higher scores, and the authors therefore could only recommend its online version as a general index of mental disorder symptoms but not as an assessment instrument. The Neuropsych Questionnaire is another example of a computerized assessment instrument that can be delivered by computer either over the Internet or in an office ( Gualtieri, 2007 ). The instrument, which evaluates 20 different clusters of symptoms, was found to be reliable, to effectively discriminate different diagnoses, and to produce results comparable with those produced by other well-known scales.

A number of screening and assessment instruments have also been developed specifically for delivery over the Internet. For example, the Web-Based Screening Questionnaire is a brief screening instrument developed for use online; it screens for major depression, alcohol use disorders, GAD, PTSD, social anxiety disorder, panic disorder, agoraphobia, specific phobia, and OCD ( Donker, van Straten, Marks, & Cuijpers, 2009 ). The specificity and sensitivity of the instrument varied according to the particular diagnosis, but after modifications were carried out following this initial research, all scores fell within an acceptable range (0.72–1.00 for sensitivity and 0.44–0.77 for specificity). Another example is the Web-Based Depression and Anxiety Text, which is a self-report screening instrument originally developed for use in primary care settings. It screens for most anxiety disorders (except GAD) and depression with good sensitivity (0.71– 0.95) and specificity (0.87–0.97; Farvolden, Cunningham, & Selby, 2009 ; Farvolden, McBride, Bagby, & Ravitz, 2003 ).

One potential concern with Internet screening instruments is whether users will seek treatment if they screen positive for a disorder. Van Ameringen, Mancini, Simpson, and Patterson (2010) evaluated the use of MACSCREEN, an online screening program for mood and anxiety disorders, and found that the majority (60.3 percent) of the 302 users in their evaluation said they would use the information provided to seek further assessment/treatment. The majority of users (62.3 percent) also indicated that they used the assessment Web site because they were already concerned that they had a problem with anxiety. Computer- or Web-delivered screening instruments can be more easily tailored to specific populations or for specific disorders and can reach a wide audience at a relatively low cost compared with interventions delivered in person. Houston et al. (2001) evaluated a Web-based depression screening program that used a computerized version of the CES-D to screen a large ( N =24,479) general population sample. The screening program used a cutoff score of 22 or greater on the CES-D to indicate a high probability of clinical depression, and individuals with scores in that range were advised to seek professional help. More than half (58 percent) of those who took the screening test had scores of 22 or higher. The Web-based screening—compared with earlier, in-person, general population depression screenings—had a larger percentage of younger participants but a smaller percentage of people of color. The overall cost per person for the screening was quite low.

Le, Perry, and Sheng (2009) evaluated both in-person and Web-delivered versions of a screening instrument (the Postpartum Depression Screening Scale) with a convenience sample of women who had recently given birth and found that it had excellent internal consistency and construct validity. They also found that women who chose online rather than in-person screening were significantly more likely to be screened as at risk for postpartum depression and were more likely to be Asian or Latina.

Screening and assessment using mobile technology

The use of handheld devices allows for ecological momentary assessment, also known as structured momentary assessment, a process whereby clients can report real-time information about symptoms, experiences, feelings, cravings, or other factors associated with their disorder(s). This technique has been used to assist in the treatment of schizophrenia ( Granholm, Loh, & Swedsen, 2008 ), PTSD ( Smith, Harms, et al., 2012 ), and drug use disorders ( Freedman et al., 2006 ; Lin, Vahabzadeh, Mezghanni, Epstein, & Preston, 2005 ), as well as nicotine dependence, anxiety symptoms, and eating disorder symptoms ( Heron & Smyth, 2010 ). Handheld devices have also been used in research studies to evaluate symptoms among people with major depression or BPD ( aan het Rot, Hogenelst, & Schoevers, 2012 ; Santangelo, Bohus, & Ebner-Priemer, 2014 ).

Ecological momentary assessment may be more effective for some clients than others. A French study involving 109 clients of an outpatient substance use disorder treatment program found that individuals in treatment for cannabis use disorders were significantly less likely than those in treatment for alcohol, opioid, or tobacco use to agree to participate in such an assessment and were less likely to respond to requests for information after agreeing ( Serre et al., 2012 ). Individuals with tobacco dependence had the highest rate of participation and compliance. Another study comparing daily mood assessment using a mobile phone application with the assessment using paper and pencil ( N =40) for individuals with BPD found significantly worse compliance and significantly greater variability among individuals who used the application than among those who used a more traditional method, suggesting some potential problems using the technology with that population ( Depp, Kim, de Dios, Wang, & Ceglowski, 2012 ). However, the authors also found greater agreement between self-assessments reported using the mobile phones and clinicians' rating of depression and manic symptoms than with those reported using paper and pencil, suggesting that there was also a potential benefit to the phone-based method.

Use in Treatment of Anxiety Disorders

A variety of computerized and phone-delivered interventions have been developed to aid in the treatment of anxiety disorders, and in general, they have been found effective in reducing anxiety and superior to waitlist controls or placebos, with some studies indicating a level of effectiveness comparable with therapist-delivered treatment ( Andersson, 2009 ; Cuijpers, Donker, van Straten, Li, & Andersson, 2010 ; Furmark et al., 2009 ; Griffiths, Farrer, & Christensen, 2010 ; Morgan & Jorm, 2009 ; Newman et al., 2010 ; Newman et al., 2011a ; Reger & Gahm, 2009 ). A meta-analytic review on the use of computers in therapy for anxiety disorders by Cuijpers et al. (2009) found that such interventions did not differ significantly in effectiveness from in-person interventions according to a number of different comparisons, but they also noted that other factors relating to differences in the studies could have affected this finding.

Newman and colleagues (2010) , in their review of the research on computer/Web-delivered interventions for anxiety disorders, also noted that there are good data supporting the use of such interventions for treating panic disorder and SAD/social phobia, and fewer, but also good, studies indicating that computerized interventions (in the form of VRET) helped in the treatment of some specific phobias (particularly fear of flying and acrophobia, but not spider phobia). Also, some limited research has found computerized interventions useful for PTSD and GAD, but not enough data are available to make recommendations for the treatment of those disorders. For treating OCD, data indicate that computerized interventions may be useful if integrated with some therapist contact, but not as stand-alone treatments.

A review by Newman, Szkodny, Llera, & Przeworski (2011b) summarized a larger group of research findings regarding the use of computer interventions that require little or no therapist contact. Specifically, the authors concluded that there was good evidence supporting the use of computerized interventions for mixed anxiety disorders, panic disorder, and SAD. For panic disorder, the best strategy appeared to be a combination of minimal therapist contact (as little as 150 minutes for a course of treatment) and Web-based or computer-based treatment, which could be enhanced by also including a stress management component. In the case of SAD, predominantly self-guided computerized interventions focusing on attention retraining and exposure appear to be effective if delivered in a clinical setting, but some minimal level of therapist contact is probably needed for therapy delivered in the client's home. Although some evidence supported the effectiveness of computerized interventions for GAD, OCD, and PTSD, the authors cautioned that more thorough research on the utility of such approaches is needed.

In the treatment of simple phobias, Web-delivered or computer-delivered interventions (usually using VR hardware and software) show promise as treatments for claustrophobia, driving phobia, flying phobia, and spider phobia, but more research is needed, as studies to date have had significant flaws ( Newman et al., 2011b ). Newman and colleagues (2011b) cautioned, however, that the research also indicates a greater problem with compliance for clients who use these technologies with little or no contact. They also noted that many of the studies they reviewed had serious methodological problems, including a lack of follow-up assessments, small sample sizes, a failure to compare computerized treatments with other active treatments as well as with no-treatment control groups, a tendency to evaluate changes in symptom levels but not clinically significant changes (i.e., whether the individual obtains remission from the disorder), a failure to provide sufficient information about the hardware and software used, and a failure to use a diverse (culturally and in other ways) population of participants.

Another review by Reger and Gahm (2009) of 19 studies of computerized interventions for anxiety (including both those that did and those that did not use the Internet) also found support for their use; such interventions were superior to no-treatment or placebo controls for treating panic disorder and specific phobias, and limited data suggested that they were more effective than no treatment or placebo for PTSD. In addition, computerized interventions for panic disorder appeared to be as effective as treatment as usual involving CBT delivered in person. These interventions also appeared to be as effective for subclinical anxiety symptoms as they were for treating anxiety disorders (for more information on this topic, see the forthcoming TIP, Managing Anxiety Symptoms in Behavioral Health Services [SAMHSA, planned a]). A number of phone and Web interventions for treating anxiety symptoms and/or anxiety disorders fall into the category of self-guided interventions (in which case, a staff person, either in person or by telephone, guides the client in the use of a computer program or printed materials; Morgan & Jorm, 2009 ). A review by Cuijpers et al. (2010) found that self-guided interventions (involving interventions that typically made use of phone contact with a therapist) were at least as effective, and possibly slightly more effective, than therapist-delivered treatment for panic disorder, social phobia, and specific phobias.

Research suggests that clients receiving online treatment for anxiety are able to establish relationships with counselors that are as stable and positive as those found in treatment provided in person, but the quality of the working alliance between client and counselor may have less effect on clients receiving Web-based treatment ( Knaevelsrud & Maercker, 2006 ). However, depending on the type of anxiety disorder being treated and client characteristics, it may be preferable to have at least some counselor contact ( Newman et al., 2011a ). Research involving palmtop computers in the treatment of both GAD and panic disorder has indicated that using such technology reduces costs compared with standard, in-person CBT ( Newman, Kenardy, Herman, & Taylor, 1997 ; Newman, Consoli, & Taylor, 1999 ; Newman et al., 2010 ). The sections that follow discuss some specific approaches to treating anxiety that can be delivered using phone or computer technology and how such technology can be used to treat specific anxiety disorders. Studies are included if they were not included in the reviews discussed previously or if they had results that warranted particular attention.

Most of the interventions available for treating anxiety via computer use CBT techniques that have been found effective at reducing anxiety/treating anxiety disorders in other settings and that typically adapt well to computer delivery ( Newman et al., 2010 ). Reviews that focus on Web-and computer-based CBT interventions for anxiety have found them to be superior to control conditions. Griffiths et al. (2010) reviewed randomized controlled trials of Web-based CBT interventions and determined that, with one exception, these interventions were associated with better outcomes than no-treatment control groups in at least one area evaluated. For interventions treating panic disorder or SAD/social phobia, effect sizes were comparable with those seen in studies of interventions delivered in person. However, the authors cautioned that in many of the studies evaluated, self-selected participants were used. A major focus of Andersson's ( 2009 ) article on Web-delivered CBT is the use of such techniques for anxiety disorders. He also found good evidence to support the use of such interventions for panic disorder and SAD and promising evidence supporting their use to treat PTSD.

Researchers in Australia developed and evaluated a Web-based CBT treatment for GAD (the Worry programme; Titov, Andrews, Robinson, et al., 2009 ). They randomly assigned 48 individuals to treatment or to a no-treatment control group and assessed them after the 9 weeks required for the delivery of the intervention (an assessment completed by 20 individuals in the treatment group and 19 in the control group). At the conclusion of treatment, participants who received the intervention had significantly lower symptoms of worry (as measured by both the Generalized Anxiety Disorder–7 Item Scale and the Penn State Worry Questionnaire) and symptoms of depression (according to the PHQ-9) than did those in the control group. Individuals with GAD ( N =29) who used an attentional bias modification software program experienced significant reductions in anxiety and anxiety symptoms, according to both the State and Trait measures from the Spielberger State–Trait Anxiety Inventory, the SCID-IV Anxiety Disorders Module, the Worry Domains Questionnaire, and the Penn State Worry Questionnaire ( Amir, Beard, Burns, & Bomyea, 2009 ).

Social anxiety disorder/social phobia

Populations who might otherwise avoid contact with others, such as people with SAD ( Andersson, 2009 ; Erwin, Turk, Heimberg, Fresco, & Hantula, 2004 ) or high levels of agoraphobic avoidance ( Andersson et al., 2008 ), may be more comfortable seeking treatment over the Internet than in person. Thus, Web-based interventions may be especially promising for treating SAD. Andersson (2009) reviewed studies by three groups of researchers of Web-based treatments for SAD that use CBT principles and therapist guidance. All three have had positive results. He noted that although some may believe that Web-based therapy is not appropriate for people with this disorder, the fact that people with SAD are extensive users of the Internet, combined with the ability of these approaches to make them more comfortable in the early stages of treatment, suggests that they may provide effective treatment for clients who would not otherwise seek it.

Research on Web-based treatments for SAD includes a 30-month follow-up on a Swedish study included in Andersson's ( 2009 ) review, in which 77.2 percent of the original 57 participants were contacted for a follow-up interview ( Carlbring, Nordgren, Furmark, & Andersson, 2009 ). At the 30-month follow-up, the authors found that significant improvements in all primary measures observed at the conclusion of the intervention continued with large effect sizes for those measures. In another Swedish study, Furmark et al. (2009) conducted two trials of a Web-based CBT treatment for SAD, comparing it with three different groups: bibliotherapy with online participant discussions, a waiting list, and (in the second trial only) Web-delivered applied relaxation training. Both the online CBT and the bibliotherapy were associated with significantly better outcomes in terms of social anxiety symptoms, general anxiety symptoms, depression, and quality of life compared with the waitlist control. Improvements in those areas continued at a 1-year posttreatment assessment. Effect sizes were somewhat larger for online CBT than for bibliotherapy and were comparable with those found in trials of online CBT with additional support from therapists, leading the researchers to conclude that feedback from therapists was not necessary for the interventions to be effective.

In a Swiss study, Berger, Hohl, and Caspar (2009) evaluated a Web-delivered treatment for SAD using a sample of 53 individuals randomly assigned to the treatment or to a waitlist control group. They found that significant improvements occurred more often with intervention participants (58 percent) than with individuals in the control group (20 percent). Titov, Andrews, Choi, et al. (2009) found that individuals with social phobia disorder had significantly better compliance with treatment and better outcomes when using a Web-based CBT treatment if they also received weekly phone calls from a physician than if they only received automated email messages. However, in another study of Web-based CBT treatment for social phobia disorder, Titov, Andrews, Schwenke, et al. (2009) compared outcomes for those who, in addition to the online treatment, received weekly phone calls from a research assistant with those who were given the opportunity to participate in an ongoing, therapist-monitored discussion group. Both groups had comparable effect sizes with respect to reductions in social phobia symptoms on multiple measures, with effect sizes being somewhat (but not significantly) larger across measures for the Internet forum group. The time spent by staff members was also similar for the two groups (38 minutes per participant for the phone contact and 37 minutes per participant for the Internet forum).

An Australian study ( Andrews et al., 2011 ) that compared Web-delivered CBT for SAD with CBT delivered in person also found no significant differences in social anxiety outcomes (both interventions achieved similar improvements), but it did find that substantially fewer clinician hours were required for the Web-based intervention (an average of 18 minutes per client) than for the intervention provided in person (an average of 240 minutes per client). In another European trial of Web-delivered CBT ( Berger, Caspar, et al., 2011 ), 81 individuals with SAD were randomly assigned to receive a self-guided intervention alone, the intervention with additional email support from a therapist, or a self-guided intervention with optional email and/or telephone support according to client demand. These researchers found significant symptom reductions for all three treatment arms, but there were no significant differences among the groups in terms of social anxiety symptom levels, remission from SAD, dropout rates, treatment compliance, or the probability of clinically significant change.

Some research has found that cognitive bias modification software (which is intended to change users' cognitive biases, as discussed more fully in the “ Interventions To Reduce Trait Anxiety and Subclinical Anxiety Symptoms ” section) shows promise as a treatment for SAD, but other studies have failed to replicate these findings ( Emmelkamp, 2012 ). For example, a small ( N =36) study of a cognitive bias modification program for SAD did find significantly greater reductions in trait anxiety and social anxiety symptoms in the treatment group than in a placebo control at the time of a 4-month follow-up assessment, with 72 percent of those in the treatment group no longer meeting criteria for SAD at the conclusion of the treatment compared with 11 percent of those in the control group ( Schmidt, Richey, Buckner, & Timpano, 2009 ). Beard, Weisberg, and Amir (2011) , in a small ( N =32) randomized controlled study, compared individuals receiving another computerized bias modification intervention for treating SAD (Attention and Interpretation Modification for SAD) with a group who used a similar software program without the same therapeutic content. Compared with those in the control group, participants who received the intervention had significantly greater reductions in social anxiety symptoms and performed better on a behavioral task intended to evaluate their level of social anxiety.

However, one study was unable to find any significant improvements associated with a cognitive bias modification intervention for SAD ( Julian, Beard, Schmidt, Powers, & Smits, 2012 ), and another found no additional benefit in adding cognitive bias modification to standard CBT ( McEvoy & Perini, 2009 ). Based on a review of this literature, Emmelkamp (2012) concluded that although cognitive bias modification software may be effective at treating subclinical SAD symptoms, there are more effective computerized interventions already available for SAD.

Research suggests that Web-based interventions are a cost-effective means of treating individuals with SAD. One Swedish study compared cost-effectiveness for a Web-based CBT treatment for SAD with a similar CBT intervention delivered in groups and found that, although both were about equally effective in reducing social anxiety symptoms, costs were significantly lower for the Web intervention ( Hedman et al., 2011 ).

Panic disorder

Researchers in the Netherlands evaluated 3-year outcomes for clients who received a Web-based CBT treatment for panic symptoms, which included computerized homework assignments and individualized therapist feedback ( Ruwaard, Broeksteeg, Schrieken, Emmelkamp, & Lange, 2010 ). Individuals who had chronic panic attacks were randomly assigned to a treatment group ( n =27) or to a waitlist control group ( n =31), and the authors used an intent-to-treat analysis to evaluate outcomes. Following treatment, participants who received the intervention had significantly better outcomes in a number of areas, including panic attack intensity (rated on a 10-item Likert scale; p< .001), panic disorder severity (Panic Disorder Severity Scale-Self Rate [PDSS-SR]; p< .027), anxiety symptoms (Depression Anxiety Stress Scales [DASS]; p< .023), depressive symptoms (DASS; p< .015), and past-week panic attack intensity ( p< .005). Following this initial trial, waitlist participants also received the intervention, and results were pooled for the two groups in evaluating outcomes 3 years later. At that time, using data obtained prior to follow-up for missing individuals, the authors found that 68 percent of participants reported no panic attacks during the prior week; 83 percent had PDSS-SR scores that did not indicate a current panic disorder.

In an Australian study of a Web-delivered intervention, researchers randomly assigned individuals who had panic disorder with agoraphobia to an online CBT intervention group ( n =32) or to a waitlist control group ( n =27; Wims, Titov, Andrews, & Choi, 2010 ). In each group, 22 individuals completed the posttreatment assessment, and 21 in the intervention group completed an assessment 1 month later. The researchers found that participants who received the intervention, compared with those in the control group, had significantly lower panic symptoms as measured by the PDSS-SR, the Body Sensation Questionnaire, and the Agoraphobic Cognitions Questionnaire. Also, panic attacks declined for participants in the intervention group from a mean of 8.03 in the prior 4 weeks at the start of treatment to 3.72 at the end of treatment and 2.76 at the 1-month posttreatment assessment (by comparison, the number of attacks for those in the control group rose from 5.84 to 8.20 at the first assessment and 8.6 at the second). The effect sizes for the intervention were comparable with those seen in other online CBT treatments and some in-person CBT treatments for panic disorder.

In an earlier Australian study, Kiropoulos et al. (2008) randomly assigned 86 individuals with panic disorder (some with and some without agoraphobia) to receive either a Web-based CBT treatment (Panic Online) or standard, best-practice CBT for panic disorder delivered in person. They found both interventions to be about equally effective, and participants in both groups showed significant improvements in terms of therapist-rated panic disorder severity; self-reported panic severity; frequency of panic attacks; anxiety, depression, and stress (measured with the DASS); and quality of life (World Health Organization Quality of Life-Brief). Participants who received in-person treatment did find greater enjoyment in their therapeutic interactions, and their therapists gave them higher ratings for treatment compliance and understanding. However, the online treatment required significantly less time from therapists.

Yet another study of Web-based CBT for panic disorder ( N =55), conducted in Australia by Klein, Richards, and Austin (2006) , compared a Web-based intervention that featured limited email contact with a printed CBT self-guided manual combined with phone contact with a therapist; they found no significant differences in terms of improvements in self-rated physical health and decreases in panic disorder symptoms, panic-related thought, and negative affect. However, the Web-based intervention was associated with better outcomes in terms of agoraphobia (as rated by staff people), and both interventions were superior in all outcomes compared with an information-only control group. As noted previously, other research from this group did not find any significant benefit in increasing the frequency of therapist email contact from once a week to three times a week when using Web-based CBT for panic disorder ( Klein et al., 2010 ).

Carlbring and colleagues (2005) studied an intervention providing Web-based CBT with minimal therapist email contact and compared it with in-person CBT in a group of 49 clients with panic disorder and found the two about equally effective. In a reanalysis of data from this earlier study, Andersson and colleagues (2008) determined that individuals with high levels of agoraphobic avoidance (on the Agoraphobic Cognitions Questionnaire) had significantly better outcomes in Web-based treatment than when receiving in-person treatment. The authors suggested that this is likely because in the early phases of treatment, the use of the Internet helped these clients overcome resistance to help-seeking and/or leaving home for appointments, but they noted that in later phases, both treatment conditions exposed individuals to anxiety-provoking situations. They also found that individuals with co-occurring personality disorders fared better with in-person treatment than Internet-delivered treatment.

Amstadter, Bromon-Fulks, Zinzow, Ruggiero, and Cercone (2009) reviewed research concerning Web-based interventions for problems that can stem from trauma exposure, including subdiagnostic levels of PTSD symptoms, substance use disorders, and depression. However, only four of the interventions they reviewed specifically treated PTSD symptoms. They concluded that, although the results of these studies are promising, more research is needed. Another review by McLean et al. (2010) was more thorough in its consideration of technology as useful in the treatment of PTSD, and it addressed such issues as the use of VRET in addition to Web- and computer-based interventions. These authors observed that such interventions have been found to be associated with significantly better outcomes than waitlist controls or supportive therapy, but they were unable to identify any studies that compared in-person and computerized CBT intervention for PTSD. They concluded that computer technology can increase access to care for people with PTSD by bridging geographical distances, making care available for people who would otherwise have to wait for open treatment slots, providing cost-effective interventions that can reach people who do not have the economic resources or insurance coverage for care, and providing a greater sense of protection for clients who might prefer the more anonymous interactions of a computerized intervention. Additionally, they noted that computerized treatment elements can enhance and improve existing interventions (e.g., tailored psychoeducation, computerized self-monitoring, VRET).

Specific computer- and Web-delivered interventions for treating PTSD include the Interapy program available in Dutch, a CBT treatment that provides exposure therapy over computers with supervision from a therapist, which is included in the reviews discussed in the “ Treatments for Multiple Anxiety Disorders ” section ( Lange, van de Ven, & Schrieken, 2003 ). Early research found that participants who received the intervention had significantly greater reductions in trauma-related symptoms and improvements in mood and anxiety compared with individuals assigned to a waitlist-only control group ( Lange, Rietdijk et al., 2003 ). More recent reports appear to confirm the utility of the intervention in reducing PTSD symptoms and improving functioning, but those studies are only available in Dutch.

Another partially computerized CBT treatment for PTSD, which is being used by VA and the U.S. military, is the Delivery of Self-Training and Education for Stressful Situations (DE-STRESS) intervention (which also makes use of some in-person or telephone interviews). This intervention is also included in the previously cited reviews. The intervention teaches clients techniques to help them cope with situations that bring on recall of traumatic events and to assist them in managing negative affect ( Litz, Engel, Bryant, & Papa, 2007 ). In a research trial, DE-STRESS ( n =24) was compared with another Web-based intervention that used supportive counseling ( n =21); among treatment completers (about 30 percent of participants dropped out from both treatment arms), those who used DE-STRESS had, at a 6-month follow-up assessment, significantly lower anxiety (measured with the Beck Anxiety Inventory [BAI]), depression (measured with the BDI), and PTSD symptoms (using the PTSD Symptom Scale–Interview Version), and they were significantly less likely to meet clinical criteria for a PTSD diagnosis ( Litz et al., 2007 ).

Klein et al. (2010) conducted an evaluation of an online CBT treatment for PTSD in Australia with 22 clients who had PTSD and were assessed at baseline, immediately after treatment, and 3 months after treatment. Treatment consisted of a 12-week psychoeducational intervention provided online with support from a trained therapist. The authors found that, at the 3-month assessment, 77 percent of participants no longer met diagnostic criteria for PTSD. The majority (69 percent) of participants rated the intervention positively, and the average therapist time required per client was relatively low (194.5 minutes).

In another Australian study, individuals with PTSD were randomly assigned to either a waitlist control group ( n =21) or to a group using a Web-based PTSD treatment ( n =23), which included a therapist-moderated online discussion forum, automatic reminder emails, and instant messaging chats with a therapist, with a mean therapist time of 104 minutes per client (Spence et al., 2011). Using an intent-to-treat analysis, the authors found that those who received the intervention had significantly lower levels of PTSD, depressive, and GAD symptoms following treatment than did members of the control group, but no significant differences were found in terms of psychosocial impairment (as measured with the Sheehan Disability Scale). Another study of a Web-based PTSD treatment conducted with German-speaking clients ( N =34) provided data from a 1.5-year follow-up period ( Knaevelsrud & Maercker, 2010 ). The authors found that significant reductions in intrusion, avoidance, and hyperarousal PTSD symptoms (evaluated using the Impact of Event Scale) measured at the end of treatment were maintained at the 1.5-year follow-up, as were significant reductions in anxiety and depression symptoms (evaluated with the SCL-90-R).

Possemato, Ouimette, and Knowlton (2011) conducted a pilot study of a brief, self-directed, Web-based intervention (written emotional disclosure [WED]) with 31 combat veterans who had screened as likely having PTSD. Participants were randomly assigned to use the intervention or to a control condition that used a Web page to discuss time management. Both intervention and control conditions involved written text and took place in three 20-minute sessions. There were no significant differences between the two groups in terms of reductions in PTSD symptoms, but the authors observed that participants who expressed more emotional content and thoughts related to their traumatic experiences were significantly more likely to have decreases in PTSD symptoms if they used the WED intervention.

McLean and colleagues (2010) and Gerardi et al. (2010) both reviewed research on VRET for treating PTSD and noted that case studies, nonrandomized studies, and one small randomized controlled trial suggest that this is a useful method for delivering exposure therapy. VRET, which uses computers and special equipment to create a three-dimensional simulation of the world, was originally used to assist in the treatment of specific phobias ( Rothbaum, Rizzo, & Difede, 2010 ). In addition to providing audio and visual content, new innovations in VRET allow for the addition of olfactory and tactile stimuli ( Rizzo, Rothbaum, & Graap, 2007 ). Developers have created specific programs to simulate environments in which trauma may have originally occurred, including the Vietnam War, the Iraq and Afghanistan Wars, the World Trade Center attack, terrorist bus attacks in Israel, and the Portuguese colonial wars in Africa ( Rothbaum et al., 2010 ).

As an example of VRET research, Difede, Cukor, Patt, Giosan, and Hoffman (2006) compared outcomes for nine individuals who used a VRET program that simulated the attack on the World Trade Center and eight who were on a waitlist for treatment. Most participants were disaster relief workers, and all had experienced trauma as a result of the attack; six of the nine who used VRET had failed to have a response to exposure therapy in prior treatment attempts. They found that those who received the intervention had significant decreases in PTSD symptoms (measured with the Clinician-Administered PTSD Scale), and five participants no longer met criteria for PTSD at the end of treatment, whereas individuals in the control group had no significant decreases in PTSD symptoms, and all were still diagnosed with PTSD.

Preliminary data on the first 20 treatment completers of the Virtual Iraq VRET program also suggest that it is an effective intervention ( Rizzo et al., 2010 ). Participants had significant decreases in PTSD symptoms (as measured with the PTSD Checklist–Military Version), and 16 of the 20 no longer met diagnostic criteria for PTSD at the conclusion of treatment. An evaluation of VRET for 24 active-duty soldiers who had been deployed to either Iraq or Afghanistan also found significant decreases in PTSD symptoms following its use, with 62 percent ( n =15) experiencing at least an 11-point decrease in PTSD Checklist–Military Version scores ( Reger et al., 2011 ). A Spanish study also evaluated a VR CBT intervention for PTSD as well as other stress-related disorders (i.e., adjustment disorder, pathological grief) with a group of 39 clients (10 with PTSD) and found the intervention to be as effective as CBT delivered in person ( Baños et al., 2011 ).

Advantages of VRET over other forms of exposure therapy include the therapist's ability to share the experience of the trauma with the client, the ability to recreate traumatic experiences that would be too difficult and/or costly to do in real life, the ability to subtly adjust the client's experience, and the ability to recreate a more realistic simulation ( Rothbaum et al., 2010 ). Potential disadvantages of the technology include the possibility of hardware or software malfunctions that could interrupt the therapy, the expense involved in creating a specific simulated environment, and the possibility that some clients may be distracted by the technology instead of focusing on the experience it presents ( Rothbaum et al., 2010 ). See Part 1, Chapter 1 of this TIP for more information on VR.

Telephone-delivered CBT has been used successfully to treat PTSD symptoms and reduce psychological distress among people who have undergone a potentially traumatizing medical procedure (hematopoietic stem-cell transplantation), although individuals did not necessarily meet criteria for a PTSD diagnosis ( DuHamel et al., 2010 ). Participants in the study ( N =89) received either 10 sessions of phone-delivered CBT or assessment only; compared with the control group, those who received the intervention reported significantly fewer PTSD symptoms as well as fewer symptoms of depression, even after controlling for demographic and medical factors. These benefits persisted at a final follow-up assessment 1 year after the study began. For information on PTSD Coach, a smartphone app for veterans developed by VA's National Center for PTSD and DoD's National Center for Telehealth and Technology, see Part 1, Chapter 1 of this TIP.

PTSD treatment can also be delivered using video conferencing. In a Canadian study, 16 clients with PTSD receiving CBT via video conferencing were compared with 32 clients who were treated in person ( Germain, Marchand, Bouchard, Drouin, & Guay, 2009 ). Both groups experienced significant improvements in PTSD symptoms and overall functioning, and no significant differences in these outcomes were observed between the groups. In a nonrandomized study, Gros, Yoder, Tuerk, Lozano, and Acierno (2011) evaluated exposure therapy for PTSD delivered using video conferencing. They compared outcomes for 62 veterans with PTSD who received exposure therapy using video conferencing and 27 who received it in person. Despite both groups experiencing significant reductions in measures of PTSD symptoms (PTSD Checklist–Military) and symptoms of depression (BDI) following treatment, those who received treatment in person had significantly greater reductions, after controlling for pretreatment scores, than did those who used video conferencing. However, the lack of random assignments may have skewed the authors' findings. The authors also noted that the effect sizes for reductions in symptoms among those who used video conferencing were comparable with those typically reported for exposure therapy, whereas the effect sizes for those who received treatment in person were more robust than had been expected.

A few different computer- and phone-assisted interventions have been developed for treating OCD and are reviewed by Moritz, Wittekind, Hauschildt, and Timpano (2011) . These authors briefly described interventions with published literature that have been found effective at reducing OCD symptoms or otherwise assisting in OCD treatment (e.g., by reducing the required length of time for traditional treatments). The interventions included in the review make use of a variety of approaches and include a VR software program ( Kirkby et al., 2000 ); a Web-based CBT treatment for hoarding behaviors ( Muroff, Steketee, Himle, & Frost, 2010 ); and a behavioral intervention, known originally as BT Steps and more recently as OCFighter, that makes use of computer and phone technology ( Bachofen et al., 1999 ; Greist et al., 2002 ; Kenwright et al., 2005 ; Marks et al., 1998 ). They also described an evaluation of a program for children and adolescents with OCD that uses Web cams to deliver treatment over the Internet. Of the programs intended for adults that are included in the review, BT Steps is the only one to have been evaluated in a randomized controlled study, which found it superior to relaxation training but inferior to therapist-guided behavioral therapy ( Greist et al., 2002 ). Another study of BT Steps found a therapist-guided version to be superior to a version that relied solely on client motivation to initiate therapist contact ( Kenwright et al., 2005 ).

A review by Lack & Storch (2008) also found computer-based interventions effective in both assessment and treatment of OCD. Lovell and Bee (2011) conducted another review of TAC for OCD. They discussed most of the same studies cited in this section and also provided more. Included in their review are five studies of computerized CBT (four of which involve the BT Steps/OCFighter program), two studies of interventions using telephones to deliver CBT, one study using video conferencing (for youth), and five studies of bibliotherapy.

Herbst et al. (2012) conducted a review of research involving the use of phone and computer technology to assist in treating OCD. They reviewed 17 studies using telephone and computer technology (15 of which were for adults). Based on their review, they concluded that these interventions were effective at reducing OCD symptom severity and had effect sizes comparable with in-person interventions. They observed that dropout rates were relatively high for these interventions, but BT Steps was able to lower dropout rates and increase effect sizes by introducing therapist-initiated phone contacts with clients. Computerized interventions, in particular, may reach clients who would not otherwise seek treatment, often because of a high degree of shame (a major barrier to treatment for people with OCD). They also observed that studies typically excluded participants with certain co-occurring disorders (e.g., major depression) and therefore these interventions may not be effective for those individuals. Some of the studies reviewed did find smaller effect sizes for clients with hoarding symptoms; these interventions may not be optimal for them.

Treatments for multiple anxiety disorders

Although most computer/Internet interventions are designed to treat a specific anxiety disorder, a few programs are available that can treat multiple anxiety disorders, and in some cases, depression or other mental disorders as well. For example, in Australia, Titov, Andrews, Johnston, Robinson, and Spence (2010) developed and evaluated a Web-based CBT intervention for anxiety symptoms and disorders, the Anxiety Program, which involved online lessons, homework assignments, an online discussion forum, and weekly email or phone contact from a therapist. They evaluated the intervention with individuals who had diagnoses of GAD, panic disorder, and/or social phobia. Participants were randomly assigned to the intervention ( n =42) or to a waitlist control group ( n =44). At a 3-month follow-up assessment, 32 individuals from the treatment group and 36 from the control group were available; those who received the intervention had significantly lower anxiety symptoms as measured by the Generalized Anxiety Disorder–7 Item Scale, the Social Phobia Screening Questionnaire, and the PDSS-SR, although they were not significantly lower on the Penn State Worry Questionnaire. At the end of treatment, 40 percent of those who received the intervention were in remission from their anxiety disorders compared with 8 percent of those in the control group.

More recently, the Anxiety Program was expanded to address depression as well as anxiety disorders and was retitled the Wellbeing Program ( Titov et al., 2011 ). In an initial evaluation, 77 participants with diagnoses of panic disorder, GAD, SAD, and/or major depression were randomly assigned to an intervention group that used the program or to a waitlist control group. A relatively high number of those who received the intervention (81 percent) completed all treatment modules, and participants reported a high level of satisfaction with the program.

Therapists spent a mean of 87.6 minutes per client. At the conclusion of treatment, participants who used the Wellbeing Program had significantly lower levels of both anxiety and depression symptoms than did those in the control group. At a 3-month posttreatment follow-up assessment, 62 percent of those who used the intervention no longer met diagnostic criteria for their primary disorder, and 54 percent no longer met criteria for any of the four disorders treated by the program. Dear et al. (2011) also evaluated a brief version of the Wellbeing Program that presented material in five sessions rather than the eight originally required. This version of the program was also very well-received by participants, who showed improvements in anxiety and depression that were of a similar magnitude to those seen in evaluations of the full program, and the mean amount of time per client spent by therapists for the entire intervention was reduced to 44.6 minutes.

In Sweden, Carlbring et al. (2011) evaluated yet another Web-based treatment for multiple anxiety disorders (i.e., SAD, GAD, panic disorder) that included personalized feedback and guidance from a therapist. The authors randomly assigned 54 participants to the intervention or to a control group that required participation in an online moderated discussion group. At the end of treatment, participants who used the program had significantly greater reductions in measures of anxiety and depressive symptoms and significantly greater improvements in quality of life compared with those in the control group. At a 1-year follow-up assessment, differences in depressive symptoms and quality of life remained significant, but at a 2-year follow-up assessment, no significant differences between the groups remained.

Ruwaard, Lange, Schrieken, and Emmelkamp (2011) reported on multiple trials conducted in the Netherlands of a Web-based CBT intervention (Interapy) that uses asynchronous text messages from therapists (similar to email exchanges) to treat a variety of mental disorders, symptoms, and related problems. Although the review includes one study each involving people with bulimic symptoms and depression, the majority of studies involved people with anxiety disorder symptoms (i.e., panic disorder, PTSD symptoms) or general anxiety/stress. Using an intent-to-treat analysis, the authors found that, across multiple trials (with a combined N =840), Interapy was associated with significantly larger reductions in primary symptom measures than were controls, and effect sizes for the treatment were comparable with those observed for CBT interventions delivered in person. Interapy was also well-received by clients, and 82 percent of participants in these trials completed treatment, which ran from 5 to 16 weeks depending on the disorder addressed and other factors. Also, 79 percent of participants completed all follow-up assessments conducted from 1 to 3 years after treatment.

Another group of Australian researchers evaluated Anxiety Online, a fully automated, free, Web-based intervention intended to assess and treat a variety of anxiety disorders ( Klein, Meyer, Austin, & Kyrios, 2011 ). Participants were diagnosed by the program with GAD ( n =88), SAD ( n =50), panic disorder ( n =40), PTSD ( n =30), or OCD ( n =17) and completed the online program and a reassessment 12 weeks after completion. At the follow-up assessment, the researchers found significant reductions in disorder severity across all five disorders and significant decreases in the number of participants meeting diagnostic criteria for all of the disorders except panic disorder (for which there was a decrease in the number, but not a significant one). For participants with GAD, SAD, OCD, and PTSD, the authors also found significant improvements in quality of life; for those with GAD, PTSD, or panic disorder, they also found significant reductions in general psychological distress.

Another well-evaluated computerized intervention available to treat a number of different anxiety disorders is the Coordinated Anxiety Learning and Management (CALM) program. CALM uses computers to deliver CBT to treat panic disorder, SAD, GAD, and/or PTSD for patients in a collaborative primary care setting ( Craske et al., 2009 ; Roy-Byrne et al., 2010 ; Sullivan et al., 2007 ). Although the intervention includes in-person contact with an anxiety clinical specialist (usually a nurse or social worker) as well as the patient's primary care doctor, a large component is the computer-delivered CBT program, which guides both patients and providers ( Craske et al., 2009 ). At a follow-up assessment 18 months after treatment, the CALM intervention was associated with significantly greater improvements in global anxiety symptoms (as measured by the Brief Symptom Inventory-12) and significantly higher rates of remission from anxiety disorders compared with standard primary care treatment for anxiety disorders ( Roy-Byrne et al., 2010 ). The program has also been well-received by both clinicians and patients ( Craske et al., 2009 ). Research continues on CALM; for example, Wetherell et al. (2013) explored the difference in effects for younger versus older adults, but such research is outside the scope of this TIP, as the CALM research concentrates on CALM as a whole (e.g., in-person contacts, medications, computerized CBT), rather than on the technological innovations.

Interventions to reduce trait anxiety and subclinical anxiety symptoms

Computer software is available to train individuals with high levels of anxiety to either interpret ambiguous events in a more positive manner or focus their attention differently to reduce anxiety (typically measured as trait anxiety, but focused on specific anxiety disorder symptoms in some more recent studies). This is a process known as cognitive bias modification or interpretive bias modification ( MacLeod & Mathews, 2012 ). In interpretive bias modification, the software uses imagery and/or verbal messages to help people imagine themselves in situations that might potentially provoke anxiety. These situations are first presented in an ambiguous manner but are then resolved in a positive manner, demonstrating that any assumed negative interpretation was wrong. Participants then answer a short multiple-choice question about the nature of the events. In programs that modify attentional bias, individuals are retrained to focus less attention on negative information and more on positive information, thus also affecting their interpretation of events. As noted previously, cognitive bias modification programs have been used to treat GAD and SAD, but their most common use has been to reduce trait anxiety and/or subclinical anxiety symptoms.

MacLeod and Mathews (2012) reviewed the theoretical basis for using cognitive bias modification to address anxiety and the research demonstrating its effectiveness in doing so. A number of small, controlled studies included in that review have found that users of interpretive bias modification software experienced significant reductions in trait and/or state anxiety, as well as improvements in other, more general measures of behavioral health ( Mathews, Ridgeway, Cook, & Yiend, 2007 ; Salemink, van den Hout, & Kindt, 2009 ). This software has been successfully delivered via the Internet ( MacLeod, Soong, Rutherford, & Campbell, 2007 ; Salemink et al., 2009 ) as well as on computers in research settings. MacLeod and Mathews (2012) also cited studies showing that this type of software can reduce subclinical levels of OCD symptoms ( Najmi & Amir, 2010 ), social anxiety symptoms ( Amir, Beard, Taylor et al., 2009 ; Amir, Weber, Beard, Bomyea, & Taylor, 2008 ; Beard et al., 2011 ; Schmidt et al., 2009 ), extreme worrying ( Hazen, Vasey, & Schmidt, 2009 ), and GAD symptoms ( Amir, Beard, Burns et al., 2009 ). However, research has not found these methods effective for treating specific phobias, where changes in attentional bias do not appear to be associated with reduction in fear ( Reese, McNally, Najmi, & Amir, 2010 ; Van Bockstaele et al., 2011 ).

Use in Treatment of Mood Disorders

Although phone- and Web-based services can be used in the treatment of any mood disorder, all of the studies included in this section address major depression or depressive symptoms. A few programs addressing BPD are also discussed in the “ Use in Continuing Care/Symptom Monitoring for Mental Illness ” section.

Video conferencing interventions to treat depression

Video conferencing has been successfully used for some time to treat mood disorders, most often major depression ( Backhaus et al., 2012 ). García-Lizana and Muñoz-Mayorga (2010a) , however, found only four randomized controlled trials of video conferencing interventions for depression (and one of those was for children and adolescents). They observed that such approaches are promising but found that the research evidence is insufficient to make a conclusion regarding effectiveness.

Trials with different populations have found video conferencing to be an effective way to deliver treatment for depression. For example, in a VA study, Ruskin et al. (2004) randomly assigned 119 veterans with depressive disorders to receive in-person mental health services or similar treatment delivered via video conferencing technology. The authors found no significant differences in outcomes, with both groups experiencing significant improvements in depressive symptoms. There were also no differences between the groups in terms of dropout rates, participant satisfaction, medication compliance, or keeping scheduled appointments.

In a more recent study, Moreno, Chong, Dumbauld, Humke, and Byreddy (2012) compared treatment for depression delivered by a psychiatrist using video conferencing technology for 80 Latinos with depression with standard care delivered by a primary care physician for 87 similar individuals. Both groups experienced significant improvements in disability, quality of life, and depression symptoms (according to both self-report and independent clinical evaluations), but those receiving the video conferencing intervention experienced significantly greater improvements in all areas. The video conferencing intervention was delivered using standard, commercially available Web cams and computers.

Telephone-based therapies to treat depression

Some research suggests that telephone-based therapies can also effectively treat depression. Mohr et al. (2008) conducted a meta-analysis of 12 trials of telephone-based interventions that addressed depression and found that such interventions reduced symptoms of depression significantly more than did control conditions and were associated with significant reductions in those symptoms from baseline to posttreatment assessments (reductions comparable with those observed in many in-person treatment evaluations). They also found a much lower attrition rate (7.6 percent across all studies) than typically found in trials of interventions for depression delivered in person. However, many of the participants in the studies reviewed had other mental or substance use disorders or serious health problems that may have affected outcomes. Telephone-based CBT has also been used in conjunction with pharmacotherapy, and in a number of studies, its use has been associated with greater reductions in depressive symptoms as well as improved medication compliance ( Leach & Christensen, 2006 ; Mohr et al., 2008 ).

A later study of a telephone-based CBT intervention (prior studies of which were included in the review by Mohr et al., 2008 ) conducted with 85 veterans did not find any significant differences in depression outcomes between those who received the intervention and those who received standard care from community-based outpatient clinics, although both groups improved ( Mohr et al., 2011 ). The authors noted that the small sample size, the low level of participation in the study among those who were recruited, and the fact that this group may have had more severe depression than those treated in previous trials of this telephone-based intervention may have affected outcomes. Yet another study from the same researchers compared CBT for depression delivered by telephone with that delivered in person to a group of 325 primary care patients ( Mohr et al., 2012 ). At the end of treatment, there were no significant differences in depressive symptoms between the intervention and control groups, with both showing significant improvements. Participants who received the telephone-delivered intervention were also significantly more likely to remain in treatment for the duration. However, at the 6-month follow-up, depressive symptoms were significantly lower for those who received in-person services, although they remained significantly lower than at baseline for those in the telephone-administered CBT group.

Another recent trial of a telephone-based CBT intervention was conducted with 291 patients with diabetes who had elevated symptoms of depression indicated by BDI scores of 14 or greater ( Piette et al., 2011 ). The authors found that the intervention added to treatment as usual was associated with significantly greater reductions in symptoms of depression than was the addition of printed information about depression to standard treatment. The group who received the intervention had significantly better health outcomes and greater involvement in recommended exercise activities compared with the control group.

Another phone-based intervention not included in the previously cited review was conducted with 604 employees covered by a managed behavioral health plan who had previously been screened as potentially suffering from depression ( Wang et al., 2007 ). Participants were randomly assigned to receive either standard care alone (consisting of referral and information about services covered by their health plan) or standard care with the addition of care management provided by phone and eight sessions of phone-based CBT. At assessments conducted 6 and 12 months after the study began, participants who received the phone-based services, compared with those in the control group, had significantly lower level of depressive symptoms (according to the Quick Inventory of Depressive Symptomatology), were significantly more likely to have retained their jobs, and had spent significantly more hours working.

According to a Cochrane review by Dale, Caramlau, Lindenmeyer, and Williams (2008) , peer support delivered by telephone has also been associated with significant decreases in depressive symptoms for women experiencing postnatal depression, as well as improvements in behaviors, such as getting health screenings and changing their diets. Computerized interventions for depression have generally been found to be effective at reducing depressive symptoms, particularly if the computerized intervention is supported by a clinical professional who is also working with the client ( Andersson & Cuijpers, 2009 ; Griffiths & Christensen, 2007 ; Griffiths et al., 2010 ; Newman et al., 2011a ). Research, although limited, also indicates that computer-based/Web-based interventions for depression are well-received by clients ( Kaltenthaler et al., 2008 ).

Computerized interventions to treat depression

A meta-analysis of 12 computerized interventions for depression (10 of which were delivered over the Internet) found them to have a moderate effect size in relation to waitlist controls ( Andersson & Cuijpers, 2009 ). However, when only those eight interventions that provided professional support in addition to the computerized intervention were considered, the effect size was significantly greater. Newman et al. (2011a) conducted another review of computerized interventions for depression that required little or no therapist involvement. They found that minimal or entirely self-guided interventions were of some benefit (greater than no treatment) to people with subclinical levels of depression, but many users of such programs do not complete treatment. They speculated that adding a motivational component delivered by a therapist could improve such treatment for people with subclinical depression. For people with major depression, the most effective intervention evaluated was one that involved minimal therapist contact (250 minutes for the entire treatment). The authors also cautioned about a number of methodological problems with studies they reviewed.

In the United Kingdom, Button, Wiles, Lewis, Peters, and Kessler (2012) evaluated factors associated with treatment response for 297 individuals with depression (according to the criteria provided in the International Statistical Classification of Diseases and Related Health Problems, tenth revision) who received Web-based CBT for depression. They found no significant effects relating to participant age, education, or history of prior treatment response. Of the factors included in their evaluation, only greater pretreatment severity of depression and being separated, widowed, or divorced were associated with significantly greater treatment response with the Web-based interventions than with the waitlist control group.

Specific Web-based programs to treat depression

A number of specific Web-based interventions have been developed to treat depression, most of which make use of CBT principles. For example, the Overcoming Depression on the Internet (ODIN) program is a self-guided CBT intervention in addition to standard care that, in an initial trial, was not found to be more effective than usual care alone. However, in a post hoc analysis, individuals with CES-D scores of 20 or less showed modest improvement ( Clarke et al., 2002 ). Clarke et al. (2005) conducted a randomized controlled trial comparing ODIN, with the addition of reminders provided via either postcards or telephone calls, with treatment as usual. Participants who received either version of the intervention had greater reductions in depressive symptoms, especially for individuals who had CES-D scores of 28 or higher at the start of the study. Remission rates also appeared to be higher for those who received ODIN, as 20 percent more participants in the two intervention groups moved from the moderate or high levels of CES-D scores to the normal range than did participants in the control group.

De Graaf, Hollon, and Huibers (2010) compared a computerized CBT intervention for depression (Color Your Life) with treatment as usual, either by itself or in combination with the online intervention, for 303 individuals who had been previously screened as having mild to moderate levels of depressive symptoms according to the BDI-II. The researchers found no significant differences among treatment groups in terms of depression severity, remission to elevated symptoms following improvement, or rates of reliable improvement during the follow-up period. However, when they evaluated outcomes for participants who were considered extremely positive responders (i.e., who had scores on the Dysfunctional Attitudes Scale-Form A and the Questionnaire Without Questions that indicated higher levels of optimism), those who had a high level of positive responding had significantly greater reductions in depressive symptoms if they received the online intervention than if they had treatment as usual. Also, participants who had a diagnosis of major depression or had parents with a mental disorder (typically associated with more severe psychopathology) had significantly better outcomes if they received the combined Web-based and standard treatments than if they received either one alone. According to another analysis of data from this study, participants who received the online intervention also made significantly less use of primary care services, specialized mental health services, and antidepressant medication during the first quarter following treatment, but those differences disappeared over time ( de Graaf et al., 2011 ).

Another fully computerized intervention, Blues Begone, was designed to treat both depression and co-occurring anxiety. For this study, 100 individuals who were referred to the study by their primary care doctors used the program (which was provided on a CD-ROM), and 58 completed the program. Of program completers, 62 percent were found to have had clinically significant changes in BDI-II scores, and 45 percent had meaningful reductions in BAI scores ( Purves, Bennett, & Wellman, 2009 ). Griffiths and Christensen (2007) reviewed a number of studies involving two Web-based interventions for depression, MoodGYM and BluePages, that were developed in Australia. Both programs demonstrated significantly greater reductions in symptoms of depression and anxiety than did no-treatment controls, and those effects continued for at least 1 year after treatment.

Mackinnon, Griffiths, and Christensen (2008) provided a 1-year follow-up evaluation of MoodGYM based on 60 percent of the 525 individuals who had been randomly assigned to use a Web site that provided psychoeducation about depression, to receive the online MoodGYM intervention, or to participate in a placebo control group. Both intervention groups had significant declines in depressive symptoms (measured with the CES-D), and those reductions in symptoms continued through the 1-year follow-up. Both intervention groups also had significantly greater reductions in symptoms than did participants in the control group, but there were no significant differences between the two interventions at the 1-year follow-up.

In yet another evaluation of MoodGYM, Farrer et al. (2011) compared 6-month follow-up outcomes for callers to an Australian counseling hotline who had high levels of psychological distress (i.e., scores of 22 or higher on the 10-item Kessler Psychological Distress Scale) and who received MoodGYM for depression ( n =36), weekly telephone follow-up calls from a lay counselor ( n =36), a combination of the two ( n =41), or access to standard care only ( n =35). Participants who received MoodGYM, either with or without the telephone component, had significantly lower levels of depression at follow-up than did participants in the other two groups, but there were no significant differences between the two MoodGYM groups in terms of compliance or outcomes.

In Germany, a Web-based CBT program to treat depression (Deprexis) was evaluated with a cohort of 320 individuals who received the intervention in addition to treatment as usual and 76 who received the intervention after a 9-week delay while receiving treatment as usual throughout the study period ( Meyer et al., 2009 ). Participants who received the intervention immediately had significant improvements in depressive symptoms (measured with the BDI) and social functioning (according to the Work and Social Adjustment Scale), and these improvements continued to be maintained 6 months after the intervention ended.

In a later evaluation of Deprexis (improved over its prior form by the addition of email reminders and nonfinancial incentives), 210 participants were randomly assigned to receive one version of the intervention or to a waitlist control group ( Moritz et al., 2012 ). At the conclusion of their treatment, participants who used Deprexis had, relative to those in the control group, significant improvements in symptoms of depression, dysfunctional attitudes, quality of life, and self-esteem. The largest improvements were observed among individuals who had baseline BDI scores between 19 and 29. The intervention had an 82 percent completion rate.

Another Web-based CBT intervention for depression is the Sadness program, which includes psychoeducation, online homework, an online discussion forum, and regular email contact with a therapist ( Perini, Titov, & Andrews, 2009 ). This program was evaluated in Australia with 45 individuals who were randomly assigned to the intervention or to a waitlist control group. At the end of treatment, participants in the intervention group had significant decreases in depressive symptoms from baseline (as measured by both the BDI and the PHQ-9) and had significantly lower levels of symptoms than did participants in the control group. The intervention had effect sizes comparable with other Web-delivered interventions for depression that included therapist support; effect sizes were larger than those of interventions that did not have such support.

Titov, Andrews, Davies, et al. (2010) compared two versions of the Sadness program—one that was moderated by a psychiatrist ( n =45) and included access to an online discussion forum and another that was moderated by a technician with no healthcare or counseling experience ( n =41) with a waitlist control group ( n =40). At the conclusion of treatment, participants in both Sadness interventions had significantly better outcomes according to the BDI-II and the PHQ-9 than did those who were in the control group, but there were no significant differences in outcomes from one another. However, at 4 months after treatment, participants in the technician-led group had further improvements in PHQ-9 scores and scores that were significantly better than those in the psychiatrist-led group. Completion rates were also good, with 80 percent of those in the technician-led group and 70 percent of those in the psychiatrist-led group completing all treatment sessions within the required time. The authors concluded that this intervention, which does require supportive messages from a staff person, can be conducted effectively by nonclinical staff members. In Great Britain, Kessler and colleagues (2009) randomly assigned 297 individuals with depression (indicated by BDI scores of 14 or more) to receive an online CBT intervention in addition to standard treatment or to receive standard treatment alone. Significantly more participants who received the online intervention (38 percent) had BDI scores of 10 or less (indicating remission from depression) at a 4-month assessment than participants in the control group (24 percent).

Comparing depression treatment programs

Dutch researchers compared two different approaches to Web-based treatment ( Warmerdam, van Straten, Jongsma, Twisk, & Cuijpers, 2010 ). These authors randomly assigned 263 participants to a CBT online treatment ( n =88), to a problem-solving therapy online treatment ( n =88), or to a waitlist control group ( n =87). At an 8-week posttreatment assessment, both treatment groups had, compared with the control group, larger reductions in depressive symptoms and significantly greater improvements in terms of dysfunctional attitudes, worry, negative problems orientation, and feelings of control. However, only participants in the problem-solving therapy group had significantly greater improvements relating to positive problem orientation and avoidance behavior compared with the control group, suggesting somewhat better results than those achieved with CBT.

In a comparison study conducted in Sweden of treatments for major depression, 88 individuals who met the inclusion criteria and agreed to participate were randomly assigned to a guided self-treatment based on CBT principles delivered over the Internet, to a personalized treatment delivered by email, or to a waitlist control group ( Vernmark et al., 2010 ). The self-guided intervention consisted of weekly modules and homework assignments. Standard CBT components were presented, and brief support was provided during the treatment. The other intervention group received email therapy that was tailored for each unique participant. Both groups had significant improvements in depressive symptoms at the end of the 8 weeks of treatment and at a 6-month follow-up assessment, but so did the control group. At the end of treatment, 34.5 percent of the self-guided group and 30 percent of the email group had BDI scores of less than 9, and at follow-up, the percentages of those with BDI scores below 9 increased to 47.4 percent for those in the self-guided group and 43.4 percent for those who received email therapy. The email therapy participants received an average of 8 treatment emails (with a range from 3 to 11); the authors noted that almost 10 times as much therapist time was required for the email therapy as for the guided Internet intervention.

Interventions for subclinical depression

A number of the trials of Web-based interventions for depression include individuals with a level of depressive symptoms that do not meet diagnostic criteria for a depressive disorder; thus, those interventions may be effective for individuals with subclinical depression. In their review of self-guided TAC interventions and minimal contact interventions for anxiety and mood disorders, Newman et al. (2011a) concluded that the research, although limited, indicated that these types of interventions were more effective in treating subclinical depression than in treating depressive disorders.

In a meta-analytic review of 12 randomized controlled trials of Web-based interventions for adults with symptoms of depression and/or anxiety, Spek et al. (2007) found that such interventions produced meaningful improvements, and they concluded that interventions with therapist support were especially effective. In an evaluation of their own Web-based randomized controlled trial of interventions for adults with subclinical depression ( N =301), Spek, Cuijpers, et al. (2008) found that, 1 year after the interventions, 62 percent of the participants who had received eight Internet-based CBT intervention modules scored below 13 on the BDI, and 45 percent of those who had received 10 weekly group CBT sessions scored below 13. Of those in the waitlist control group, 38 percent had BDI scores below 13 at the 1-year follow-up. They also estimated that for every four clients with subthreshold depression who were treated with the Web-based intervention rather than being on a waitlist, one would have symptoms reduced below the cutoff level. Thus, not only was the outcome for the Internet-based CBT treatment adult group significantly better than for those adults in the waitlist group, but the results also demonstrated that meaningful reductions in the burden of depressive episodes can be achieved by reaching more individuals through Internet-based efforts.

More recently, Morgan, Jorm, and Mackinnon (2012) conducted a randomized controlled trial of an email-based treatment specifically for subclinical depression, which they also conceptualized as a prevention intervention for major depression. In the study, 1,326 individuals were randomly assigned either to receive the automated email intervention, which provided advice on self-guided methods to reduce symptoms of depression, or to a control group, which received emails containing general information about depression. At the end of the study, participants who received the intervention had significantly lower levels of depressive symptoms, although the difference was not large.

Medication management interventions to treat depression

A few interventions are available that use phones or computers to help individuals with depression properly manage their medication. Simon, Ludman, Tutty, Operskalski, and Von Korff (2004) randomly assigned 600 individuals who had recently started on antidepressant medication as prescribed by a primary care doctor to receive usual primary care alone, primary care with coordinated phone-based care management intended to improve medication compliance, or usual primary care with coordinated phone-based care management and eight sessions of telephone-based CBT. Compared with those who received standard care alone, participants who received the CBT plus phone-based management had significantly lower levels of depressive symptoms (on the Hopkins Symptom Checklist Depression Scale [HSCL] and the PHQ-9), were significantly more likely to report that they felt their depression was “much improved,” and were significantly more likely to indicate that they were “very satisfied” with their treatment 6 months into the study. Those who had received the coordinated care management and primary care were also significantly more likely than those in primary care alone to report feeling “much improved” and being “very satisfied,” although not to the same degree as those in the more intense interventions with the added CBT. Both phone-based interventions also had a high participation rate (more than 90 percent).

Ludman, Simon, Tutty, and Von Korff (2007) evaluated a CBT plus care management intervention for 393 clients beginning antidepressant treatment with a primary care doctor. They were followed for 18 months and had significantly greater improvements in depression symptoms, with average HSCL scores in the treatment group being 0.68 (standard deviation [SD]=0.65) and in the usual care group being 0.85 (SD=0.55), and phone therapy clients were significantly more likely to report feeling “much improved” or “very much improved” at both 12 and 18 months (77.5 percent versus 63 percent), compared with the usual care group. Simon, Ralston et al. (2011) adapted telephone-based CBT care management for delivery via the Internet using text messages. They randomly assigned 208 individuals with a depressive disorder who had recently been prescribed antidepressant medication by a primary care doctor to receive the intervention in addition to standard care or to receive standard care only. At a follow-up assessment 5 months after the study began, participants who used the online messaging intervention had significantly higher rates of medication compliance, significantly lower levels of depressive symptoms (according to the HSCL), and significantly greater satisfaction with their depression treatment than did those in the control group.

Use in Treatment of Eating Disorders

A variety of interventions using a range of new technologies have been implemented with people who have eating disorders, although most are intended to treat bulimia nervosa (BN) specifically. A nonsystematic review of the use of new technologies in the treatment of eating disorders and body image problems found that interventions delivered via CD-ROM or the Internet were effective treatments for eating disorders and that a variety of other technologies (e.g., email, text messaging, palmtop computers) had been shown to be effective adjuncts to in-person treatment ( Paxton & Franko, 2010 ).

Wilson and Zandberg (2012) reviewed research on self-guided CBT for eating disorders. They found that in many studies, therapist-guided, computer-based treatments for BN are superior to nontreatment controls, such as waitlist controls. Mitchell et al. (2011) indicated that such interventions can be as effective as CBT delivered in person. Wilson and Zandberg (2012) also observed that self-guided interventions using computers were effective at treating binge eating disorders, more effective than nontreatment controls, and comparable with a number of active treatments (e.g., behavioral weight loss treatment, treatment as usual). Similar findings from a smaller number of studies also suggested that these approaches were effective for treating eating disorders not otherwise specified. In addition, Wilson and Zandberg (2012) reviewed literature showing that:

• Self-guided interventions, although they were not as well-received as interpersonal therapy according to one comparative study, were generally well-rated by users in terms of their suitability and helpfulness.
• Attrition rates for these interventions were comparable with those of other treatments; controlled studies found that about one-third of clients drop out of treatment.
• These interventions were cost-effective.

Some individuals with eating disorders may prefer computer-based/Internet-based interventions to those delivered in person ( Johnston, Startup, Lavender, Godfrey, & Schmidt, 2010 ; Shapiro et al., 2007 ). However, Paxton and Franko (2010) cautioned that one potential problem with many interventions that deliver care for eating disorders at a distance is the inability of the provider to accurately visually monitor the client's physical condition and thereby ensure that it has not worsened. Mitchell and colleagues (2008) compared CBT treatments for BN delivered in person ( n =66) and by videophone technology using dedicated terminals and T1 lines ( n =61) for women in eastern North Dakota and northwestern Minnesota. At an assessment 1 year after treatment, the authors found slightly, but not significantly, higher rates of abstinence from binging and purging and significantly greater reductions in thoughts related to binging and purging (measured with the Eating Disorder Examination) for those who received the intervention delivered in person. Both groups had a high dropout rate, but there were no significant differences in retention. The authors concluded that the differences in outcomes between the two groups were not large, and that, in general, the study supported the idea that CBT interventions could be delivered using telemedicine technology.

A Spanish study evaluated an online psychoeducation and self-guided intervention based on CBT principles in which participants had weekly contact with a coach/therapist using a secure messaging program and unlimited contact with peers through an online discussion forum ( Fernández-Aranda et al., 2009 ). At the end of the 4-month intervention, compared with individuals in a waitlist control group, those who received the intervention were significantly more likely to be abstinent from both binging and purging during the 2 weeks prior to assessment (22.6 percent of those in the intervention group were abstinent compared with none of those in the control group). Another Web-based, self-guided intervention using CBT principles, which included an online discussion forum for participants and weekly contact with a graduate student counselor, was evaluated in Sweden ( Ljotsson et al., 2007 ). The authors found significant improvements at the end of treatment for those who received the intervention compared with those in the control group, with 46 percent of treatment completers reporting abstinence from binging and purging during the 28 days prior to assessment.

Yet another CBT-based online self-guided intervention for BN was evaluated in Switzerland ( Carrard et al., 2006 ; Carrard, Crépin, et al., 2011 ). Carrard, Crépin, et al. (2011) , in a study involving 127 individuals with BN or subclinical levels of BN symptoms, found that the participants who used the intervention had significantly greater improvements in terms of eating disorder symptoms, body dissatisfaction, drive for thinness, and binge eating episodes than did participants who were in a waitlist control group. They also found that the improvements were maintained 6 months after the intervention concluded. In another trial, this Web-based CBT intervention for BN was evaluated in four different European countries, and participants who used the intervention experienced significant improvements in eating disorder symptoms and general symptoms of psychopathology (as measured by the SCL-90-R) following its use ( Carrard, Fernández-Aranda, et al., 2011 ).

In Great Britain, Sánchez-Ortiz et al. (2011) evaluated another Web-based CBT intervention for BN that was designed for female college students ( N =76). Participants who used the intervention had significantly better outcomes in terms of eating disorder symptoms and behaviors according to a number of different measures than did participants in a waitlist control group. In a pilot study, Shapiro and colleagues (2007) compared a CBT treatment for binge eating disorder delivered via CD-ROM with an in-person CBT intervention and a waitlist control group using a group of 66 individuals with anorexia nervosa (AN) or BN. The authors found no significant differences in terms of frequency of binging between the two interventions, with both interventions associated with significant declines in the number of days of binge eating. Many participants who received the CD-ROM intervention reported continued use of the program after the intervention concluded, and the majority of participants in the waitlist control group chose to receive CD-ROM treatment rather than the in-person treatment after the waiting period concluded.

Email has also been used with some effectiveness to deliver treatment for eating disorders. For example, Robinson and Serfaty (2008) compared outcomes for 110 individuals with BN or binge eating disorders assigned to an email treatment, to a self-directed writing intervention with minimal therapist contact, or to a waitlist control group. At the end of the 3-month treatment, a significantly greater percentage of participants in the two treatment groups, compared with those in the control group, no longer met criteria for an eating disorder. There were, however, no significant differences between the two treatment groups.

Other technologies have also been used to support treatment for eating disorders, including VRET ( Perpiñá, Botella, & Baños, 2003 ; Riva, Bacchetta, Cesa, Conti, & Molinari, 2004 ), text messages ( Robinson et al., 2006 ), and palmtop computers ( Norton, Wonderlich, Myers, Mitchell, & Crosby, 2003 ). Ferrer-García and Gutiérrez-Maldonado (2012) reviewed research on the use of VR to assess body image distortion and in the treatment of eating disorders. They concluded that although research of better quality is needed, the use of VR in addition to other treatment improves outcomes, especially in relation to body image dissatisfaction.

Use in Treatment of Pathological Gambling

A few interventions for pathological gambling have been developed for use via phones or computers. Gainsbury and Blaszczynski (2011) conducted a literature review of Web-based, self-guided interventions for problem gambling and concluded that such interventions can be successful, are cost-effective, and may reach potential clients who would not otherwise seek traditional treatment for pathological gambling. Carlbring, Degerman, Jonsson, and Andersson (2012) evaluated the effectiveness of a Web-based CBT intervention for pathological gambling with 284 Swedish individuals who had gambling disorders. The intervention involved minimal contact with a therapist via email or phone in addition to the Web-based component, with an average of 4 hours of staff person time per participant. Participants experienced significant decreases in symptoms of pathological gambling, depression, and anxiety, as well as improvements in quality of life; significant improvements were found up to 36 months after treatment. Individuals were significantly more likely to respond to the treatment if they did not engage in heavy drinking (i.e., typically consumed two drinks or less during drinking episodes) and did not experience dissociative symptoms during gambling (i.e., did not completely lose track of time).

Fewer phone-based interventions are available for problem gambling, but in Australia, callers to a gamblers' helpline ( N =230) were recruited for a telephone-based intervention that used motivational interviewing and CBT principles in scheduled phone calls ( Rodda & Lubman, 2012 ). The program has been well-received by participants, many of whom would not otherwise have sought in-person treatment.

Use in Treatment of Other Mental Disorders/Problems

Phone-based and computer-based interventions have also been developed for BPD, trichotillomania, psychotic disorders, and chronic insomnia.

Rizvi et al. (2011) evaluated the use of an interactive mobile phone application to help in dialectical behavior therapy, an established treatment for BPD, by providing ongoing assessment and coaching. A pilot study involving 22 individuals with co-occurring BPD and substance use disorders found significant decreases in substance-related cravings and emotional intensity. Participants also experienced significant decreases in depression and general psychological distress over the course of the study.

German researchers have developed a CD-ROM program to improve client skill acquisition in the use of dialectical behavior therapy ( Wolf et al., 2011 ). In a small ( N =24) pilot study, participants who used the program showed significantly greater improvements in demonstrating knowledge related to skills taught in treatment and a greater ability to categorize concepts related to skills than did those in the control group, who received dialectical behavior therapy alone.

Also in Germany, Moritz and Rufer (2011) randomly assigned 42 individuals with trichotillomania to receive a self-guided intervention (i.e., decoupling) delivered via email or to participate in a control group who received instructions via email for progressive muscle relaxation. Participants who received the self-guided decoupling intervention had significantly greater reductions in trichotillomania behaviors, but declines in depressive and OCD symptoms did not differ significantly between the two groups.

Video conferencing has been successfully used with clients with schizophrenia and other psychotic disorders ( Sharp et al., 2011 ). Although some clinicians have found that people with psychosis have difficulty interacting with video conferencing, Grady and Singleton (2011) found that people with psychosis did not react negatively to video conferencing, nor did they experience exacerbations of symptoms.

Computer-based psychoeducational interventions have also been developed for people with schizophrenia and their families/friends. A Cochrane review conducted by Välimäki et al. (2012) that included six randomized trials found evidence that psychoeducation delivered using computer technology was associated with short-term improvements in mental state and a perception by clients that, compared with control groups, they had greater social support. Also, such psychoeducation was associated with long-term, but not short-term, improvements in medication compliance. However, no significant differences were found in quality of life or functional disability between people with schizophrenia who received computer-based psychoeducation and those who received traditional printed education materials, although both groups had significantly better outcomes than did those who received no psychoeducation intervention ( Pitkänen et al., 2012 ). Further information on programs for clients with BPD or schizophrenia that use mobile technologies for continuing care and for family psychoeducation and assistance is provided in the two sections that follow.

A number of Web-based interventions have been developed for chronic insomnia, a symptom associated with multiple mental and substance use disorders. For example, Vincent and Lewycky (2009) evaluated a brief, fully automated, online treatment for insomnia that included multimedia content in a group of 118 individuals with chronic insomnia who were either referred by physicians or recruited from the community. All participants were also receiving standard care for any diagnosed mental or substance use disorders, including sleep disorders, but individuals with mania, schizophrenia, or elevated substance use were excluded from the study. Compared with participants assigned to a waitlist control group, those who used the intervention had significantly greater improvements in sleep quality, severity of insomnia, and daytime fatigue. About one-third of participants dropped out of treatment, but individuals who had been referred rather than recruited from the community were significantly more likely to drop out.

Ritterband et al. (2009) evaluated an online CBT intervention for insomnia (Sleep Healthy Using the Internet [SHUTi ] ) with a group of 45 individuals who met criteria for a diagnosis of primary insomnia. Participants were randomly assigned to the intervention or to a waitlist control, and the authors found that participants who used SHUTi had significant improvements in insomnia symptoms and sleep efficacy, whereas those in the control group did not show any significant change. Improved outcomes were also found at the 6-month follow-up assessment. In Great Britain, Espie et al. (2012) evaluated another online CBT treatment for chronic insomnia that used an animated virtual therapist, automated mobile text messages and/or email reminders, and a moderated online discussion board. Participants in the study were randomly assigned to receive the intervention ( n =55), to receive another online treatment (imagery relief therapy) presented in much the same way ( n =55), or to participate in a treatment-as-usual control group that involved receiving standard services for insomnia from primary care doctors ( n =54). Compared with the other two groups, participants who received online CBT had significantly greater improvements in sleep efficacy.

Use in Continuing Care/Symptom Monitoring for Mental Illness

A number of programs are currently in development or being tested to help people with mental illness monitor their recovery and alert providers when intervention is required. One of the largest of these programs is VA's My Recovery Plan. Through a VA Web site ( My health e Vet; http://www.myhealth.va.gov ), veterans and their family members can receive educational materials about mental and substance use disorders; use screening tools to help them (and their family members) determine whether they have a problem that may require treatment; and monitor and relay to providers information on pertinent issues, such as mental disorder symptoms, medication compliance, progress toward recovery goals, and use of coping skills ( Bush, Bosmajian, Fairall, McCann, & Ciulla, 2011 ; Cucciare, Weingardt, & Humphreys, 2009 ; Weingardt & Lysell, 2007 ).

Phone technologies can also be used for ongoing care and symptom monitoring. For example, the Connecticut VA system implemented a behavioral health management system that used electronic messaging devices connected to phone lines that enabled participants (who had been diagnosed with schizophrenia, PTSD, depression, or substance use disorders) to report daily information about their behavioral health ( Godleski, Darkins, et al., 2012 ). When reported data indicated a potential problem, participants were contacted by phone for further assessment and, if necessary, scheduling of appointments. In the initial 2 years of the program, 76 patients with a variety of mental and substance use disorders used the system. The number of participants who were hospitalized went down from 42 in the 6 months prior to the intervention to 6 in the first 6 months of the intervention, and the number of emergency room visits went down from 80 to 16.

A number of other interventions using telephones or computers for medication monitoring have been successfully used with people who have schizophrenia. For example, Frangou, Sachpazidis, Stassinakis, and Sakas (2005) evaluated a computerized medication dispenser, which transmits information when the dispenser is used, with a group of 108 individuals with schizophrenia whose medication was monitored by the device, by a pill counting system, or by self-report. Participants who used the computerized dispensers had significantly greater reductions in emergency room use and medical appointments and significantly greater improvements on the Global Clinical Impression Scale compared with those in the control groups. The dispensers were well-received by users and required only minimal training for use.

Ruskin and colleagues (2003) used the interactive, computerized Med-eMonitor system for prompting medication compliance; monitoring symptoms, signs, and medication use; and providing educational information in their pilot study of 22 outpatient clients with schizophrenia or a schizoaffective disorder and 16 of the clients' caregivers. Ruskin and colleagues (2003) found a wide range of responses to the Med-eMonitor, from one client believing it made his symptoms worse to most of the participants finding it easy to use and helpful with compliance. Also, the three clinicians whose clients were in the pilot study found the Med-eMonitor system useful in terms of gaining information about whether clients actually did or did not take their medications and how client symptoms waxed and waned, as well as seeming to assist some clients in developing positive habits related to medication use.

Sachpazidis and Majadas (2006) have developed another system for medication management and reminders using cell phones. Also, software is available that uses a relational agent (i.e., an animated image of a person) that interacts with users to motivate them and help improve their medication compliance ( Puskar, Schlenk, Callan, Bickmore, & Sereika, 2011 ). Other uses of technology in relation to medication management for people with schizophrenia include software to aid in making decisions about medication use ( Deegan, 2007 ) and a VR simulation to assess clients' medication management skills ( Kurtz, Baker, Pearlson, & Astur, 2007 ).

Another smartphone application, called FOCUS—which combines medication management, symptom management, and components that assist with sleep hygiene and social functioning—has been evaluated in a pilot study with 33 individuals with psychotic disorders ( Ben-Zeev et al., 2014 ). More than 90 percent of participants found FOCUS to be highly acceptable and usable, and all but one participant completed the trial successfully. The authors found no significant relationships between use of the program and participants' cognitive functioning, severity of negative symptoms, reading level, or the presence of persecutory ideation. A pretest-posttest evaluation found that participants had significant improvements in positive symptoms, general psychopathology, and depressive symptoms following 1 month's use of the program.

Eating disorders

In Germany, Fichter et al. (2012) evaluated a Web-based relapse prevention intervention for women with AN or subclinical AN behaviors who had completed inpatient treatment for the disorder. Participants were randomly assigned to receive the 9-month intervention plus standard continuing care or continuing care alone. During the 9-month period, those who received the intervention gained significantly more body weight and showed significantly more improvement in a number of eating-related behaviors and cognitions (e.g., sexual anxieties, bulimic symptoms, social anxieties).

Gulec et al. (2011) conducted a small pilot study ( N =20) in Hungary evaluating a Web-based, therapist-supported continuing care program for people with BN or eating disorders not otherwise specified. The program was well-received, with 71.4 percent of participants stating that they found it mostly helpful or very helpful.

Web-based CBT has been successfully used for continuing care with clients who have had major depression. In a Swedish study, 84 individuals with partially remitted major depression who had previously been treated for that disorder were randomly assigned to a 10-week Web-based continuing care intervention (which used a self-guided intervention combined with email contact with a therapist) or to a no-treatment control group ( Holländare et al., 2011 ). Participants who received the intervention were significantly less likely, compared with those in the control group, to experience a relapse to major depression during treatment and up to 6 months later.

Bipolar disorder

A few different Web-based programs have been developed to help individuals with bipolar disorder and their families with ongoing symptom monitoring and long-term recovery. For example, a multimedia Web site from the Black Dog Institute in Australia was developed to provide education for clients and their families with modules on such topics as psychosocial treatments, medication, and symptom monitoring ( Proudfoot et al., 2007 ). In addition, the site provides self-testing for depression as well as bipolar disorder. At the time of the study, the program was being evaluated to determine whether it improves client self-management and psychosocial outcomes ( Proudfoot et al., 2007 ). Another Australian Web site ( http://MoodSwings.net.au ) was developed to help individuals with bipolar disorder manage the disorder and includes self-guided modules, a user discussion board, and an online mood diary ( Lauder et al., 2012 ). The evaluation of the program is also ongoing.

Beating Bipolar is a Web-based program developed in the United Kingdom to aid in the management of bipolar disorder by improving quality of life and reducing relapses ( Smith, Griffiths, et al., 2011 ). However, its use in addition to standard treatment was not found to be associated with significantly better outcomes on most measures than standard treatment alone. The one exception to this is in the scores on the psychological subsection of the World Health Organization Quality of Life scale, where participants who received the intervention showed modest improvements that were significantly greater than those observed in members of the control group.

Another online recovery program for people with bipolar disorder is MyRecoveryPlan (not to be confused with the VA program of the same name), which provides psychoeducation, recovery guidance, self-monitoring tools, social networking tools, and tools to help clients become more active participants in behavioral health services ( Simon, Ralston, et al., 2011 ). In a pilot study, 118 participants recruited from online and in-person support groups for people with bipolar disorder were randomly assigned either to receive the intervention alone or to the intervention with an additional peer coaching component. Those participants who had email communication with a peer coach were significantly more likely to return to use the program after they registered and were significantly more likely to continue using the program for more than 3 weeks.

Todd, Jones, and Lobban (2012) reported on focus group discussions conducted with people with bipolar disorder in the United Kingdom to identify their preferences concerning Web-based self-management tools. They found that individuals with bipolar disorder who sought behavioral health services reported a preference for Internet-based services because of their perception that evidence-based interventions were more accessible via this medium. However, focus group participants did discuss lack of motivation and procrastination as potential problems in the use of such interventions over the Web.

Schizophrenia

Mobile phone interventions are available for ongoing symptom monitoring and continuing care for people with schizophrenia. Granholm et al. (2012) reported on a noncontrolled study of three components of the Mobile Assessment and Treatment for Schizophrenia intervention, which uses text messages and assessment questions delivered via mobile phones to assist with medication management, socialization, and symptom monitoring. They found that participants significantly improved in medication compliance, significantly increased social interaction, and had significant reductions in auditory hallucinations after initiating use of these programs. About a quarter (23.6 percent) of participants—those with more severe negative symptoms and worse functioning—did not complete the intervention. Another mobile intervention for people with schizophrenia, which provides a schedule manager and allows for symptom monitoring, was developed in Canada, but client satisfaction in a pilot study was low, and further evaluation has been called for ( Sablier et al., 2012 ). Also, in using the Med-eMonitor system mentioned previously, Ruskin et al. (2005) found that 36 percent of the 22 clients with schizophrenia or schizoaffective disorders were unable or unwilling to use the device in their home setting.

Peer support/mutual-help groups

An important part of continuing care for many people with mental and substance use disorders is the use of mutual-help groups, and a variety of mutual-help Web sites, chat rooms, and services are available for this purpose. Although the use of mutual help, because it is not treatment per se, is beyond the scope of this TIP, articles are available that discuss the use of Web-based mutual help and its potential benefits and problems ( Barak et al., 2010 ; Bennett, Reynolds, Christensen, & Griffiths, 2010 ; Griffiths et al., 2009 ; Melling & Houguet-Pincham, 2011 ).

Barak et al. (2010) cited evidence, mostly from outside the behavioral health field, indicating that online support groups can be effective at improving participants' sense of empowerment and well-being. They identified research that indicates that the following factors, which are involved in online peer support, can help participants:

• The exchange of relevant knowledge
• The psychological effects of writing
• Giving and receiving emotional support
• Discovering that one's experiences are not unique
• Gaining social recognition
• Creating and sustaining interpersonal relationships
• Helping others
• Getting assistance in making decisions
• Sharing personal experiences
• Experiencing fun/amusement

Although data are limited, the available data suggest that many individuals with either mental illness or concerns about mental illness symptoms do seek help and support online. For example, in an Australian survey of 74 college students, 53 percent of participants reported using the Internet for mental health support ( O'Dea & Campbell, 2011 ).

Many individuals with mental illness, especially those who are younger, also commonly use online social networking sites and chat forums, suggesting that they would use peer support Web sites. For example, according to a National Alliance on Mental Illness survey of 274 young adult members, 94 percent of members with mental illness used social networking Web sites ( Gowen, Deschaine, Gruttadara, & Markey, 2012 ). Respondents with a mental disorder were also significantly more likely to use such sites to make friends or connect with people with similar interests than were other respondents (typically family members of individuals with mental illness).

In most cases, there are few or no data on the potential benefits or drawbacks of Internet support groups for people with mental or substance use disorders. For example, in their review of such groups for individuals with depression, Griffiths et al. (2009) concluded that “the evidence is not of sufficient quality or strength to inform decision making” (p. 9). In a later review of Internet peer support for depression, Melling and Houguet-Pincham (2011) reached essentially the same conclusion, although they found anecdotal evidence to support the use of such groups.

Haug et al. (2008) reported positive anecdotal evidence from a German program that used an Internet chat group for continuing care following inpatient treatment for severe psychosomatic or neurotic disorders. Internet chat groups enabled geographically dispersed individuals to continue group interactions that began while they were in treatment. Houston et al. (2002) followed a group of users of one of five online depression support groups who self-selected to participate in the study for a 1-year period to evaluate changes in depressive symptoms (according to the CES-D). They found that individuals who were frequent users of the group (i.e., used it for 5 or more hours in a 2-week period) were more likely to gain remission from depression (i.e., had CES-D scores of 22 or lower) than were less frequent users, even after controlling for demographic factors and baseline CES-D scores.

However, there is some evidence that such groups may not be beneficial for clients with serious mental illness, particularly if they are unmoderated. Kaplan et al. (2011) randomly assigned 300 individuals with a schizophrenia spectrum or affective disorder to an unmoderated peer-support electronic mailing list, to an unmoderated peer-support Internet bulletin board, or to a waitlist control group. Although an intent-to-treat analysis did not find significant differences in outcomes among the three groups, a post hoc analysis did find that participants who reported greater use of either form of Internet peer support reported significantly more psychological distress than did individuals with low levels of participation. Also, among peer support users, there was a significant correlation between having more positive experiences with the support group and having more psychological distress. The authors observed that most of these participants, who were recruited from Web sites and email newsletters for people with mental illness, were currently engaged in treatment for their mental illness and had relatively high rates of employment and marriage/being partnered, which may have marked the group as different from other treatment-seeking samples. Also, they noted that the unmoderated nature of the two support groups may have made a difference in how those interactions affected participants.

Interventions for family members

Web-based services aimed at family members of people with mental illness may improve family members' abilities to address problems, such as the return of symptoms as well as other aspects of family members' behavioral health ( Stjernswärd, Östman, & Löwgren, 2012 ). Grover et al. (2011) reported on a Web-based intervention for family members of people with AN in the United Kingdom. Compared with a control group who received standard support services, participants who used the program had significantly greater reductions in anxiety and depression as measured with the Hospital Anxiety and Depression Scale and in levels of expressed emotion (i.e., overinvolvement and criticism in response to their family member's eating disorder), as evaluated with the Level of Expressed Emotion Scale.

Rotondi (2010) described and provided some data for a Web-based intervention for clients with schizophrenia and their families called Schizophrenia Online Access to Resources (SOAR). The intervention provides multifamily psychoeducation; teaches skills to improve self-efficacy, self-management of symptoms and effects of the disorder, and problem-solving abilities; provides mechanisms for peer support; and helps users identify and meet their own goals. In developing the SOAR Web site, designers considered the types of cognitive deficits people with schizophrenia often face to produce a more usable site (e.g., by using memory aids to help with navigation, requiring a minimal amount of abstract thinking, not using decorative features that might distract users). Specific components of the SOAR program included three therapy groups (one for clients with schizophrenia only, one for a client's family members and others in his or her support network, and a multifamily group for all site users), a module that allowed users to ask questions of experts anonymously, educational resources, and a list of community resources. Preliminary data indicate that the site was well-received and often used by clients and their families, and users with schizophrenia had significantly less perceived stress and significantly greater reductions in positive symptoms of schizophrenia compared with a treatment-as-usual control group.

In another publication, Rotondi et al. (2010) reported on 1-year outcomes for participants in the SOAR pilot study (31 people with schizophrenia and 24 support people). They found that, compared with those who received usual care alone, those who used the intervention had significantly greater reductions in positive symptoms and significantly greater increases in knowledge about their disorder. Support people who used the intervention, compared with those connected to participants in the control group, also had significantly greater increases in knowledge about the disorder. Glynn, Randolph, Garrick, and Lui (2010) conducted a small ( N =36) preliminary evaluation of a Web-based program (the Online Relative Support Program) to provide the families of people with schizophrenia with psychoeducation and support. The program was well-received by users, but it appeared to have only a minimal impact on outcome measures for the person who had schizophrenia compared with data from comparable clients. Dausch et al. (2009) provided a case history of a client with a schizoaffective disorder who was participating in family-focused video conferencing therapy with his mother.

The U.S. military and VA have developed Web sites to help family members of veterans with mental or substance use disorders. For example, VA has developed an online psychoeducational program for family members of individuals with schizophrenia that has been well-received in pilot testing ( Glynn et al., 2010 ). Another psychoeducational Web site for family members of military personnel/veterans with PTSD has been pilot tested, and 57 percent of users reported taking action to discuss PTSD symptoms with a family member; between 82 and 95 percent of those individuals believed that discussion was beneficial ( Roy et al., 2012 ).

• Promoting Compliance, Engagement, and Retention

Data are limited concerning potential problems with treatment compliance and retention for phone- and computer-based interventions, but there are some indications that there may be more, or at least different, problems in these areas than are found with traditional methods of treatment delivery. For example, Kiluk et al. (2011) found that only 15 percent of the 75 randomized trials of computer-assisted therapies they evaluated had retention rates of 80 percent or higher. Many studies also failed to report compliance data. A review by Waller and Gillbody (2009) did find higher, but not significantly so, dropout rates for participants using Web-based interventions compared with those receiving services in person, but they also observed that this was not the case in every trial they reviewed. Personal circumstances (e.g., a lack of time) were most often cited as the reason for dropping out of Web-based programs. On the other hand, a number of users of Web-based interventions, in giving their reasons for use, stated that they would not have bothered to seek services from a healthcare provider (given that most of the interventions reviewed took place in Europe, this usually referred to primary care doctors).

Research suggests that telephonic interventions may reduce attrition relative to in-person interventions. A meta-analysis of telephone-based interventions for depression found a relatively low rate of attrition (7.6 percent) across 12 different studies ( Mohr et al., 2008 ). Studies that used a mental health professional rather than a professional from outside the behavioral health field, those that provided individual rather than group services, and those that used CBT rather than another orientation for treatment all had significantly lower attrition rates.

A more recent study found significantly higher attrition for people receiving CBT treatment for depression in person than for those receiving such treatment by phone ( Mohr et al., 2012 ). That study found the largest difference in attrition during the first 5 weeks of treatment. Some video conferencing interventions have also had relatively high retention rates. For example, a Spanish study of video conferencing for a variety of mental disorders retained 130 of 140 participants for the full 24 weeks of treatment ( De Las Cuevas, Arrendondo, Cabrera, Sulzenbacher, & Meise, 2006 ). VA's small ( N =27) pilot study of an intervention that combined two-way interactive text messaging and ecological momentary assessment was able to increase participant involvement significantly (going from a rate of 23 percent in the first cohort to 89 percent in the second cohort) by changing to scheduled messages rather than random ones and providing a single electronic device to do both tasks rather than two devices ( Smith, Harms, et al., 2012 ).

Researchers have found some ways to improve retention for Web-based treatments. For example, a research group in Germany, by using email reminders and nonfinancial incentives, was able to maintain a relatively high rate of retention across interventions for depression ( Moritz et al., 2012 ), OCD ( Moritz, Jelinek, Hauschildt, & Naber, 2010 ), and trichotillomania ( Moritz & Rufer, 2011 ). In an article discussing best practices for online therapy, Abbott, Klein, and Ciechomski (2008) suggested that incorporating graphics and media files into treatment materials and requiring homework exercises can improve client engagement and retention.

Compliance with Web-based treatment may also be improved by using guided rather than unguided interventions. For example, research comparing guided and unguided online treatments for SAD found significantly better compliance among participants who received the guided intervention ( Nordgreen et al., 2012 ). Participants who perceived their treatment as highly credible were also significantly more likely to comply with treatment protocols than were those who perceived the treatment as less credible. Titov, Andrews, Choi, Schwencke, & Mahoney (2008) also found greater compliance among participants who received a guided online intervention for social anxiety symptoms than among those receiving an unguided intervention.

In a series of studies concerning factors associated with Web-based treatment compliance for both anxiety and depressive disorders, Hilvert-Bruce, Rossouw, Wong, Sutherland, and Andrews (2012) found that greater contact with clinicians was associated with greater treatment compliance, even if that contact was only in the form of email to motivate the client to complete the entire course of treatment. In addition, they found that after implementing four practices to improve compliance, based on findings from prior research (i.e., allowing clients more choice over the course and timing of online modules, requiring a higher initial fee to use the program and less money per session, providing customized reminders in addition to automatic reminders, and ensuring that clients had clinician contact), full compliance with the treatment increased significantly from 37.9 percent to 60 percent of users. Despite participants who completed these treatments having significantly greater decreases in mental distress (as measured by the Kessler 10-Item Scale) than noncompleters, noncompleters also had significant improvements in this area, and both completers and noncompleters showed a similar pattern of correlation between the number of treatment modules completed and lower ratings of mental distress.

New technology can also be used to improve engagement and retention for clients receiving traditional behavioral health services. Seal et al. (2012) reported on a VA program in which 73 people who had been screened via a telephone screening system as possibly having one or more mental disorders were either given four sessions of motivational interviewing via phone and referred to treatment or referred and given four neutral phone calls providing logistical information about behavioral health services. Using an intent-to-treat analysis, the authors found that participants who received the intervention were more than twice as likely to enter a behavioral health program as those in the control group and had significantly higher rates of retention after receiving services.

Researchers conducted a small pilot study where 24 adult children of people with an alcohol use disorder were randomly assigned to 10 weeks of group therapy; to use of a computer program (Comprehensive Health Enhancement Support System [CHESS]) that provided resources to improve social relatedness, coping competence, and motivation; or to both group therapy and CHESS ( Gustafson, Shaw, et al., 2011 ). Average attendance at weekly group sessions increased from 38 percent to 82 percent when CHESS was added to group psychotherapy. Also, users of the CHESS program had, on average, a 10 percent improvement in self-reported quality of life, but those who received group therapy only experienced a decline in quality of life during the same period. However, given the small sample size, no tests of significance were performed.

• Technology in the Treatment of Substance Use Disorders

New technologies have many potential uses for substance use disorder treatment programs, both as stand-alone interventions and as ways of improving existing services. They may also reach populations who might not otherwise seek such services. A number of authors have observed that clients seeking Web- and phone-based substance use disorder treatment services are more likely to be younger and often have responsibilities that make use of traditional treatment services difficult ( Swan & Tyssen, 2009 ). However, only a couple of studies have attempted to evaluate how individuals using such services may differ from those who use traditional substance use disorder treatment services. Data from Australia indicated that clients seeking substance use disorder treatment services online or by phone ( n =277) were more likely than those seeking such services in person to be female, and those seeking services online were significantly younger and more likely to be employed than those seeking services by phone or in person ( Swan & Tyssen, 2009 ).

Lieberman and Huang (2008) surveyed 1,060 individuals who sought an online assessment of their drinking, who had AUDIT scores of eight or higher (indicating at least problematic drinking), and who agreed to participate in the study. They compared demographic and other factors for this group with those of 952 clients in the outpatient treatment arm of a large national study (Project MATCH). Participants who used the Web site were significantly younger, more likely to be female, and more likely to be employed than the outpatient group. The mean AUDIT score for Web site users was 20, which, despite being significantly lower than that of the treatment group (who had a mean score of 26.1), still indicated a high level of alcohol-related problems. Users of the Web site had a significantly lower level of problem recognition and greater level of ambivalence about drinking (both according to the Stages of Change Readiness and Treatment Eagerness Scale) compared with those in outpatient treatment. The authors concluded that a Web-based intervention to increase motivation for treatment would be useful and could reach individuals who were not yet prepared to enter traditional treatment.

Cucciare et al. (2009) reviewed research on a number of areas where Internet technology can support existing substance use disorder treatment services, including in the provision of brief motivational interventions, concurrent recovery monitoring, and long-term recovery management. An earlier review by Copeland and Martin (2004) concluded that, in spite of limited data and methodological problems with some studies, Web-based interventions were effective alternatives to standard substance use disorder treatment for clients who were less severely dependent and for brief interventions. Rooke, Thorsteinsson, Karpin, Copeland, and Allsop (2010) conducted a meta-analysis of computer- and Web-delivered interventions for either alcohol or tobacco use and found an overall, weighted average effect size of 20 (p<0.001) for the 42 studies they identified, with the tobacco interventions having significantly lower effect sizes compared with interventions targeting alcohol. They also determined that many common components of these interventions (e.g., group discussion, entertaining features, normative feedback, a focus on relapse prevention) did not significantly influence effect size.

Computer-based interventions for drug use disorders are less common than those for tobacco or alcohol use. In a review of such interventions, Moore et al. (2011) found 12 studies that they considered of at least moderate quality (11 of which used control conditions). They concluded that, compared with control conditions (most often treatment as usual, followed by assessment only), computerized interventions were associated with significantly greater reductions in drug use according to self-report and urinalysis during the treatment period, with some evidence that the effects continued after treatment. They also observed that clients were generally satisfied with computerized interventions and had similar levels of engagement and retention to those found in interventions delivered in person by a counselor. In addition to its use as a way to deliver treatment, computer and Internet technology can also be used to support treatment in a variety of ways. For example, Alemi et al. (2005) discussed the use of email communication to increase family involvement in a client's substance use disorder treatment and, in particular, using the family to improve a client's motivation for change.

In general, screening for substance use disorders and/or potentially harmful use conducted by computer, whether at a terminal in a treatment facility or over the Internet, appears to be as effective as self-administered screening instruments that use paper and pen (Miller et al., 2009). Some research also indicates that clients may prefer revealing certain types of information when using a computer rather than directly to another person ( Butler et al., 2001 ; Erdman, Klein, & Greist, 1985 ; Lessler et al., 2000 ). For example, research on the computer-delivered Addiction Severity Index-Multimedia Version (ASI-MV) found that individuals were significantly more likely to disclose about a number of different things (e.g., sedative use, the importance of substance use disorder treatment, legal difficulties, family conflict) on the ASI-MV than when another person administered the index ( Butler et al., 2009 ). Research also indicates that automated assessments are well-received by substance use disorder treatment clients ( Hile & Adkins, 1997 ).

Miller, Kraus, Kaak, Sprang, and Burton (2002) compared three different methods for completing a variety of alcohol screening instruments: paper and pencil, delivery via the Internet, and delivery via the Internet with an interruption to simulate conditions that might occur for some individuals using a Web-based screener. They found no significant differences among the three assessment methods. The instruments they evaluated were the Alcohol Dependence Scales, the AUDIT, the Rutgers Alcohol Problem Index, and the University of Rhode Island Change Assessment. Many commonly used substance use disorder screening and assessment instruments have been adapted for computer/Internet use and found to be valid in that format. For example, the ASI has been adapted for Internet and automated telephone use and was found to be reliable compared with in-person assessments and based on polls that gauged general satisfaction among users ( Brodey et al., 2004 ). The ASI-MV has also been found to be reliable, valid, and well-received by users when administered by computer ( Butler et al., 2001 ; Butler et al., 2009 ). Researchers have also found the Alcohol Timeline Followback instrument to be reliable when administered by telephone or computer ( Sobell, Brown, Leo, & Sobell, 1996 ).

There are also some substance use disorder screening and assessment instruments that have been developed specifically for delivery over the Internet. For example, the Dynamic Assessment and Referral System for Substance Abuse (DARSSA) is a computerized screening and assessment program that provides personalized feedback and referrals to treatment when appropriate. DARSSA was evaluated with a group of 85 patients from an emergency department (ED) and progressive care unit at a large urban hospital, and it identified 48 percent of participants as engaging in risky substance use, including a number of participants who had not been so identified during their medical assessments ( Boudreaux et al., 2009 ). In that pilot study, 42 percent of the participants identified as engaging in risky substance use accepted the DARSSA referrals to treatment.

Vernon (2010) reviewed seven studies of online drinking assessments not discussed here, most of which use the AUDIT or Stages of Change Readiness and Treatment Eagerness Scale for screening purposes, but did not provide any conclusions about the effectiveness of those instruments when administered online. He did, however, note that those studies found that between 12 and 63 percent of site visitors completed the assessment instrument, with an average across instruments of 56.4 percent of visitors.

Video conferencing can be used to assess substance use disorders, but not all studies have found that assessments made using this technology were as effective as those made in person, at least with specific populations. For example, a study conducted with 53 American Indian veterans found that substance use disorders were significantly more likely to be assessed following an in-person interview than they were when the same clients were assessed by different interviewers via video conferencing, whereas there were no significant differences in other behavioral health diagnoses ( Shore et al., 2007 ).

Use in Early Intervention

The line between prevention and early intervention is not always clear, and a number of the prevention programs discussed previously may also be considered early intervention programs, because they provide a brief intervention aimed at decreasing alcohol use or else helping people whose drinking is risky to determine whether they have a problem warranting further treatment. Similarly, some programs ( Williams, Herman-Stahl, Calvin, Pemberton, & Bradshaw, 2009 ) have been used with general population samples, and in that application, they have a preventive function. As Cucciare and colleagues (2009) noted, technology facilitates the collection of personal data on substance use patterns and can be used to generate personalized feedback with much less time and effort than was needed in the past. Web sites that offer screening are also a good way to interest people who are concerned about their drinking in considering treatment and in receiving information about treatments and links to resources ( Lieberman & Massey, 2008 ; Linke, Murray, Butler, & Wallace, 2007 ; Vernon, 2010 ).

Vaca, Winn, Anderson, Kim, and Arcila (2011) conducted a 6-month telephone interview follow-up of 221 emergency room patients (57 percent of the 385 participants who had enrolled). The study was designed to increase and enhance the use of ED-based alcohol screening, brief intervention, and referral to treatment (SBIRT) by introducing ways that computers could reduce the burden of SBIRT on ED staff members and facilitate ED-SBIRT by organizing and making use of the information gathered through computerized screening with the development of a computer-guided brief negotiated interview and a computer-generated personal alcohol reduction plan. Excluding individuals who were intoxicated at the time, medically unstable, and/or being held for psychiatric evaluation, 4,375 patients were screened by computer using questions from the AUDIT; 781 (18 percent) screened as at risk for alcohol use problems and received both a computerized brief intervention (including feedback, reasons to cut down, goal setting, and readiness to change) and a personalized printed alcohol reduction plan. (Patients whose AUDIT scores were 19 or greater were seen by a social worker and referred to treatment, and they were thus not eligible for the study.) Of 586 eligible subjects, 385 participants agreed to be interviewed by phone after 6 months, but only 221 completed the interview. At the follow-up assessment, 47 percent of those who had screened as engaging in risky drinking no longer met criteria for risky drinking. Readiness to change, as assessed by the program, was a good predictor of who would lower their drinking beneath recommended limits.

Cunningham, Wild, Cordingly, Van Mierlo, and Humphreys (2009) described and evaluated a Canadian Web site ( http://www.checkyourdrinking.net/CYD/CYDScreenerP1_0.aspx ) using participants ( N =185) who had previously been screened as meeting at least minimum criteria for risky drinking (i.e., AUDIT-Consumption [AUDIT-C] scores of 4 or higher). Participants were randomly assigned either to use the site or to fill out a written survey about the features they believed should be included on the site. Users of the site answered a brief series of questions and then received individualized feedback concerning their drinking, its potential effects, and related factors (such as drinking norms for people in their age group from their country). Participants completed questionnaires before using the site and then 3 and 6 months after. For individuals included in the study whose drinking was problematic (i.e., those who had AUDIT-C scores of 11 or higher at baseline), use of the Web site was associated with significant reductions in weekly alcohol consumption and in AUDIT-C scores at the 3- and 6-month assessments, with a typical reduction of between six and seven drinks per week.

A similar self-assessment Web site available in Finland was evaluated by Koski-Jännes, Cunningham, Tolonen, & Bothas (2007) , using a sample of 343 self-selected users of the Web site. Participants completed a short set of questions that incorporated the AUDIT as well as other questions about the psychosocial consequences of their drinking. They were then provided with personalized feedback about their drinking and information about normal drinking in Finland. Participants were self-selected, so the authors compared their drinking patterns and AUDIT scores to those of a random sampling of other users of the Web. Those who agreed to participate in the study had significantly higher AUDIT scores as well as greater levels of consumption.

Participants were assessed at baseline and again 3 months later. At the follow-up assessment, AUDIT scores dropped significantly from a mean of 16.6 to 13.8; additionally, drinks consumed per week, drinks consumed on the last drinking occasion, and drinking consequences all declined significantly.

Another study of computerized screening and brief intervention for problem drinking was conducted at Boston University with 235 first-year students who had been screened as engaging in unhealthy drinking (i.e., had AUDIT scores of 8 or greater) and who completed both an initial and a 1-month postintervention assessment ( Saitz et al., 2007 ). Participants were assigned to receive either a minimal or a more extensive (but still brief) intervention (the latter including three additional Web-based screenings) delivered over the Web. There were no significant differences between the two brief interventions; participants in both groups decreased their alcohol use, with 15 percent of the men and 33 percent of the women no longer meeting criteria for unhealthy alcohol use at the follow-up assessment. The interventions had a greater effect on women than men, and female (but not male) participants had significantly fewer drinks per week and fewer heavy drinking episodes after the intervention.

In an Australian study, 2,435 college students who had previously been screened as potentially having problems with drinking (i.e., had AUDIT scores of 8 or more and exceeded the Australian Government's drinking guidelines at least once in the prior month) were randomly assigned to an online screening and brief intervention site or to an online assessment-only control group ( Kypri et al., 2009 ). At both 1 and 6 months after the intervention, those who used the intervention drank significantly less overall and significantly less frequently than did individuals in the control group. Web site users drank 17 percent less than control group members at the 1-month assessment and 11 percent less at the 6-month assessment. The follow-up rates and demographic/clinical profiles were comparable for intervention and control groups, with 65 percent of both groups available for the 6-month follow-up. In another article on the development of this program, Hallett, Maycock, Kypri, Howat, and McManus (2009) noted that 99 percent of users found it easy to complete, 76 percent said it gave them information that was relevant to their lives, and 55 percent stated that they would recommend it to others.

In another Australian study, Kay-Lambkin, Baker, Kelly, and Lewin (2012) randomly assigned individuals who were experiencing moderate or greater levels of depressive symptoms (i.e., had BDI scores of 17 or higher) and who used more alcohol than suggested by Australian national guidelines and/or used marijuana more than once per week to receive one in-person introductory treatment session followed by nine sessions of motivational interviewing/CBT delivered in person, supportive counseling delivered in person, or motivational interviewing/CBT delivered mostly by computer. Although 274 participants were assigned to treatment arms, only 163 completed an assessment 3 months after the start of the study, and outcome data were only reported for those 163. At the 3-month assessment, participants who received the computerized treatment had significantly greater reductions in alcohol use and reported significantly less alcohol use than did participants in either of the other treatment groups. However, a comparative trial of an identical brief intervention with personalized feedback delivered either in person or by computer with 84 college students who had previously screened as being at risk for alcohol use disorder found no significant differences in outcomes between the two methods of delivery ( Butler & Correia, 2009 ). Participants receiving either version of the intervention did have significantly better outcomes (in terms of quantity and frequency of alcohol consumption) than did members of a no-treatment control group.

DoD has conducted research evaluating factors associated with better outcomes for military personnel who used either a Web-based early intervention for drinking or a Web-based alcohol prevention intervention ( Williams et al., 2009 ). In this study, personnel selected from all service branches were randomly assigned to the Drinker's Check-Up program (a brief motivational interview that included personalized feedback about drinking; n =1,483), the Alcohol Savvy program (an alcohol use disorder prevention program developed for workplace delivery; n =688), or a no-treatment control group ( n =919). Use of the Drinker's Check-Up was found to be associated with significantly better drinking-related outcomes during the 6-month follow-up period, whereas the Alcohol Savvy program was not. In evaluating mediating factors, the authors found that there was strong support for the effect of perceived descriptive norms concerning others' drinking on alcohol outcomes, but only the Drinker's Check-Up intervention had the expected effect on participants' perceptions of drinking norms. In fact, users of the Alcohol Savvy program began to perceive others as drinking more than they had prior to using the program.

Computer technology can help improve motivation for treatment. Lieberman and Massey (2008) evaluated an online motivational intervention with 244 people whose drinking had been identified as problematic. Prior to intervention, 46 participants had stated that they were very interested in some form of treatment for their drinking, and 69 people said they were somewhat interested; after the intervention, those numbers rose to 68 and 90, respectively. Computer or Internet approaches offering early intervention for illicit drug use are less common than those for alcohol. Ondersma et al. (2007) evaluated a brief motivational intervention to reduce illicit drug use among postpartum women. For the evaluation, 107 women who had recently delivered children at a large urban hospital were randomly assigned to receive the 20-minute computer-based intervention along with some printed educational materials or to participate in an assessment-only control group. At a 4-month follow-up assessment, women in the control group had increased their drug use, but women who received the intervention had decreased use; for drugs other than marijuana, decreases were significant ( p< 0.05).

Use in the Treatment of Alcohol Use Disorder and Problem Drinking

A number of Web-delivered interventions for problem alcohol use are available; some overlap with interventions for alcohol prevention or early intervention and have thus been discussed previously. Additionally, some interventions address both alcohol and illicit drug use and are discussed under “Use in the Treatment of Illicit Drug Use Disorders.” Interventions that target clients with a diagnosable alcohol use disorder are somewhat less common. Such interventions, though, have the potential to reach clients who might not otherwise seek treatment ( Lieberman & Huang, 2008 ). Vernon (2010) reviewed eight studies of online treatments for problematic alcohol use in nonclinical samples. None of the studies were limited to individuals with alcohol use disorders, but those studies that did provide information on alcohol consumption suggested that clients were consuming amounts similar to those seen in standard treatment settings, although other data suggested that participants were experiencing fewer negative consequences than typically seen in standard treatment settings. With one exception, participants showed significant improvements in at least one outcome measure, but given wide variations in measures, types of interventions, and varied use of control groups, it was not possible to draw conclusions about the relative effectiveness of the interventions studied.

Squires and Bryant (2010) reviewed research on six different TAC interventions for problematic alcohol use/alcohol use disorders. They found that, in general, these interventions achieved reductions in alcohol consumption and/or alcohol-related problems, but most had only preliminary/modest data supporting their use. Only two interventions reviewed are available commercially: the Behavioral Self-Control Program for Windows and the Drinker's Check-Up.

Dutch researchers compared a Web-based, therapist-delivered intervention ( n =68); a fully automated, Web-based self-guided intervention ( n =68); and a waitlist control group ( n =69). Using an intent-to-treat analysis, they found that outcomes were significantly better for participants in the therapist-delivered intervention than for those in the self-guided intervention and significantly better for participants in the self-guided intervention than for those in the control group ( Blankers et al., 2011 ).

Interventions to reduce alcohol use/treat alcohol use disorders can also be implemented in medical settings. Hasin et al. (2013) evaluated a phone-based intervention (HealthCall) that used interactive voice response technology plus motivational interviewing to reduce heavy drinking among primary care patients with HIV ( N =258). The intervention was associated with significantly fewer drinks per day in comparison with a treatment-as-usual control group and fewer, but not significantly so, drinks per day in comparison with motivational interviewing alone. However, when only those participants who had alcohol dependence were included (about half the sample), use of the phone-based intervention was associated with significantly fewer drinks per day than motivational interviewing plus standard treatment as well as standard treatment alone.

Hasin et al. (2014) evaluated a version of HealthCall adapted for smartphones with a group of 41 people being treated for HIV who had alcohol dependence and compared results with 43 individuals with alcohol dependence who used the standard phone-based version of the intervention, who had been included in the Hasin et al. (2013) study. Participants used HealthCall on significantly more days when it was available on smartphones than on standard phones, but there were no significant differences in the number of drinks per day.

Dulin, Gonzalez, and Campbell (2014) conducted a pilot test ( N =28) of another smartphone intervention known as Buddy Steps, which helps reduce alcohol consumption in people with less severe alcohol use disorders. Buddy Steps was designed to enhance motivation for change, help participants avoid high-risk situations, improve social support networks, assist participants in managing cravings, enhance problem-solving skills, promote engagement in nondrinking activities, and teach assertive communication skills. Participants significantly decreased heavy drinking (from 46 percent of their days prior to treatment to 25 percent while using the program) and reduced the average drinks per day by 25 percent.

Barnett, Tidey, Murphy, Swift, and Colby (2011) reported on a small ( N =20) trial that evaluated a contingency management intervention to reduce alcohol consumption that used a Secure Continuous Remote Alcohol Monitoring (SCRAM) ankle bracelet. The SCRAM anklet is locked onto the ankle and has sensors that evaluate ethanol vapor from the skin to assess alcohol use as well as tampering. Participants received cash incentives for days of abstinence following 1 week of normal drinking. In the second and third week of the study, participants had significant reductions in days abstinent and in measures of average and peak alcohol concentration.

Cognitive bias modification software (see the “ Interventions to Reduce Trait Anxiety and Subclinical Anxiety Symptoms ” section) has also been used with clients who have alcohol use disorders; preliminary research suggests that it is effective in altering clients' attitudes toward alcohol and alcohol-related cues, but more data are needed to determine how changes in attitudes translate into behavioral changes ( Wiers, Eberl, Rinck, Becker, & Lindenmeyer, 2011 ).

Cunningham and Van Mierlo (2009) provided a different type of review of studies of Web-based interventions for problem drinking/alcohol use disorders, focusing not on results, but rather on positive and negative aspects of study design. Among the problems they identified with research to date on this type of intervention are difficulties in generalizing results to real-world settings, reliance on self-selection for samples, lack of suitable control groups, and poor follow-up rates.

Video conferencing-based group therapy has been used for people with alcohol use disorders. In a pilot study, the intervention was well-received by participants and had a relatively high level of attendance and low level of attrition ( Frueh, Henderson, & Myrick, 2005 ). Many interventions for alcohol use disorders or problem drinking have high dropout rates ( Vernon, 2010 ). On the plus side, Dutch research indicates that a Web-based program for problem drinking attracted clients who might not otherwise seek treatment ( Postel, De Haan, & De Jong, 2010 ).

Use in the Treatment of Smoking/Smokeless Tobacco Use

Interventions that use modern communication technology to help people stop using tobacco (either smoked or smokeless) are relatively widespread and able to reach a large client population in a timely and cost-effective manner ( Leykin, Barrera, & Muñoz, 2010 ). Research and numerous reviews of phone- and Web-based interventions for smoking cessation have generally found the interventions to be effective, especially compared with no-treatment controls or printed materials ( Chen et al., 2012 ; Myung, McDonnell, Kazinets, Seo, & Moskowitz, 2009 ; Shahab & McEwen, 2009 ; Stead et al., 2007 ; Whittaker et al., 2012 ). Most Web- and phone-based smoking cessation studies, however, do rely on self-report for evaluating smoking rates, which may mean that such research undercounts the amount of actual smoking ( Kiluk et al., 2011 ). However, An, Klatt, et al. (2008) found a relatively low underreporting rate when they used carbon monoxide tests to confirm self-reports.

Phone-based interventions

Phone-based interventions such as quitlines are widely used and well-evaluated. A few large-scale reviews have been conducted that support the effectiveness of quitlines and other phone-based counseling services for smoking cessation ( Fiore, Bailey, & Cohen, 2000 ; Lichtenstein, Glasgow, Lando, Ossip-Klein, & Boles, 1996 ; Stead et al., 2007 ; Tzelepis, Paul, Walsh, McElduff, & Knight, 2011 ). Quitlines—in which a caller initiates a request for help with cessation—are available in all 50 states, plus Puerto Rico and the District of Columbia ( Cummins et al., 2007 ). A review of the 52 U.S. quitlines ( Cummins et al., 2007 ) found that most are capable of handling calls in more than one language (with a mean of two languages), receive calls a mean of 97 hours per week, and provide counseling services for a mean of 86 hours per week.

All of the U.S. quitlines provide some form of multisession counseling, with 61.5 percent allowing for follow-up sessions initiated by the caller and 28.8 percent providing proactive follow-up (i.e., calls initiated by the counselor). These quitlines offer a mean of five sessions of counseling, with a typical first session lasting about 30 minutes and follow-up sessions lasting about 17 minutes. Approximately one third (34.6 percent) also provide free nicotine replacement medications. From 2004 to 2005, approximately 1 percent of people who smoked in the United states called such quitlines; the rates for particular states varied from .01 to 4.28 percent, with a strong correlation between the amount of funding for the state's quitline and the percentage of people who smoked in the state who used it. In general, researchers have not investigated different components of quitlines, but in an update of a meta-analysis originally conducted for the Cochrane Collaboration, Stead et al. (2007) determined that quitlines that provided proactive counseling increased the odds of successful smoking cessation 1.41 times (95 percent confidence interval [CI]: 1.27–1.57).

In a meta-analytic review of 24 randomized controlled studies specifically involving proactive telephone counseling for smoking cessation, Tzelepis et al. (2011) concluded that, compared with mutual-help materials or no-intervention control groups, such interventions were associated with significantly better point prevalence abstinence outcomes 6 to 9 months after study entry, but not 12 to 15 months after. Compared with no-treatment and mutual-help material controls, such interventions were also associated with significantly higher rates of 3 or more months of continuous abstinence at both 6 to 9 months and 12 to 15 months after entry. Study quality and recruitment methods did not affect results in regard to point prevalence abstinence, but studies of worse quality were associated with higher rates of continuous abstinence at 6 to 9 months, although not at 12 to 15 months. Quitlines and telephone-based cessation counseling also appear to be effective for young adults, a population difficult to reach with cessation initiatives ( Rabius et al., 2004 ; Sims et al., 2012 ).

Mobile phone interventions

Mobile phones can also be used in smoking cessation efforts, typically using text messages or video instead of audio communication. Whittaker et al. (2012) reviewed five randomized or quasi-randomized trials that provided at least 6 months of follow-up data on mobile phone interventions (including Free et al., 2011 ). They found that such interventions were associated with significantly higher rates of abstinence from smoking for 6 months or more compared with control conditions. However, results varied a good deal across studies, with much stronger results from three of the five studies than from the other two.

Three of the studies included in the Whittaker et al. (2012) review used the TXT2STOP program (which involves sending text messages to mobile phones), as delivered in either New Zealand or Great Britain. In the most recent of these studies, Free et al. (2011) randomly assigned 5,800 individuals who smoked to receive regular, automated, tailored text messages sent as part of the TXT2STOP program or to receive regular messages thanking them for their participation (the control group). They found those who used TXT2STOP were significantly more likely to have biochemically verified continuous abstinence for the 6-month follow-up period than were participants in the control group, and this held true when participants lost to follow-up were counted as smoking. For more information on TXT2STOP, see Part 1, Chapter 1 of this TIP.

As might be expected, not all aids for smoking cessation can be considered adequate. Abroms, Padmanabhan, Thaweethai, and Phillips (2011) reviewed iPhone applications sold as cessation aids and found that few, if any, complied with recommended guidelines.

Web-based interventions

A variety of Web-delivered interventions have been developed to help individuals quit smoking (see description of major cessation Web sites in Leykin et al., 2010 ). In general, such interventions have been found to be effective, but the evidence supporting their use is not as strong as the evidence supporting the use of telephone-based interventions ( Chen et al., 2012 ; Leykin et al., 2010 ; Myung et al., 2009 ; Shahab & McEwen, 2009 ; Zbikowski et al., 2008 ). Web and phone technology can also be combined, which may improve effectiveness ( Brendryen, Drozd, & Kraft, 2008 ; Graham et al., 2011 ; Zbikowski et al., 2008 ). Web-based cessation programs also appear to provide effective assistance for people trying to quit smokeless tobacco (e.g., Severson, Gordon, Danaher, & Akers, 2008 ).

In a comprehensive meta-analytic review, Chen et al. (2012) reviewed 77 publications reporting on 60 randomized or quasi-randomized controlled trials involving computers and other electronic aids, including automated mobile phone applications (but not counseling by phone), for adult smoking cessation. The authors concluded that, overall, such interventions increase the odds of achieving cessation to a small degree over no treatment or generic self-help materials and that factors such as the concurrent use of other, nonelectronic interventions and the specific mode of delivery do not affect those odds significantly. However, they concluded that such interventions are still highly cost-effective.

Shahab and McEwen (2009) reviewed 11 randomized controlled trials of Web-based interventions published between 2005 and 2008 and concluded that tailored, Web-based interventions were more effective than printed psychoeducational materials or email interventions and that their impact is similar to that of telephone intervention or physician advice. The authors noted that research generally indicates that only those interventions that are aimed at people who smoke but are already motivated to quit are effective. Another meta-analytic review of 22 interventions from around the world (9 Web-based and 13 other computer-based) also concluded that these interventions, regardless of whether they used the Internet, were significantly more effective than control groups; their use was associated with cessation rates similar to those seen in studies of in-person cessation counseling ( Myung et al., 2009 ). The authors found that, for the seven studies that provided long-term follow-up data (i.e., 1 year or more), the cessation rate for users of a Web-based cessation program was 9.9 percent (95 percent CI: 8.9–10.9) compared with 5.7 percent for control groups (95 percent CI: 5.1–6.3).

Leykin et al. (2010) provided a more informal but far-ranging review of research on smoking cessation programs delivered via the Internet. They summarized research and briefly described 25 such Web sites from around the world. More recent studies by other researchers have continued to find Web-based cessation programs effective at increasing abstinence, especially compared with information-only ( Graham et al., 2011 ) or assessment-only ( Haug, Meyer, & John, 2011 ) control groups.

Contingency management has been used to improve abstinence rates in an online smoking cessation program for people in rural areas. Stoops et al. (2009) compared outcomes for 35 people who smoked and received monetary incentives contingent on providing breath carbon monoxide samples (while on camera to ensure no false samples) indicating abstinence from smoking with a control group of 33 who received monetary incentives regardless of their abstinence status. Individuals in the abstinence-contingent group were significantly more likely to be abstinent during the 6-week intervention.

Dallery and Raiff (2011) discussed some concerns involved in using contingency management in Web-based interventions for smoking cessation. For example, they emphasized the importance of intensive monitoring and frequent reinforcement during the initial days and weeks of quit efforts, as these early periods of cessation efforts are often the most difficult for those trying to quit; the ability to maintain abstinence throughout the early period increases the likelihood of long-term treatment success.

Not all Web-based cessation programs have been found to be effective. For example, McKay, Danaher, Seeley, Lichtenstein, and Gau (2008) found no significant differences in cessation, at both 3- and 6-month posttreatment assessments, between individuals ( N =2,138) randomly assigned to the Quit Smoking Network smoking cessation program and those assigned to a Web-based exercise program that gave some encouragement to quit smoking. The authors attributed the lack of a significant effect to a small number (i.e., one) of median visits to the site for participants and to a high dropout rate (approximately 61 percent by the 6-month assessment).

Many individuals searching for information about or help with smoking cessation may be reaching Web sites that are not helpful (e.g., sites run by tobacco companies, sites that use unproven methods; Bock, Graham, Whiteley, & Stoddard, 2008 ; Etter, 2006 ). Bock et al. (2008) found that only 26 percent of the sites found in searches for smoking cessation information were actually intended for that purpose, and 31 percent of those had at least some inaccurate information.

Tailored content

Researchers have found that Web-based smoking cessation programs that are able to tailor their content to the individual user have significantly better outcomes than those that present static content, although not for every population ( Rabius, Pike, Wiatrek, & McAlister, 2008 ; Strecher et al., 2008 ; Strecher, Shiffman, & West, 2005 ). A study comparing outcomes for 2,468 individuals who were randomly assigned to use one of six smoking cessation Web sites and for whom 13-month follow-up data were available found that outcomes for those using interactive or tailored sites were not significantly better for participants as a whole, but when individuals who had symptoms possibly indicative of depression (and who also had significantly worse outcomes) were removed from the sample, the use of more interactive sites was associated with a significantly ( p =.04) higher cessation rate at the follow-up assessment ( Rabius et al., 2008 ).

Severson et al. (2008) compared two versions of the ChewFree Web site ( https://chewfree.com ), one providing tailored feedback based on interactions with users and another using a more static and linear approach. Participants who used the interactive site had significantly higher rates of abstinence from all tobacco products at both 3- and 6-month postintervention assessments. Similarly, Strecher et al. (2005) evaluated 3-month cessation rates for 3,971 individuals in Great Britain and Ireland who were randomly assigned to use either a tailored or a nontailored smoking cessation Web site (while also using a nicotine patch) and found that those who had the tailored intervention were significantly more likely to report 10 or more weeks of continuous abstinence from tobacco use. They also found that the perceived relevance of the cessation program, assessed 6 weeks after starting the program, was a partial mediator of outcomes that might account for better results for those using the tailored intervention.

In another study, Stretcher et al. (2008) evaluated 6-month outcomes for 1,866 people who used a smoking cessation Web site where five different factors (the depth of tailoring of messages relating to outcome expectations, the detail of success stories provided to the user, the detail and amount of attention paid to the user's perceived barriers to cessation, the degree of personalization in the introductory session, and the schedule for exposure to key elements of the Web site) were randomized in 16 different combinations. The authors found that the detail provided in success stories and the depth of personalization in messages relating to outcome expectations were both significantly related to cessation outcomes ( p< 0.04 and p< 0.02, respectively). Also, the detail of success stories interacted with the participant's education level, so that for individuals with less than a high school education (but not those with a greater level of education), highly tailored success stories had a significant relation to outcomes.

Research suggests that messages should be tailored to the client's stage of readiness for tobacco cessation to maximize effectiveness ( Etter, 2005 ). Although it is not as common as Web-based interventions, delivery of tailored content by email is also possible. Dutch researchers found that individuals who smoked and received tailored email messages regarding cessation ( n =224) reported significantly higher past-week cessation rates than did those who received generic, nontailored emails ( n =234) and reported greater appreciation for the intervention and a stronger belief in its credibility as well ( Te Poel et al., 2009 ).

Targeted populations

Cessation sites can target specific populations. For example, the RealU site was developed for students at the University of Minnesota. In an evaluation of the site, An, Klatt, et al. (2008) randomly assigned students who had previously reported smoking cigarettes in the prior month to a control group that received a confirmation email that included links to online health resources or to an intervention group that paid participants with gift vouchers to motivate them to sign on to the study Web site once a week. At the site, they would report on health and lifestyle factors, take an interactive quiz on smoking-related or general interest topics, view a student-written online magazine that included at least one article on smoking or smoking cessation in college, and receive weekly emails from peer coaches. The rate of past-month abstinence reported at a 30-week assessment was significantly higher for those who participated in the intervention; carbon monoxide tests showed a low rate of underreporting, but the difference in abstinence rates between the two groups was no longer significant at a 6-month assessment.

Another smoking cessation intervention targeting college students using both Internet and mobile phone text messages was found to have a high success rate in a small ( N =31) study of students who smoked on a daily basis ( Riley, Obermayer, & Jean-Mary, 2008 ). The authors found that 6 weeks after beginning the intervention, 45 percent of participants reported past-week abstinence from smoking, and 42 percent had prior-week abstinence according to biochemical validation.

Another cessation intervention that uses computer technology targeted toward a specific population is a program at the University of Texas M. D. Anderson Cancer Center, which uses Web-based screening of new cancer patients to identify those who currently smoke as well as telephone technology to conduct treatment sessions with healthcare providers ( Mallen, Blalock, & Cinciripini, 2006 ). Preliminary findings from the program evaluation suggested that the program was effective at significantly reducing tobacco use.

Jones et al. (2010) conducted a pilot test of a Web-based smoking cessation site for people who are Deaf, which included features such as an online video chat room for sign language discussions among participants and a trained moderator. The program was well-received by a small group ( N =5) of participants.

Brunette et al. (2011) conducted a pilot study of a computer-based intervention intended to help motivate people with serious mental illness to quit smoking ( N =41). They used a convenience sample and assigned participants to the intervention or to a waitlist control group. Those who used the intervention were significantly more likely to show some motivation to quit (e.g., talking to a counselor about cessation, starting a cessation program) and to show multiple signs of motivation than were members of the control group. A later evaluation of this program, involving 128 people with serious mental illness who smoked, found that 32 percent began recommended treatment after using the system; 51 percent engaged in cessation-related behavior ( Ferron et al., 2012 ). In a model that controlled for smoking behavior and other factors (e.g., cognitive functioning), self-reported readiness to quit was the only variable significantly associated with cessation behavior in the 2 months following use of the site.

Web site components

Because smoking cessation Web sites typically include multiple components, An, Schillo, et al. (2008) evaluated the use of different features of the QuitPlan smoking cessation site ( https://www.quitplan.com ) and their relationship to outcomes for 607 registered users of the site. The features evaluated were:

• Interactive quit planning tools (used by 77 percent of study participants).
• Other informational resources, such as the quit guide and medication guide (used by 60 percent).
• Passive community interaction (e.g., reading other members' journal posts, viewing the discussion board; used by 38 percent).
• Interactive diagnostic tools (used by 34.7 percent).
• Active community engagement (e.g., posting to discussion board, writing journal entries; used by 24 percent).
• One-on-one messaging (used by 11 percent).
• Online counseling (used by 5 percent).

At a 6-month follow-up assessment, 9.7 percent of participants had been abstinent from smoking for 1 month or more according to self-report. The authors found that people who used the quit planning tools more than once were more likely to be abstinent; those who used them two or three times were 1.87 times more likely, and those who used them four or more times were 2.35 times more likely. The use of one-on-one messaging was also associated with increased odds of being abstinent at the 6-month assessment, but the difference was not significant.

Another possibly helpful feature of smoking cessation programs is the use of peer support through an online bulletin board or a similar mechanism, which can enable more rapid response to participant problems/concerns than can be provided by staff members alone ( Selby, Van Mierlo, Voci, Parent, & Cunningham, 2010 ). Stoddard et al. (2008) compared two versions of the same Web site ( http://www.smokefree.gov ), one of which had an online discussion or bulletin board and the other of which did not. They randomly assigned 1,375 participants to use one of the two sites, and although those who used the site with the bulletin board did spend significantly more time on the site, doing so did not translate into significantly better outcomes, perhaps because only a small percentage of those participants (11.8 percent) chose to view or post messages.

Adding a phone component to a Web-based intervention may also improve outcomes. In a large ( N =2,005) evaluation of three different formats of the Quit Using Internet and Telephone Treatment, Graham et al. (2011) randomly assigned people who smoked to use a basic, static Web site; an enhanced Web site that was interactive in format; or an advanced site with the addition of proactive phone calls from trained counselors. Although past-month abstinence rates at a final assessment 18 months after the intervention did not vary significantly, participants who used the enhanced site and received counselor phone calls were significantly more likely than those in the other groups to have been abstinent at all assessments (conducted at 3, 6, 12, and 18 months); those who used the enhanced site without phone calls were significantly more likely than those who used the basic site to have been abstinent at all assessments.

Automated interventions

Despite some smoking cessation interventions making use of human counselors, others are fully automated, and automated interventions have been found to be significantly more effective than no-treatment controls ( Balmford, Borland, & Benda, 2008 ; Swartz, Noell, Schroeder, & Ary, 2006 ). For example, Swartz et al. (2006) found that at a 90-day postintervention assessment, the past-week cessation rate for participants using a fully automated site was more than twice as high (12.3 percent) as that in a no-treatment control group (5 percent). Mobile devices have also been used for automated cessation interventions. A large ( N =5,524) trial conducted in Great Britain compared an automated text messaging intervention delivered via mobile phone with a control group who received text messages unrelated to smoking cessation ( Free et al., 2011 ). Participants who received the intervention were significantly more likely to have been continuously abstinent, as verified using saliva cotinine testing, for 6 months (10.7 percent) than were those in the control group (4.9 percent).

Brendryen et al. (2008) reported on a Norwegian program that used automated email, Web, interactive voice, and text messages (all accessed using cell phones). Compared with a control group who only received printed materials about cessation, those in the intervention group had significantly higher repeated point abstinence rates during the 1-year program.

Some studies have found that interactive Web-based interventions, which can provide tailored feedback, can have a significant positive effect on smoking outcomes and are more effective than self-guided interventions using printed materials or email, but evidence is mixed regarding their effectiveness relative to static Web-based interventions ( Shahab & McEwen, 2009 ). Shahab and McEwen (2009) , in their review of Web-based cessation programs, concluded that evidence is stronger for the effectiveness of interactive interventions than for static ones, but they also noted that specific studies included in their review had greatly varying results and varied considerably in terms of their specific components. Other research found higher abstinence rates for people receiving counselor contact, dependent on the type and form of contact ( Graham et al., 2011 ).

Other factors affecting outcomes

An Australian study evaluated demographic factors associated with Web site use for 28,247 users of the Quitcoach smoking cessation site ( http://www.quitcoach.org.au ; Balmford, Borland, Li, & Ferretter, 2009 ). Compared with Australians who smoked in general (according to a separate general population survey) and users of the Quitline service (a service that offers cessation advice by phone), users of the Web site were more likely to be male and were, on average, younger. Compared with Quitline users alone, Quitcoach site users were more likely to have attempted quitting before, were less likely to use pharmacotherapy, and were less likely to smoke 30 or more cigarettes per day. According to another study of Quitcoach data, women, older adults, and people using pharmacotherapy were significantly more likely than others to return to the site after an initial viewing ( Balmford et al., 2008 ).

Zbikowski et al. (2008) evaluated the Free & Clear Quit for Life Program, a large ( N =11,143) cessation intervention that offered both phone- and Web-based services. They found that women were significantly more likely than men to use both the phone and Web components of the program, people ages 26 and older who smoked were more likely to use both than were those under 26, and individuals who smoked moderately (15 to 20 cigarettes a day) were more likely to use them than those who smoked on a light or heavy basis. Clients who completed a greater number of phone counseling sessions and those who used the Web-based component on at least one occasion were significantly more likely to report successful tobacco cessation at a 6-month postintervention follow-up after controlling for age, gender, and cigarettes smoked per day. For each completed call, the odds of successfully quitting increased by 56 percent; for each time the individual logged in and used the Web-based component, those odds increased by 14 percent. The overall successful cessation rate for respondents to the 6-month follow-up survey was 41 percent; if nonresponders were factored in as participants who had not achieved cessation, the rate would have been 21 percent.

In another study, users of a Web-based smoking cessation program developed at the University of Michigan were assigned to have access either to the Web site as a whole or to specific parts of it on a sequential basis ( Strecher et al., 2008 ). The authors found that participants were less likely to engage with the site (defined as opening more parts of the Web site) if they were male, were under 40 years of age, and had no higher education. Participants who were provided with more personalized feedback on how to deal with barriers to successful cessation (based on factors such as their home environment, family life, stress/coping skills, and level of physical activity) were more likely to view more parts of the Web site.

Brendryen et al. (2008) evaluated possible mediators for the effectiveness of a Norwegian Web-based smoking cessation program (Happy Ending). None of the factors evaluated mediated long-term cessation, but coping planning skills and self-efficacy were partial mediators for abstinence at an assessment 1 month after each participant's quit date. Similarly, Danaher, Smolkowski, Seeley, and Severson (2008) evaluated possible mediators for successful abstinence from the use of smokeless tobacco for participants in the ChewFree Web-based program. They found that program exposure did not mediate positive outcomes when self-efficacy was also considered, suggesting that self-efficacy was a more important mediator than program exposure. They hypothesized that a certain degree of program exposure could increase self-efficacy, thus leading to better outcomes, but they noted that it is also possible that individuals with greater self-efficacy are more likely to engage with the program.

Co-occurring mental and substance use disorders may also affect cessation for Web-based program users. In a bilingual international program, past, but not current, major depression was associated with significantly higher cessation rates, whereas current depression was associated with significantly lower rates ( Muñoz et al., 2006 ). Similarly, in a multisite trial, individuals who reported at least one symptom indicative of possible depression had significantly worse cessation rates at a follow-up assessment 13 months after study entry ( Rabius et al., 2008 ).

Research concerns

One problem with a great deal of research on Web-based smoking cessation is that interventions often have high rates of attrition, and this may skew some of the data presented in the studies thus far considered ( Kiluk et al., 2011 ; Webb, 2009 ). For example, among studies reviewed by Shahab and McEwen (2009) , anywhere from 7 to 73 percent of participants were lost to follow-up. Saul et al. (2007) made a special effort to lower attrition by using the telephone as well as email for surveys and providing cash incentives for survey completion, but they still obtained only a 77.6 percent response rate for their 6-month follow-up survey.

Also, most cessation rates in research on Web- and phone-based tobacco cessation programs are based on participant self-report, which may inflate rates somewhat. A New Zealand Web-based cessation program (Smokestop) used carbon monoxide tests 1, 3, and 6 months after program entry to confirm reported cessation rates and found that 12 percent of participants ( n =12) achieved 6 months of continuous cessation (a percentage comparable with other studies; Fraser, McRobbie, Bullen, Whittaker, & Barlow, 2010 ). However, the program began with an initial 30-minute in-person interview to instruct participants in use of the Web site and also provided free nicotine replacement (as some American Web-based programs also do). Graham and Papandonatos (2008) found that results from Web-administered smoking surveys (of the kind typically used in the research thus far described) did not differ significantly from telephone-administered surveys, and, more importantly, the race/ethnicity and income level of respondents did not significantly affect the psychometric properties of such instruments.

Another potential problem in research on Web-based interventions that was raised by Danaher, Lichtenstein, McKay, and Seeley (2009) is the use of additional smoking cessation aids that were not recommended or assigned as part of the study. In their analysis of 1,028 participants in a Web-based randomized clinical trial, 24.1 percent of participants used methods that were not part of the study (12.6 percent consulted books/pamphlets, 4.5 percent used hypnotherapy or acupuncture, 2.3 percent used outside group counseling, and 1.7 percent used outside individual counseling). The authors recommended that studies evaluating Web-based smoking cessation interventions view them as part of a larger network of behavioral change efforts for participants and that study participants be asked to identify other factors involved in those efforts.

Little research has been conducted evaluating cultural differences in the use of Web-based smoking cessation programs, but one study that evaluated outcomes from an international group of Spanish ( n =500) and English ( n =500) speakers did not find any significant differences in outcomes between the two groups. In other research conducted involving the same program, Muñoz and colleagues (2006) found that Spanish-speaking participants were more likely to be male than female, whereas the reverse was true for English speakers; Spanish speakers also had a higher mean level of education. However, these differences likely reflect differences in smoking and Internet access among the two groups.

Use in the Treatment of Illicit Drug Use Disorders

Moore et al. (2011) reviewed 12 studies of moderate quality that evaluated computer-based interventions for drug use disorders, most using onsite computers and 11 including active controls (typically, treatment as usual). They concluded that such interventions are more effective at reducing substance use during treatment than are control conditions (such as treatment as usual) and that they continue to be effective afterward. Also, these studies suggested that these interventions are well-received by clients and that clients using them have levels of engagement and retention similar to levels seen in clients receiving traditional treatments.

Carroll et al. (2008) gave early results from a randomized trial of a multimedia computer-delivered CBT treatment for substance use disorders known as CBT4CBT. Individuals who received the intervention in addition to standard treatment ( n =35), compared with those who received standard treatment alone ( n =38), provided significantly more urine samples that tested negative for drugs. In addition, for those who received the intervention, there was a stronger association between outcome and treatment engagement; completion of homework assignments was a significant predictor of treatment engagement, highlighting the importance of homework completion for those receiving this intervention.

In a later article, Carroll et al. (2009) provided 6-month follow-up outcomes for 73 of the original 77 participants. In that assessment, the authors found that, after controlling for treatment retention, posttreatment outcomes, and exposure to other treatment during follow-up, individuals who received the intervention tended to continue to reduce their drug use over the follow-up period, whereas those who were in the control group tended to increase it during that period. In another analysis of these data, Sugarman, Nich, and Carroll (2010) found no significant differences in the use of coping strategies between the intervention and control groups, but they did find a significant association between the use of coping strategies and decreased use of drugs and observed that the strength of this association was stronger for those participants who used CBT4CBT. Kiluk, Nich, Babuscio, and Carroll (2010) reanalyzed the data for 52 participants who completed all assessments and found that those who had the additional CBT4CBT treatment had significantly greater improvements (according to independent ratings of role-plays) than did those receiving standard treatment alone. Also based on these data, Olmstead, Ostrow, and Carroll (2010) calculated the additional cost of using CBT4CBT to be $39 for programs and $27 for patients and calculated that the program would be cost-effective if the perceived value of one drug-free urine specimen were $21 or greater.

Carroll et al. (2014) conducted a randomized trial comparing CBT4CBT with standard treatment alone for 101 individuals in a methadone maintenance program who had cocaine dependence. Participants who used the CBT4CBT program were significantly more likely to have 3 or more consecutive weeks of cocaine abstinence during the 8-week trial than were those in the control group (36 percent of the CBT4CBT group had 3 or more weeks of abstinence compared with 17 percent of the control group). In addition, CBT4CBT groups had better, but not significantly so, outcomes in other areas, including more urine samples free of all drugs. When considering only those participants who completed the trial ( n =69), the results were significantly better for those who used CBT4CBT.

Internet-based video conferencing has been used to conduct group therapy for clients in methadone maintenance who were required to enter stepped-up treatment as a result of testing positive for illicit drug use ( King et al., 2009 ). In the pilot evaluation of the program, 37 clients were randomly assigned to a standard in-person group at the methadone maintenance facility or to an online group that used group counselors who were experienced in providing treatment using this technology. The authors found no significant differences between the two groups in terms of achieving the 2 consecutive weeks of abstinence required before participants could leave stepped-up care. Although treatment satisfaction was comparably high for participants in both treatment conditions, all of those in the online group who responded to the question said they would prefer to use it over the traditional group.

Computer and telephone interventions have also been developed for clients receiving buprenorphine treatment. Bickel, Marsch, Buchhalter, and Badger (2008) randomly assigned 135 clients with opioid dependence who were also receiving buprenorphine to one of three groups: a computer-assisted community reinforcement intervention with contingency management using vouchers, the same treatment delivered only in person, or a standard counseling approach (reflecting treatment provided in methadone maintenance programs). Participants were followed for 23 weeks of treatment, and retention rates were comparable across the three groups (62 percent retention for the computer-assisted care, 58 percent for the standard care, and 53 percent for the in-person intervention groups). The authors found that participants in both intervention groups had significantly more ( p< .05) weeks of abstinence from cocaine and opioids (according to thrice-weekly urine drug screens) than did those in the control groups and comparable mean amounts of continuous abstinence (7.98 weeks for those who received the in-person intervention and 7.78 weeks for those who received the computer-assisted intervention). However, the computer-assisted intervention required considerably less time from therapists, as 80 percent of the intervention was delivered by the computer program.

Ruetsch, Tkacz, McPherson, and Cacciola (2012) provided follow-up data for 1,426 patients receiving buprenorphine who were randomly assigned to receive either standard care or standard care plus a telephone-based support program (HereToHelp), which involved calls from care coaches who gave information and encouragement but not counseling. At assessments 1 year after beginning the program, participants who received HereToHelp and accepted at least three phone calls were significantly more likely to be compliant with their buprenorphine treatment, and participants who were compliant with treatment had significantly better outcomes according to all seven scales of the ASI. The authors concluded that intervention has a positive effect on substance use, mental health, and other outcomes indirectly by improving compliance. See Ruetsch, Cacciola, and Tkacz (2010) for more information on the specifics of the intervention.

In another study, 160 participants in a methadone maintenance program were assigned to receive standard treatment either alone or with the addition of a Web-based intervention based on the community reinforcement approach ( Acosta et al., 2012 ). The researchers evaluated the effects of cognitive functioning on treatment response for participants in the two groups. They found that a number of measures (subscales of the Short Form of the MicroCog Assessment of Cognitive Functioning) representing greater impairment in different areas of cognitive functioning were associated with significantly lower levels of abstinence for recipients of standard care alone, but not for those who received the additional Web-based intervention. They observed that this finding may indicate that Web-based interventions can help reduce the negative effects of impaired cognitive functioning on substance use disorder treatment outcomes.

Specialized treatment programs that use computer/Internet technology can also be created for specific underserved populations. For example, SAMHSA funded a Web-based substance use disorder treatment program for people who are Deaf that was able to successfully treat this underserved population. According to preliminary data, 66.7 percent of 36 participants reported no past-month alcohol use, and 63.9 percent reported no past-month illicit drug use at a 6-month follow-up assessment ( Moore et al., 2009 ). A couple of computerized interventions have also been evaluated for the treatment of cannabis use disorders. A pilot study comparing computerized treatment using motivational enhancement therapy, CBT, and abstinence-based contingency management ( n =16) with a similar treatment delivered by therapists ( n =22) found no significant differences between the two groups in terms of attendance, retention, or cannabis use outcomes evaluated 12 weeks after the intervention. In both groups, rates of daily use dropped sharply from 4 out of every 5 days (roughly 80 percent) to 1 out of every 9 days (roughly 11 percent), but rates of continuous abstinence for 10 to 12 weeks were low (about 30 percent for the group delivered the CBT by the computer and about 18 percent for the group with therapists delivering the CBT; Budney et al., 2010 ).

In a German study, Tossmann, Jonas, Tensil, Lang, and Strüber (2011) compared outcomes for users of a 50-day Web-based treatment for cannabis use disorders ( n =360, 100 of whom completed the intervention and provided appropriate data) with those for individuals assigned to a waitlist control group ( n =360, 106 of whom provided full data). Participants who used the intervention had significantly greater reductions in cannabis use than did those in the control group. In addition, the intervention had small-to-moderate effects on a number of secondary outcomes, including self-efficacy, life satisfaction, anxiety, and depression. VanDeMark et al. (2010) evaluated factors associated with active engagement for users of an online substance use disorder treatment intervention (E-TREAT), a motivational interviewing approach where clients worked with a recovery coach using a variety of communication methods (including telephone, email, text messages, and Web-based communication). Clients were more likely to engage in the program if they were female, had children, reported a positive relationship with their recovery coach, and had not completed a substance use disorder treatment program in the past.

Use in Continuing Care/Symptom Monitoring for Substance Use Disorders

Telephone and Internet technologies have been used in continuing care for people with substance use disorders for some time and to a fairly large extent, but much of this use is informal; few studies have evaluated the use of these technologies. Gustafson, Boyle, et al. (2011) reviewed randomized trials that use these technologies to provide continuing care for chronic diseases, including substance use disorders. They found that monitoring and proactive outreach by computer were particularly effective. However, small sample sizes and poor designs limit the conclusions that can be drawn. Both this article and another article ( Gustafson, Shaw, et al., 2011 ) described Addiction-CHESS (A-CHESS), a mobile technology application intended to promote long-term recovery from substance use disorders by providing monitoring of recovery activities, decision-making tools, support services (including discussion groups and feedback from experts), and information. This application is based on the CHESS system (see discussion under “ Promoting Compliance, Engagement, and Retention ”) that has been found effective in promoting other kinds of behavioral change, including smoking cessation and the management of physical illnesses, such as asthma and HIV ( Gustafson, Boyle, et al., 2011 ; Gustafson, Shaw et al., 2011 ).

More recently, Gustafson et al. (2014) conducted a randomized trial comparing treatment as usual for clients who attended a residential treatment program that did not feature coordinated care following treatment ( n =179) with standard residential treatment with the addition of A-CHESS to help clients manage drinking following treatment ( n =170). Participants who used A-CHESS reported significantly fewer days of risky drinking (i.e., consuming more than four standard drinks for men and three for women in a 2-hour period) than did those in the control group during the 4-month follow-up period (with means of 1.39 days and 2.75 days, respectively).

Telephone monitoring for continuing care following substance use disorder treatment has been found to be effective in a series of studies, particularly for clients who are not considered at high risk for relapse. McKay, Lynch, Shepard, & Pettinati (2005) compared 12 weeks of weekly telephone monitoring and brief counseling (consisting of a 15-minute call in which goals were reviewed, problems discussed, and new goals set) combined with 4 weeks of weekly supportive group sessions with either 12 weeks of twice-a-week CBT relapse prevention sessions or twice-a-week supportive group counseling with a group of clients who had completed an intensive outpatient treatment program ( N =359). At assessments 24 months after the intervention, the authors found that rates of total abstinence were significantly higher for those who received telephone checkups compared with those who received standard group counseling for continuing care. However, further analysis revealed that, although clients who had low to moderate levels of risk for relapse (assessed using a composite risk indicator measure the authors developed based on other research; McKay, Lynch, Shepard, Morgenstern, et al., 2005 ) had overall higher abstinence rates when receiving telephone monitoring, those who were assessed as being at greater risk for relapse had higher overall abstinence rates in standard supportive group treatment. In another article, the authors noted that no other client characteristics, besides the composite risk for relapse, marked clients as unsuitable for telephone-based counseling ( McKay, Lynch, Shepard, & Pettinati, 2005 ).

Another analysis from these researchers, which used data only from the telephone-based and supportive group interventions, tried to identify the mediating factors that might account for the better outcomes seen in telephone-based continuing care compared with standard supportive group counseling for continuing care ( Mensinger, Lynch, TenHave, & McKay, 2007 ). Participants in the telephone-based intervention had greater involvement in mutual-help groups during follow-up than did those in standard care, and they also had higher self-efficacy scores and a greater commitment to abstinence at the 6-month follow-up assessment. The authors concluded that these factors accounted for higher overall abstinence rates among those who received the telephone-based intervention.

More recently, McKay et al. (2011) reported 2-year follow-up outcomes for 252 individuals with alcohol use disorders (49 percent of whom also had cocaine dependence) who had completed an outpatient treatment program and received standard continuing care consisting of a stepped-care counseling program ( n =86), telephone monitoring and feedback only ( n =83), or a combination of the two ( n =83) for 18 months. During at least some of the assessments conducted during the 18-month period while the intervention occurred, alcohol-related outcomes (e.g., use of any alcohol, days of heavy drinking) were significantly better for participants in the combined treatment group than in either of the other groups. Also, participants who received telephone counseling alone had significantly fewer days of alcohol use during months 10 through 15 than did those who received standard continuing care. However, at an assessment conducted 24 months after treatment started, differences in outcomes were no longer significant.

To reduce costs, telephone monitoring can also use interactive voice response systems, which have a computer-operated voice that asks questions and collects keypad responses from clients. Helzer, Badger, Rose, Mongeon, and Searles (2002) used such a system to collect 2 years of daily information from a sample of heavy drinkers who did not receive substance use disorder treatment services ( N =33) about their alcohol consumption and found that this act of reporting by itself was associated with reductions in alcohol use. Stout et al. (1999) developed a continuing care intervention, known as Case Monitoring, that used a stepped-care approach to telephone monitoring. The intervention started with a 30-minute interview during treatment to establish rapport and identify problems that might lead to relapse and then involved counselor-initiated telephone contact that started 1 week after discharge and continued monthly for 3 months, then tapered to two contacts at 6-week intervals, followed by nine contacts at 2-month intervals (for a total of 15 contacts over 2 years for clients who were doing well). If a client was at risk for relapse or had already relapsed, then the contact schedule was restarted.

In their evaluation of this case monitoring process conducted with 342 clients, Stout et al. (2001) found that telephone monitoring was associated with fewer relapses and, to a greater extent, less severe relapses for those who did drink compared with a control group who received standard continuing care only. At the 3-year follow-up, people who received telephone monitoring had a significantly smaller percentage of days of heavy drinking (12 percent) compared with those in the control group (24 percent). The authors also found that the intervention was associated with savings in treatment costs after the first year.

Another approach to telephone monitoring for people in recovery from substance use disorders was evaluated in a series of National Institute on Drug Abuse Clinical Trials Network studies ( Farabee et al., 2012 ; Hubbard et al., 2007 ; Karno, Farabee, Brecht, & Rawson, 2012 ). The Hubbard et al. (2007) study used telephone calls to encourage clients ( N =339) to comply with continuing care plans, but the results failed to clearly evidence the intervention's efficacy when attendance at continuing care of those who received phone calls was compared with that of participants in the control group, who did not receive any phone calls. Farabee et al. (2012) developed a more elaborate telephone support program with phone referrals to continuing care conducted in four different styles (unstructured/nondirective, structured/nondirective, unstructured/directive, and structured/directive). Clients ( N =301) were randomized to one of the four styles of telephone support or to a control group who received only a referral to continuing care. At 3-month follow-up, participants who received any type of telephone intervention had a decrease in the mean group score on the drug use composite of the ASI, whereas those in the control group had, on average, an increase in composite ASI scores for drug use. There were no significant differences among the four telephone intervention groups. Karno et al. (2012) , using the same data as Farabee et al. (2012) , also found that, at the 3-month follow-up, the directive versions of the telephone calls were less effective with clients who were assessed as being high in “reactance” at baseline before the interventions, whereas the directive approaches were more effective for those clients assessed as low in reactance. The nondirective styles of telephone interventions did increase the likelihood of abstinence at 3 months in the high reactance group of clients (although the difference did not reach statistical significance).

Ongoing monitoring can also be conducted using email, but research is lacking regarding this approach. Collins, McAllister, and Ford (2007) discussed their own experiences and client responses to regular email communication for continuing care as well as for outpatient treatment. They noted that, although they have found this approach to be beneficial and well-received, a lack of reimbursement for the service and privacy issues are concerns that need to be addressed. New Web-based technologies are already replacing phone and email monitoring, and some trial projects reported on their use for recovery monitoring ( Cucciare et al., 2009 ). For example, in VA's My Recovery Plan, clients receiving VA substance use disorder treatment services can monitor progress toward recovery goals and receive feedback and reminders relating to medication compliance, behavioral health appointments, mutual-help meetings, specific tasks relating to their goals, and relapse triggers and the use of coping skills ( Cucciare et al., 2009 ; Weingardt & Lysell, 2007 ). Site users are also able to complete online self-change modules intended to aid them in their recovery ( Weingardt & Lysell, 2007 ).

Internet technology can also be used to collect data regularly about clients' treatment experiences, satisfaction with and participation in treatment and mutual help, and ongoing substance use; these data can then be displayed in a convenient format to monitor client progress. Cucciare et al. (2009) described a few programs (including their own developed for VA) that enable such monitoring for outpatient clients. As with mental illness (see the “ Peer Support/Mutual-Help Groups ” section), online mutual-help groups may also be of help to clients in recovery from substance use disorders. Research shows that mutual help improves long-term recovery from substance use disorders (see the planned TIP, Relapse Prevention and Recovery Promotion in Behavioral Health Services [ SAMHSA, planned b ]), but little has been done to evaluate the effects of online recovery services specifically. Hall and Tidwell (2003) conducted a descriptive survey of users of Web-based mutual-help and peer services and found that users were twice as likely to be female as male, were predominantly White Americans, and had a mean age of 46.

• Administrative Issues in the Use of New Technologies

Computer and phone technologies offer many opportunities for behavioral health services, enabling them to potentially improve cost-effectiveness while also reaching individuals who might not otherwise receive such services. However, some authors have observed that the behavioral health system is not prepared for what are likely to be large-scale increases in the use of Web-delivered care in coming years ( Alleman, 2002 ). Cartreine, Ahern, and Locke (2010) discussed some of the systemic and programmatic changes needed to meet the need for more computer-based treatment, as well as addressing related business and legal issues. Kraus and Zack (2004) explored some of the business-related aspects of setting up and conducting online therapy. The sections that follow touch on legal and ethical issues, questions of reimbursement, and potential problems with implementing new interventions using these technologies, as well asthe use of such technologies to provide supervision and training. However, this section does not provide a detailed discussion of those topics; the reader should consult Part 2 of this TIP for more information, as well as for information on other topics of interest to program administrators.

Legal and Ethical Issues

There is little if any research on legal and ethical issues in this context, but a number of authors have published opinions on this topic ( Baker & Bufka, 2011 ; Barnett & Scheetz, 2003 ; Cartreine et al., 2010 ; Kanani & Regehr, 2003 ; Kraus, 2004 ; Mallen, Vogel, & Rochlen, 2005 ; Midkiff & Wyatt, 2008 ; Ragusea, 2012 ; Recupero, 2006 ; Shaw & Shaw, 2006 ; Welfel & Heinlen, 2010 ). Many have also explored self-disclosure and confidentiality in Web-based behavioral health interactions, including Joinson and Paine (2007) ; Ragusea (2012) ; and Zur, Williams, Lehavot, and Knapp (2009) . The specifics of their recommendations are beyond the scope of this literature review, but readers are referred to Part 2 of this TIP, which discusses these issues at length.

McAdams and Wyatt (2010) explored issues relating to the regulation of both therapy and clinical supervision conducted using telecommunication technologies, and they reported on information gleaned from interviews with licensure board members. Many of the legal issues that might affect Web-delivered or similar interventions have yet to be determined—for example, the degree to which the developer of a computer-based intervention can be held accountable for malpractice (see Cartreine et al., 2010 , for a discussion of this and similar issues). One area of particular legal and ethical concern is the safety of clients being treated remotely using phone and Web technologies. Luxton, O'Brien, McCann, and Mishkind (2012) reviewed literature regarding, and discussed issues related to, client safety for people treated via video conferencing while in clinically unsupervised settings.

State regulations concerning the use of Internet and phone technologies for providing counseling and supervision/training are changing quickly, and providers should seek appropriate guidance for their state regulations and any appropriate regulations concerning the provision of services across state lines. In 2008, McAdams and Wyatt (2010) surveyed state boards responsible for counselor certification and found that only 14 states had regulations concerning TAC; only 6 had regulations regarding technology-assisted supervision. Another 20 states had regulations that were either under discussion or in development. In some cases, it may be possible to obtain state waivers so that therapy conducted over the Internet or videophone will qualify as an in-person service, as was the case for clients who were Deaf in the Wright State University e-therapy substance use disorder treatment program ( Moore et al., 2009 ). Laws are rapidly changing in this area, and readers should seek legal advice from professionals regarding state laws and regulations, risk and liability, and insurance coverage, especially in regard to risk management issues associated with TAC.

Implementing New Technologies

Technical Assistance Publication 31, Implementing Change in Substance Abuse Treatment Programs ( CSAT, 2009b ), provides more detailed information about the process of implementing new programs and interventions in substance use disorder treatment settings, much of it applicable to other behavioral health settings. This section addresses some additional literature that discusses implementation of specific types of interventions and the potential challenges involved.

The process of implementation should begin with a weighing of the pros and cons of integrating these technologies into a program. Pros include increasing cost-effectiveness, extending the reach of services, and facilitating information exchange with some clients; cons include higher dropout rates compared with services delivered in person, the possibility that certain interpersonal factors that can positively affect treatment may be removed from services, and the possibility that such services may eventually reduce the need (and payment) for in-person services ( Cucciare, 2010 ).

Lovejoy, Demireva, Grayson, and McNamara (2009) , in an article exploring the implementation of online therapies in light of Rogers' diffusion of innovations theory, identified a number of potential barriers to such implementation and some of the ways in which adaption of this technology might be accelerated. Jones, Leonard, and Birmingham (2006) also reviewed some of the barriers involved in setting up an online program for mental health services. They discussed necessary hardware, treatment protocols, and some of the technical issues that need to be addressed (e.g., camera positioning). However, such issues need to be evaluated in light of technology advancements, which are rapidly occurring. Brooks, Ryder, Carise, and Kirby (2010) observed that computerized interventions may be especially vulnerable to partial implementation, because counselors/therapists may not understand how to integrate those services into existing practice and/or have reservations about clients' abilities to make use of such interventions. Program managers may also want to consider applying an implementation framework, such as Promoting Action on Research Implementation in Health Services or Predisposing, Reinforcing, and Enabling Constructs in Educational Diagnosis and Evaluation. Both of these were discussed by Cucciare (2010) in the context of implementing new technology in behavioral health settings.

Reimbursement for Services

In the past, problems getting reimbursement for services or having access to services from a given payer kept many programs from using phone- and/or Web-delivered interventions. A 2005 survey of 62 organizations providing telehealth services (not necessarily behavioral health service providers) found that 58 percent were receiving reimbursement from private payers, and 81 percent (apparently of those that billed private payers) stated that there was no difference in the amount of reimbursement between telehealth and traditional services ( Whitten & Buis, 2007 ). Around the same time, other data indicated that many health plans only used Internet services for limited purposes. In a survey of 368 health plans across the country, two-thirds reported that they provided online behavioral health self-assessment tools; about half offered online referrals; but only 2 percent provided online counseling ( Horgan, Merrick, Reif, & Stewart, 2007 ).

Behavioral health programs offering services online are increasing, but many still do not do so. Hall and Hall (2009) , based on a review of both policies and initiatives, identified potential problems with the current funding situations for Web- and phone-based services, including:

• Not having single state sources of information on funding and conducting such services.
• Complex and sometimes confusing rules concerning such services.
• Changes in policies and laws that may not affect changes in service delivery and the funding of such services.
• A limited impetus to change the current status quo in regard to such services.

However, recent changes may make it easier for programs to receive reimbursement for behavioral health services provided via the Internet. As of 2012, 15 states had passed legislation requiring that private payers reimburse for telehealth services under certain circumstances ( Billings, 2012 ). Under certain circumstances and for approved providers, Medicare will approve reimbursement for some telehealth services, including behavioral health services. However, the Medicare regulations are complex and are being amended regularly, especially in terms of the use of technologies and in light of the changes related to the expansion of healthcare coverage under the Affordable Care Act.

Hall and Hall (2009) noted the difficulty of sorting out the complexities involved with Medicare and Medicaid reimbursement. The American Psychological Association (APA) and other behavioral health organizations track changes and initiatives of interest to their membership ( APA Practice Organization, 2011 ). States decide whether or not Medicaid will allow for reimbursement of telehealth services, which services will be covered, and under what conditions. As of 2012, 40 states provided Medicaid reimbursement for at least some electronically delivered behavioral health services ( Secure Telehealth, 2012 ). The ATA (2010) provided more detailed state-by-state information about telehealth services that are reimbursable under Medicaid as well as other state laws and policies relating to telehealth services, but as these policies are still changing, it is important to consult state entities to determine current policies. See Part 2, Chapter 1 of this TIP for more on reimbursement issues.

Supervision and Training Using New Technologies

Many counselor training and education activities are already conducted using computers and the Internet, and research generally indicates that these technologies are effective for this purpose ( Ferreira, 2005 ; Liebowitz, 2003 ; Murdock, Williams, Becker, Bruce, & Young, 2012 ). Computer technologies also offer a number of potential benefits for the training of counselors, such as the ability to provide real-time feedback to trainees who are conducting practice sessions ( Rosenberg, 2006 ). Trepal, Haberstroh, Duffey, and Evans (2007) discussed some of the issues involved in teaching counseling skills via the Internet, especially in terms of establishing a relationship. A review by Hayes (2008) discussed the use of computers in training and supervising counselors, including such factors as use of computer-based simulations, student attitudes toward new technology, and ethical issues. Individual and group instruction can be conducted using Web-based technology; at least one study has found the latter to be an effective training platform for teaching CBT to counselors ( Weingardt, Cucciare, Bellotti, & Lai, 2009 ).

Different types of technology may have different specific applications to training and supervision, just as they do to counseling. Video conferencing and text-based interactions, such as using instant messaging or online chat forums, can be effective ways to improve counselor attitudes and skills ( Abbass et al., 2011 ). Carlson-Sabelli (2010) discussed the use of Internet forums as an adjunct to counselor training and supervision. Coursol, Lewis, & Seymour (2010) discussed the application of video conferencing technology to counselor training and supervision. However, not all studies have found Web-based training as effective as that delivered in person. For example, Sholomskas et al. (2005) found the effectiveness of a training Web site with written materials superior to written materials alone, but somewhat less effective than an in-person seminar with supervised casework for the teaching of CBT. The Internet can also be used to train auxiliary staff members and peer assistants. Worrall and Fruzzetti (2009) discussed the use of a Web-based training program using online videos for peer supervisors working with therapists delivering dialectical behavior therapy. Vaccaro and Lambie (2007) reviewed options for conducting computer-based training and supervision, as well as advantages and disadvantages and ethical concerns for this type of supervision/training.

Smith, Carpenter, et al. (2012) randomly assigned 97 substance use disorder treatment counselors who were enrolled in a 2-day motivational interviewing workshop to receive live supervision conducted using video conferencing technology, supervision using videotaped practice sessions, or the workshop alone without an additional supervision component. Participants' sessions with clients were rated 1, 8, and 20 weeks after the workshop using the Motivation Interviewing Treatment Integrity Coding System. Participants who used teleconferencing for supervision had significantly better compliance compared with those who used the workshop alone, and they did a significantly better job in maintaining a proper ratio between questions and reflections than did those in either of the other groups.

Clinical supervision can also be conducted using phone and Internet technologies. Abbass et al. (2011) reviewed literature on the use of Web conferencing technology to supervise psychotherapists. They noted its benefits in terms of reducing costs, enabling long-distance supervision, and integrating supervision with training and educational materials. They also reviewed some potential problems, such as technical difficulties, the absence of local support during times of crisis, and possible difficulties/anxieties relating to the supervisory alliance. Wood, Miller, and Hargrove (2005) provided a model for a four-part training process for counselors and supervisors and discussed the use of telephone and computer technology to provide clinical supervision to counselors working in rural areas.

Peer supervision and support can also be provided to counselors via Internet or phone. Yeh et al. (2008) suggested that an online peer supervision group is a viable alternative to in-person groups, and they found that participants in an online peer supervision group for counselors felt comfortable and confident using this form of interaction. A related issue is the need to train therapists in the use of electronic media to conduct therapy. As Abbott et al. (2008) observed, training is needed to communicate effectively via computer, with attention to tasks such as communicating empathy via text instead of in person and handling ethical issues that might arise in the e-therapy situations.

• aan het Rot M, Hogenelst K, Schoevers RA. Mood disorders in everyday life: A systematic review of experience sampling and ecological momentary assessment studies. Clinical Psychology Review. 2012; 32 (6):510–523. [ PubMed : 22721999 ]
• Abbass A, Arthey S, Elliott J, Fedak T, Nowoweiski D, Markovski J, Nowoweiski S. Web-conference supervision for advanced psychotherapy training: A practical guide. Psychotherapy: Theory, Research, Practice, Training. 2011; 48 :109–118. [ PubMed : 21639654 ]
• Abbott JA, Klein B, Ciechomski L. Best practices in online therapy. Journal of Technology in Human Services. 2008; 26 :360–375.
• Abroms LC, Padmanabhan N, Thaweethai L, Phillips T. iPhone apps for smoking cessation: A content analysis. American Journal of Preventive Medicine. 2011; 40 :279–285. [ PMC free article : PMC3395318 ] [ PubMed : 21335258 ]
• Acosta MC, Marsch LA, Xie H, Guarino H, Aponte-Melendez Y. A Web-based between behavior therapy program influences the association cognitive functioning and retention and abstinence in clients receiving methadone maintenance treatment. Journal of Dual Diagnosis. 2012; 8 :283–293. [ PMC free article : PMC3650891 ] [ PubMed : 23671409 ]
• Alcañiz M, Lozano JA, Rey B. Technological background about VR. In: Riva G, Botella C, Legeron P, Optale G, editors. Cybertherapy: Internet and virtual reality as assessment and rehabilitation tools for clinical psychology and neuroscience. Amsterdam: Ios Press; 2004. pp. 199–216.
• Alemi F, Haack MR, Dill R, Harge A. Engaging client's family and friends in online counseling. Journal of Addictions Nursing. 2005; 16 :47–55. [ PMC free article : PMC2789596 ] [ PubMed : 19997537 ]
• Alemi F, Haack MR, Nemes S, Aughburns R, Sinkule J, Neuhauser D. Therapeutic emails. Substance Abuse Treatment, Prevention, and Policy. 2007; 2 [ PMC free article : PMC1805747 ] [ PubMed : 17302991 ]
• Alleman JR. Online counseling: The Internet and mental health treatment. Psychotherapy: Theory, Research, Practice, Training. 2002; 39 :199–209.
• Almlöv J, Carlbring P, Berger T, Cuijpers P, Andersson G. Therapist factors in Internet-delivered cognitive behavioural therapy for major depressive disorder. Cognitive Behaviour Therapy. 2009; 38 :247–254. [ PubMed : 19802751 ]
• American Telemedicine Association. Evidence-based practice for telemental health. Washington, DC: Author; 2009.
• American Telemedicine Association. Category: US states. Washington, DC: Author; 2010.
• Amir N, Beard C, Burns M, Bomyea J. Attention modification program in individuals with generalized anxiety disorder. Journal of Abnormal Psychology. 2009; 118 :28–33. [ PMC free article : PMC2645540 ] [ PubMed : 19222311 ]
• Amir N, Beard C, Taylor CT, Klumpp H, Elias J, Burns M, Chen X. Attention training in individuals with generalized social phobia: A randomized controlled trial. Journal of Consulting and Clinical Psychology. 2009; 77 :961–973. [ PMC free article : PMC2796508 ] [ PubMed : 19803575 ]
• Amir N, Weber G, Beard C, Bomyea J, Taylor CT. The effect of a single-session attention modification program on response to a public-speaking challenge in socially anxious individuals. Journal of Abnormal Psychology. 2008; 117 :860–868. [ PMC free article : PMC3569035 ] [ PubMed : 19025232 ]
• Amstadter AB, Broman-Fulks J, Zinzow H, Ruggiero KJ, Cercone J. Internet-based interventions for traumatic stress-related mental health problems: A review and suggestion for future research. Clinical Psychology Review. 2009; 29 :410–420. [ PMC free article : PMC2704915 ] [ PubMed : 19403215 ]
• An LC, Klatt C, Perry CL, Lein EB, Hennrikus DJ, Pallonen UE, Ehlinger EP. The RealU online cessation intervention for college smokers: A randomized controlled trial. Preventive Medicine. 2008; 47 :194–199. [ PubMed : 18565577 ]
• An LC, Schillo BA, Saul JE, Wendling AH, Klatt CM, Berg CJ, Luxenberg MG. Utilization of smoking cessation informational, interactive, and online community resources as predictors of abstinence: Cohort study. Journal of Medical Internet Research. 2008; 10 :e55. [ PMC free article : PMC2630836 ] [ PubMed : 19103587 ]
• Andersson G. Using the Internet to provide cognitive behavior therapy. Behaviour Research and Therapy. 2009; 47 (3):175–180. [ PubMed : 19230862 ]
• Andersson G, Carlbring P, Grimlund A. Predicting treatment outcome in Internet versus face to face treatment of panic disorder. Computers in Human Behavior. 2008; 24 :1790–1801.
• Andersson G, Cuijpers P. Internet-based and other computerized psychological treatments for adult depression: A meta-analysis. Cognitive Behaviour Therapy. 2009; 38 :196–205. [ PubMed : 20183695 ]
• Andrews G, Cuijpers P, Craske MG, McEvoy P, Titov N. Computer therapy for the anxiety and depressive disorders is effective, acceptable and practical health care: A meta-analysis. PLOS One. 2010; 5 :e13196. [ PMC free article : PMC2954140 ] [ PubMed : 20967242 ]
• Andrews G, Davies M, Titov N. Effectiveness of a randomized controlled trial of face to face versus Internet cognitive behaviour therapy for social phobia. Australian and New Zealand Journal of Psychiatry. 2011; 45 :337–340. [ PubMed : 21323490 ]
• Anthony K. Therapy online—The therapeutic relationship in typed text. In: Bolton G, Howlett S, Lago C, Wright JK, editors. Writing cures: An introductory handbook of writing in counselling and psychotherapy. Hove, United Kingdom: Brunner-Routledge; 2004. pp. 133–141.
• APA Practice Organization. Reimbursement for telehealth services. Washington, DC: Author; 2011.
• Armfield NR, Gray LC, Smith AC. Clinical use of Skype: A review of the evidence base. Journal of Telemedicine and Telecare. 2012; 18 :125–127. [ PubMed : 22362829 ]
• Atherton H, Sawmynaden P, Sheikh A, Majeed A, Car J. Email for clinical communication between patients/caregivers and healthcare professionals. Cochrane Database of Systematic Reviews. 2012; 11 CD007978. [ PubMed : 23152249 ]
• Bachofen M, Nakagawa A, Marks IM, Park JM, Greist JH, Baer L, Dottl SL. Home self-assessment and self-treatment on obsessive–compulsive disorder using a manual and a computer-conducted telephone interview: Replication of a US-UK study. Journal of Clinical Psychiatry. 1999; 60 :545–549. [ PubMed : 10485637 ]
• Backhaus A, Agha Z, Maglione ML, Repp A, Ross B, Zuest D, Thorp SR. Videoconferencing psychotherapy: A systematic review. Psychological Services. 2012; 9 :111–131. [ PubMed : 22662727 ]
• Baker DC, Bufka LF. Preparing for the telehealth world: Navigating legal, regulatory, reimbursement, and ethical issues in an electronic age. Professional Psychology: Research and Practice. 2011; 42 :405–411.
• Balmford J, Borland R, Benda P. Patterns of use of an automated interactive personalized coaching program for smoking cessation. Journal of Medical Internet Research. 2008; 10 :e54. [ PMC free article : PMC2630829 ] [ PubMed : 19097975 ]
• Balmford J, Borland R, Li L, Ferretter I. Usage of an Internet smoking cessation resource: The Australian QuitCoach. Drug and Alcohol Review. 2009; 28 :66–72. [ PMC free article : PMC4493749 ] [ PubMed : 19320678 ]
• Baños RM, Guillen V, Quero S, Garcia-Palacios A, Alcaniz M, Botella C. A virtual reality system for the treatment of stress-related disorders: A preliminary analysis of efficacy compared to a standard cognitive behavioral program. International Journal of Human Computer Studies. 2011; 69 :602–613.
• Barak A, Boneh O, Dolev-Cohen M. Factors underlying participants' gains in online support groups. In: Blachnio AE, Przepiórka AE, Rowinski T, editors. Internet in psychological research. Warsaw, Poland: Cardinal Stefan Wyszyński University Press; 2010. pp. 17–38.
• Barak A, Buchanan T. Internet-based psychological testing and assessment. In: Kraus R, Zack JS, Stricker G, editors. Online counseling: A handbook for mental health professionals. New York: Elsevier Science; 2004. pp. 217–239.
• Barak A, Hen L, Boniel-Nissim M, Shapira N. A comprehensive review and a meta-analysis of the effectiveness of Internet-based psychotherapeutic interventions. Journal of Technology in Human Services. 2008; 26 :109–160.
• Barak A, Klein B, Proudfoot JG. Defining Internet-supported therapeutic interventions. Annals of Behavioral Medicine. 2009; 38 :4–17. [ PubMed : 19787305 ]
• Barnett JE, Scheetz K. Technological advances and telehealth: Ethics, law, and the practice of psychotherapy. Psychotherapy: Theory, Research, Practice, Training. 2003; 40 :86–93.
• Barnett NP, Tidey J, Murphy JG, Swift R, Colby SM. Contingency management for alcohol use reduction: A pilot study using a transdermal alcohol sensor. Drug and Alcohol Dependence. 2011; 118 (2–3):391–399. [ PMC free article : PMC3190068 ] [ PubMed : 21665385 ]
• Bauer S, Moessner M, Wolf M, Haug S, Kordy H. ES[S]PRIT: An Internet-based programme for the prevention and early intervention of eating disorders in college students. British Journal of Guidance & Counselling. 2009; 37 :327–336.
• Bauer S, Percevic R, Okon E, Meermann R, Kordy H. Use of text messaging in the aftercare of patients with bulimia nervosa. European Eating Disorders Review. 2003; 11 :279–290.
• Beard C, Weisberg RB, Amir N. Combined cognitive bias modification treatment for social anxiety disorder: A pilot trial. Depression and Anxiety. 2011; 28 :981–988. [ PMC free article : PMC3215809 ] [ PubMed : 21898705 ]
• Bellafiore DR, Colon Y, Rosenberg P. Online counseling groups. In: Kraus R, Zack JS, Stricker G, editors. Online counseling: A handbook for mental health professionals. New York: Elsevier Science; 2004. pp. 197–216.
• Bennett K, Reynolds J, Christensen H, Griffiths KM. E-hub: An online self-help mental health service in the community. Medical Journal of Australia. 2010; 192 :S48–S52. [ PubMed : 20528710 ]
• Ben-Zeev D, Brenner CJ, Begale M, Duffecy J, Mohr DC, Mueser KT. Feasibility, acceptability, and preliminary efficacy of a smartphone intervention for schizophrenia. Schizophrenia Bulletin. 2014; 40 (6):1244–1253. [ PMC free article : PMC4193714 ] [ PubMed : 24609454 ]
• Berger T, Caspar F, Richardson R, Kneubuler B, Sutter D, Andersson G. Internet-based treatment of social phobia: A randomized controlled trial comparing unguided with two types of guided self-help. Behaviour Research and Therapy. 2011; 49 :158–169. [ PubMed : 21255767 ]
• Berger T, Hämmerli K, Gubser N, Andersson G, Caspar F. Internet-based treatment of depression: A randomized controlled trial comparing guided with unguided self-help. Cognitive Behaviour Therapy. 2011; 40 :251–266. [ PubMed : 22060248 ]
• Berger T, Hohl E, Caspar F. Internet-based treatment for social phobia: A randomized controlled trial. Journal of Clinical Psychology. 2009; 65 :1021–1035. [ PubMed : 19437505 ]
• Bersamin M, Paschall MJ, Fearnow-Kenney M, Wyrick D. Effectiveness of a Web-based alcohol-misuse and harm-prevention course among high- and low-risk students. Journal of American College Health. 2007; 55 :247–254. [ PubMed : 17319331 ]
• Bickel WK, Christensen DR, Marsch LA. A review of computer-based interventions used in the assessment, treatment, and research of drug addiction. Substance Use & Misuse. 2011; 46 :4–9. [ PMC free article : PMC3092368 ] [ PubMed : 21190401 ]
• Bickel WK, Marsch LA, Buchhalter AR, Badger GJ. Computerized behavior therapy for opioid-dependent outpatients: A randomized controlled trial. Experimental and Clinical Psychopharmacology. 2008; 16 :132–143. [ PMC free article : PMC2746734 ] [ PubMed : 18489017 ]
• Billings G. Michigan becomes 15th state to pass private payer telehealth reimbursement. Washington, DC: Center for Telehealth and e-Health Law; 2012.
• Bischoff RJ. Considerations in the use of telecommunications as a primary treatment medium: The application of behavioral telehealth to marriage and family therapy. American Journal of Family Therapy. 2004; 32 :173–187.
• Bishop D, Bryant KS, Giles SM, Hansen WB, Dusenbury L. Simplifying the delivery of a prevention program with Web-based enhancements. Journal of Primary Prevention. 2006; 27 :433–444. [ PubMed : 16763766 ]
• Blankers M, Koeter MWJ, Schippers GM. Internet therapy versus Internet self-help versus no treatment for problematic alcohol use: A randomized controlled trial. Journal of Consulting and Clinical Psychology. 2011; 79 :330–341. [ PubMed : 21534652 ]
• Bledsoe TS, Simmerok BD. A multimedia-rich platform to enhance student engagement and learning in an online environment. Journal of Asynchronous Learning Networks. 2014; 17 (4):57–66.
• Bock BC, Graham AL, Whiteley JA, Stoddard JL. A review of Web-assisted tobacco interventions (WATIs). Journal of Medical Internet Research. 2008; 10 :e39. [ PMC free article : PMC2630838 ] [ PubMed : 19000979 ]
• Boeschoten RE, Nieuwenhuis MM, van Oppen OP, Uitdehaag BM, Polman CH, Collette EH, Dekker J. Feasibility and outcome of a Web-based self-help intervention for depressive symptoms in patients with multiple sclerosis: A pilot study. Journal of the Neurological Sciences. 2012; 315 :104–109. [ PubMed : 22133479 ]
• Bopp JM, Miklowitz DJ, Goodwin GM, Stevens W, Rendell JM, Geddes JR. The longitudinal course of bipolar disorder as revealed through weekly text messaging: A feasibility study. Bipolar Disorders. 2010; 12 :327–334. [ PMC free article : PMC2921773 ] [ PubMed : 20565440 ]
• Boschen MJ, Casey LM. The use of mobile telephones as adjuncts to cognitive behavioral psychotherapy. Professional Psychology: Research and Practice. 2008; 39 :546–552.
• Boudreaux ED, Bedek KL, Gilles D, Baumann BM, Hollenberg S, Lord SA, Grissom G. The Dynamic Assessment and Referral System for Substance Abuse (DARSSA): Development, functionality, and end-user satisfaction. Drug and Alcohol Dependence. 2009; 99 :37–46. [ PMC free article : PMC2636871 ] [ PubMed : 18775606 ]
• Brendryen H, Drozd F, Kraft P. A digital smoking cessation program delivered through Internet and cell phone without nicotine replacement (happy ending): Randomized controlled trial. Journal of Medical Internet Research. 2008; 10 (5):e51. [ PMC free article : PMC2630841 ] [ PubMed : 19087949 ]
• Brodey BB, Rosen CS, Brodey IS, Sheetz BM, Steinfeld RR, Gastfriend DR. Validation of the Addiction Severity Index (ASI) for Internet and automated telephone self-report administration. Journal of Substance Abuse Treatment. 2004; 26 :253–259. [ PubMed : 15182889 ]
• Brooks AC, Ryder D, Carise D, Kirby KC. Feasibility and effectiveness of computer-based therapy in community treatment. Journal of Substance Abuse Treatment. 2010; 39 :227–235. [ PubMed : 20667682 ]
• Brouwer W, Kroeze W, Crutzen R, de Nooijer J, de Vries NK, Brug J, Oenema A. Which intervention characteristics are related to more exposure to Internet-delivered healthy lifestyle promotion interventions? A systematic review. Journal of Medical Internet Research. 2011; 13 :e2. [ PMC free article : PMC3221341 ] [ PubMed : 21212045 ]
• Brunette MF, Ferron JC, McHugo GJ, Davis KE, Devitt TS, Wilkness SM, Drake RE. An electronic decision support system to motivate people with severe mental illnesses to quit smoking. Psychiatric Services. 2011; 62 :360–366. [ PubMed : 21459986 ]
• Budney AJ, Fearer S, Walker DD, Stanger C, Thostenson J, Grabinski M, Bickel WK. An initial trial of a computerized behavioral intervention for cannabis use disorder. Drug and Alcohol Dependence. 2010; 115 (1–2):74–79. [ PMC free article : PMC3071451 ] [ PubMed : 21131143 ]
• Burgess N, Christensen H, Leach LS, Farrer L, Griffiths KM. Mental health profile of callers to a telephone counselling service. Journal of Telemedicine and Telecare. 2008; 14 :42–47. [ PubMed : 18318929 ]
• Bush NE, Bosmajian CP, Fairall JM, McCann RA, Ciulla RP. Afterdeployment ​.org : A Web-based multimedia wellness resource for the postdeployment military community. Professional Psychology: Research and Practice. 2011; 42 :455–462.
• Butcher JN, Perry JN, Atlis MM. Validity and utility of computer-based test interpretation. Psychological Assessment. 2000; 12 :6–18. [ PubMed : 10752359 ]
• Butcher JN, Perry J, Hahn J. Computers in clinical assessment: Historical developments, present status, and future challenges. Journal of Clinical Psychology. 2004; 60 :331–345. [ PubMed : 14981795 ]
• Butler LH, Correia CJ. Brief alcohol intervention with college student drinkers: Face-to-face versus computerized feedback. Psychology of Addictive Behaviors. 2009; 23 :163–167. [ PubMed : 19290702 ]
• Butler SF, Budman SH, Goldman RJ, Newman FL, Beckley KE, Trottier D, Cacciola JS. Initial validation of a computer-administered Addiction Severity Index: The ASI-MV. Psychology of Addictive Behaviors. 2001; 15 :4–12. [ PubMed : 11255937 ]
• Butler SF, Villapiano A, Malinow A. The effect of computer-mediated administration on self-disclosure of problems on the Addiction Severity Index. Journal of Addiction Medicine. 2009; 3 :194–203. [ PMC free article : PMC2794410 ] [ PubMed : 20161486 ]
• Button KS, Wiles NJ, Lewis G, Peters TJ, Kessler D. Factors associated with differential response to online cognitive behavioural therapy. Social Psychiatry and Psychiatric Epidemiology. 2012; 47 :827–833. [ PubMed : 21541696 ]
• Buzza C, Ono SS, Turvey C, Wittrock S, Noble M, Reddy G, Reisinger HS. Distance is relative: Unpacking a principal barrier in rural healthcare. Journal of General Internal Medicine. 2011; 26 (Suppl 2):648–654. [ PMC free article : PMC3191222 ] [ PubMed : 21989617 ]
• Callan JA, Wright JH. Mood disorders. In: Cucciare MA, Weingardt KR, editors. Using technology to support evidence-based behavioral health practices: A clinician's guide. New York: Routledge/Taylor & Francis Group; 2010. pp. 3–26.
• Carey KB, Scott-Sheldon LA, Elliott JC, Bolles JR, Carey MP. Computer-delivered interventions to reduce college student drinking: A meta-analysis. Addiction. 2009; 104 (11):1807–1819. [ PMC free article : PMC2763045 ] [ PubMed : 19744139 ]
• Carlbring P, Degerman N, Jonsson J, Andersson G. Internet-based treatment of pathological gambling with a three-year follow-up. Cognitive Behaviour Therapy. 2012; 41 (4):321–334. [ PMC free article : PMC3516818 ] [ PubMed : 22620990 ]
• Carlbring P, Maurin L, Torngren C, Linna E, Eriksson T, Sparthan E, Andersson G. Individually-tailored, Internet-based treatment for anxiety disorders: A randomized controlled trial. Behaviour Research and Therapy. 2011; 49 :18–24. [ PubMed : 21047620 ]
• Carlbring P, Nilsson-Ihrfelt E, Waara J, Kollenstam C, Buhrman M, Kaldo V, Andersson G. Treatment of panic disorder: Live therapy vs. self-help via the Internet. Behaviour Research and Therapy. 2005; 43 :1321–1333. [ PubMed : 16086983 ]
• Carlbring P, Nordgren LB, Furmark T, Andersson G. Long-term outcome of Internet-delivered cognitive–behavioural therapy for social phobia: A 30-month follow-up. Behaviour Research and Therapy. 2009; 47 :848–850. [ PubMed : 19631312 ]
• Carlson-Sabelli L. Using forums to enrich counselor training and supervision. In: Anthony K, Nagel DM, Goss S, editors. The use of technology in mental health: Applications, ethics and practice. Springfield, IL: Charles C. Thomas Publisher; 2010. pp. 415–420.
• Carr AC, Ghosh A, Marks IM. Computer-supervised exposure treatment for phobias. Canadian Journal of Psychiatry. 1988; 33 :112–117. [ PubMed : 3365636 ]
• Carrard I, Crepin C, Rouget P, Lam T, Golay A, Van der Linden M. Randomised controlled trial of a guided self-help treatment on the Internet for binge eating disorder. Behaviour Research and Therapy. 2011; 49 :482–491. [ PubMed : 21641580 ]
• Carrard I, Fernández-Aranda F, Lam T, Nevonen L, Liwowsky I, Volkart AC, Norring C. Evaluation of a guided Internet self-treatment programme for bulimia nervosa in several European countries. European Eating Disorders Review. 2011; 19 :138–149. [ PubMed : 20859989 ]
• Carrard I, Rouget P, Fernández-Aranda F, Volkart AC, Damoiseau M, Lam T. Evaluation and deployment of evidence based patient self-management support program for bulimia nervosa. International Journal of Medical Informatics. 2006; 75 :101–109. [ PubMed : 16115793 ]
• Carroll KM, Ball SA, Martino S, Nich C, Babuscio TA, Nuro KF, Rounsaville BJ. Computer-assisted delivery of cognitive–behavioral therapy for addiction: A randomized trial of CBT4CBT. American Journal of Psychiatry. 2008; 165 :881–888. [ PMC free article : PMC2562873 ] [ PubMed : 18450927 ]
• Carroll KM, Ball SA, Martino S, Nich C, Babuscio TA, Rounsaville BJ. Enduring effects of a computer-assisted training program for cognitive behavioral therapy: A 6-month follow-up of CBT4CBT. Drug and Alcohol Dependence. 2009; 100 :178–181. [ PMC free article : PMC2742309 ] [ PubMed : 19041197 ]
• Carroll KM, Kiluk BD, Nich C, Gordon MA, Portnoy GA, Marino DR, Ball SA. Computer-assisted delivery of cognitive-behavioral therapy: Efficacy and durability of CBT4CBT among cocaine-dependent individuals maintained on methadone. American Journal of Psychiatry. 2014; 171 (4):436–444. [ PMC free article : PMC4042674 ] [ PubMed : 24577287 ]
• Cartreine JA, Ahern DK, Locke SE. A roadmap to computer-based psychotherapy in the United States. Harvard Review of Psychiatry. 2010; 18 :80–95. [ PubMed : 20235773 ]
• Cavanagh K, Shapiro DA, Van Den Berg S, Swain S, Barkham M, Proudfoot J. The acceptability of computer-aided cognitive behavioural therapy: A pragmatic study. Cognitive Behaviour Therapy. 2009; 38 :235–246. [ PubMed : 19306147 ]
• Center for Substance Abuse Treatment. Considerations for the provision of e-therapy. Rockville, MD: Center for Substance Abuse Treatment, Substance Abuse and Mental Health Services Administration; 2009. HHS Publication. No. (SMA) 09-4450.
• Center for Substance Abuse Treatment. Implementing change in substance abuse treatment programs. Rockville, MD: Substance Abuse and Mental Health Services Administration; 2009. (Technical Assistance Publication (TAP) Series 31). HHS Publication No. (SMA) 09-4377. 2009.
• Chen YF, Madan J, Welton N, Yahaya I, Aveyard P, Bauld L, Munafò MR. Effectiveness and cost-effectiveness of computer and other electronic aids for smoking cessation: A systematic review and network meta-analysis. Health Technology Assessment. 2012; 16 :i–v. [ PubMed : 23046909 ]
• Chiauzzi E, Brevard J, Thurn C, Decembrele S, Lord S. MyStudentBody-stress: An online stress management intervention for college students. Journal of Health Communication. 2008; 13 :555–572. [ PubMed : 18726812 ]
• Chiauzzi E, Green TC, Lord S, Thum C, Goldstein M. My Student Body: A high-risk drinking prevention Web site for college students. Journal of American College Health. 2005; 53 :263–274. [ PMC free article : PMC1885481 ] [ PubMed : 15900990 ]
• Chinman M, Hassell J, Magnabosco J, Nowlin-Finch N, Marusak S, Young AS. The feasibility of computerized patient self-assessment at mental health clinics. Administration and Policy in Mental Health. 2007; 34 :401–409. [ PubMed : 17453332 ]
• Chinman M, Tremain B, Imm P, Wandersman A. Strengthening prevention performance using technology: A formative evaluation of interactive Getting to Outcomes. American Journal of Orthopsychiatry. 2009; 79 (4):469–481. [ PMC free article : PMC2859836 ] [ PubMed : 20099938 ]
• Choi I, Zou J, Titov N, Dear BF, Li S, Johnston L, Hunt C. Culturally attuned Internet treatment for depression amongst Chinese Australians: A randomised controlled trial. Journal of Affective Disorders. 2012; 136 :459–468. [ PubMed : 22177742 ]
• Chou SW, Hunt MY, Beckjord BE, Moser PR, Hesse WB. Social media use in the United States: Implications for health communication. Journal of Medical Internet Research. 2009; 11 :e48. [ PMC free article : PMC2802563 ] [ PubMed : 19945947 ]
• Clarke G, Eubanks D, Reid E, Kelleher C, O'Connor E, DeBar LL, Gullion C. Overcoming Depression on the Internet (ODIN) (2): A randomized trial of a self-help depression skills program with reminders. Journal of Medical Internet Research. 2005; 7 :e16. [ PMC free article : PMC1550641 ] [ PubMed : 15998607 ]
• Clarke G, Reid E, Eubanks D, O'Connor E, DeBar LL, Kelleher C, Nunley S. Overcoming depression on the Internet (ODIN): A randomized controlled trial of an Internet depression skills intervention program. Journal of Medical Internet Research. 2002; 4 :E14. [ PMC free article : PMC1761939 ] [ PubMed : 12554545 ]
• Collins GB, McAllister MS, Ford DB. Patient–provider e-mail communication as an adjunctive tool in addiction medicine. Journal of Addictive Diseases. 2007; 26 :45–52. [ PubMed : 17594997 ]
• Cook JE, Doyle C. Working alliance in online therapy as compared to face-to-face therapy: Preliminary results. CyberPsychology and Behavior. 2002; 5 :95–105. [ PubMed : 12025884 ]
• Copeland J, Martin G. Web-based interventions for substance use disorders: A qualitative review. Journal of Substance Abuse Treatment. 2004; 26 :109–116. [ PubMed : 15050088 ]
• Coursol DH, Lewis J, Seymour JW. The use of videoconferencing to enrich counselor training and supervision. In: Anthony K, Nagel DM, Goss S, editors. The use of technology in mental health: Applications, ethics and practice. Springfield, IL: Charles C. Thomas Publisher; 2010. pp. 280–288.
• Craske MG, Rose RD, Lang A, Welch SS, Campbell-Sills L, Sullivan G, Roy-Byrne PP. Computer-assisted delivery of cognitive behavioral therapy for anxiety disorders in primary-care settings. Depression and Anxiety. 2009; 26 :235–242. [ PMC free article : PMC2906395 ] [ PubMed : 19212970 ]
• Crawley A, Fetzner M, editors. Journal of Asynchronous Learning Networks. 2013; 17 :3.
• Cucciare MA. Implementation. In: Cucciare MA, Weingardt KR, editors. Using technology to support evidence-based behavioral health practices: A clinician's guide. New York: Routledge/Taylor & Francis Group; 2010. pp. 225–245.
• Cucciare MA, Weingardt KR, Humphreys K. How Internet technology can improve the quality of care for substance use disorders. Current Drug Abuse Reviews. 2009; 2 :256–262. [ PubMed : 20443772 ]
• Cuijpers P, Donker T, van Straten A, Li J, Andersson G. Is guided self-help as effective as face-to-face psychotherapy for depression and anxiety disorders? A systematic review and meta-analysis of comparative outcome studies. Psychological Medicine. 2010:1–15. [ PubMed : 20406528 ]
• Cuijpers P, Marks IM, van Straten A, Cavanagh K, Gega L, Andersson G. Computer-aided psychotherapy for anxiety disorders: A meta-analytic review. Cognitive Behaviour Therapy. 2009; 38 :66–82. [ PubMed : 20183688 ]
• Cummins SE, Bailey L, Campbell S, Koon-Kirby C, Zhu SH. Tobacco cessation quitlines in North America: A descriptive study. Tobacco Control. 2007; 16 (Suppl 1):i9–15. [ PMC free article : PMC2598516 ] [ PubMed : 18048639 ]
• Cunningham JA, Van Mierlo T. Methodological issues in the evaluation of Internet-based interventions for problem drinking. Drug and Alcohol Review. 2009; 28 :12–17. [ PubMed : 19320670 ]
• Cunningham JA, Wild TC, Cordingley J, Van Mierlo T, Humphreys K. A randomized controlled trial of an Internet-based intervention for alcohol abusers. Addiction. 2009; 104 :2023–2032. [ PMC free article : PMC2779998 ] [ PubMed : 19922569 ]
• Dale J, Caramlau IO, Lindenmeyer A, Williams SM. Peer support telephone calls for improving health. Cochrane Database of Systematic Reviews. 2008 CD006903. [ PMC free article : PMC7386897 ] [ PubMed : 18843736 ]
• Dallery J, Raiff BR. Contingency management in the 21st century: Technological innovations to promote smoking cessation. Substance Use & Misuse. 2011; 46 :10–22. [ PMC free article : PMC3085931 ] [ PubMed : 21190402 ]
• Danaher BG, Lichtenstein E, McKay GH, Seeley RJ. Use of non-assigned smoking cessation programs among participants of a Web-based randomized controlled trial. Journal of Medical Internet Research. 2009; 11 :e26. [ PMC free article : PMC2762799 ] [ PubMed : 19632976 ]
• Danaher BG, McKay HG, Seeley JR. The information architecture of behavior change Websites. Journal of Medical Internet Research. 2005; 7 :e12. [ PMC free article : PMC1550648 ] [ PubMed : 15914459 ]
• Danaher BG, Smolkowski K, Seeley JR, Severson HH. Mediators of a successful Web-based smokeless tobacco cessation program. Addiction. 2008; 103 :1706–1712. [ PMC free article : PMC3782377 ] [ PubMed : 18715238 ]
• Dausch BM, Miklowitz DJ, Nagamoto HT, Adler LE, Shore JH. Family-focused therapy via videoconferencing. Journal of Telemedicine and Telecare. 2009; 15 :211–214. [ PubMed : 19471035 ]
• Day SX, Schneider PL. The subjective experiences of therapists in face-to-face, video, and audio sessions. In: Bloom JW, Walz GR, editors. Cybercounseling and cyberlearning: Strategies and resources for the millennium. Alexandria, VA: American Counseling Association; 2000. pp. 203–218.
• Day SX, Schneider PL. Psychotherapy using distance technology: A comparison of face-to-face, video, and audio treatment. Journal of Counseling Psychology. 2002; 49 :499–503.
• Dear BF, Titov N, Schwencke G, Andrews G, Johnston L, Craske MG, McEvoy P. An open trial of a brief transdiagnostic Internet treatment for anxiety and depression. Behaviour Research and Therapy. 2011; 49 :830–837. [ PubMed : 21993153 ]
• Deegan PE. The lived experience of using psychiatric medication in the recovery process and a shared decision-making program to support it. Psychiatric Rehabilitation Journal. 2007; 31 :62–69. [ PubMed : 17694717 ]
• de Graaf LE, Gerhards SAH, Arntz A, Riper H, Metsemakers JFM, Evers SM, Huibers MJ. One-year follow-up results of unsupported online computerized cognitive behavioural therapy for depression in primary care: A randomized trial. Journal of Behavior Therapy and Experimental Psychiatry. 2011; 42 :89–95. [ PubMed : 20723885 ]
• de Graaf LE, Hollon SD, Huibers MJ. Predicting outcome in computerized cognitive behavioral therapy for depression in primary care: A randomized trial. Journal of Consulting and Clinical Psychology. 2010; 78 :184–189. [ PubMed : 20350029 ]
• De Las Cuevas C, Arredondo MT, Cabrera MF, Sulzenbacher H, Meise U. Randomized clinical trial of telepsychiatry through videoconference versus face-to-face conventional psychiatric treatment. Telemedicine Journal and e-Health. 2006; 12 :341–350. [ PubMed : 16796502 ]
• Delrahim-Howlett K, Chambers CD, Clapp JD, Xu R, Duke K, Moyer RJ III, Van Sickle D. Web-based assessment and brief intervention for alcohol use in women of childbearing potential: A report of the primary findings. Alcoholism: Clinical and Experimental Research. 2011; 35 :1331–1338. [ PubMed : 21410488 ]
• Depp CA, Kim DH, de Dios LV, Wang V, Ceglowski J. A pilot study of mood ratings captured by mobile phone versus paper-and-pencil mood charts in bipolar disorder. Journal of Dual Diagnosis. 2012; 8 :326–332. [ PMC free article : PMC3640573 ] [ PubMed : 23646035 ]
• Derrig-Palumbo K. Using chat and instant messaging (IM) to conduct a therapeutic relationship. In: Anthony K, Nagel DM, Goss S, editors. The use of technology in mental health: Applications, ethics and practice. Springfield, IL: Charles C. Thomas Publisher; 2010. pp. 15–28.
• Difede J, Cukor J, Patt I, Giosan C, Hoffman H. The application of virtual reality to the treatment of PTSD following the WTC attack. Annals of the New York Academy of Sciences. 2006; 1071 :500–501. [ PubMed : 16891607 ]
• Dolan D. New guideline for treating psychiatric disorders in multiple sclerosis. Neurology Now. 2014; 10 (1):12–14.
• Donker T, van Straten A, Marks I, Cuijpers P. A brief Web-based screening questionnaire for common mental disorders: Development and validation. Journal of Medical Internet Research. 2009; 11 :e19. [ PMC free article : PMC2763401 ] [ PubMed : 19632977 ]
• Donovan E, Wood M, Frayjo K, Black RA, Surette DA. A randomized, controlled trial to test the efficacy of an online, parent-based intervention for reducing the risks associated with college-student alcohol use. Addictive Behaviors. 2012; 37 :25–35. [ PMC free article : PMC3227221 ] [ PubMed : 21963316 ]
• Dorstyn DS, Mathias JL, Denson LA. Psychosocial outcomes of telephone-based counseling for adults with an acquired physical disability: A meta-analysis. Rehabilitation Psychology. 2011; 56 :1–14. [ PubMed : 21401281 ]
• Doumas DM, Andersen LL. Reducing alcohol use in first-year university students: Evaluation of a Web-based personalized feedback program. Journal of College Counseling. 2009; 12 :18–32.
• Doumas DM, Kane CM, Navarro B, Roman J. Decreasing heavy drinking in first-year students: Evaluation of a Web-based personalized feedback program administered during orientation. Journal of College Counseling. 2011; 14 :5–20.
• Doumas DM, McKinley LL, Book P. Evaluation of two Web-based alcohol interventions for mandated college students. Journal of Substance Abuse Treatment. 2009; 36 :65–74. [ PubMed : 18657941 ]
• DuHamel KN, Mosher CE, Winkel G, Labay LE, Rini C, Meschian YM, Redd WH. Randomized clinical trial of telephone-administered cognitive–behavioral therapy to reduce post-traumatic stress disorder and distress symptoms after hematopoietic stem-cell transplantation. Journal of Clinical Oncology. 2010; 28 :3754–3761. [ PMC free article : PMC2917309 ] [ PubMed : 20625129 ]
• Dulin PL, Gonzalez VW, Campbell K. Results of a pilot test of a self-administered smartphone-based treatment system for alcohol use disorders: Usability and early outcomes. Substance Abuse. 2014; 35 (2):168–175. [ PMC free article : PMC4019942 ] [ PubMed : 24821354 ]
• Eakin EG, Lawler SP, Vandelanotte C, Owen N. Telephone interventions for physical activity and dietary behavior change: A systematic review. American Journal of Preventive Medicine. 2007; 32 :419–434. [ PubMed : 17478269 ]
• Ehrenreich B, Righter B, Rocke DA, Dixon L, Himelhoch S. Are mobile phones and handheld computers being used to enhance delivery of psychiatric treatment? A systematic review. Journal of Nervous and Mental Disease. 2011; 199 :886–891. [ PubMed : 22048142 ]
• Eichhorn KC. Soliciting and providing social support over the Internet: An investigation of online eating disorder support groups. Journal of Computer-Mediated Communication. 2008; 14 :67–78.
• Emmelkamp PM. Attention bias modification: The emperor's new suit? BMC Medicine. 2012; 10 :63. [ PMC free article : PMC3391176 ] [ PubMed : 22731990 ]
• Enos G. Technology [online virtual platforms such as Second Life could help clients model recovery-affirming behaviors] Addiction Professional. 2008; 6 :53–54.
• Eonta AM, Christon LM, Hourigan SE, Ravindran N, Vrana SR, Southam-Gerow MA. Using everyday technology to enhance evidence-based treatments. Professional Psychology: Research and Practice. 2011; 42 :513–520.
• Erdman HP, Klein MH, Greist JH. Direct patient computer interviewing. Journal of Consulting and Clinical Psychology. 1985; 53 :760–773. [ PubMed : 3841354 ]
• Erwin BA, Turk CL, Heimberg RG, Fresco DM, Hantula DA. The Internet: Home to a severe population of individuals with social anxiety disorder? Journal of Anxiety Disorders. 2004; 18 :629–646. [ PubMed : 15275943 ]
• Espie CA, Kyle SD, Williams C, Ong JC, Douglas NJ, Hames P, Brown JS. A randomized, placebo-controlled trial of online cognitive behavioral therapy for chronic insomnia disorder delivered via an automated media-rich Web application. Sleep. 2012; 35 :769–781. [ PMC free article : PMC3353040 ] [ PubMed : 22654196 ]
• Etter JF. Comparing the efficacy of two Internet-based, computer-tailored smoking cessation programs: A randomized trial. Journal of Medical Internet Research. 2005; 7 :e2. [ PMC free article : PMC1550632 ] [ PubMed : 15829474 ]
• Etter JF. A list of the most popular smoking cessation Web sites and a comparison of their quality. Nicotine and Tobacco Research. 2006; 8 (Suppl 1):S27–S34. [ PubMed : 17491168 ]
• Eyrich-Garg KM. Mobile phone technology: A new paradigm for the prevention, treatment, and research of the non-sheltered “street” homeless? Journal of Urban Health. 2010; 87 :365–380. [ PMC free article : PMC2871091 ] [ PubMed : 20397058 ]
• Farabee D, Cousins SJ, Brecht ML, Antonini VP, Lee AB, Brummer J, Rawson RA. A comparison of four telephone-based counseling styles for recovering stimulant users. Psychology of Addictive Behaviors. 2012; 27 (1):223–229. [ PMC free article : PMC3500433 ] [ PubMed : 22867295 ]
• Farrer L, Christensen H, Griffiths KM, Mackinnon A. Internet-based CBT for depression with and without telephone tracking in a national helpline: Randomised controlled trial. PLOS ONE. 2011; 6 :e28099. [ PMC free article : PMC3227639 ] [ PubMed : 22140514 ]
• Farvolden P, Cunningham J, Selby P. Using e-health programs to overcome barriers to the effective treatment of mental health and addiction problems. Journal of Technology in Human Services. 2009; 27 :5–22.
• Farvolden P, McBride C, Bagby RM, Ravitz P. A Web-based screening instrument for depression and anxiety disorders in primary care. Journal of Medical Internet Research. 2003; 5 :e23. [ PMC free article : PMC1550568 ] [ PubMed : 14517114 ]
• Ferguson SG, Shiffman S. Using the methods of ecological momentary assessment in substance dependence research—Smoking cessation as a case study. Substance Use & Misuse. 2011; 46 :87–95. [ PubMed : 21190409 ]
• Fernández-Aranda F, Nunez A, Martinez C, Krug I, Cappozzo M, Carrard I, Lam T. Internet-based cognitive–behavioral therapy for bulimia nervosa: A controlled study. CyberPsychology and Behavior. 2009; 12 :37–41. [ PubMed : 19006463 ]
• Ferreira CJ. Effectiveness of teaching basic microskills and empathy development in an online environment. Virginia Beach: VA: Regent University; 2005.
• Ferrer-García M, Gutiérrez-Maldonado J. The use of virtual reality in the study, assessment, and treatment of body image in eating disorders and nonclinical samples: A review of the literature. Body Image. 2012; 9 :1–11. [ PubMed : 22119329 ]
• Ferron JC, Brunette MF, McGurk SR, Xie H, Frounfelker R, Cook JA, McHugo G. Do symptoms and cognitive problems affect the use and efficacy of a Web-based decision support system for smokers with serious mental illness? Journal of Dual Diagnosis. 2012; 8 :315–325.
• Fichter MM, Quadflieg N, Nisslmuller K, Lindner S, Osen B, Huber T, Wünsch-Leiteritz W. Does Internet-based prevention reduce the risk of relapse for anorexia nervosa? Behaviour Research and Therapy. 2012; 50 :180–190. [ PubMed : 22317754 ]
• Finfgeld-Connett D. Web-based treatment for rural women with alcohol problems: Preliminary findings. CIN: Computers, Informatics, Nursing. 2009; 27 :345–353. [ PMC free article : PMC2819034 ] [ PubMed : 19901570 ]
• Finkelstein J, Lapshin O. Reducing depression stigma using a Web-based program. International Journal of Medical Informatics. 2007; 76 :726–734. [ PubMed : 16996299 ]
• Finkelstein J, Lapshin O, Cha E. Feasibility of promoting smoking cessation among methadone users using multimedia computer-assisted education. Journal of Medical Internet Research. 2008; 10 :e33. [ PMC free article : PMC2630834 ] [ PubMed : 18984556 ]
• Fiore MC, Bailey WC, Cohen SJ. Treating tobacco use and dependence: Clinical practice guideline. Rockville, MD: U.S. Department of Health and Human Services; 2000.
• Fjeldsoe BS, Marshall AL, Miller YD. Behavior change interventions delivered by mobile telephone short-message service. American Journal of Preventive Medicine. 2009; 36 :165–173. [ PubMed : 19135907 ]
• Forducey PG, Glueckauf RL, Bergquist TF, Maheu MM, Yutsis M. Telehealth for persons with severe functional disabilities and their caregivers: Facilitating self-care management in the home setting. Psychological Services. 2012; 9 :144–162. [ PMC free article : PMC3375593 ] [ PubMed : 22662729 ]
• Fox S, Duggan M. Mobile health 2012: Half of smartphone owners use their devices to get health information and one-fifth of smartphone owners have health apps. Washington, DC: Pew Research Center 's Internet & American Life Project; 2012.
• Frangou S, Sachpazidis I, Stassinakis A, Sakas G. Telemonitoring of medication adherence in patients with schizophrenia. Telemedicine Journal and e-Health. 2005; 11 :675–683. [ PubMed : 16430387 ]
• Franko DL, Mintz LB, Villapiano M, Green TC, Mainelli D, Folensbee L, Budman SH. Food, mood, and attitude: Reducing risk for eating disorders in college women. Health Psychology. 2005; 24 :567–578. [ PubMed : 16287402 ]
• Fraser T, McRobbie H, Bullen C, Whittaker R, Barlow D. Acceptability and outcome of an Internet-based smoking cessation programme. International Journal of Tuberculosis and Lung Disease. 2010; 14 :113–118. [ PubMed : 20003704 ]
• Free C, Knight R, Robertson S, Whittaker R, Edwards P, Zhou W, Roberts I. Smoking cessation support delivered via mobile phone text messaging (txt2stop): A single-blind, randomised trial. The Lancet. 2011; 378 :49–55. [ PMC free article : PMC3143315 ] [ PubMed : 21722952 ]
• Freedman MJ, Lester KM, McNamara C, Milby JB, Schumacher JE. Cell phones for ecological momentary assessment with cocaine-addicted homeless patients in treatment. Journal of Substance Abuse Treatment. 2006; 30 :105–111. [ PubMed : 16490673 ]
• Frueh BC, Henderson S, Myrick H. Telehealth service delivery for persons with alcoholism. Journal of Telemedicine and Telecare. 2005; 11 :372–375. [ PubMed : 16238840 ]
• Furmark T, Carlbring P, Hedman E, Sonnenstein A, Clevberger P, Bohman B, Andersson G. Guided and unguided self-help for social anxiety disorder: Randomised controlled trial. British Journal of Psychiatry. 2009; 195 :440–447. [ PubMed : 19880935 ]
• Gainsbury S, Blaszczynski A. Online self-guided interventions for the treatment of problem gambling. International Gambling Studies. 2011; 11 :289–308.
• García-Lizana F, Muñoz-Mayorga I. Telemedicine for depression: A systematic review. Perspectives in Psychiatric Care. 2010; 46 :119–126. [ PubMed : 20377799 ]
• García-Lizana F, Muñoz-Mayorga I. What about telepsychiatry? A systematic review. Primary Care Companion to the Journal of Clinical Psychiatry. 2010; 12 (2) [ PMC free article : PMC2911004 ] [ PubMed : 20694116 ]
• Gerardi M, Cukor J, Difede J, Rizzo A, Rothbaum BO. Virtual reality exposure therapy for post-traumatic stress disorder and other anxiety disorders. Current Psychiatry Reports. 2010; 12 :298–305. [ PubMed : 20535592 ]
• Gerber BS, Stolley MR, Thompson AL, Sharp LK, Fitzgibbon ML. Mobile phone text messaging to promote healthy behaviors and weight loss maintenance: A feasibility study. Health Informatics Journal. 2009; 15 :17–25. [ PMC free article : PMC2730665 ] [ PubMed : 19218309 ]
• Germain V, Marchand A, Bouchard S, Drouin MS, Guay S. Effectiveness of cognitive behavioural therapy administered by videoconference for posttraumatic stress disorder. Cognitive Behaviour Therapy. 2009; 38 :42–53. [ PubMed : 19235601 ]
• Germain V, Marchand A, Bouchard S, Guay S, Drouin MS. Assessment of the therapeutic alliance in face-to-face or videoconference treatment for posttraumatic stress disorder. Cyberpsychology, Behavior, and Social Networking. 2010; 13 :29–35. [ PubMed : 20528290 ]
• Gibbons RD, Weiss DJ, Kupfer DJ, Frank E, Fagiolini A, Grochocinski VJ, Immekus JC. Using computerized adaptive testing to reduce the burden of mental health assessment. Psychiatric Services. 2008; 59 :361–368. [ PMC free article : PMC2916927 ] [ PubMed : 18378832 ]
• Glynn SM, Randolph ET, Garrick T, Lui A. A proof of concept trial of an online psychoeducational program for relatives of both veterans and civilians living with schizophrenia. Psychiatric Rehabilitation Journal. 2010; 33 :278–287. [ PubMed : 20374986 ]
• Godleski L, Cervone D, Vogel D, Rooney M. Home telemental health implementation and outcomes using electronic messaging. Journal of Telemedicine and Telecare. 2012; 18 :17–19. [ PubMed : 22052966 ]
• Godleski L, Darkins A, Peters J. Outcomes of 98, 609 U.S. Department of Veterans Affairs patients enrolled in telemental health services, 2006–2010. Psychiatric Services. 2012; 63 :383–385. [ PubMed : 22476305 ]
• Godleski L, Nieves JE, Darkins A, Lehmann L. VA telemental health: Suicide assessment. Behavioral Sciences & the Law. 2008; 26 :271–286. [ PubMed : 18548515 ]
• Golkaramnay V, Bauer S, Haug S, Wolf M, Kordy H. The exploration of the effectiveness of group therapy through an Internet chat as aftercare: A controlled naturalistic study. Psychotherapy and Psychosomatics. 2007; 76 :219–225. [ PubMed : 17570960 ]
• Gorini A, Fasano AI, Gaggioli A, Vigna C, Riva G. A second life for telehealth: Prospects for the use of virtual online worlds in clinical psychology. Annual Review of CyberTherapy and Telemedicine. 2008; 6 :13–19.
• Gould MS, Kalafat J, Harris Munfakh JL, Kleinman M. An evaluation of crisis hotline outcomes. Part 2: Suicidal callers. Suicide and Life-Threatening Behavior. 2007; 37 :338–352. [ PubMed : 17579545 ]
• Gowen K, Deschaine M, Gruttadara D, Markey D. Young adults with mental health conditions and social networking Websites: Seeking tools to build community. Psychiatric Rehabilitation Journal. 2012; 35 :245–250. [ PubMed : 22246123 ]
• Grady B, Singleton M. Telepsychiatry “coverage” to a rural inpatient psychiatric unit. Telemedicine Journal and e-Health. 2011; 17 :603–608. [ PubMed : 21939381 ]
• Graham AL, Cobb NK, Papandonatos GD, Moreno JL, Kang H, Tinkelman DG, Abrams DB. A randomized trial of Internet and telephone treatment for smoking cessation. Archives of Internal Medicine. 2011; 171 :46–53. [ PMC free article : PMC3319451 ] [ PubMed : 21220660 ]
• Graham AL, Papandonatos DG. Reliability of Internet- versus telephone-administered questionnaires in a diverse sample of smokers. Journal of Medical Internet Research. 2008; 10 :e8. [ PMC free article : PMC2483847 ] [ PubMed : 18364345 ]
• Granholm E, Ben-Zeev D, Link PC, Bradshaw KR, Holden JL. Mobile Assessment and Treatment for Schizophrenia (MATS): A pilot trial of an interactive text-messaging intervention for medication adherence, socialization, and auditory hallucinations. Schizophrenia Bulletin. 2012; 38 :414–425. [ PMC free article : PMC3329971 ] [ PubMed : 22080492 ]
• Granholm E, Loh C, Swendsen J. Feasibility and validity of computerized ecological momentary assessment in schizophrenia. Schizophrenia Bulletin. 2008; 34 :507–514. [ PMC free article : PMC2632427 ] [ PubMed : 17932087 ]
• Green KE, Iverson KM. Computerized cognitive–behavioral therapy in a stepped care model of treatment. Professional Psychology: Research and Practice. 2009; 40 :96–103.
• Greist JH, Gustafson DH, Stauss FF, Rowse GL, Laughren TP, Chiles JA. Suicide risk prediction: A new approach. Suicide and Life-Threatening Behavior. 1974; 4 :212–223. [ PubMed : 4613980 ]
• Greist JH, Marks IM, Baer L, Kobak KA, Wenzel KW, Hirsch MJ, Clary CM. Behavior therapy for obsessive–compulsive disorder guided by a computer or by a clinician compared with relaxation as a control. Journal of Clinical Psychiatry. 2002; 63 :138–145. [ PubMed : 11874215 ]
• Griffiths KM, Christensen H. Internet-based mental health programs: A powerful tool in the rural medical kit. Australian Journal of Rural Health. 2007; 15 :81–87. [ PubMed : 17441815 ]
• Griffiths KM, Farrer L, Christensen H. The efficacy of Internet interventions for depression and anxiety disorders: A review of randomised controlled trials. Medical Journal of Australia. 2010; 192 :S4–11. [ PubMed : 20528707 ]
• Griffiths MK, Calear LA, Banfield M, Tam A. Systematic review on Internet Support Groups (ISGs) and depression (2): What is known about depression ISGs? Journal of Medical Internet Research. 2009; 11 :e41. [ PMC free article : PMC2802257 ] [ PubMed : 19793718 ]
• Grohol JM. Online counseling: A historical perspective. In: Kraus R, Stricker G, editors. Online counseling: A handbook for mental health professionals. New York: Elsevier Science; 2004. pp. 51–68.
• Gros DF, Veronee K, Strachan M, Ruggiero KJ, Acierno R. Managing suicidality in home-based telehealth. Journal of Telemedicine and Telecare. 2011; 17 :332–335. [ PubMed : 21844180 ]
• Gros DF, Yoder M, Tuerk PW, Lozano BE, Acierno R. Exposure therapy for PTSD delivered to veterans via telehealth: Predictors of treatment completion and outcome and comparison to treatment delivered in person. Behavior Therapy. 2011; 42 :276–283. [ PubMed : 21496512 ]
• Grover M, Naumann U, Mohammad-Dar L, Glennon D, Ringwood S, Eisler I, Schmidt U. A randomized controlled trial of an Internet-based cognitive-behavioural skills package for carers of people with anorexia nervosa. Psychological Medicine. 2011; 41 :2581–2591. [ PubMed : 21733215 ]
• Gualtieri CT. An Internet-based symptom questionnaire that is reliable, valid, and available to psychiatrists, neurologists, and psychologists. Medscape General Medicine. 2007; 9 :3. [ PMC free article : PMC2234274 ] [ PubMed : 18311353 ]
• Gulec H, Moessner M, Mezei A, Kohls E, Tury F, Bauer S. Internet-based maintenance treatment for patients with eating disorders. Professional Psychology: Research and Practice. 2011; 42 :479–486.
• Gustafson DH, Boyle MG, Shaw BR, Isham A, McTavish F, Richards S, Johnson K. An e-health solution for people with alcohol problems. Alcohol Research and Health. 2011; 33 :327–337. [ PMC free article : PMC3536059 ] [ PubMed : 23293549 ]
• Gustafson DH, Greist JH, Stauss FF, Erdman H, Laughren T. A probabilistic system for identifying suicide attemptors. Computers and Biomedical Research. 1977; 10 :83–89. [ PubMed : 322942 ]
• Gustafson DH, McTavish FM, Chih M-Y, Atwood AK, Johnson RA, Boyle MG, Shah D. A smartphone application to support recovery from alcoholism: A randomized clinical trial. JAMA Psychiatry. 2014; 71 (5):566–572. [ PMC free article : PMC4016167 ] [ PubMed : 24671165 ]
• Gustafson DH, Shaw BR, Isham A, Baker T, Boyle MG, Levy M. Explicating an evidence-based, theoretically informed, mobile technology-based system to improve outcomes for people in recovery for alcohol dependence. Substance Use & Misuse. 2011; 46 :96–111. [ PMC free article : PMC3179272 ] [ PubMed : 21190410 ]
• Haas A, Koestner B, Rosenberg J, Moore D, Garlow SJ, Sedway J, Nemeroff CB. An interactive Web-based method of outreach to college students at risk for suicide. Journal of American College Health. 2008; 57 :15–22. [ PubMed : 18682341 ]
• Hailey D, Roine R, Ohinmaa A. The effectiveness of telemental health applications: A review. Canadian Journal of Psychiatry. 2008; 53 :769–778. [ PubMed : 19087471 ]
• Hall M, Hall P. Telemedicine reimbursement: A national scan of current policies and emerging initiatives. Sacramento, CA: California Telemedicine and eHealth Center; 2009.
• Hall MJ, Tidwell WC. Internet recovery for substance abuse and alcoholism: An exploratory study of service users. Journal of Substance Abuse Treatment. 2003; 24 :161–167. [ PubMed : 12745033 ]
• Hallett J, Maycock B, Kypri K, Howat P, McManus A. Development of a Web-based alcohol intervention for university students: Processes and challenges. Drug and Alcohol Review. 2009; 28 :31–39. [ PubMed : 19320673 ]
• Hanley T, Reynolds DJ Jr. Counselling psychology and the Internet: A review of the quantitative research into online outcomes and alliances within text-based therapy. Counselling Psychology Review. 2009; 24 :4–13.
• Hasin DS, Aharonovich E, Greenstein E. HealthCall for the smartphone: Technology enhancement of brief intervention in HIV alcohol dependent patients. Addiction Science & Clinical Practice. 2014; 9 :5. [ PMC free article : PMC3943503 ] [ PubMed : 24533631 ]
• Hasin DS, Aharonovich E, O'Leary A, Greenstein E, Pavlicova M, Arunajadai S, Johnston B. Reducing heavy drinking in HIV primary care: A randomized trial of brief intervention, with and without technological enhancement. Addiction. 2013; 108 (7):1230–1240. [ PMC free article : PMC3755729 ] [ PubMed : 23432593 ]
• Haug S, Meyer C, John U. Efficacy of an Internet program for smoking cessation during and after inpatient rehabilitation treatment: A quasi-randomized controlled trial. Addictive Behaviors. 2011; 36 :1369–1372. [ PubMed : 21907496 ]
• Haug S, Sedway J, Kordy H. Group processes and process evaluations in a new treatment setting: Inpatient group psychotherapy followed by Internet-chat aftercare groups. International Journal of Group Psychotherapy. 2008; 58 :35–53. [ PubMed : 18211213 ]
• Hayes BG. The use of multimedia instruction in counselor education: A creative teaching strategy. Journal of Creativity in Mental Health. 2008; 3 :243–253.
• Hazen RA, Vasey MW, Schmidt NB. Attentional retraining: A randomized clinical trial for pathological worry. Journal of Psychiatric Research. 2009; 43 :627–633. [ PubMed : 18722627 ]
• Hedman E, Andersson E, Ljotsson B, Andersson G, Ruck C, Lindefors N. Cost-effectiveness of Internet-based cognitive behavior therapy vs. cognitive behavioral group therapy for social anxiety disorder: Results from a randomized controlled trial. Behaviour Research and Therapy. 2011; 49 :729–736. [ PubMed : 21851929 ]
• Helzer JE, Badger GJ, Rose GL, Mongeon JA, Searles JS. Decline in alcohol consumption during two years of daily reporting. Journal of Studies on Alcohol. 2002; 63 :551–558. [ PubMed : 12380851 ]
• Herbst N, Voderholzer U, Stelzer N, Knaevelsrud C, Hertenstein E, Schlegl S, Külz AK. The potential of telemental health applications for obsessive–compulsive disorder. Clinical Psychology Review. 2012; 32 :454–466. [ PubMed : 22705583 ]
• Heron KE, Smyth JM. Ecological momentary interventions: Incorporating mobile technology into psychosocial and health behaviour treatments. British Journal of Health Psychology. 2010; 15 :1–39. [ PMC free article : PMC2800172 ] [ PubMed : 19646331 ]
• Hester RK, Delaney HD, Campbell W. The College Drinker's Check-Up: Outcomes of two randomized clinical trials of a computer-delivered intervention. Psychology of Addictive Behaviors. 2012; 26 :1–12. [ PMC free article : PMC3246033 ] [ PubMed : 21823769 ]
• Hester RK, Squires DD, Delaney HD. The Drinker's Check-up: 12-month outcomes of a controlled clinical trial of a stand-alone software program for problem drinkers. Journal of Substance Abuse Treatment. 2005; 28 :159–169. [ PubMed : 15780546 ]
• Hile MG, Adkins RE. Do substance abuse and mental health clients prefer automated assessments? Behavior Research Methods, Instruments & Computers. 1997; 29 :146–150.
• Hilvert-Bruce Z, Rossouw PJ, Wong N, Sunderland M, Andrews G. Adherence as a determinant of effectiveness of Internet cognitive behavioural therapy for anxiety and depressive disorders. Behaviour Research and Therapy. 2012; 50 :463–468. [ PubMed : 22659155 ]
• Holländare F, Johnsson S, Randestad M, Tillfors M, Carlbring P, Andersson G, Engström I. Randomized trial of Internet-based relapse prevention for partially remitted depression. Acta Psychiatrica Scandinavica. 2011; 124 :285–294. [ PubMed : 21401534 ]
• Horgan CM, Merrick EL, Reif S, Stewart M. Datapoints: Internet-based behavioral health services in health plans. Psychiatric Services. 2007; 58 :307. [ PubMed : 17325101 ]
• Houston TK, Cooper LA, Ford DE. Internet support groups for depression: A 1-year prospective cohort study. American Journal of Psychiatry. 2002; 159 :2062–2068. [ PubMed : 12450957 ]
• Houston TK, Cooper LA, Vu HT, Kahn J, Toser J, Ford DE. Screening the public for depression through the Internet. Psychiatric Services. 2001; 52 :362–367. [ PubMed : 11239106 ]
• Hubbard RL, Leimberger JD, Haynes L, Patkar AA, Holter J, Liepman MR., National Institute on Drug Abuse. Telephone enhancement of long-term engagement (TELE) in continuing care for substance abuse treatment: A NIDA clinical trials network (CTN) study. American Journal of Addiction. 2007; 16 :495–502. [ PubMed : 18058417 ]
• Huppert JD, Kivity Y, Barlow DH, Gorman JM, Shear MK, Woods SW. Therapist effects and outcomes-alliance correlation in cognitive behavior therapy for panic disorder with agoraphobia. Behaviour Research and Therapy. 2014; 52 :26–34. [ PMC free article : PMC4098660 ] [ PubMed : 24275067 ]
• Hustad JTP, Barnett NP, Borsari B, Jackson KM. Web-based alcohol prevention for incoming college students: A randomized controlled trial. Addictive Behaviors. 2010; 35 :183–189. [ PMC free article : PMC2815230 ] [ PubMed : 19900763 ]
• Hyler SE, Gangure DP, Batchelder ST. Can telepsychiatry replace in-person psychiatric assessments? A review and meta-analysis of comparison studies. CNS Spectrums. 2005; 10 :403–413. [ PubMed : 15858458 ]
• International Telecommunication Union. Key global telecom indicators for the world telecommunication service sector. Geneva: International Telecommunication Union; 2012.
• Jencius M, Sager DE. The practice of marriage and family counseling in cyberspace. Family Journal. 2001; 9 :295–301.
• Johnston O, Startup H, Lavender A, Godfrey E, Schmidt U. Therapeutic writing as an intervention for symptoms of bulimia nervosa: Effects and mechanism of change. International Journal of Eating Disorders. 2010; 43 :405–419. [ PubMed : 19544556 ]
• Joinson AN, Paine CB. Self-disclosure, privacy and the Internet. In: Joinson A, McKenna K, Postmes T, Reips UD, editors. The Oxford handbook of internet psychology. Oxford, United Kingdom: Oxford University Press; 2007. pp. 237–252.
• Jones EG, Goldsmith M, Effken J, Button K, Crago M. Creating and testing a deaf-friendly, stop-smoking Web site intervention. American Annals of the Deaf. 2010; 155 :96–102. [ PubMed : 20503910 ]
• Jones RM, Leonard S, Birmingham L. Setting up a telepsychiatry service. Psychiatric Bulletin. 2006; 30 :464–467.
• Julian K, Beard C, Schmidt NB, Powers MB, Smits JA. Attention training to reduce attention bias and social stressor reactivity: An attempt to replicate and extend previous findings. Behaviour Research and Therapy. 2012; 50 :350–358. [ PMC free article : PMC3327818 ] [ PubMed : 22466022 ]
• Juzang I, Fortune T, Black S, Wright E, Bull S. A pilot programme using mobile phones for HIV prevention. Journal of Telemedicine and Telecare. 2011; 17 :150–153. [ PubMed : 21270049 ]
• Kaltenthaler E, Brazier J, De Nigris E, Tumur I, Ferriter M, Beverley C, Sutcliff P. Computerised cognitive behaviour therapy for depression and anxiety update: A systematic review and economic evaluation. Health Technology Assessment. 2006; 10 :iii, xi–xiv, 1–168. [ PubMed : 16959169 ]
• Kaltenthaler E, Sutcliffe P, Parry G, Beverley C, Rees A, Ferriter M. The acceptability to patients of computerized cognitive behaviour therapy for depression: A systematic review. Psychological Medicine: A Journal of Research in Psychiatry and the Allied Sciences. 2008; 38 :1521–1530. [ PubMed : 18205964 ]
• Kanani K, Regehr C. Clinical, ethical, and legal issues in e-therapy. Families in Society. 2003; 84 :155–162.
• Kang SH, Gratch J. People like virtual counselors that highly-disclose about themselves. Studies in Health Technology and Informatics. 2011; 167 :143–148. [ PubMed : 21685657 ]
• Kaplan K, Salzer MS, Solomon P, Brusilovskiy E, Cousounis P. Internet peer support for individuals with psychiatric disabilities: A randomized controlled trial. Social Science and Medicine. 2011; 72 :54–62. [ PubMed : 21112682 ]
• Karno M, Farabee D, Brecht M-L, Rawson R. Patient reactance moderates the effect of directive telephone counseling for methamphetamine users. Journal of Studies on Alcohol. 2012; 73 :844–850. [ PMC free article : PMC3410952 ] [ PubMed : 22846250 ]
• Kay-Lambkin FJ, Baker AL, Kelly BJ, Lewin TJ. It's worth a try: The treatment experiences of rural and urban participants in a randomized controlled trial of computerized psychological treatment for comorbid depression and alcohol/other drug use. Journal of Dual Diagnosis. 2012; 8 :262–276.
• Kenardy J, McCafferty K, Rosa V. Internet-delivered indicated prevention for anxiety disorders: Six-month follow-up. Clinical Psychologist. 2006; 10 :39–42.
• Kenwright M, Marks I, Graham C, Franses A, Mataix-Cols D. Brief scheduled phone support from a clinician to enhance computer-aided self-help for obsessive–compulsive disorder: Randomized controlled trial. Journal of Clinical Psychology. 2005; 61 :1499–1508. [ PubMed : 16173084 ]
• Kessler D, Lewis G, Kaur S, Wiles N, King M, Weich S, Peters T. Therapist-delivered Internet psychotherapy for depression in primary care: A randomised controlled trial. The Lancet. 2009; 374 :628–634. [ PubMed : 19700005 ]
• Kiluk BD, Nich C, Babuscio T, Carroll KM. Quality versus quantity:Acquisition of coping skills following computerized cognitive behavioral therapy for substance use disorders. Addiction. 2010; 105 :2120–2127. [ PMC free article : PMC2975828 ] [ PubMed : 20854334 ]
• Kiluk BD, Sugarman DE, Nich C, Gibbons CJ, Martino S, Rounsaville BJ, Carroll KM. A methodological analysis of randomized clinical trials of computer-assisted therapies for psychiatric disorders: Toward improved standards for an emerging field. American Journal of Psychiatry. 2011; 168 :790–799. [ PMC free article : PMC3607199 ] [ PubMed : 21536689 ]
• King VL, Stoller KB, Kidorf M, Kindbom K, Hursh S, Brady T, Brooner RK. Assessing the effectiveness of an Internet-based videoconferencing platform for delivering intensified substance abuse counseling. Journal of Substance Abuse Treatment. 2009; 36 :331–338. [ PubMed : 18775625 ]
• Kirkby KC, Berrios GE, Daniels BA, Menzies RG, Clark A, Romano A. Process-outcome analysis in computer-aided treatment of obsessive–compulsive disorder. Comprehensive Psychiatry. 2000; 41 :259–265. [ PubMed : 10929793 ]
• Kiropoulos LA, Klein B, Austin DW, Gilson K, Pier C, Mitchell J, Ciechomski L. Is Internet-based CBT for panic disorder and agoraphobia as effective as face-to-face CBT? Journal of Anxiety Disorders. 2008; 22 :1273–1284. [ PubMed : 18289829 ]
• Klein B, Austin D, Pier C, Kiropoulos L, Shandley K, Mitchell J, Ciechomski L. Internet-based treatment for panic disorder: Does frequency of therapist contact make a difference? Cognitive Behaviour Therapy. 2009; 38 :100–113. [ PubMed : 19306149 ]
• Klein B, Meyer D, Austin DW, Kyrios M. Anxiety online: A virtual clinic:preliminary outcomes following completion of five fully automated treatment programs for anxiety disorders and symptoms. Journal of Medical Internet Research. 2011; 13 :e89. [ PMC free article : PMC3222205 ] [ PubMed : 22057287 ]
• Klein B, Mitchell J, Abbott J, Shandley K, Austin D, Gilson K, Redman T. A therapist-assisted cognitive behavior therapy Internet intervention for posttraumatic stress disorder: Pre-, post- and 3-month follow-up results from an open trial. Journal of Anxiety Disorders. 2010; 24 :635–644. [ PubMed : 20447802 ]
• Klein B, Richards JC, Austin DW. Efficacy of Internet therapy for panic disorder. Journal of Behavior Therapy and Experimental Psychiatry. 2006; 37 :213–238. [ PubMed : 16126161 ]
• Knaevelsrud C, Maercker A. Does the quality of the working alliance predict treatment outcome in online psychotherapy for traumatized patients? Journal of Medical Internet Research. 2006; 8 :e31. [ PMC free article : PMC1794007 ] [ PubMed : 17213049 ]
• Knaevelsrud C, Maercker A. Long-term effects of an Internet-based treatment for posttraumatic stress. Cognitive Behaviour Therapy. 2010; 39 :72–77. [ PubMed : 19675958 ]
• Kobak KA, Greist JH, Jefferson JW, Mundt JC, Katzelnick DJ. Computerized assessment of depression and anxiety over the telephone using interactive voice response. MD Computing. 1999; 16 :64–68. [ PubMed : 10439605 ]
• Kobak KA, Williams JB, Engelhardt N. A comparison of face-to-face and remote assessment of inter-rater reliability on the Hamilton Depression Rating Scale via videoconferencing. Psychiatry Research. 2008; 158 :99–103. [ PubMed : 17961715 ]
• Koski-Jännes A, Cunningham JA, Tolonen K, Bothas H. Internet-based self-assessment of drinking–3-month follow-up data. Addictive Behaviors. 2007; 32 :533–542. [ PubMed : 16822619 ]
• Kramer GM, Shore JH, Mishkind MC, Friedl KE, Poropatich RK, Gahm GA. A standard telemental health evaluation model: The time is now. Telemedicine Journal and e-Health. 2012; 18 :309–313. [ PubMed : 22424077 ]
• Kraus R. Ethical and legal considerations for providers of mental health services online. In: Kraus R, Zack JS, Stricker G, editors. Online counseling: A handbook for mental health professionals. New York: Elsevier Science; 2004. pp. 123–144.
• Kraus R, Zack JS. The business aspects of online counseling. In: Kraus R, Zack JS, Stricker G, editors. Online counseling:A handbook for mental health professionals. New York: Elsevier Science; 2004. pp. 145–160.
• Kurtz MM, Baker E, Pearlson GD, Astur RS. A virtual reality apartment as a measure of medication management skills in patients with schizophrenia: A pilot study. Schizophrenia Bulletin. 2007; 33 :1162–1170. [ PMC free article : PMC2632360 ] [ PubMed : 16956984 ]
• Kypri K, Hallett J, Howat P, McManus A, Maycock B, Bowe S, Horton NJ. Randomized controlled trial of proactive Web-based alcohol screening and brief intervention for university students. Archives of Internal Medicine. 2009; 169 :1508–1514. [ PubMed : 19752409 ]
• Lack CW, Storch EA. The use of computers in the assessment and treatment of obsessive–compulsive disorder. Computers in Human Behavior. 2008; 24 :917–929.
• LaMendola WF. Telemental health services in frontier areas: Provider and consumer perspectives. Letter to the field no. 19. Boulder, CO: Western Interstate Commission for Higher Education; 2000.
• Lane DJ, Lindemann DF, Schmidt JA. A comparison of computer-assisted and self-management programs for reducing alcohol use among students in first year experience courses. Journal of Drug Education. 2012; 42 :119–135. [ PubMed : 23185834 ]
• Lange A, Rietdijk D, Hudcovicova M, van de Ven J-P, Schrieken B, Emmelkamp PMG. Interapy: A controlled randomized trial of the standardized treatment of posttraumatic stress through the Internet. Journal of Consulting and Clinical Psychology. 2003; 71 :901–909. [ PubMed : 14516238 ]
• Lange A, van de Ven J-P, Schrieken B. Interapy: Treatment of post-traumatic stress via the Internet. Cognitive Behaviour Therapy. 2003; 32 :110–124. [ PubMed : 16291543 ]
• Lauder S, Chester A, Castle D, Dodd S, Berk L, Klein B, Berk M. Development of an online intervention for bipolar disorder. Psychology Health & Medicine. 2012; 18 (2):155–165. www ​.moodswings.net.au . [ PubMed : 22712771 ]
• Le HN, Perry DF, Sheng X. Using the Internet to screen for postpartum depression. Maternal and Child Health Journal. 2009; 13 :213–221. [ PubMed : 18278545 ]
• Leach LS, Christensen H. A systematic review of telephone-based interventions for mental disorders. Journal of Telemedicine and Telecare. 2006; 12 :122–129. [ PubMed : 16638233 ]
• Lee JH, Kwon H, Choi J, Yang BH. Cue-exposure therapy to decrease alcohol craving in virtual environment. CyberPsychology & Behavior. 2007; 10 :617–623. [ PubMed : 17927528 ]
• Lessler JT, Caspar RA, Penne MA, Barker PR. Developing computer assisted interviewing (CAI) for the National Household Survey on Drug Abuse. Journal of Drug Issues. 2000; 30 :9–34.
• Levine D, Madsen A, Wright E, Barar RE, Santelli J, Bull S. Formative research on MySpace: Online methods to engage hard-to-reach populations. Journal of Health Communication. 2011; 16 :448–454. [ PubMed : 21391040 ]
• Lewis MA, Neighbors C, Oster-Aaland L, Kirkeby BS, Larimer ME. Indicated prevention for incoming freshmen: Personalized normative feedback and high-risk drinking. Addictive Behaviors. 2007; 32 :2495–2508. [ PMC free article : PMC2969166 ] [ PubMed : 17658695 ]
• Leykin Y, Barrera AZ, Muñoz RF. Smoking cessation via the Internet. In: Cucciare MA, Weingardt KR, editors. Using technology to support evidence-based behavioral health practices: A clinician's guide. New York: Routledge/Taylor & Francis; 2010. pp. 109–132.
• Lichtenstein E, Glasgow RE, Lando HA, Ossip-Klein DJ, Boles SM. Telephone counseling for smoking cessation: Rationales and meta-analytic review of evidence. Health Education Research. 1996; 11 :243–257. [ PubMed : 10163409 ]
• Lieberman DZ, Huang SW. A technological approach to reaching a hidden population of problem drinkers. Psychiatric Services. 2008; 59 :297–303. [ PubMed : 18308911 ]
• Lieberman DZ, Massey SH. Pathways to change: The effect of a Web application on treatment interest. American Journal of Addiction. 2008; 17 :265–270. [ PubMed : 18612880 ]
• Liebowitz J. Teaching people skills totally online? College Teaching. 2003; 51 :82–85.
• Lin JL, Vahabzadeh M, Mezghanni M, Epstein DH, Preston KL. A high-level specification for adaptive ecological momentary assessment: Real-time assessment of drug craving, use and abstinence. AMIA Annual Symposium Proceedings. 2005:455–459. Retrieved November 5, 2015, from http://www ​.ncbi.nlm.nih ​.gov/pmc/articles/PMC1560797 . [ PMC free article : PMC1560797 ] [ PubMed : 16779081 ]
• Linke S, Murray E, Butler C, Wallace AP. Internet-based interactive health intervention for the promotion of sensible drinking: Patterns of use and potential impact on members of the general public. Journal of Medical Internet Research. 2007; 9 :e10. [ PMC free article : PMC1874715 ] [ PubMed : 17513281 ]
• Lipman EL, Kenny M, Marziali E. Providing Web-based mental health services to at-risk women. BMC Womens Health. 2011; 11 :38. [ PMC free article : PMC3179921 ] [ PubMed : 21854563 ]
• Litz BT, Engel CC, Bryant RA, Papa A. A randomized, controlled proof-of-concept trial of an Internet-based, therapist-assisted self-management treatment for posttraumatic stress disorder. American Journal of Psychiatry. 2007; 164 :1676–1683. [ PubMed : 17974932 ]
• Ljotsson B, Lundin C, Mitsell K, Carlbring P, Ramklint M, Ghaderi A. Remote treatment of bulimia nervosa and binge eating disorder: A randomized trial of Internet-assisted cognitive behavioural therapy. Behaviour Research and Therapy. 2007; 45 :649–661. [ PubMed : 16899213 ]
• Lovecchio CP, Wyatt TM, DeJong W. Reductions in drinking and alcohol-related harms reported by first-year college students taking an online alcohol education course: A randomized trial. Journal of Health Communication. 2010; 15 :805–819. [ PubMed : 21104507 ]
• Lovejoy TI, Demireva PD, Grayson JL, McNamara JR. Advancing the practice of online psychotherapy: An application of Rogers' diffusion of innovations theory. Psychotherapy: Theory, Research, Practice, Training. 2009; 46 :112–124. [ PubMed : 22122574 ]
• Lovell K, Fullalove L, Garvey R, Brooker C. Telephone treatment of obsessive–compulsive disorder. Behavioural and Cognitive Psychotherapy. 2000; 28 :87–91.
• Low KG, Charanasomboon S, Lesser J, Reinhalter K, Martin R, Jones H, Taylor CB. Effectiveness of a computer-based interactive eating disorders prevention program at long-term follow-up. Eating Disorders: The Journal of Treatment & Prevention. 2006; 14 :17–30. [ PubMed : 16757446 ]
• Lucey R. SAMHSA's Current and Future Direction for Prevention in Higher Education. Rockville, MD: Substance Abuse and Mental Health Services Administration; 2015.
• Ludman EJ, Simon GE, Tutty S, Von Korff M. A randomized trial of telephone psychotherapy and pharmacotherapy for depression: Continuation and durability of effects. Journal of Consulting and Clinical Psychology. 2007; 75 :257–266. [ PubMed : 17469883 ]
• Lustria ML, Cortese J, Noar SM, Glueckauf RL. Computer-tailored health interventions delivered over the Web: Review and analysis of key components. Patient Education and Counseling. 2009; 74 :156–173. [ PubMed : 18947966 ]
• Luxton DD, Kinn JT, June JD, Pierre LW, Reger MA, Gahm GA. Caring letters project: A military suicide-prevention pilot program. Crisis. 2012; 33 :5–12. [ PubMed : 21940244 ]
• Luxton DD, O'Brien K, McCann RA, Mishkind MC. Home-based telemental healthcare safety planning: What you need to know. Telemedicine and e-Health. 2012; 18 :629–633. [ PubMed : 23061644 ]
• Mackinnon A, Griffiths KM, Christensen H. Comparative randomised trial of online cognitive-behavioural therapy and an information Website for depression: 12-month outcomes. British Journal of Psychiatry. 2008; 192 :130–134. [ PubMed : 18245031 ]
• MacLeod C, Mathews A. Cognitive bias modification approaches to anxiety. Annual Review of Clinical Psychology. 2012; 8 :189–217. [ PubMed : 22035241 ]
• MacLeod C, Soong LY, Rutherford EM, Campbell LW. Internet-delivered assessment and manipulation of anxiety-linked attentional bias: Validation of a free-access attentional probe software package. Behavior Research Methods. 2007; 39 :533–538. [ PubMed : 17958165 ]
• Magaletta PR, Fagan TJ, Peyrot MF. Telehealth in the Federal Bureau of Prisons: Inmates' perceptions. Professional Psychology: Research and Practice. 2000; 31 :497–502.
• Maheu MM, Pulier ML, Wilhelm FH, McMenamin JP, Brown-Connolly NE. The mental health professional and the new technologies: A handbook for practice today. Mahwah, NJ: Lawrence Erlbaum Associates; 2004.
• Mallen MJ, Blalock JA, Cinciripini PM. Using technology to serve patients and practitioners: A comprehensive tobacco-cessation program for cancer patients. Counselling and Psychotherapy Research: Linking Research with Practice. 2006; 6 :196–201.
• Mallen MJ, Vogel DL, Rochlen AB. The practical aspects of online counseling: Ethics, training, technology, and competency. Counseling Psychologist. 2005; 33 :776–818.
• Marks IM, Baer L, Greist JH, Park JM, Bachofen M, Nakagawa A, Mantle JM. Home self-assessment of obsessive–compulsive disorder. Use of a manual and a computer-conducted telephone interview: Two UK–US studies. British Journal of Psychiatry. 1998; 172 :406–412. [ PubMed : 9747402 ]
• Marks IM, Kenwright M, McDonough M, Whittaker M, Mataix-Cols D. Saving clinicians' time by delegating routine aspects of therapy to a computer: A randomized controlled trial in phobia/panic disorder. Psychological Medicine. 2004; 34 :9–17. [ PubMed : 14971623 ]
• Martin-Khan M, Wootton R, Gray L. A systematic review of the reliability of screening for cognitive impairment in older adults by use of standardised assessment tools administered via the telephone. Journal of Telemedicine and Telecare. 2010; 16 :422–428. [ PubMed : 21030488 ]
• Mathews A, Ridgeway V, Cook E, Yiend J. Inducing a benign interpretational bias reduces trait anxiety. Journal of Behavior Therapy and Experimental Psychiatry. 2007; 38 :225–236. [ PubMed : 17141179 ]
• Maust DT, Mavandadi S, Benson A, Streim JE, Difilippo S, Snedden T, Oslin DW. Telephone-based care management for older adults initiated on psychotropic medication. International Journal of Geriatric Psychiatry. 2012; 28 (4):410–416. [ PMC free article : PMC3514587 ] [ PubMed : 22678956 ]
• McAdams CR III, Wyatt KL. The regulation of technology-assisted distance counseling and supervision in the United States: An analysis of current extent, trends, and implications. Counselor Education and Supervision. 2010; 49 :179–192.
• McDonnell DD, Kazinets G, Lee HJ, Moskowitz JM. An Internet-based smoking cessation program for Korean Americans: Results from a randomized controlled trial. Nicotine & Tobacco Research. 2011; 13 :336–343. [ PubMed : 21330285 ]
• McEvoy PM, Perini SJ. Cognitive behavioral group therapy for social phobia with or without attention training: A controlled trial. Journal of Anxiety Disorders. 2009; 23 :519–528. [ PubMed : 19059753 ]
• McGinty KL, Saeed SA, Simmons SC, Yildirim Y. Telepsychiatry and e-mental health services: Potential for improving access to mental health care. Psychiatric Quarterly. 2006; 77 :335–342. [ PubMed : 16927161 ]
• McKay GH, Danaher BG, Seeley RJ, Lichtenstein E, Gau MJ. Comparing two Web-based smoking cessation programs: Randomized controlled trial. Journal of Medical Internet Research. 2008; 10 :e40. [ PMC free article : PMC2630830 ] [ PubMed : 19017582 ]
• McKay JR, Lynch KG, Shepard DS, Morgenstern J, Forman RF, Pettinati HM. Do patient characteristics and initial progress in treatment moderate the effectiveness of telephone-based continuing care for substance use disorders? Addiction. 2005; 100 :216–226. [ PubMed : 15679751 ]
• McKay JR, Lynch KG, Shepard DS, Pettinati HM. The effectiveness of telephone-based continuing care for alcohol and cocaine dependence: 24-month outcomes. Archives of General Psychiatry. 2005; 62 :199–207. [ PubMed : 15699297 ]
• McKay JR, Van Horn HD, Oslin DW, Ivey M, Drapkin ML, Coviello DM, Lynch KG. Extended telephone-based continuing care for alcohol dependence: 24-month outcomes and subgroup analyses. Addiction. 2011; 106 :1760–1769. [ PMC free article : PMC3174323 ] [ PubMed : 21545667 ]
• McLean CP, Steenkamp MM, Levy HC, Litz BT. Posttraumatic stress disorder. In: Cucciare MA, Weingardt KR, editors. Using technology to support evidence-based behavioral health practices: A clinician's guide. New York: Routledge/Taylor & Francis Group; 2010. pp. 45–68.
• McNeill LH, Puleo E, Bennett GG, Emmons KM. Exploring social contextual correlates of computer ownership and frequency of use among urban, low-income, public housing adult residents. Journal of Medical Internet Research. 2007; 9 :e35. [ PMC free article : PMC2223187 ] [ PubMed : 18093903 ]
• McPherson TL, Cook RF, Back AS, Hersch RK, Hendrickson A. A field test of a Web-based substance abuse prevention training program for health promotion professionals. American Journal of Health Promotion. 2006; 20 :396–400. [ PubMed : 16871819 ]
• McTavish FM, Chih M-Y, Shah D, Gustafson DH. How patients recovering from alcoholism use a smartphone intervention. Journal of Dual Diagnosis. 2012; 8 :294–304. [ PMC free article : PMC3541672 ] [ PubMed : 23316127 ]
• Melling B, Houguet-Pincham T. Online peer support for individuals with depression: A summary of current research and future considerations. Psychiatric Rehabilitation Journal. 2011; 34 :252–254. [ PubMed : 21208865 ]
• Mensinger JL, Lynch KG, TenHave TR, McKay JR. Mediators of telephone-based continuing care for alcohol and cocaine dependence. Journal of Consulting and Clinical Psychology. 2007; 75 :775–784. [ PMC free article : PMC2677065 ] [ PubMed : 17907859 ]
• Meyer B, Berger T, Caspar F, Beevers GC, Andersson G, Weiss M. Effectiveness of a novel integrative online treatment for depression (Deprexis): Randomized controlled trial. Journal of Medical Internet Research. 2009; 11 :e15. [ PMC free article : PMC2762808 ] [ PubMed : 19632969 ]
• Meyerbröker K, Emmelkamp PM. Virtual reality exposure therapy in anxiety disorders: A systematic review of process-and-outcome studies. Depression and Anxiety. 2010; 27 :933–944. [ PubMed : 20734361 ]
• Midkiff DM, Wyatt WJ. Ethical issues in the provision of online mental health services (etherapy). Journal of Technology in Human Services. 2008; 26 :310–332.
• Miller TW, Kraus RF, Kaak O, Sprang R, Burton D. Telemedicine: A child psychiatry case report. Telemedicine Journal and e-Health. 2002; 8 :139–141. [ PubMed : 12020414 ]
• Mitchell J, Stanimirovic R, Klein B, Vella-Brodrick D. A randomised controlled trial of a self-guided Internet intervention promoting well-being. Computers in Human Behavior. 2009; 25 :749–760.
• Mitchell JE, Agras S, Crow S, Halmi K, Fairburn CG, Bryson S, Kraemer H. Stepped care and cognitive-behavioural therapy for bulimia nervosa: Randomised trial. British Journal of Psychiatry. 2011; 198 :391–397. [ PMC free article : PMC3093678 ] [ PubMed : 21415046 ]
• Mitchell JE, Crosby RD, Wonderlich SA, Crow S, Lancaster K, Simonich H, Myers TC. A randomized trial comparing the efficacy of cognitive–behavioral therapy for bulimia nervosa delivered via telemedicine versus face-to-face. Behaviour Research and Therapy. 2008; 46 :581–592. [ PMC free article : PMC2633728 ] [ PubMed : 18374304 ]
• Mohr DC, Carmody T, Erickson L, Jin L, Leader J. Telephone-administered cognitive behavioral therapy for veterans served by community-based outpatient clinics. Journal of Consulting and Clinical Psychology. 2011; 79 :261–265. [ PubMed : 21299274 ]
• Mohr DC, Ho J, Duffecy J, Reifler D, Sokol L, Burns MN, Siddique J. Effect of telephone-administered vs. face-to-face cognitive behavioral therapy on adherence to therapy and depression outcomes among primary care patients: A randomized trial. JAMA. 2012; 307 :2278–2285. [ PMC free article : PMC3697075 ] [ PubMed : 22706833 ]
• Mohr DC, Siddique J, Ho J, Duffecy J, Jin L, Fokuo JK. Interest in behavioral and psychological treatments delivered face-to-face, by telephone, and by Internet. Annals of Behavioral Medicine. 2010; 40 :89–98. [ PMC free article : PMC2914835 ] [ PubMed : 20652466 ]
• Mohr DC, Vella L, Hart S, Heckman T, Simon G. The effect of telephone-administered psychotherapy on symptoms of depression and attrition: A meta-analysis. Clinical Psychology: Science and Practice. 2008; 15 :243–253. [ PMC free article : PMC3045729 ] [ PubMed : 21369344 ]
• Moon J, Lee JH. Cue exposure treatment in a virtual environment to reduce nicotine craving: A functional MRI study. CyberPsychology & Behavior. 2009; 12 :43–45. [ PubMed : 19025463 ]
• Moore BA, Fazzino T, Garnet B, Cutter CJ, Barry DT. Computer-based interventions for drug use disorders: A systematic review. Journal of Substance Abuse Treatment. 2011; 40 :215–223. [ PMC free article : PMC3073660 ] [ PubMed : 21185683 ]
• Moore D, Guthmann D, Rogers N, Fraker S, Embree J. E-therapy as a means for addressing barriers to substance use disorder treatment for persons who are deaf. Journal of Sociology & Social Welfare. 2009; 36 :75–92.
• Moreno FA, Chong J, Dumbauld J, Humke M, Byreddy S. Use of standard Webcam and Internet equipment for telepsychiatry treatment of depression among underserved Hispanics. Psychiatric Services. 2012; 63 (12):1213–1217. [ PubMed : 23026854 ]
• Morgan AJ, Jorm AF. Outcomes of self-help efforts in anxiety disorders. Expert Review of Pharmacoeconomics and Outcomes Research. 2009; 9 :445–459. [ PubMed : 19817528 ]
• Morgan AJ, Jorm AF, Mackinnon AJ. Email-based promotion of self-help for subthreshold depression: Mood Memos randomised controlled trial. British Journal of Psychiatry. 2012; 200 :412–418. [ PubMed : 22442102 ]
• Morgan S, Polowy CI. Social workers and Skype - Part I. Washington, DC: National Association of Social Workers; 2011.
• Morgan S, Polowy CI. Social workers and Skype - Part II. Washington, DC: National Association of Social Workers; 2012.
• Morie JF. Re-entry: Online virtual worlds as a healing space for veterans. Marina del Rey, CA: University of Southern California Institute for Creative Technologies; 2009.
• Morie JF, Chance E. Extending the reach of health care for obesity and diabetes using virtual worlds. Journal of Diabetes Science and Technology. 2011; 5 :272–276. [ PMC free article : PMC3125916 ] [ PubMed : 21527093 ]
• Moritz S, Jelinek L, Hauschildt M, Naber D. How to treat the untreated: effectiveness of a self-help metacognitive training program (myMCT) for obsessive–compulsive disorder. Dialogues in Clinical Neuroscience. 2010; 12 :209–220. [ PMC free article : PMC3181961 ] [ PubMed : 20623925 ]
• Moritz S, Rufer M. Movement decoupling: A self-help intervention for the treatment of trichotillomania. Journal of Behavior Therapy and Experimental Psychiatry. 2011; 42 :74–80. [ PubMed : 20674888 ]
• Moritz S, Schilling L, Hauschildt M, Schröder J, Treszl A. A randomized controlled trial of Internet-based therapy in depression. Behaviour Research and Therapy. 2012; 50 :513–521. [ PubMed : 22677231 ]
• Moritz S, Wittekind CE, Hauschildt M, Timpano KR. Do it yourself? Self-help and online therapy for people with obsessive–compulsive disorder. Current Opinion in Psychiatry. 2011; 24 :541–543. [ PubMed : 21897252 ]
• Morland LA, Greene CJ, Grubbs K, Kloezeman K, Mackintosh MA, Rosen C, Frueh BC. Therapist adherence to manualized cognitive-behavioral therapy for anger management delivered to veterans with PTSD via videoconferencing. Journal of Clinical Psychology. 2011; 67 :629–638. [ PubMed : 21360528 ]
• Mucic D. Transcultural telepsychiatry and its impact on patient satisfaction. Journal of Telemedicine and Telecare. 2010; 16 :237–242. [ PubMed : 20423935 ]
• Muller I, Yardley L. Telephone-delivered cognitive behavioural therapy: A systematic review and meta-analysis. Journal of Telemedicine and Telecare. 2011; 17 :177–184. [ PubMed : 21357672 ]
• Muñoz RF, Lenert LL, Delucchi K, Stoddard J, Perez JE, Penilla C, Pérez-Stable EJ. Toward evidence-based Internet interventions: A Spanish/English Web site for international smoking cessation trials. Nicotine & Tobacco Research. 2006; 8 :77–87. [ PubMed : 16497602 ]
• Murdock J, Williams A, Becker K, Bruce MA, Young S. Online versus on-campus: A comparison study of counseling skills courses. Journal of Human Resource and Adult Learning. 2012; 8 :105–118.
• Muroff J, Steketee G, Himle J, Frost R. Delivery of Internet treatment for compulsive hoarding (D.I.T.C.H.). Behaviour Research and Therapy. 2010; 48 :79–85. [ PubMed : 19800051 ]
• Myung SK, McDonnell DD, Kazinets G, Seo HG, Moskowitz JM. Effects of Web- and computer-based smoking cessation programs: Meta-analysis of randomized controlled trials. Archives of Internal Medicine. 2009; 169 :929–937. [ PubMed : 19468084 ]
• Najmi S, Amir N. The effect of attention training on a behavioral test of contamination fears in individuals with subclinical obsessive–compulsive symptoms. Journal of Abnormal Psychology. 2010; 119 :136–142. [ PMC free article : PMC2866445 ] [ PubMed : 20141250 ]
• National Telecommunications and Information Administration. The Broadband Technology Opportunities Program: Expanding broadband access and adoption in communities across America: Overview of grant awards. Washington, DC: Author; 2010.
• Neighbors C, Lee CM, Lewis MA, Fossos N, Walter T. Internet-based personalized feedback to reduce 21st-birthday drinking: A randomized controlled trial of an event-specific prevention intervention. Journal of Consulting and Clinical Psychology. 2009; 77 :51–63. [ PMC free article : PMC2682322 ] [ PubMed : 19170453 ]
• Newman MG, Consoli AJ, Taylor CB. A palmtop computer program for the treatment of generalized anxiety disorder. Behavior Modification. 1999; 23 :597–619. [ PubMed : 10533442 ]
• Newman MG, Kenardy J, Herman S, Taylor CB. Comparison of palmtop-computer-assisted brief cognitive–behavioral treatment to cognitive–behavioral treatment for panic disorder. Journal of Consulting and Clinical Psychology. 1997; 65 :178–183. [ PubMed : 9103747 ]
• Newman MG, Koif DF, Przeworski A, Llera SJ. Anxiety disorders. In: Cucciare MA, Weingardt KR, editors. Using technology to support evidence-based behavioral health practices: A clinician's guide. New York: Routledge/Taylor & Francis Group; 2010. pp. 27–44.
• Newman MG, Szkodny LE, Llera SJ, Przeworski A. A review of technology-assisted self-help and minimal contact therapies for anxiety and depression: Is human contact necessary for therapeutic efficacy? Clinical Psychology Review. 2011; 31 :89–103. [ PubMed : 21130939 ]
• Newman MG, Szkodny LE, Llera SJ, Przeworski A. A review of technology-assisted self-help and minimal contact therapies for drug and alcohol abuse and smoking addiction: Is human contact necessary for therapeutic efficacy? Clinical Psychology Review. 2011; 31 :178–186. [ PubMed : 21095051 ]
• Newton MS, Ciliska D. Internet-based innovations for the prevention of eating disorders: A systematic review. Eating Disorders: The Journal of Treatment & Prevention. 2006; 14 :365–384. [ PubMed : 17062448 ]
• Nordgreen T, Havik OE, Ost LG, Furmark T, Carlbring P, Andersson G. Outcome predictors in guided and unguided self-help for social anxiety disorder. Behaviour Research and Therapy. 2012; 50 :13–21. [ PubMed : 22134140 ]
• Norman S. The use of telemedicine in psychiatry. Journal of Psychiatric and Mental Health Nursing. 2006; 13 :771–777. [ PubMed : 17087682 ]
• Norton M, Wonderlich SA, Myers T, Mitchell JE, Crosby RD. The use of palmtop computers in the treatment of bulimia nervosa. European Eating Disorders Review. 2003; 11 :231–242.
• Nundy S, Dick JJ, Chou C-H, Nocon RS, Chin MH, Peek ME. Mobile phone diabetes project led to improved glycemic control and net savings for Chicago plan participants. Health Affairs. 2014; 33 :265–272. [ PMC free article : PMC4034376 ] [ PubMed : 24493770 ]
• O'Dea B, Campbell A. Healthy connections: Online social networks and their potential for peer support. Studies in Health Technology and Informatics. 2011; 168 :133–140. [ PubMed : 21893921 ]
• Olmstead TA, Ostrow CD, Carroll KM. Cost-effectiveness of computer-assisted training in cognitive–behavioral therapy as an adjunct to standard care for addiction. Drug and Alcohol Dependence. 2010; 110 (3):200–207. [ PMC free article : PMC3033701 ] [ PubMed : 20392575 ]
• Ondersma SJ, Svikis DS, Schuster CR. Computer-based brief intervention: A randomized trial with postpartum women. American Journal of Preventive Medicine. 2007; 32 :231–238. [ PMC free article : PMC1858656 ] [ PubMed : 17236741 ]
• Optale G, Pastore M, Marin S, Bordin D, Nasta A, Pianon C. Male sexual dysfunctions: Immersive virtual reality and multimedia therapy. In: Riva G, Botella C, Legeron P, Optale G, editors. Cybertherapy: Internet and virtual reality as assessment and rehabilitation tools for clinical psychology and neuroscience. Amsterdam: Ios Press; 2004. pp. 165–180.
• O'Reilly R, Bishop J, Maddox K, Hutchinson L, Fisman M, Takhar J. Is telepsychiatry equivalent to face-to-face psychiatry? Results from a randomized controlled equivalence trial. Psychiatric Services. 2007; 58 :836–843. [ PubMed : 17535945 ]
• Palmqvist B, Carlbring P, Andersson G. Internet-delivered treatments with or without therapist input: Does the therapist factor have implications for efficacy and cost? Expert Review of Pharmacoeconomics and Outcomes Research. 2007; 7 :291–297. [ PubMed : 20528315 ]
• Paschall MJ, Antin T, Ringwalt CL, Saltz RF. Evaluation of an Internet-based alcohol misuse prevention course for college freshmen: Findings of a randomized multi-campus trial. American Journal of Preventive Medicine. 2011; 41 :300–308. [ PMC free article : PMC3173258 ] [ PubMed : 21855745 ]
• Paschall MJ, Bersamin M, Fearnow-Kenney M, Wyrick D. Effectiveness of a Web-based alcohol-misuse and harm-prevention course among high- and low-risk students. Journal of American College Health. 2007; 55 :247–254. [ PubMed : 17319331 ]
• Patrick K, Raab F, Adams AM, Dillon L, Zabinski M, Rock LC, Norman GJ. A text message-based intervention for weight loss: Randomized controlled trial. Journal of Medical Internet Research. 2009; 11 :e1. [ PMC free article : PMC2729073 ] [ PubMed : 19141433 ]
• Patten SB. Prevention of depressive symptoms through the use of distance technologies. Psychiatric Services. 2003; 54 :396–398. [ PubMed : 12610251 ]
• Paxton SJ, Franko DL. Body image and eating disorders. In: Cucciare MA, Weingardt KR, editors. Using technology to support evidence-based behavioral health practices: A clinician's guide. New York: Routledge/Taylor & Francis Group; 2010. pp. 151–168.
• Paxton SJ, McLean SA, Gollings EK, Faulkner C, Wertheim EH. Comparison of face-to-face and Internet interventions for body image and eating problems in adult women: An RCT. International Journal of Eating Disorders. 2007; 40 :692–704. [ PubMed : 17702020 ]
• Perini S, Titov N, Andrews G. Clinician-assisted Internet-based treatment is effective for depression: Randomized controlled trial. Australian and New Zealand Journal of Psychiatry. 2009; 43 :571–578. [ PubMed : 19440890 ]
• Perpiñá C, Botella C, Baños RM. Virtual reality in eating disorders. European Eating Disorders Review. 2003; 11 :261–278.
• Pew Research Center. The Web at 25 in the U.S. Washington, DC: Pew Research Center; 2014.
• Piette JD, Richardson C, Himle J, Duffy S, Torres T, Vogel M, Valenstein M. A randomized trial of telephonic counseling plus walking for depressed diabetes patients. Medical Care. 2011; 49 :641–648. [ PMC free article : PMC3117978 ] [ PubMed : 21478777 ]
• Pitkänen A, Välimäki M, Kuosmanen L, Katajisto J, Koivunen M, Hätönen H, Knapp M. Patient education methods to support quality of life and functional ability among patients with schizophrenia: A randomised clinical trial. Quality of Life Research. 2012; 21 :247–256. [ PubMed : 21655934 ]
• Pollard RQ, Dean RK, O'Hearn A, Haynes SL. Adapting health education material for deaf audiences. Rehabilitation Psychology. 2009; 54 :232–238. [ PubMed : 19469615 ]
• Polosa R, Russo C, Di Maria A, Arcidiacono G, Morjaria JB, Piccillo GA. Feasibility of using e-mail counseling as part of a smoking-cessation program. Respiratory Care. 2009; 54 :1033–1039. [ PubMed : 19650944 ]
• Portnoy DB, Scott-Sheldon LAJ, Johnson BT, Carey MP. Computer-delivered interventions for health promotion and behavioral risk reduction: A meta-analysis of 75 randomized controlled trials, 1988–2007. Preventive Medicine: An International Journal Devoted to Practice and Theory. 2008; 47 :3–16. [ PMC free article : PMC2572996 ] [ PubMed : 18403003 ]
• Possemato K, Ouimette P, Knowlton P. A brief self-guided telehealth intervention for post-traumatic stress disorder in combat veterans: A pilot study. Journal of Telemedicine and Telecare. 2011; 17 :245–250. [ PubMed : 21636687 ]
• Postel MG, De Haan HA, De Jong CAJ. Evaluation of an e-therapy program for problem drinkers: A pilot study. Substance Use & Misuse. 2010; 45 :2059–2075. [ PubMed : 20438314 ]
• Postel MG, De Haan HA, Ter Huurne ED, Becker ES, de Jong CA. Characteristics of problem drinkers in e-therapy versus face-to-face treatment. American Journal of Drug and Alcohol Abuse. 2011; 37 :537–542. [ PubMed : 21797813 ]
• Price J, Sapci H. Law & psychiatry: Telecourt: the use of videoconferencing for involuntary commitment hearings in academic health centers. Psychiatric Services. 2007; 58 :17–18. [ PubMed : 17215408 ]
• Proudfoot J, Parker G, Hyett M, Manicavasagar V, Smith M, Grdovic S, Greenfield L. Next generation of self-management education: Web-based bipolar disorder program. Australian and New Zealand Journal of Psychiatry. 2007; 41 :903–909. [ PubMed : 17924243 ]
• Purves DG, Bennett M, Wellman N. An open trial in the NHS of Blues Begone: A new home based computerized CBT program. Behavioural and Cognitive Psychotherapy. 2009; 37 :541–551. [ PubMed : 19703330 ]
• Puskar K, Schlenk EA, Callan J, Bickmore T, Sereika S. Relational agents as an adjunct in schizophrenia treatment. Journal of Psychosocial Nursing and Mental Health Services. 2011; 49 :22–29. [ PubMed : 21766736 ]
• Rabinowitz T, Murphy KM, Amour JL, Ricci MA, Caputo MP, Newhouse PA. Benefits of a telepsychiatry consultation service for rural nursing home residents. Telemedicine Journal and e-Health. 2010; 16 :34–40. [ PubMed : 20070161 ]
• Rabius V, McAlister AL, Geiger A, Huang P, Todd R. Telephone counseling increases cessation rates among young adult smokers. Health Psychology. 2004; 23 :539–541. [ PubMed : 15367074 ]
• Rabius V, Pike JK, Wiatrek D, McAlister LA. Comparing Internet assistance for smoking cessation: 13-month follow-up of a six-arm randomized controlled trial. Journal of Medical Internet Research. 2008; 10 :e45. [ PMC free article : PMC2630842 ] [ PubMed : 19033146 ]
• Ragusea AS. The more things change, the more they stay the same: Ethical issues in the provision of telehealth. In: Knapp SJ, Gottlieb MC, Handelsman MM, VandeCreek LD, editors. APA handbook of ethics in psychology, Vol 2: Practice, teaching, and research. Washington, DC: American Psychological Association; 2012. pp. 183–198.
• Ramos-Ríos R, Mateos R, Lojo D, Conn DK, Patterson T. Telepsychogeriatrics: A new horizon in the care of mental health problems in the elderly. International Psychogeriatrics. 2012; 24 :1708–1724. [ PubMed : 22687259 ]
• Recupero PR. Legal concerns for psychiatrists who maintain Web sites. Psychiatric Services. 2006; 57 :450–452. [ PubMed : 16603737 ]
• Recupero PR, Harms S. Using email to conduct a therapeutic relationship. In: Anthony K, Nagel DM, Goss S, editors. The use of technology in mental health: Applications, ethics and practice. Springfield, IL: Charles C. Thomas Publisher; 2010. pp. 3–14.
• Reese RJ, Conoley CW, Brossart DF. Effectiveness of telephone counseling: A field-based investigation. Journal of Counseling Psychology. 2002; 49 :233–242.
• Reese RJ, Conoley CW, Brossart DF. The attractiveness of telephone counseling: An empirical investigation of client perceptions. Journal of Counseling & Development. 2006; 84 :54–60.
• Reese HE, McNally RJ, Najmi S, Amir N. Attention training for reducing spider fear in spider-fearful individuals. Journal of Anxiety Disorders. 2010; 24 :657–662. [ PMC free article : PMC4038305 ] [ PubMed : 20510577 ]
• Regan S, Reyen M, Lockhart AC, Richards AE, Rigotti NA. An interactive voice response system to continue a hospital-based smoking cessation intervention after discharge. Nicotine and Tobacco Research. 2011; 13 :255–260. [ PMC free article : PMC3107613 ] [ PubMed : 21330278 ]
• Reger GM, Holloway KM, Candy C, Rothbaum BO, Difede J, Rizzo AA, Gahm GA. Effectiveness of virtual reality exposure therapy for active duty soldiers in a military mental health clinic. Journal of Traumatic Stress. 2011; 24 :93–96. [ PubMed : 21294166 ]
• Reger MA, Gahm GA. A meta-analysis of the effects of Internet- and computer-based cognitive–behavioral treatments for anxiety. Journal of Clinical Psychology. 2009; 65 :53–75. [ PubMed : 19051274 ]
• Reynolds DJ Jr., Stiles WB, Grohol JM. An investigation of session impact and alliance in Internet based psychotherapy: Preliminary results. Counselling & Psychotherapy Research. 2006; 6 :164–168.
• Rice E, Lee A, Taitt S. Cell phone use among homeless youth: Potential for new health interventions and research. Journal of Urban Health. 2011; 88 :1175–1182. [ PMC free article : PMC3232411 ] [ PubMed : 22076445 ]
• Richardson LK, Frueh BC, Grubaugh AL, Egede L, Elhai JD. Current directions in videoconferencing tele-mental health research. Clinical Psychology: Science and Practice. 2009; 16 :323–338. [ PMC free article : PMC2758653 ] [ PubMed : 20161010 ]
• Riley W, Obermayer J, Jean-Mary J. Internet and mobile phone text messaging intervention for college smokers. Journal of American College Health. 2008; 57 :245–248. [ PubMed : 18809542 ]
• Rimoldi G, Lewis P, Jampala C. Help at the hip: Increasing resiliency in high risk suicidal veterans through interactive text messaging. New York: Veterans Health Administration; 2012.
• Riper H, Kramer J, Keuken M, Smit F, Schippers G, Cuijpers P. Predicting successful treatment outcome of Web-based self-help for problem drinkers: Secondary analysis from a randomized controlled trial. Journal of Medical Internet Research. 2008; 10 :e46. [ PMC free article : PMC2629366 ] [ PubMed : 19033150 ]
• Ritter P, Lorig K, Laurent D, Matthews K. Internet versus mailed questionnaires: A randomized comparison. Journal of Medical Internet Research. 2004; 6 :e29. [ PMC free article : PMC1550608 ] [ PubMed : 15471755 ]
• Ritterband LM, Thorndike FP, Cox DJ, Kovatchev BP, Gonder-Frederick LA. A behavior change model for Internet interventions. Annals of Behavioral Medicine. 2009; 38 :18–27. [ PMC free article : PMC2878721 ] [ PubMed : 19802647 ]
• Ritterband LM, Thorndike FP, Gonder-Frederick LA, Magee JC, Bailey ET, Saylor DK, Morin CM. Efficacy of an Internet-based behavioral intervention for adults with insomnia. Archives of General Psychiatry. 2009; 66 :692–698. [ PMC free article : PMC3723339 ] [ PubMed : 19581560 ]
• Riva G, Bacchetta M, Cesa G, Conti S, Molinari E. The use of VR in the treatment of eating disorders. In: Riva G, Botella C, Legeron P, Optale G, editors. Cybertherapy:Internet and virtual reality as assessment and rehabilitation tools for clinical psychology and neuroscience. Amsterdam: Ios Press; 2004. pp. 121–164.
• Riva G, Gaggioli A, Villani D, Preziosa A, Morganti F, Corsi R, Vezzadini L. NeuroVR: An open source virtual reality platform for clinical psychology and behavioral neurosciences. Studies in Health Technology and Informatics. 2007; 125 :394–399. [ PubMed : 17377310 ]
• Rizvi SL, Dimeff LA, Skutch J, Carroll D, Linehan MM. A pilot study of the DBT coach: An interactive mobile phone application for individuals with borderline personality disorder and substance use disorder. Behavior Therapy. 2011; 42 :589–600. [ PubMed : 22035988 ]
• Rizzo AS, Difede J, Rothbaum BO, Reger G, Spitalnick J, Cukor J, McLay L. Development and early evaluation of the virtual Iraq/Afghanistan exposure therapy system for combat-related PTSD. Annals of the New York Academy of Sciences. 2010; 1208 :114–125. [ PubMed : 20955333 ]
• Rizzo A, Rothbaum BO, Graap K. Virtual reality applications for the treatment of combat-related PTSD. In: Figley CR, Nash WP, editors. Combat stress injury: Theory, research, and management. New York: Routledge/Taylor & Francis Group; 2007. pp. 183–204.
• Robinson P, Serfaty M. Getting better byte by byte: A pilot randomised controlled trial of email therapy for bulimia nervosa and binge eating disorder. European Eating Disorders Review. 2008; 16 :84–93. [ PubMed : 17879223 ]
• Robinson S, Perkins S, Bauer S, Hammond N, Treasure J, Schmidt U. Aftercare intervention through text messaging in the treatment of bulimia nervosa–Feasibility pilot. International Journal of Eating Disorders. 2006; 39 :633–638. [ PubMed : 16937381 ]
• Rodda S, Lubman D. Ready to Change: A scheduled telephone counselling programme for problem gambling. Australasian Psychiatry. 2012; 20 (4):338–342. [ PubMed : 22767931 ]
• Rohde P, Lewinsohn PM, Seeley JR. Comparability of telephone and face-to-face interviews in assessing axis I and II disorders. American Journal of Psychiatry. 1997; 154 :1593–1598. [ PubMed : 9356570 ]
• Rooke S, Thorsteinsson E, Karpin A, Copeland J, Allsop D. Computer-delivered interventions for alcohol and tobacco use: A meta-analysis. Addiction. 2010; 105 :1381–1390. [ PubMed : 20528806 ]
• Rosenberg JI. Real-time training: Transfer of knowledge through computer-mediated, real-time feedback. Professional Psychology: Research and Practice. 2006; 37 :539–546.
• Roth JD. Group psychotherapy: 12 steps into cyberspace. Addiction Professional. 2005; 3 :46–47.
• Rothbaum BO, Rizzo AS, Difede J. Virtual reality exposure therapy for combat-related posttraumatic stress disorder. Annals of the New York Academy of Sciences. 2010; 1208 :126–132. [ PubMed : 20955334 ]
• Rotondi AJ. Schizophrenia. In: Cucciare MA, Weingardt KR, editors. Using technology to support evidence-based behavioral health practices: A clinician's guide. New York: Routledge/Taylor & Francis Group; 2010. pp. 69–89.
• Rotondi AJ, Anderson CM, Haas GL, Eack SM, Spring MB, Ganguli R, Rosenstock J. Web-based psychoeducational intervention for persons with schizophrenia and their supporters: One-year outcomes. Psychiatric Services. 2010; 61 :1099–1105. [ PubMed : 21041348 ]
• Rotondi AJ, Spring M, Grady J, Simpson R, Luther J, Abebe KZ, Haas GL. Use of a fractional factorial experiment to assess the e-healthcare application design needs of persons with dual diagnosis. Journal of Dual Diagnosis. 2012; 8 :277–282.
• Roy MJ, Taylor P, Runge W, Grigsby E, Woolley M, Torgeson T. Web-based post-traumatic stress disorder education for military family members. Military Medicine. 2012; 177 :284–290. [ PubMed : 22479915 ]
• Roy-Byrne P, Craske MG, Sullivan G, Rose RD, Edlund MJ, Lang AJ, Stein MB. Delivery of evidence-based treatment for multiple anxiety disorders in primary care: A randomized controlled trial. JAMA. 2010; 303 :1921–1928. [ PMC free article : PMC2928714 ] [ PubMed : 20483968 ]
• Ruetsch C, Cacciola J, Tkacz J. A national study of a telephone support service for patients receiving office-based buprenorphine medication-assisted treatment: Study feasibility and sample description. Journal of Substance Abuse Treatment. 2010; 39 :307–317. [ PubMed : 20728299 ]
• Ruetsch C, Tkacz J, McPherson TL, Cacciola J. The effect of telephonic patient support on treatment for opioid dependence: Outcomes at one year follow-up. Addictive Behaviors. 2012; 37 :686–689. [ PubMed : 22348921 ]
• Ruskin PE, Silver-Aylaian M, Kling MA, Reed SA, Bradham DD, Hebel JR, Hauser P. Treatment outcomes in depression: Comparison of remote treatment through telepsychiatry to in-person treatment. American Journal of Psychiatry. 2004; 161 :1471–1476. [ PubMed : 15285975 ]
• Ruskin PE, Van Der Wende J, Clark CR, Fenton J, Deveau J, Thapar R. Feasibility of using the Med-eMonitor system in the treatment of schizophrenia: A pilot study. Drug Information Journal. 2003; 37 :283–291.
• Ruwaard J, Broeksteeg J, Schrieken B, Emmelkamp P, Lange A. Web-based therapist-assisted cognitive behavioral treatment of panic symptoms: A randomized controlled trial with a three-year follow-up. Journal of Anxiety Disorders. 2010; 24 :387–396. [ PubMed : 20227241 ]
• Ruwaard J, Lange A, Schrieken B, Emmelkamp P. Efficacy and effectiveness of online cognitive behavioral treatment: A decade of interapy research. Studies in Health Technology and Informatics. 2011; 167 :9–14. [ PubMed : 21685634 ]
• Sablier J, Stip E, Jacquet P, Giroux S, Pigot H, Franck N. Ecological assessments of activities of daily living and personal experiences with Mobus, an assistive technology for cognition: A pilot study in schizophrenia. Assistive Technology. 2012; 24 :67–77. [ PubMed : 22876729 ]
• Sachpazidis I, Majadas M. A medication adherence monitoring system based on HL7 messages. Journal on Information Technology in Healthcare. 2006; 4 :49–58.
• Saitz R, Palfai TP, Freedner N, Winter MR, MacDonald A, Lu J, Dejong W. Screening and brief intervention online for college students: The iHealth study. Alcohol and Alcoholism. 2007; 42 :28–36. [ PubMed : 17130139 ]
• Salemink E, van den Hout M, Kindt M. Effects of positive interpretive bias modification in highly anxious individuals. Journal of Anxiety Disorders. 2009; 23 :676–683. [ PubMed : 19272750 ]
• Sánchez-Ortiz VC, Munro C, Stahl D, House J, Startup H, Treasure J, Schmidt U. A randomized controlled trial of Internet-based cognitive-behavioural therapy for bulimia nervosa or related disorders in a student population. Psychological Medicine. 2011; 41 :407–417. [ PubMed : 20406523 ]
• Santangelo P, Bohus M, Ebner-Priemer UW. Ecological momentary assessment in borderline personality disorder: A review of recent findings and methodological challenges. Journal of Personality Disorders. 2014; 28 (4):555–576. [ PubMed : 22984853 ]
• Saul JE, Schillo BA, Evered S, Luxenberg MG, Kavanaugh A, Cobb N, An LC. Impact of a statewide Internet-based tobacco cessation intervention. Journal of Medical Internet Research. 2007; 9 :e28. [ PMC free article : PMC2047287 ] [ PubMed : 17942390 ]
• Schmidt NB, Richey JA, Buckner JD, Timpano KR. Attention training for generalized social anxiety disorder. Journal of Abnormal Psychology. 2009; 118 :5–14. [ PubMed : 19222309 ]
• Seal KH, Abadjian L, McCamish N, Shi Y, Tarasovsky G, Weingardt K. A randomized controlled trial of telephone motivational interviewing to enhance mental health treatment engagement in Iraq and Afghanistan veterans. General Hospital Psychiatry. 2012; 34 (5):450–459. [ PubMed : 22632925 ]
• Secure Telehealth. Medicaid reimburses for telepsychiatry in 40 states. Pittsburgh, PA: Author; 2012.
• Selby P, Van Mierlo MT, Voci SC, Parent D, Cunningham JA. Online social and professional support for smokers trying to quit: An exploration of first time posts from 2562 members. Journal of Medical Internet Research. 2010; 12 :e34. [ PMC free article : PMC2956324 ] [ PubMed : 20719739 ]
• Seligman ME, Rashid T, Parks AC. Positive psychotherapy. American Psychologist. 2006; 61 :774–788. [ PubMed : 17115810 ]
• Seligman MEP, Steen TA, Park N, Peterson C. Positive psychology progress Empirical validation of interventions. American Psychologist. 2005; 60 :410–421. [ PubMed : 16045394 ]
• Selmi PM, Klein MH, Greist JH, Sorrell SP, Erdman HP. Computer-administered cognitive-behavioral therapy for depression. American Journal of Psychiatry. 1990; 147 :51–56. [ PubMed : 2403473 ]
• Serre F, Fatseas M, Debrabant R, Alexandre JM, Auriacombe M, Swendsen J. Ecological momentary assessment in alcohol, tobacco, cannabis and opiate dependence: A comparison of feasibility and validity. Drug and Alcohol Dependence. 2012; 126 :118–123. [ PubMed : 22647899 ]
• Severson HH, Gordon JS, Danaher BG, Akers L. ChewFree ​.com : Evaluation of a Web-based cessation program for smokeless tobacco users. Nicotine and Tobacco Research. 2008; 10 :381–391. [ PubMed : 18236303 ]
• Shahab L, McEwen A. Online support for smoking cessation: A systematic review of the literature. Addiction. 2009; 104 :1792–1804. [ PubMed : 19832783 ]
• Shapiro JR, Reba-Harrelson L, Dymek-Valentine M, Woolson SL, Hamer RM, Bulik CM. Feasibility and acceptability of CD-ROM-based cognitive-behavioural treatment for binge-eating disorder. European Eating Disorders Review. 2007; 15 :175–184. [ PubMed : 17676687 ]
• Sharp IR, Kobak KA, Osman DA. The use of videoconferencing with patients with psychosis: A review of the literature. Annals of General Psychiatry. 2011; 10 :14. [ PMC free article : PMC3101132 ] [ PubMed : 21501496 ]
• Shaw HE, Shaw SF. Critical ethical issues in online counseling: Assessing current practices with an ethical intent checklist. Journal of Counseling and Development. 2006; 84 :41–53.
• Shiffman S. Ecological momentary assessment (EMA) in studies of substance use. Psychological Assessment. 2009; 21 :486–497. [ PMC free article : PMC2846437 ] [ PubMed : 19947783 ]
• Sholomskas DE, Syracuse-Siewert G, Rounsaville BJ, Ball SA, Nuro KF, Carroll KM. We don't train in vain: A dissemination trial of three strategies of training clinicians in cognitive–behavioral therapy. Journal of Consulting and Clinical Psychology. 2005; 73 :106–115. [ PMC free article : PMC2367057 ] [ PubMed : 15709837 ]
• Shore J, Kaufmann LJ, Brooks E, Bair B, Dailey N, Richardson WJ, Manson S. Review of American Indian veteran telemental health. Telemedicine and e-Health. 2012; 18 :87–94. [ PubMed : 22283396 ]
• Shore JH, Savin D, Novins D, Manson SM. Cultural aspects of telepsychiatry. Journal of Telemedicine and Telecare. 2006; 12 :116–121. [ PubMed : 16638232 ]
• Shore JH, Savin D, Orton H, Beals J, Manson SM. Diagnostic reliability of telepsychiatry in American Indian veterans. American Journal of Psychiatry. 2007; 164 :115–118. [ PubMed : 17202552 ]
• Simon GE, Ludman EJ, Goodale LC, Dykstra DM, Stone E, Cutsogeorge D, Pabiniak C. An online recovery plan program: Can peer coaching increase participation? Psychiatric Services. 2011; 62 :666–669. [ PMC free article : PMC4176611 ] [ PubMed : 21632737 ]
• Simon GE, Ludman EJ, Tutty S, Operskalski B, Von Korff KM. Telephone psychotherapy and telephone care management for primary care patients starting antidepressant treatment: A randomized controlled trial. JAMA. 2004; 292 :935–942. [ PubMed : 15328325 ]
• Simon GE, Ralston JD, Savarino J, Pabiniak C, Wentzel C, Operskalski BH. Randomized trial of depression follow-up care by online messaging. Journal of General Internal Medicine. 2011; 26 :698–704. [ PMC free article : PMC3138593 ] [ PubMed : 21384219 ]
• Simon GE, Revicki D, VonKorff M. Telephone assessment of depression severity. Journal of Psychiatric Research. 1993; 27 :247–252. [ PubMed : 8295157 ]
• Simpson S. Psychotherapy via videoconferencing: A review. British Journal of Guidance and Counselling. 2009; 37 :271–286.
• Simpson S, Morrow E. Using videoconferencing for conducting a therapeutic relationship. In: Anthony K, Nagel DM, Goss S, editors. The use of technology in mental health: Applications, ethics and practice. Springfield, IL: Charles C. Thomas Publisher; 2010. pp. 94–103.
• Sims TH, McAfee T, Fraser DL, Baker TB, Fiore MC, Smith SS. Quitline cessation counseling for young adult smokers: A randomized clinical trial. Nicotine and Tobacco Research. 2012; 15 (5):932–941. [ PMC free article : PMC3621584 ] [ PubMed : 23080378 ]
• Singh SP, Arya D, Peters T. Accuracy of telepsychiatric assessment of new routine outpatient referrals. BMC Psychiatry. 2007; 7 :55. [ PMC free article : PMC2194760 ] [ PubMed : 17919329 ]
• Skinner AEG, Latchford G. International and multicultural issues in online counseling. In: Kraus R, Zack JS, Stricker G, editors. Online counseling: A handbook for mental health professionals. 2nd ed. New York: Elsevier Science; 2011. pp. 257–269.
• Smith A, Rainie L, Zickuhr K. College students and technology. Washington, DC: Pew Internet and American Life Project; 2011.
• Smith B, Harms WD, Burres S, Korda H, Rosen H, Davis J. Enhancing behavioral health treatment and crisis management through mobile ecological momentary assessment and SMS messaging. Health Informatics Journal. 2012; 18 :294–308. [ PubMed : 23257059 ]
• Smith DJ, Griffiths E, Poole R, di Florio A, Barnes E, Kelly MJ, Simpson S. Beating bipolar: Exploratory trial of a novel Internet-based psychoeducational treatment for bipolar disorder. Bipolar Disorders. 2011; 13 :571–577. [ PubMed : 22017225 ]
• Smith JL, Carpenter KM, Amrhein PC, Brooks AC, Levin D, Schreiber EA, Nunes EV. Training substance abuse clinicians in motivational interviewing using live supervision via teleconferencing. Journal of Consulting and Clinical Psychology. 2012; 80 :450–464. [ PMC free article : PMC3365649 ] [ PubMed : 22506795 ]
• Sobell LC, Brown J, Leo GI, Sobell MB. The reliability of the Alcohol Timeline Followback when administered by telephone and by computer. Drug and Alcohol Dependence. 1996; 42 :49–54. [ PubMed : 8889403 ]
• Sorvaniemi M, Ojanen E, Santamäki O. Telepsychiatry in emergency consultations: A follow-up study of sixty patients. Telemedicine Journal and e-Health. 2005; 11 :439–441. [ PubMed : 16149889 ]
• Spek V, Cuijpers P, Nyklíček I, Smits N, Riper H, Keyzer J, Pop V. One-year follow-up results of a randomized controlled clinical trial on Internet-based cognitive behavioural therapy for subthreshold depression in people over 50 years. Psychological Medicine. 2008; 38 :635–639. [ PubMed : 18205965 ]
• Spek V, Nyklíček I, Cuijpers P, Pop V. Predictors of outcome of group and Internet-based cognitive behavior therapy. Journal of Affective Disorders. 2008; 105 :137–145. [ PubMed : 17543392 ]
• Spek V, Nyklíček I, Smits N, Cuijpers P, Riper H, Keyzer J, Pop V. Internet-based cognitive behavioural therapy for subthreshold depression in people over 50 years old: A randomized controlled clinical trial. Psychological Medicine. 2007; 37 :1797–1806. [ PubMed : 17466110 ]
• Squires DD, Bryant MD. Using technology to support evidence-based behavioral health practices: A clinician's guide. New York: Routledge/Taylor & Francis Group; 2010. Substance use disorders; pp. 91–108.
• Stead LF, Perera R, Lancaster T. A systematic review of interventions for smokers who contact quitlines. Tobacco Control. 2007; 16 (Suppl 1):i3–i8. [ PMC free article : PMC2598525 ] [ PubMed : 18048627 ]
• Stellefson M, Hanik B, Chaney B, Chaney D, Tennant B, Chavarria EA. eHealth literacy among college students: A systematic review with implications for eHealth education. Journal of Medical Internet Research. 2011; 13 :e102. [ PMC free article : PMC3278088 ] [ PubMed : 22155629 ]
• Stjernswärd S, Östman M, Löwgren J. Online self-help tools for the relatives of persons with depression - A feasibility study. Scandinavian Journal of Caring Sciences. 2012; 26 :70–80. [ PubMed : 21692824 ]
• Stoddard JL, Augustson EM, Moser RP. Effect of adding a virtual community (bulletin board) to smokefree ​.gov : Randomized controlled trial. Journal of Medical Internet Research. 2008; 10 :e53. [ PMC free article : PMC2630832 ] [ PubMed : 19097974 ]
• Stoops WW, Dallery J, Fields NM, Nuzzo PA, Schoenberg NE, Martin CA, Wong CJ. An Internet-based abstinence reinforcement smoking cessation intervention in rural smokers. Drug and Alcohol Dependence. 2009; 105 :56–62. [ PMC free article : PMC2743786 ] [ PubMed : 19615830 ]
• Stout RL, Rubin A, Zwick W, Zywiak W, Bellino L. Optimizing the cost-effectiveness of alcohol treatment: A rationale for extended case monitoring. Addictive Behaviors. 1999; 24 :17–35. [ PubMed : 10189970 ]
• Stout RL, Zywiak W, Rubin A, Zwick W, Larson MJ, Shepard D. Case monitoring for alcoholics: One year clinical and health cost effects. Alcoholism: Clinical and Experimental Research. 2001; 25 (1):128–135.
• Strecher VJ, McClure JB, Alexander GL, Chakraborty B, Nair VN, Konkel JM, Pomerleau OF. Web-based smoking-cessation programs: Results of a randomized trial. American Journal of Preventive Medicine. 2008; 34 :373–381. [ PMC free article : PMC2697448 ] [ PubMed : 18407003 ]
• Strecher VJ, Shiffman S, West R. Randomized controlled trial of a Web-based computer-tailored smoking cessation program as a supplement to nicotine patch therapy. Addiction. 2005; 100 :682–688. [ PubMed : 15847626 ]
• Substance Abuse and Mental Health Services Administration. Managing anxiety symptoms in behavioral health services. Rockville, MD: Substance Abuse and Mental Health Services Administration; (Treatment Improvement Protocol (TIP) Series). (planned a)
• Substance Abuse and Mental Health Services Administration. Relapse prevention and recovery promotion in behavioral health services. Rockville, MD: Substance Abuse and Mental Health Services Administration; (Treatment Improvement Protocol (TIP) Series). (planned b)
• Sucala M, Schnur JB, Constantino MJ, Miller SJ, Brackman EH, Montgomery GH. The therapeutic relationship in e-therapy for mental health: A systematic review. Journal of Medical Internet Research. 2012; 14 :e110. [ PMC free article : PMC3411180 ] [ PubMed : 22858538 ]
• Sugarman DE, Nich C, Carroll KM. Coping strategy use following computerized cognitive–behavioral therapy for substance use disorders. Psychology of Addictive Behaviors. 2010; 24 :689–695. [ PMC free article : PMC3059590 ] [ PubMed : 21198228 ]
• Suler J. Assessing a person's suitability for online therapy: The ISMHO Clinical Case Study Group. CyberPsychology and Behavior. 2001; 4 :675–679. [ PubMed : 11800174 ]
• Suler J. The psychology of text relationships. In: Kraus R, Zack JS, Stricker G, editors. Online counseling: A handbook for mental health professionals. New York: Elsevier Science; 2004. pp. 19–50.
• Sullivan G, Craske MG, Sherbourne C, Edlund MJ, Rose RD, Golinelli D, Roy-Byrne PP. Design of the Coordinated Anxiety Learning and Management study: Innovations in collaborative care for anxiety disorders. General Hospital Psychiatry. 2007; 29 :379–387. [ PMC free article : PMC2095116 ] [ PubMed : 17888803 ]
• Sunderland M, Wong N, Hilvert-Bruce Z, Andrews G. Investigating trajectories of change in psychological distress amongst patients with depression and generalised anxiety disorder treated with Internet cognitive behavioural therapy. Behaviour Research and Therapy. 2012; 50 :374–380. [ PubMed : 22498311 ]
• Swan AJ, Tyssen EG. Enhancing treatment access: Evaluation of an Australian Web-based alcohol and drug counselling initiative. Drug and Alcohol Review. 2009; 28 :48–53. [ PubMed : 19320675 ]
• Swartz LH, Noell JW, Schroeder SW, Ary DV. A randomised control study of a fully automated Internet based smoking cessation programme. Tobacco Control. 2006; 15 :7–12. [ PMC free article : PMC2563640 ] [ PubMed : 16436397 ]
• Tate DF. A series of studies examining Internet treatment of obesity to inform Internet interventions for substance use and misuse. Substance Use & Misuse. 2011; 46 :57–65. [ PubMed : 21190406 ]
• Tate DF, Finkelstein EA, Khavjou O, Gustafson A. Cost-effectiveness of Internet interventions: Review and recommendations. Annals of Behavioral Medicine. 2009; 38 :40–45. [ PMC free article : PMC2772952 ] [ PubMed : 19834778 ]
• Taylor CB, Bryson S, Luce KH, Cunning D, Doyle AC, Abascal LB, Wilfley DE. Prevention of eating disorders in at-risk college-age women. Archives of General Psychiatry. 2006; 63 :881–888. [ PMC free article : PMC3837629 ] [ PubMed : 16894064 ]
• Tenkku LE, Mengel MB, Nicholson RA, Hile MG, Morris DS, Salas J. A Web-based intervention to reduce alcohol-exposed pregnancies in the community. Health Education & Behavior. 2011; 38 :563–573. [ PubMed : 21471438 ]
• Te Poel F, Bolman C, Reubsaet A, de Vries H. Efficacy of a single computer-tailored e-mail for smoking cessation: Results after 6 months. Health Education Research. 2009; 24 :930–940. [ PubMed : 19574405 ]
• Titov N, Andrews G, Choi I, Schwencke G, Johnston L. Randomized controlled trial of Web-based treatment of social phobia without clinician guidance. Australian and New Zealand Journal of Psychiatry. 2009; 43 :913–919.
• Titov N, Andrews G, Choi I, Schwencke G, Mahoney A. Shyness 3: Randomized controlled trial of guided versus unguided Internet-based CBT for social phobia. Australian and New Zealand Journal of Psychiatry. 2008; 42 :1030–1040. [ PubMed : 19016091 ]
• Titov N, Andrews G, Davies M, McIntyre K, Robinson E, Solley K. Internet treatment for depression: A randomized controlled trial comparing clinician vs. technician assistance. PLOS ONE. 2010; 5 :e10939. [ PMC free article : PMC2882336 ] [ PubMed : 20544030 ]
• Titov N, Andrews G, Johnston L, Robinson E, Spence J. Transdiagnostic Internet treatment for anxiety disorders: A randomized controlled trial. Behaviour Research and Therapy. 2010; 48 :890–899. [ PubMed : 20561606 ]
• Titov N, Andrews G, Robinson E, Schwencke G, Johnston L, Solley K. Clinician-assisted Internet-based treatment is effective for generalized anxiety disorder: Randomized controlled trial. Australian and New Zealand Journal of Psychiatry. 2009; 43 :905–912.
• Titov N, Andrews G, Schwencke G, Solley K, Johnston L, Robinson E. An RCT comparing effect of two types of support on severity of symptoms for people completing Internet-based cognitive behaviour therapy for social phobia. Australian and New Zealand Journal of Psychiatry. 2009; 43 :920–926.
• Titov N, Dear BF, Schwencke G, Andrews G, Johnston L, Craske MG, McEvoy P. Transdiagnostic Internet treatment for anxiety and depression: A randomised controlled trial. Behaviour Research and Therapy. 2011; 49 (8):441–452. [ PubMed : 21679925 ]
• Titus JC, Guthmann D. Addressing the black hole in substance abuse treatment for deaf and hard of hearing individuals: Technology to the rescue. Journal of the American Deafness and Rehabilitation Association. 2010; 43 :92–100.
• Todd NJ, Jones SH, Lobban FA. What do service users with bipolar disorder want from a Web-based self-management intervention? A qualitative focus group study. Clinical Psychology & Psychotherapy. 2012; 20 (6):531–543. [ PubMed : 22715161 ]
• Tossmann HP, Jonas B, Tensil MD, Lang P, Strüber E. A controlled trial of an Internet-based intervention program for cannabis users. Cyberpsychology, Behavior, and Social Networking. 2011; 14 :673–679. [ PubMed : 21651419 ]
• Trepal H, Haberstroh S, Duffey T, Evans M. Considerations and strategies for teaching online counseling skills: Establishing relationships in cyberspace. Counselor Education and Supervision. 2007; 46 :266–279.
• Tsan JY, Day SX. Personality and gender as predictors of online counseling use. Journal of Technology in Human Services. 2007; 25 :39–55.
• Tunstall N, Prince M, Mann A. Concurrent validity of a telephone-administered version of the Gospel Oak instrument (including the SHORT-CARE). International Journal of Geriatric Psychiatry. 1997; 12 :1035–1038. [ PubMed : 9395936 ]
• Turner JW. Telepsychiatry as a case study of presence: Do you know what you are missing? Journal of Computer Mediated Communication. 2001; 6 (4)
• Tzelepis F, Paul CL, Walsh RA, McElduff P, Knight J. Proactive telephone counseling for smoking cessation: Meta-analyses by recruitment channel and methodological quality. Journal of the National Cancer Institute. 2011; 103 :922–941. [ PubMed : 21666098 ]
• Vaca FE, Winn D, Anderson CL, Kim D, Arcila M. Six-month follow-up of computerized alcohol screening, brief intervention, and referral to treatment in the emergency department. Substance Abuse. 2011; 32 :144–152. [ PubMed : 21660874 ]
• Vaccaro N, Lambie GW. Computer-based counselor-in-training supervision: Ethical and practical implications for counselor educators and supervisors. Counselor Education and Supervision. 2007; 47 :46–57.
• Välimäki M, Hätönen H, Lahti M, Kuosmanen L, Adams CE. Information and communication technology in patient education and support for people with schizophrenia. Cochrane Database of Systematic Reviews. 2012; 10 CD007198. [ PubMed : 23076932 ]
• Vallejo MA, Jordan CM, Diaz MI, Comeche MI, Ortega J. Psychological assessment via the Internet: A reliability and validity study of online (vs. paper-and-pencil) versions of the General Health Questionnaire-28 (GHQ-28) and the Symptoms Check-List-90-Revised (SCL-90-R). Journal of Medical Internet Research. 2007; 9 :e2. [ PMC free article : PMC1794673 ] [ PubMed : 17478411 ]
• Van Ameringen M, Mancini C, Simpson W, Patterson B. Potential use of Internet-based screening for anxiety disorders: A pilot study. Depression and Anxiety. 2010; 27 :1006–1010. [ PubMed : 20721902 ]
• van Bastelaar KM, Pouwer F, Cuijpers P, Riper H, Snoek FJ. Web-based depression treatment for type 1 and type 2 diabetic patients: A randomized, controlled trial. Diabetes Care. 2011; 34 :320–325. [ PMC free article : PMC3024341 ] [ PubMed : 21216855 ]
• Van Bockstaele B, Verschuere B, Koster EHW, Tibboel H, De Houwer J, Crombez G. Effects of attention training on self-reported, implicit, physiological and behavioural measures of spider fear. Journal of Behavior Therapy and Experimental Psychiatry. 2011; 42 :211–218. [ PubMed : 21315884 ]
• VanDeMark NR, Burrell NR, LaMendola WF, Hoich CA, Berg NP, Medina E. An exploratory study of engagement in a technology-supported substance abuse intervention. Substance Abuse Treatment, Prevention, and Policy. 2010; 5 :10. [ PMC free article : PMC2898791 ] [ PubMed : 20529338 ]
• van der Zanden RA, Speetjens PA, Arntz KS, Onrust SA. Online group course for parents with mental illness: Development and pilot study. Journal of Medical Internet Research. 2010; 12 :e50. [ PMC free article : PMC3057319 ] [ PubMed : 21169178 ]
• Vernmark K, Lenndin J, Bjarehed J, Carlsson M, Karlsson J, Oberg J, Andersson G. Internet administered guided self-help versus individualized e-mail therapy: A randomized trial of two versions of CBT for major depression. Behaviour Research and Therapy. 2010; 48 :368–376. [ PubMed : 20152960 ]
• Vernon ML. A review of computer-based alcohol problem services designed for the general public. Journal of Substance Abuse Treatment. 2010; 38 :203–211. [ PMC free article : PMC2835799 ] [ PubMed : 20015607 ]
• Villani D, Riva G. Does interactive media enhance the management of stress? Suggestions from a controlled study. Cyberpsychology, Behavior, and Social Networking. 2012; 15 :24–30. [ PubMed : 22032797 ]
• Vincent N, Lewycky S. Logging on for better sleep: RCT of the effectiveness of online treatment for insomnia. Sleep: Journal of Sleep and Sleep Disorders Research. 2009; 32 :807–815. [ PMC free article : PMC2690569 ] [ PubMed : 19544758 ]
• Wall AF. Evaluating a health education Web site: The case of AlcoholEdu. NASPA Journal. 2007; 44 :692–714.
• Waller R, Gilbody S. Barriers to the uptake of computerized cognitive behavioural therapy: A systematic review of the quantitative and qualitative evidence. Psychological Medicine. 2009; 39 :705–712. [ PubMed : 18812006 ]
• Walters ST, Vader AM, Harris TR. A controlled trial of Web-based feedback for heavy drinking college students. Prevention Science. 2007; 8 :83–88. [ PubMed : 17136461 ]
• Wang PS, Simon GE, Avorn J, Azocar F, Ludman EJ, McCulloch J, Kessler RC. Telephone screening, outreach, and care management for depressed workers and impact on clinical and work productivity outcomes: A randomized controlled trial. JAMA. 2007; 298 :1401–1411. [ PMC free article : PMC2859667 ] [ PubMed : 17895456 ]
• Wantland DJ, Portillo CJ, Holzemer WL, Slaughter R, McGhee EM. The effectiveness of Web-based vs. non-Web-based interventions: A meta-analysis of behavioral change outcomes. Journal of Medical Internet Research. 2004; 6 :e40. [ PMC free article : PMC1550624 ] [ PubMed : 15631964 ]
• Warmerdam L, van Straten A, Jongsma J, Twisk J, Cuijpers P. Online cognitive behavioral therapy and problem-solving therapy for depressive symptoms: Exploring mechanisms of change. Journal of Behavior Therapy and Experimental Psychiatry. 2010; 41 :64–70. [ PubMed : 19913781 ]
• Watkins DC, Smith LC, Kerber K, Kuebler J, Himle JA. Email reminders as a self-management tool in depression: A needs assessment to determine patients' interests and preferences. Journal of Telemedicine and Telecare. 2011; 17 :378–381. [ PubMed : 21933895 ]
• Webb TL. Commentary on Shahab & McEwen (2009): Understanding and preventing attrition in online smoking cessation interventions: A self-regulatory perspective. Addiction. 2009; 104 :1805–1806. [ PubMed : 19832784 ]
• Webb TL, Joseph J, Yardley L, Michie S. Using the Internet to promote health behavior change: A systematic review and meta-analysis of the impact of theoretical basis, use of behavior change techniques, and mode of delivery on efficacy. Journal of Medical Internet Research. 2010; 12 :e4. [ PMC free article : PMC2836773 ] [ PubMed : 20164043 ]
• Weingardt KR, Cucciare MA, Bellotti C, Lai WP. A randomized trial comparing two models of Web-based training in cognitive–behavioral therapy for substance abuse counselors. Journal of Substance Abuse Treatment. 2009; 37 :219–227. [ PMC free article : PMC2771721 ] [ PubMed : 19339136 ]
• Weingardt KR, Lysell KM. Session 125 My recovery plan: Using My HealtheVet to support veterans with mental health and substance use disorders. Washington, DC: U.S. Department of Veterans Affairs; 2007.
• Welfel ER, Heinlen K. Ethics in technology and mental health. In: Cucciare MA, Weingardt KR, editors. Using technology to support evidence-based behavioral health practices: A clinician's guide. New York: Routledge/Taylor & Francis Group; 2010. pp. 267–290.
• Westphal A, Dingjan P, Attoe R. What can low and high technologies do for late-life mental disorders? Current Opinion in Psychiatry. 2010; 23 :510–515. [ PubMed : 20717039 ]
• Wetherell JL, Petkus AJ, Thorp SR, Stein MB, Chavira DA, Campbell-Sills L, Roy-Byrne P. Age differences in treatment response to a collaborative care intervention for anxiety disorders. British Journal of Psychiatry. 2013; 203 (1):65–72. [ PMC free article : PMC3696879 ] [ PubMed : 23580378 ]
• Whittaker R, McRobbie H, Bullen C, Borland R, Rodgers A, Gu Y. Mobile phone-based interventions for smoking cessation. Cochrane Database of Systematic Reviews. 2012; 11 CD006611. [ PubMed : 23152238 ]
• Whitten P, Buis L. Private payer reimbursement for telemedicine services in the United States. Telemedicine Journal and e-Health. 2007; 13 :15–23. [ PubMed : 17309350 ]
• Wiers RW, Eberl C, Rinck M, Becker ES, Lindenmeyer J. Retraining automatic action tendencies changes alcoholic patients' approach bias for alcohol and improves treatment outcome. Psychological Science. 2011; 22 :490–497. [ PubMed : 21389338 ]
• Wilkinson N, Ang RP, Goh DH. Online video game therapy for mental health concerns: A review. International Journal of Social Psychiatry. 2008; 54 :370–382. [ PubMed : 18720897 ]
• Williams J, Herman-Stahl M, Calvin SL, Pemberton M, Bradshaw M. Mediating mechanisms of a military Web-based alcohol intervention. Drug and Alcohol Dependence. 2009; 100 :248–257. [ PubMed : 19081206 ]
• Wilson GT, Zandberg LJ. Cognitive-behavioral guided self-help for eating disorders: effectiveness and scalability. Clinical Psychology Review. 2012; 32 :343–357. [ PubMed : 22504491 ]
• Wims E, Titov N, Andrews G, Choi I. Clinician-assisted Internet-based treatment is effective for panic: A randomized controlled trial. Australian and New Zealand Journal of Psychiatry. 2010; 44 :599–607. [ PubMed : 20560847 ]
• Wolf M, Ebner-Priemer U, Schramm E, Domsalla M, Hautzinger M, Bohus M. Maximizing skills acquisition in dialectical behavior therapy with a CD-ROM-based self-help program: Results from a pilot study. Psychopathology. 2011; 44 :133–135. [ PubMed : 21228618 ]
• Worrall JM, Fruzzetti AE. Improving peer supervisor ratings of therapist performance in dialectical behavior therapy: An Internet-based training system. Psychotherapy: Theory, Research, Practice, Training. 2009; 46 :476–479. [ PubMed : 22121843 ]
• Wyn J, Cuervo H, Woodman D, Stokes H. Young people, well being and communication technologies. Melbourne, Australia: Youth Research Centre, University of Melbourne; 2005.
• Xu R, Mei G, Zhang G, Gao P, Judkins T, Cannizzaro M, Li J. A voice-based automated system for PTSD screening and monitoring. Studies in Health Technology and Informatics. 2012; 173 :552–558. [ PubMed : 22357057 ]
• Yager Z, O'Dea JA. Prevention programs for body image and eating disorders on university campuses: A review of large, controlled interventions. Health Promotion International. 2008; 23 :173–189. [ PubMed : 18263883 ]
• Yeh CJ, Chang T, Chiang L, Drost CM, Spelliscy D, Carter RT, Chang Y. Development, content, process and outcome of an online peer supervision group for counselor trainees. Computers in Human Behavior. 2008; 24 :2889–2903.
• Yellowlees PM, Holloway KM, Parish MB. Therapy in virtual environments–Clinical and ethical issues. Telemedicine and e-Health. 2012; 18 :558–564. [ PubMed : 22823138 ]
• Yellowlees PM, Odor A, Parish MB, Iosif AM, Haught K, Hilty D. A feasibility study of the use of asynchronous telepsychiatry for psychiatric consultations. Psychiatric Services. 2010; 61 :838–840. [ PubMed : 20675845 ]
• Yuen EK, Goetter EM, Herbert JD, Forman EM. Challenges and opportunities in Internet-mediated telemental health. Professional Psychology: Research and Practice. 2012; 43 :1–8.
• Zabinski MF, Wilfley DE, Calfas KJ, Winzelberg AJ, Taylor CB. An interactive psychoeducational intervention for women at risk of developing an eating disorder. Journal of Consulting and Clinical Psychology. 2004; 72 :914–919. [ PubMed : 15482051 ]
• Zack JS. Technology of online counseling. In: Kraus R, Zack JS, Stricker G, editors. Online counseling: A handbook for mental health professionals. New York: Elsevier Science; 2004. pp. 93–121.
• Zbikowski MS, Hapgood J, Barnwell SS, McAfee T. Phone and Web-based tobacco cessation treatment: Real-world utilization patterns and outcomes for 11, 000 tobacco users. Journal of Medical Internet Research. 2008; 10 :e41. [ PMC free article : PMC2630835 ] [ PubMed : 19017583 ]
• Zur O, Williams MH, Lehavot K, Knapp S. Psychotherapist self-disclosure and transparency in the Internet age. Professional Psychology: Research and Practice. 2009; 40 :22–30.
• Cite this Page Center for Substance Abuse Treatment (US). Using Technology-Based Therapeutic Tools in Behavioral Health Services. Rockville (MD): Substance Abuse and Mental Health Services Administration (US); 2015. (Treatment Improvement Protocol (TIP) Series, No. 60.) Section 1, A Review of the Literature.
• PDF version of this title (3.0M)

In this Page

Other titles in these collections.

• SAMHSA/CSAT Treatment Improvement Protocols
• Health Services/Technology Assessment Texts (HSTAT)

Related information

• PMC PubMed Central citations
• PubMed Links to PubMed

Recent Activity

• A Review of the Literature - Using Technology-Based Therapeutic Tools in Behavio... A Review of the Literature - Using Technology-Based Therapeutic Tools in Behavioral Health Services

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

Serious games in high-stakes assessment contexts: a systematic literature review into the game design principles for valid game-based performance assessment

• Research Article
• Open access
• Published: 08 April 2024

Cite this article

You have full access to this open access article

• Aranka Bijl   ORCID: orcid.org/0000-0001-5745-1396 1 , 2 , 3 ,
• Bernard P. Veldkamp 2 ,
• Saskia Wools 3 &
• Sebastiaan de Klerk 3  

The systematic literature review (1) investigates whether ‘serious games’ provide a viable solution to the limitations posed by traditional high-stakes performance assessments and (2) aims to synthesize game design principles for the game-based performance assessment of professional competencies. In total, 56 publications were included in the final review, targeting knowledge, motor skills and cognitive skills and further narrowed down to teaching, training or assessing professional competencies. Our review demonstrates that serious games are able to provide an environment and task authentic to the target competency. Collected in-game behaviors indicate that serious games are able to elicit behavior that is related to a candidates’ ability level. Progress feedback and freedom of gameplay in serious games can be implemented to provide an engaging and enjoyable environment for candidates. Few studies examined adaptivity and some examined serious games without an authentic environment or task. Overall, the review gives an overview of game design principles for game-based performance assessment. It highlights two research gaps regarding authenticity and adaptivity and concludes with three implications for practice.

Avoid common mistakes on your manuscript.

In the years since their first introduction (ca. 1950s), videogames have only increased in popularity. In education, videogames are already widely applied as tools to support students in learning (cf. Boyle et al., 2016 ; Ifenthaler et al., 2012 ; Young et al., 2012 ). In contrast, less research has been done on the use of videogames as summative assessment environments, even though administering (high-stakes) summative assessments through games has several advantages.

First, videogames can be used to administer standardized assessments that provide richer data about candidate ability in comparison to traditional standardized assessments (e.g., multiple-choice tests; Schwartz & Arena, 2013 ; Shaffer & Gee, 2012 ; Shute & Rahimi, 2021 ). Second, assessment through videogames gives considerable freedom in recreating real-life criterion situations, which allows for authentic, situated assessment even when this is not feasible in the real working environment (Bell et al., 2008 ; Dörner et al., 2016 ; Fonteneau et al., 2020 ; Harteveld, 2011 ; Kirriemur & McFarlane, 2004 ; Michael & Chen, 2006 ). Third, videogames can offer candidates a more enjoyable test experience by providing an engaging environment where they are given a high degree of autonomy (Boyle et al., 2012 ; Jones, 1998 ; Mavridis & Tsiatsos, 2017 ). Finally, videogames allow for assessment through in-game behaviors (i.e., stealth assessment), which intends to make assessment less salient for candidates and lets them retain engagement (Shute & Ke, 2012 ; Shute et al., 2009 ).

The benefits above highlight why videogames are viable assessment environments, irrespective of the specific level of cognitive achievement (e.g., those depicted in Bloom’s revised taxonomy; Krathwohl, 2002 ). Moreover, the possibility for immersing candidates in complex, situated contexts make them especially interesting for higher-order learning outcomes such as problem solving and critical thinking (Dede, 2009 ; Shute & Ke, 2012 ). Therefore, videogames may provide a solution to the validity threats associated with traditional high-stakes performance assessments: an assessment type to evaluate competencies through a construct-relevant task in the context for which it is intended (Lane & Stone, 2006 ; Messick, 1994 ; Stecher, 2010 ), often used for the purpose of vocational certification.

The first validity threat associated with high-stakes performance assessments is the prevalence of test anxiety among candidates (Lane & Stone, 2006 ; Messick, 1994 ; Stecher, 2010 ), which is shown to be negatively correlated to test performance (von der Embse et al., 2018 ; von der Embse & Witmer, 2014 ). Although some debate exists about the causal relationship between the two (Jerrim, 2022 ; von der Embse et al., 2018 ), it is apparent that candidates who experience test anxiety are unfairly disadvantaged in high-stakes assessment contexts.

The second threat identified is caused by a need for high-stakes performance assessment to be both standardized to ensure objectivity and fairness (AERA et al., 2014 ; Kane, 2006 ) as well as include a construct-relevant task (e.g., writing an essay, participating in a roleplay; Lane & Stone, 2006 ; Messick, 1994 ). While neither rule out adaptivity (e.g., adaptive testing and open-ended assessments), the combination often restricts us to use a linear performance task that is not adaptable to candidate ability level. The potential mismatch that could occur between task difficulty and the ability level of candidates posits two disadvantages. First, the mismatch can frustrate candidates, which negatively affects their test performance (Wainer, 2000 ). Second, candidates likely receive fewer tasks that align with their ability level, which negatively affects test reliability and efficiency (Burr et al., 2023 ). High-stakes performance assessments would thus benefit from adaptive testing that is personalized and appropriately difficult, allowing candidates to be challenged enough to retain engagement (Burr et al., 2023 ; Malone & Lepper, 1987 ; Van Eck, 2006 ) while assessors are able to determine whether the candidate is at the required level efficiently and reliably (Burr et al., 2023 ; Davey, 2011 ). Additionally, adaptive testing allows for more personalized (end-of-assessment) feedback that could further boost candidate performance (Burr et al., 2023 ; Martin & Lazendic, 2018 ).

The third threat identified in high-stakes performance assessment is a lack of assessment authenticity. Logically, assessment would be administered best in the authentic context (i.e., the workplace in the case of professional competencies). This leads to a high degree of fidelity: how closely the assessment environment mirrors reality (Alessi, 1988, as cited in Gulikers et al., 2004 ). Unfortunately, this is not attainable for competencies that are dangerous or unethical to carry out (Bell et al., 2008 ; Williams-Bell et al., 2015 ). Another concern is that in the workplace, assessments are largely dependent on the workplace in which they are carried out. This would lead to considerable variations in testing conditions between candidates, but also the construct relevance of tasks they are evaluated on (Baartman & Gulikers, 2017 ). Authenticity of physical context and task are two dimensions required for mobilizing the competencies of interest (Gulikers et al., 2004 ), there is a need to achieve authenticity in other ways. Authenticity is also related to transfer: applying what is learned to new contexts. The higher the alignment between assessment and reality is, the more likely it is that the transfer of competence to the professional practice is made.

The fourth threat identified are inconsistencies between raters in scoring candidate performance. Traditional high-stakes performance assessments are often accompanied by rubrics to evaluate candidate performance; however, inconsistencies in how rubrics are interpreted and used leads to construct-irrelevant variance (Lane & Stone, 2006 ; Wools et al., 2010 ). In this study, the aim is to investigate whether ‘serious games’ (SGs)—those “used for purposes other than mere entertainment” (Susi et al., 2007 ; p. 1)—provide a viable solution to this and the other limitations posed by traditional high-stakes performance assessments.

The most important characteristic of games is that they are played with a clear goal in mind. Many games have a predetermined goal, but other games allow players to define their own objectives (Charsky, 2010 ; Prensky, 2001 ). Goals are given structure by the provision of rules, choices, and feedback (Lameras et al., 2017 ). First, rules direct players towards the goal by placing restrictions on gameplay (Charsky, 2010 ). Second, choices enable players to make decisions, for example to choose between different strategies to attain the goal (Charsky, 2010 ). The extent to which rules are restrictive for the gameplay is also closely related to the choices players have in the game (Charsky, 2010 ). Thus, rules and choices seem to be on two ends of a continuum that determines the linearity of a game. Linearity is defined as the extent to which players are given freedom of gameplay (Kim & Shute, 2015 ; Rouse, 2004 ). The third characteristic, feedback, is a well-versed topic in the field of education. In education, the main purpose of feedback is to help students get insight into their learning and get student understanding to the level of learning goals (Hattie & Timperley, 2007 ; Shute, 2008 ; van der Kleij et al., 2012 ). In games, feedback is used in a similar way to guide players towards the goal, as well as facilitate interactivity (Prensky, 2001 ). Feedback in games is provided in many modalities and gives players information about how they are progressing and where they stand with regards to the goal. For instance whether their actions have brought them closer to the goal or further away. Games are made up of a collection of game mechanics that define the game and determine how it is played (Rouse, 2004 ; Schell, 2015 ). In other words, game mechanics are how the defining features of games are translated into gameplay. To illustrate, game mechanics that provide feedback to players can include hints, gaining or losing lives, progress bars, dashboards, currencies and/or progress trees (Lameras et al., 2017 ).

When designing a game-based performance assessment, determining the information that should be collected about candidates to inform competence and designing the tasks that fulfill this information need is something that should be considered carefully for each professional competency. One way is through the use of the evidence-centered design (ECD) framework (cf. Mislevy & Riconscente, 2006 ). The ECD framework is a systematic approach to test development that relies on evidentiary arguments to move from a candidates behavior on a task to inferences about candidate ability. It is beyond the scope of the current study to examine the design of game content in relation to the target professional competencies. In this systematic literature review, the aim is to determine which game mechanics could help overcome the validity threats associated with high-stakes performance assessments and are suitable for use in such assessments.

Previous research for game design has been done for instructional SGs (e.g., dos Santos & Fraternali, 2016 ; Gunter et al., 2008 ). For SGs used in high-stakes performance assessments, emphasis is put on the potential effect of game mechanics on the validity of inferences should be considered. For instance, choices in game design can affect correlations between in-game behavior and player ability (Kim & Shute, 2015 ). Moreover, game mechanics exist that are likely to introduce construct-irrelevant variance when used in high-stakes performance assessments. To illustrate, when direct feedback about performance (e.g., points, lives, feedback messages) is given to players, at least part of the variance in test scores would be explained by the type and amount of feedback a candidate has received.

Establishing design principles for SGs for high-stakes performance assessment is important for several reasons. First, such an overview allows future developers such assessments to make more informed choices regarding game design. Second, combining and organizing the insights gained from the available empirical evidence advances the knowledge framework around the implementation of high-stakes performance assessment through games. Reviews on the use of games exist for learning (e.g., Boyle et al., 2016 ; Connolly et al., 2012 ; Young et al., 2012 ) or are targeted at specific professional domains (e.g., Gao et al., 2019 ; Gorbanev et al., 2018 ; Graafland et al., 2012 ; Wang et al., 2016 ). Nevertheless, a research gap remains as there is no knowledge of a systematic literature review that addresses the high-stakes performance assessment of professional competencies. To this end, this study begins with identifying the available literature on SGs targeted at professional competencies; then extracts the implemented game mechanics that could help to overcome the validity threats associated with high-stakes performance assessment; and finally synthesizes game design principles for game-based performance assessment in high-stakes contexts.

The scope of the current review is limited to professional competencies specifically catered to a vocation (e.g., construction hazard recognition). More generic professional competencies (e.g., programming) are not taken into consideration, as the context in which they are used can also fall outside of secondary vocational and higher education. Additionally, there is a growing body of literature that recognizes the potential of in-game behavior as a source of information about ability level in the context of game-based learning (e.g., Chen et al., 2020 ; Kim & Shute, 2015 ; Shute et al., 2009 ; Wang et al., 2015 ; Westera et al., 2014 ). As the relationship between in-game behavior and candidate ability is of equal importance in assessment, the scope of the current review includes SGs that focus not only on assessment, but also teaching and training of professional competencies.

The following section describes the procedure followed in conducting the current systematic literature review. First, a description of the inclusion criteria and search terms is given. This is followed by a description of the selection process and data extraction, together with an evaluation of the objectivity of the inclusion and quality criteria. Then, the search and selection results are presented, where two further categorizations of included studies operationalized: the type of competency and the how a successful SG is defined.

Following the guidelines described in Systematic Reviews in the Social Sciences (Petticrew & Roberts, 2005 ), the protocol below gives a description and the rationale behind the review along with a description of how different studies were identified, analyzed, and synthesized.

Databases and search terms

The databases that include most publications from the field of educational measurement ( Education Resources Information Center (ERIC) , PsycInfo , Scopus , and Web of Science) were consulted for the literature search using the following search terms:

Serious game : (serious gam* or game-based assess* or game-based learn* or game-based train*) and

Quality measure : (perform* or valid* or effect* or affect*)

Inclusion criteria and selection process

The initial search results were narrowed down by selecting only publications that were published in English and in a scientific, peer-reviewed journal. To be included, studies were required to report on the empirical research results of a study that (1) focused on a digital SG used for teaching, training, or assessment of one or more professional competencies specific to a work setting, (2) was conducted in secondary vocational education, higher education or vocational settings, and (3) included a measure to assess the dependent variable related to the quality of the SG. Studies were excluded when the focus was on simulations; while they have an overlapping role in the acquisition of professional competencies to SGs, these modalities represent distinct types of digital environments.

All results from the databases were exported to Endnote X9 (The EndNote Team, 2013 ) for screening. The selection process was conducted in three rounds. First, duplicates, and alternative document types (e.g., editorials, conference proceedings, letters) were removed. Then, the publications were screened based on the titles and abstracts; publications were removed when the title or abstract mentioned features of the study mutually exclusive with the inclusion criteria (e.g., primary school, rehabilitation, systematic literature review). Second, titles and abstracts of the remaining results were screened again. When the title or abstract lacked information, the full article was inspected. To illustrate, some titles and abstracts did not mention the target population, or whether the game was digital, or whether the professional competency was specific to a work setting. Finally, full-text articles were screened for full compliance with the inclusion criteria. Data was extracted from those publications.

The objectivity of the inclusion criteria was determined by blinded double classification on two occasions. The first occasion, after the removal of duplicates and alternative document types, 30 randomly selected publications were independently double-classified by an expert in the field of educational measurement based on the title and abstract. An agreement rate of 93% with a Cohen’s Kappa coefficient of .81 translated to a near perfect inter-rater reliability (Landis & Koch, 1977 ). On the second occasion, a random selection of 32 publications considered for data extraction were blindly double-classified based on the full-text by a master student in educational measurement which resulted in an agreement rate of 97% was with a near perfect Cohen’s Kappa coefficient (.94; Landis & Koch, 1977 ).

To assess the comprehensiveness of the systematic review and identify additional relevant studies, snowballing was conducted by backward and forward reference searching in Web of Science . For publications not available on Web of Science , snowballing was done in Scopus .

Data extraction

For the publications included, data was extracted systematically by means of a data extraction form (Supplementary Information SI1). The data extraction form includes: (1) general information, (2) details on the professional competency and research design, (3) serious game (SG) specifics and (4) a quality checklist.

The quality checklist contains 12 closed questions with three response options: the criterion is met (1), the criterion is met partly (.5), and the criterion is not met (0). Studies that scored 7 or below were considered to be of poor quality and were excluded. Studies that scored between 7.5 and 9.5 were considered to be of medium quality, while studies with scores 10 or above were considered to be of good quality (denoted with an asterisk in the data selection table; Supplementary Information SI2). These categories were determined by piloting the study quality checklist on two publications that were included, based on the inclusion criteria: one that was considered to be of a poor quality and one that was considered to be of good quality. The scores obtained by those studies were set as the lower and upper threshold, respectively.

As this systematic literature review is focused on the extraction of game mechanics to inform game design principles, all articles included in the review needed to obtain a score of at least .5 on the criteria that the game is discussed in enough detail. When publications explicitly refer to external sources for additional information, information from those sources were included in the data extraction form as well.

Blinded double coding to determine the reliability of the quality criteria for inclusion was done by the same raters described above. 24 randomly selected publications from the final review were included, with a varying overlap between three raters. The assigned scores were translated to the corresponding class (i.e., poor, medium, and good) to calculate the agreement rate. The rates ranged between 82 and 93%, which correspond to Cohen’s Kappa coefficients between substantial and near perfect (.66–.88; Landis & Koch, 1977 ; Table  1 ).

Search and selection results

In the PRISMA flow diagram of the publication selection process (Fig.  1 ; Moher et al., 2009 ), the two rounds in which titles and abstracts were screened for eligibility are combined. The databases were consulted on the 21st of December 2020 and yielded a total of 6,128 publications. After the removal of duplicates, 3,160 publications were left. On the basis of the inclusion criteria, another 2,981 publications were excluded from the review. In total, data was extracted from 179 publications. During the examination of the full-text articles, 129 studies were excluded due to insufficient quality (n = 42), lack of a detailed game description (n = 6), unavailability of the article (n = 5), not classifying the application as a game (n = 10) and an overall mismatch with the inclusion criteria (n = 66). In total, 50 publications were included. Snowballing was conducted in November of 2021 and resulted in the inclusion of six additional studies. In total, 56 publications were included in the final review.

PRISMA flow diagram of inclusion of the systematic literature review. PRISMA  preferred reporting items for systematic reviews and meta-analyses

Categorization of selected studies

Competency types.

Professional competencies are acquired and assessed in different ways. Given the variety of professional competencies, there is no universal game design that is likely to be beneficial across the board (Wouters et al., 2009 ). Other researchers (e.g., Young et al., 2012 ) even suggest that game design principles should not be generalized across games, contexts or competencies. While more content-related game design principles likely need to be defined per context, this review is conducted with the idea that generic game design principles exist that can be successfully used in multiple contexts. In that sense, the aim is to provide a starting point from where more context-specific SGs can be designed, for example through the use of ECD.

The review is organized according to the type of professional competency that is evaluated rather than the content of the SG under investigation, as this provides an idea of what researchers expect to train or assess within the SG. Different distinctions between competencies can be made. For example, Wouters et al. ( 2009 ) distinguish between cognitive, motor, affective, and communicative competencies. Moreover, Harteveld ( 2011 ) distinguishes between knowledge, skills, and attitudes. These taxonomies served as a basis to inductively categorize the targeted professional competencies into knowledge, motor skills, and cognitive skills.

The knowledge category includes studies that focus on for instance declarative knowledge (i.e., fact-based) or procedural knowledge (i.e., how to do something). For instance, the procedural steps involved in cardiopulmonary resuscitation (CPR). The motor skills category refers to motor behaviors (i.e., movements). For CPR, an example would be compression depth. The cognitive skills category encompasses skills such as reasoning, planning, and decision making. For example, studies that focus on the recognition of situations that require CPR.

Successful SGs

The scope of this systematic literature review is limited to SGs that are shown to be successful in teaching, training, or the assessment of professional competencies. As research methodologies differ between studies, there is a need to define what characterizes a successful SG. When SGs were used in teaching or training, it was deemed successful when a significant improvement in the targeted professional competency was found (e.g., through an external validated measure of the competency). Some studies compared an active control group and an experimental group that additionally received an SG (e.g., Boada et al., 2015 ; Dankbaar et al., 2016 ; Graafland et al., 2017 ; see Supplementary Information SI2 for a full account): an SG was not deemed successful in the current results when such two groups showed comparable results. When SGs were used for assessment, it was deemed successful when (1) research results showed a significant relationship between the SG and a validated measure of the targeted competency, or (2) the SG was shown to accurately distinguish between different competency levels.

The studies included in the review are discussed in two ways. First, descriptives of the included studies are given in terms of the degree to which games were successful in teaching, training, or assessment of professional competencies, the professional domains, and the competency types. Then, the game mechanics associated with the potential solutions to the validity threats in traditional performance assessment are presented.

Descriptives of the included studies

The final review includes 56 studies, published between 2006 and 2020 (consult Supplementary Information SI2 for a more detailed overview). No noteworthy differences were found between the SGs that aimed to teach, train, and assess professional competencies. Therefore, the results for the SGs included in the review are presented collectively.

Serious games with successful results

Divided over the type of professional competency evaluated, 84%, 83%, and 100% reported research results showing the SG was successful for cognitive skills, knowledge, and motor skills respectively (Table  2 ). Of the studies included in the systematic review, three studies found mixed effects of the SG under investigation between competency types (i.e., Luu et al., 2020 ; Phungoen et al., 2020 ; Tan et al., 2017 ).

Professional domains and competency types

The studies included in the review can be divided over seven professional domains (Table  3 ). These are further separated into professional competencies (see Supplementary Information SI2 for a full account). Examples include history taking (Alyami et al., 2019 ), crisis management (Steinrücke et al., 2020 ) and cultural understanding (Brown et al., 2018 ). Furthermore, the studies included in the review can be divided into three competency types: cognitive skills (n = 21), knowledge (n = 31), and motor skills (n = 4). An important note is that some studies evaluate the SG on more than one competency type, thus the sum of these categories is greater than the total number of studies included.

Game mechanics

The following section discusses the inclusion of game mechanics—all design choices within the game—for the SGs discussed in the studies included in the review. Following the aim of the current paper, the game mechanics discussed are selected for having the potential to (1) mediate the validity threats associated with traditional performance assessments, and (2) be appropriate for implementing in a game-based performance assessment.

Authenticity

Authenticity in the SGs is divided into two dimensions: authenticity of the physical context and task. First, an example of a physical context that was not representative of the real working environment was found for all three competencies (Table  4 ). Regarding the SGs targeted at cognitive skills, this was the case for Effic’ Asthme (Fonteneau et al., 2020 ). In this SG, the target population—medical students—would normally carry out pediatric asthma exacerbation in a hospital setting. The game environment used is, however, the virtual bedroom of a child. Regarding the SGs targeted at knowledge, Alyami et al. ( 2019 ) implemented the game Metaphoria to teach history taking content to medical students. Here, the game environment is inside a pyramid within a fantasy world. The final SG using a game environment that does not resemble the real working environment within the motor skill competency type studied by Jalink et al. ( 2014 ). In this SG, laparoscopic skills are trained by having players perform tasks in an underground mining environment.

Second, of the studies for which task authenticity could be determined, all but four included an authentic task for the professional competency targeted (Table  5 ). Examples of a task that was not authentic were found for all three competency types. Two SGs that targeted cognitive skills did not include an authentic task (Brown et al., 2018 ; Chee et al., 2019 ) as a result of implementing role reversals. Within these SGs, the players played in a reversed role fashion, and thus the task was not authentic for the task in the real working environment. One SG targeting knowledge did not include an authentic task (Alyami et al., 2019 ). In Metaphoria , the task for players is to interpret visual metaphors in relation to symptoms, whereas the target professional competency was history taking content. Finally, the SG studied by Drummond et al. ( 2017 ), targeting motor skills, the professional competency under investigation was not represented authentically within the game as the navigation was through point-and-click.

Unobtrusive data collection

For all three competency types, studies were found that use in-game data to make inferences about player ability (Table  6 ). While other studies did mention the collection of in-game behaviors, the results were limited to those that assessed the appropriateness of using the data in the assessment of competencies.

Different measures of in-game behaviors were found. First, 12 SGs determine competency by comparing player performance to some predetermined target, sometimes also translated to a score. In the game VERITAS (Veracity Education and Reactance Instruction through Technology and Applied Skills; Miller et al., 2019 ), for instance, players are assessed on whether they accurately assess whether the statement given by a character in the game is true or false. Second, seven SGs use time spent (i.e., completion time or playing time) as a measure of performance. For example, in the SG Wii Laparoscopy (Jalink et al., 2014 ), completion time is used to assess performance. This performance metric in the game showed a high correlation with performance on a validated measure for laparoscopic skills, but it should be noted that time penalties were included for mistakes made during the task. Finally, the use of log data was found in one SG targeted at cognitive skills (Steinrücke et al., 2020 ). In the Dilemma Game, in-game measures collected during gameplay were found to have promising relationships with competency levels.

In SGs, the difficulty level can be adapted in two ways: independent of the actions of players or dependent on the actions of players (Table  7 ). Whereas SGs that varied in difficulty level were found for professional competencies related to both knowledge and motor skills, none were found for professional competencies related to cognitive skills. Three SGs were found that adjust difficulty level based on player actions; however, none of the SGs adjusts the difficulty level down based on player actions. Three studies evaluated SGs where difficulty level was varied independent of player actions. Regarding the SGs targeted at knowledge, players either received fixed assignments (Boada et al., 2015 ) or were able to set the difficulty level prior to gameplay (Taillandier & Adam, 2018 ). The SG studied by Asadipour et al. ( 2017 ), targeting motor skills, increased challenge by building up the flying speed during the game as well as random generation of coins, but this was independent of player ability. Two SGs targeted at knowledge did mention difficulty levels, but not how they were adjusted. The SG Metaphoria (Alyami et al., 2019 ) included three difficulty levels. The SG Sustainability Challenge (Dib & Adamo-Villani, 2014 ) became more challenging as players progress to higher levels, but it is not clear when or how this was done.

Test anxiety

As described earlier, games are able to provide a more enjoyable testing experience by providing an engaging environment with a high degree of autonomy. Therefore, the way game characteristics, feedback, rules, and choices—are expressed in the studies included in the review are discussed below. To avoid confusion with linearity of assessment, the expression freedom of gameplay to describe the interaction between rules and choices.

First, seven examples were found where players are given feedback unrelated to performance (Table  8 ). Some ways feedback was given included a dashboard (Perini et al., 2018 ), remaining resources (Calderón et al., 2018 ; Taillandier & Adam, 2018 ) remaining time (Calderón et al., 2018 ; Dankbaar et al., 2017a , 2017b ; Mohan et al., 2014 ) or remaining tasks (Jalink et al., 2014 ).

Second, all studies included in the review but two include game mechanics to give some freedom of gameplay (Table  9 ). For cognitive skills and knowledge, game mechanics included the choice between multiple options (n = 14 for both), the inclusion of interactive elements (n = 8, for both) and the possibility for free exploration (n = 5 and n = 8, respectively). Two examples of customization were found: Dib and Adamo-Villani ( 2014 ) gave players the choice of avatar, whereas Alyami et al. ( 2019 ) allowed for a custom name. For the SGs that target motor skills, freedom of gameplay was given through control over the movements. For three out of four SGs in this category, special controllers were developed to give players authentic control over the movements in the game. This was not the case for Drummond et al. ( 2017 ), as their game did not explicitly train CPR; however, the researchers did assess its effect on motor skills.

Included studies

The final review included 56 studies. Of these, many reported positive results. This suggests that SGs are often successful in teaching, training, or assessing professional competencies, but could also point to a publication bias of positive results. As similar reviews to the current one (e.g., Connolly et al., 2012 ; Randel et al., 1992 ; Vansickle, 1986 ; Wouters et al., 2009 ) draw on similar databases, it is difficult to establish what is true. Some studies found mixed results for different competency types, suggesting that different approaches are warranted. Therefore, game mechanics in SGs for different competency types are discussed separately.

The review included few studies on SGs targeting motor skills compared to those targeting cognitive skills and knowledge. The low number of SGs for motor skills could be due to the need for specialized equipment to create an SG targeting motor skills. For example, Wii Laparoscopy (Jalink et al., 2014 ) is played using controllers that are specifically designed for the game. Not only does it require an extra investment, it also affects the ease of large scale implementation. There is no indication that motor skills cannot be assessed through SGs: four out of five studies have shown positive effects, both in learning effectiveness and assessment accuracy. Despite this, the benefits may only outweigh the added costs in situations where it is unfeasible to perform the professional competency in the real working environment.

Focusing on game mechanics for the authenticity of the physical context and the task, the results indicate that SGs are able to provide both. It should be noted that, while SGs are able to simulate the physical context and task with high fidelity, authenticity remains a matter of perception (Gulikers et al., 2008 ). The review focused only on those SGs that were successful when compared to validated measures of the targeted professional competency. Since these measures are considered to be accurate proxies for workplace performance, the transfer to the real working environment is likely to have been made. For all three competency types, examples were found for SGs that did not include an authentic physical context or authentic task, while still mobilizing competencies of interest. Even though the number of SGs in these categories is quite small, it does indicate that it is possible to assess professional competencies without an authentic environment or task.

The in-game measures most often used in the included SGs are those that indicate how well a player did in comparison to some standard or target. This suggests that SGs are able to elicit behavior in players that is dependent on their ability level in the target professional competency. Since the accuracy measures varied depending on the professional competency, an investigation is warranted to determine which in-game measures are indicative of ability per situation. Evidentiary frameworks such as the ECD framework can provide guidance in determining which data could be used to make inferences about candidate ability. Despite the promising results, more research should be done on the informational value of log data before claims can be made.

Some examples of studies were found where adaptivity was implemented was adaptive. In particular, some promising relationships between in-game behaviors and ability level were found. In traditional (high-stakes) testing, adaptivity has already been implemented successfully (Martin & Lazendic, 2018 ; Straetmans & Eggen, 2007 ). Although there are professional competencies for which ability levels cannot be differentiated, you are either able to do it or not. For such competencies, adaptivity does not have an added benefit. In contrast, for professional competencies where it is possible to differentiate ability levels, adaptivity should be considered.

Considering the appropriateness of game mechanics for high-stakes assessment, feedback considered in the current review was limited to progress feedback. This adds a fourth type of feedback to the feedback already recognized for assessment: knowledge of correct response, elaborated feedback, and delayed knowledge of results (van der Kleij et al., 2012 ). Although the small number of SGs that incorporated progress feedback affect the generalizability of the finding, it does indicate that feedback about progress may be the most appropriate solution.

Freedom of gameplay

A variety of game mechanics implemented in the SGs included in the review fulfill freedom of gameplay. While some studies did not elaborate on the choices given in the game, common ways players are given freedom are through choice options, interactive elements, and freedom to explore. These game mechanics were found in various studies, which raises the possibility that these findings can be generalized to new SGs targeted at assessing professional competencies. Other game mechanics related to freedom of gameplay were also found in a smaller capacity. Thus, further research should shed light on their generalizability. Moreover, the freedom of gameplay provided to the player plays a substantial role in shaping overall player experience and behavior (Kim & Shute, 2015 ; Kirginas & Gouscos, 2017 ). Therefore, future research should shed further light on whether different game mechanics influence players in different ways.

Limitations

Although the current systematic literature review provides a useful overview of the game design principles for game-based performance assessment of professional competencies, some limitations are identified.

First, the review covered a substantial amount of studies from the healthcare domain. This may be because the medical field consists of many higher order standardized tasks which may be particularly suitable to SGs. Although the large contribution of studies in the healthcare domain could limit the generalizability to other domains. The results of this systematic review were quite uniform; no indication was found that SGs in healthcare employed different game mechanics were employed. Moreover, there is a growing popularity of SGs in healthcare education (Wang et al., 2016 ), resulting in a higher number of studies that were available compared to other professional domains. It is advisable to regard the current results as a starting point for game design principles game-based performance assessment. Further research into the generalizability of game design principles across professional domains is warranted.

The second limitation is true for all systematic literature reviews: it is a cross section of the literature and may not present the full picture. The inclusion of studies is dependent on what is available in the search databases, what is accessible, and what keywords are included in the literature. Likely due to this limitation, only studies published from 2006 are included in the review, while the use of SGs dates back much further (Randel et al., 1992 ; Vansickle, 1986 ). To minimize the omission of relevant literature, snowballing was conducted on the final selection of studies. This method allowed for including related and potentially relevant studies. In total, six additional publications were included through this method out of the 2,370 considered.

After snowballing, an assessment of why these additionally included studies were not found through the search results resulted in various insights. First, three studies used the terms (educational) video game in their publication on SGs (Duque et al., 2008 ; Jalink et al., 2014 ; Mohan et al., 2017 ). Including this term in the original search would have resulted in too many hits outside of the scope of the current review. Second, Moreno-Ger et al. ( 2010 ) used the term simulation to describe the application, but refer to the application as game-like. As simulations fall outside of the scope of the current review, the absence of this study in the initial search cannot be attributed to a gap in the search terms, Third, the publication from Blanié et al. ( 2020 ) was probably not found due to a mismatch in search terms related to the quality measure. Additional search terms such as impact or improve could have been included. As only one additional study was found that presented this issue, it is unlikely to have had a great effect on the outcome of the review. Finally, it is unclear why the study by Fonteneau et al. ( 2020 ) was not found through the initial search, as it showed a match with the search terms used in the current review. Perhaps, this misclassification can be ascribed to the search databases queried.

Finally, many of the studies included in the review compare SGs to other, non-digital or digital, alternatives in terms of learning. These types of studies often include many confounding variables (Cook, 2005 ). This is because a comparison is done between interventions that are different in more ways than one. These differences affect the results in different ways: positive, negative, or even through an interaction with other features.

Suggestions for future research

Besides providing interesting insights, the current review also has implications for research. First, the review identified SGs successful in teaching, training, or assessment that did not authentically represent the physical context or task. Although in this review, too few examples were found to generalize the findings. Second, while some studies were found in which the SGs difficulty was adaptive, more studies should be conducted on the implementation of adaptivity within SGs. In particular, how in-game behavior to match the difficulty level to the ability level of the candidates. Third, Fantasy is included in many games (Charsky, 2010 ; Prensky, 2001 ) and is regarded as one of the reasons for playing them (Boyle et al., 2016 ). By including fantasy elements in game-based performance assessments, assessment can become even more engaging and enjoyable and candidates can become even less aware of being assessed. For learning, it has been suggested that fantasy should be closely connected to the learning content (Gunter et al., 2008 ; Malone, 1981 ), but further research might explore whether this holds for SGs used for the (high-stakes) assessment of professional competencies. Furthermore, while fantasy elements may blur the direct link between the SG and the professional practice, in-game behavior may still have a clear relationship with professional competencies (Kim & Shute, 2015 ; Simons et al., 2021 ). More research into the effect of authenticity on the measurement validity of SGs in assessing professional competencies is warranted.

Implications for practice

Based on the results of the review, four recommendations can be made for practice. First, regardless of the competency type: design the SG in such a way that both the task and the context are authentic. The results have shown that SGs are able to provide a representation of the physical context and task, authentic to the professional competency under investigation. Thus, in situations where the physical context or assessment task are difficult to represent in a traditional performance assessments, SGs can provide a solution. At the same time, implementing non-authentic (fantasy) contexts and tasks should be investigated further before being implemented in high-stakes performance assessment.

Second, ensure that in-game behavior within the SG is collected. This review has synthesized additional evidence for the potential of in-game behavior as a source of information about ability level. That being said, the in-game behavior that can be used to inform ability level is dependent on both the professional competency of interest and game design. While no generalized design principles regarding the collection of gameplay data can be given, evidentiary frameworks (e.g., ECD) can be used to determine which in-game behavior can be used to infer ability level. This is ultimately connected to implementation of adaptivity. While a limited number of SGs were found that implemented adaptivity, the potential to unobtrusively data about ability level underscores a missed opportunity for the wider implementation of adaptivity in SGs. Taken together with the successful implementation of adaptive testing in traditional high-stakes assessments (Martin & Lazendic, 2018 ; Straetmans & Eggen, 2007 ), a third recommendation would be to implement adaptivity where appropriate.

Finally, this review gives an overview of the game mechanics for high-stakes game-based performance assessment with little risk of affecting validity. To provide freedom of gameplay for SGs targeted at cognitive skills and knowledge, include free exploration, interactive elements and providing options. For motor skills, giving control over movements is a, perhaps straightforward, game design principle. Furthermore, feedback in SGs for high-stakes performance assessments can be done through providing progress feedback, which is different from traditional types of feedback in education (van der Kleij et al., 2012 ) but has potential to satisfy feedback as a game mechanic. These recommendations, intended for game developers, may prove useful in designing future SGs for the (high-stakes) assessment of professional competencies.

In-text citations

American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME). (2014). Standards for educational and psychological testing . American Educational Research Association.

Google Scholar  

Baartman, L., & Gulikers, J. (2017). Assessment in Dutch vocational education: Overview and tensions of the past 15 years. In E. De Bruijn, S. Billet, & J. Onstenk (Eds.), Enhancing teaching and learning in the Dutch vocational education system: Reforms enacted (pp. 245–266). Springer.

Chapter   Google Scholar  

Bell, B. S., Kanar, A. M., & Kozlowski, S. W. J. (2008). Current issues and future directions in simulation-based training in North America. The International Journal of Human Resource Management, 19 (8), 1416–1434. https://doi.org/10.1080/09585190802200173

Article   Google Scholar  

Boyle, E., Hainey, T., Connolly, T. M., Gray, G., Earp, J., Ott, M., Lim, T., Ninaus, M., Ribeiro, C., & Pereira, J. (2016). An update to the systematic literature review of empirical evidence on the impacts and outcomes of computer games and serious games. Computers & Education, 94 , 178–192. https://doi.org/10.1016/j.compedu.2015.11.003

Boyle, E. A., Connolly, T. M., Hainey, T., & Boyle, J. M. (2012). Engagement in digital entertainment games: A systematic review. Computers in Human Behavior, 28 (3), 771–780. https://doi.org/10.1016/j.chb.2011.11.020

Burr, S., Gale, T., Kisielewska, J., Millin, P., Pêgo, J., Pinter, G., Robinson, I., & Zahra, D. (2023). A narrative review of adaptive testing and its application to medical education. MedEdPublish . https://doi.org/10.12688/mep.19844.1

Charsky, D. (2010). From edutainment to serious games: A change in the use of game characteristics. Games and Culture, 5 (2), 177–198. https://doi.org/10.1177/1555412009354727

Chen, F., Cui, Y., & Chu, M.-W. (2020). Utilizing game analytics to inform and validate digital game-based assessment with evidence-centered game design: A case study. International Journal of Artificial Intelligence in Education, 30 (3), 481–503. https://doi.org/10.1007/s40593-020-00202-6

Connolly, T. M., Boyle, E. A., MacArthur, E., Hainey, T., & Boyle, J. M. (2012). A systematic literature review of empirical evidence on computer games and serious games. Computers & Education, 59 (2), 661–686. https://doi.org/10.1016/j.compedu.2012.03.004

Cook, D. A. (2005). The research we still are not doing: An agenda for the study of computer-based learning. Academic Medicine, 80 (6), 541–548. https://doi.org/10.1097/00001888-200506000-00005

Davey, T. (2011). A guide to computer adaptive testing systems . C. o. C. S. S. Officers.

Dede, C. (2009). Immersive interfaces for engagement and learning. Science, 323 (5910), 66–69. https://doi.org/10.1126/science.1167311

Dörner, R., Göbel, S., Effelsberg, W., & Wiemeyer, J. (2016). Introduction. In R. Dörner, S. Göbel, W. Effelsberg, & J. Wiemeyer (Eds.), Serious games: Foundations, concepts and practice (pp. 1–34). Springer.

dos Santos, A. D., & Fraternali, P. (2016). A Comparison of methodological frameworks for digital learning game design. Lecture notes in computer science games and learning alliance. Springer.

Gao, Y., Gonzalez, V. A., & Yiu, T. W. (2019). The effectiveness of traditional tools and computer-aided technologies for health and safety training in the in the construction sector: A systematic review. Computers & Education, 138 , 101–115. https://doi.org/10.1016/j.compedu.2019.05.003

Gorbanev, I., Agudelo-Londoño, S., González, R. A., Cortes, A., Pomares, A., Delgadillo, V., Yepes, F. J., & Muñoz, Ó. (2018). A systematic review of serious games in medical education: Quality of evidence and pedagogical strategy. Medical Education Online, 23 (1), Article 1438718. https://doi.org/10.1080/10872981.2018.1438718

Graafland, M., Schraagen, J. M., & Schijven, M. P. (2012). Systematic review of serious games for medical education and surgical skills training. British Journal of Surgery, 99 (10), 1322–1330. https://doi.org/10.1002/bjs.8819

Gulikers, J. T. M., Bastiaens, T. J., & Kirschner, P. A. (2004). A five-dimensional framework for authentic assessment. Educational Technology Research and Development, 52 (3), 67. https://doi.org/10.1007/BF02504676

Gulikers, J. T. M., Bastiaens, T. J., Kirschner, P. A., & Kester, L. (2008). Authenticity is in the eye of the beholder: Student and teacher perceptions of assessment authenticity. Journal of Vocational Education and Training, 60 (4), 401–412. https://doi.org/10.1080/13636820802591830

Gunter, G. A., Kenny, R. F., & Vick, E. H. (2008). Taking educational games seriously: using the RETAIN model to design endogenous fantasy into standalone educational games. Educational Technology Research and Development, 56 (5), 511–537. https://doi.org/10.1007/s11423-007-9073-2

Harteveld, C. (2011). Foundations. Triadic Game design: balancing reality, meaning and play (pp. 31–93). Springer.

Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77 (1), 81–112. https://doi.org/10.3102/003465430298487

Ifenthaler, D., Eseryel, D., & Ge, X. (2012). Assessment in game-based learning: Foundations, innovations, and perspectives . Springer.

Book   Google Scholar  

Jerrim, J. (2022). Test anxiety: Is it associated with performance in high-stakes examinations? Oxford Review of Education . https://doi.org/10.1080/03054985.2022.2079616

Jones, M. G. (1998). Creating engagement in computer-based learning environments . https://www.yumpu.com/en/document/read/18776351/creating-engagement-in-computer-based-learning-environments

Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 17–64). Praeger Publishers.

Kim, Y. J., & Shute, V. J. (2015). The interplay of game elements with psychometric qualities, learning, and enjoyment in game-based assessment. Computers & Education, 87 , 340–356. https://doi.org/10.1016/j.compedu.2015.07.009

Kirginas, S., & Gouscos, D. (2017). Exploring the impact of freeform gameplay on players’ experience: an experiment with maze games at varying levels of freedom of movement. International Journal of Serious Games . https://doi.org/10.17083/ijsg.v4i4.175

Kirriemur, J., & McFarlane, A. (2004). Literature review in games and learning . Sage.

Krathwohl, D. R. (2002). A revision of Bloom’s taxonomy: An overview. Theory Into Practice, 41 (4), 212–218. https://doi.org/10.1207/s15430421tip4104_2

Lameras, P., Arnab, S., Dunwell, I., Stewart, C., Clarke, S., & Petridis, P. (2017). Essential features of serious games design in higher education: Linking learning attributes to game mechanics. British Journal of Educational Technology, 48 (4), 972–994. https://doi.org/10.1111/bjet.12467

Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33 (1), 159–174.

Lane, S., & Stone, C. A. (2006). Performance assessment. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 387–431). Praeger Publishers.

Malone, T. W. (1981). Toward a theory of intrinsically motivating instruction. Cogntive Science, 4 , 333–369. https://doi.org/10.1207/s15516709cog0504_2

Malone, T. W., & Lepper, M. R. (1987). Making learning fun: A taxonomy of intrinsic motivations for learning. In R. E. Snow & M. J. Farr (Eds.), Aptitude, learning, and instruction: Conative and affective process analyses (pp. 223–253). Lawrence Erlbaum Associates, Inc.

Martin, A. J., & Lazendic, G. (2018). Computer-adaptive testing: Implications for students’ achievement, motivation, engagement, and subjective test experience. Journal of Educational Psychology, 110 , 27–45. https://doi.org/10.1037/edu0000205

Mavridis, A., & Tsiatsos, T. (2017). Game-based assessment: Investigating the impact on test anxiety and exam performance. Journal of Computer Assisted Learning, 33 (2), 137–150. https://doi.org/10.1111/jcal.12170

Messick, S. (1994). Alternative modes of assessment, uniform standards of validity. ETS Research Report Series, 1994 (2), i–22. https://doi.org/10.1002/j.2333-8504.1994.tb01634.x

Michael, D., & Chen, S. (2006). Serious games: Games that educate, train, and inform . Muska & Lipman/Premier-Trade.

Mislevy, R. J., & Riconscente, M. M. (2006). Evidence-centered assessment design. In S. M. Downing & T. M. Haladyna (Eds.), Handbook of test development (pp. 61–90). Lawrence Erlbaum Associates.

Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The, P. G. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLOS Medicine, 6 (7), e1000097. https://doi.org/10.1371/journal.pmed.1000097

Petticrew, M., & Roberts, H. (2005). Systematic reviews in the social sciences: A practical guide. Blackwell Publishing . https://doi.org/10.1002/9780470754887

Prensky, M. (2001). Fun, play and games: What makes games engaging? In M. Prensky (Ed.), Digital game-based learning (pp. 16–47). McGraw-Hill.

Randel, J. M., Morris, B. A., Wetzel, C. D., & Whitehill, B. V. (1992). The effectiveness of games for educational purposes: A review of recent research. Simulation & Gaming, 23 (3), 261–276. https://doi.org/10.1177/1046878192233001

Rouse, R. (2004). Game design: Theory and practice (2nd ed.). Jones and Bartlett Publishers, Inc.

Schell, J. (2015). The art of game design: A book of lenses (2nd ed.). CRC Press.

Schwartz, D. L., & Arena, D. (2013). Measuring what matters most: Choice-based assessment for the digital age . The MIT Press.

Shaffer, D. W., & Gee, J. P. (2012). The right kind of GATE: Computer games and the future of assessment. In M. C. Mayrath, J. Clarke-Midura, & D. H. Robinson (Eds.), Technology-based assessments for 21st century skills: Theoretical and practical implications from modern research (pp. 211–228). Information Age Publishing.

Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78 (1), 153–189. https://doi.org/10.3102/0034654307313795

Shute, V. J., & Ke, F. (2012). Games, learning, and assessment. In D. Ifenthaler, D. Eseryel, & X. Ge (Eds.), Assessment in game-based learning: foundations, innovations, and perspectives (pp. 43–58). Springer.

Shute, V. J., & Rahimi, S. (2021). Stealth assessment of creativity in a physics video game. Computers in Human Behavior, 116 , Article 106647. https://doi.org/10.1016/j.chb.2020.106647

Shute, V. J., Ventura, M., Bauer, M., & Zapata-Rivera, D. (2009). Melding the power of serious games and embedded assessment to monitor and foster learning: Flow and grow. In U. Ritterfeld, M. J. Cody, & P. Vorderer (Eds.), Serious games: Mechanisms and effects (pp. 295–321). Routledge.

Simons, A., Wohlgenannt, I., Weinmann, M., & Fleischer, S. (2021). Good gamers, good managers? A proof-of-concept study with Sid Meier’s Civilization. Review of Managerial Science, 15 (4), 957–990. https://doi.org/10.1007/s11846-020-00378-0

Stecher, B. (2010). Performance assessment in an era of standards-based educational accountability . Stanford University, Stanford Center for Opportunity Policy in Education.

Straetmans, G. J. J. M., & Eggen, T. J. H. M. (2007). WISCAT-pabo: computergestuurd adaptief toetspakket rekenen. Onderwijsinnovatie, 2017 (3), 17–27.

Susi, T., Johannesson, J., & Backlund, P. (2007). Serious game—An overview [IKI Technical Reports] . https://www.diva-portal.org/smash/get/diva2:2416/FULLTEXT01.pdf

The EndNote Team. (2013). EndNote (Version X9) Clarivate. https://endnote.com/

van der Kleij, F. M., Eggen, T. J. H. M., Timmers, C. F., & Veldkamp, B. P. (2012). Effects of feedback in a computer-based assessment for learning. Computers & Education, 58 (1), 263–272. https://doi.org/10.1016/j.compedu.2011.07.020

Van Eck, R. (2006). Digital game-based learning: It’s not just the digital natives who are restless. Educause Review, 41 (2), 16–30.

Vansickle, R. L. (1986). A quantitative review of research on instructional simulation gaming: A twenty-year perspective. Theory & Research in Social Education, 14 (3), 245–264. https://doi.org/10.1080/00933104.1986.10505525

von der Embse, N., Jester, D., Roy, D., & Post, J. (2018). Test anxiety effects, predictors, and correlates: A 30-year meta-analytic review. Journal of Affective Disorders, 483–493 , 132–156. https://doi.org/10.1016/j.jad.2017.11.048

von der Embse, N., & Witmer, S. E. (2014). High-stakes accountability: Student anxiety and large-scale testing. Journal of Applied School Psychology, 30 (2), 132–156. https://doi.org/10.1080/15377903.2014.888529

Wainer, H. (2000). Introduction and history. In H. Wainer, N. J. Dorans, R. Eignor, B. F. Green, R. J. Mislevy, L. Steinberg, & D. Thissen (Eds.), Computerized adaptive testing: A primer (2nd ed., pp. 1–21). Lawrence Erlbaum Associates Inc.

Wang, L., Shute, V., & Moore, G. R. (2015). Lessons learned and best practices of stealth assessment. International Journal of Gaming and Computer-Mediated Simulations, 7 (4), 66–87. https://doi.org/10.4018/ijgcms.2015100104

Wang, R., DeMaria, S., Jr., Goldberg, A., & Katz, D. (2016). A systematic review of serious games in training health care professionals. Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare, 11 (1), 41–51. https://doi.org/10.1097/sih.0000000000000118

Westera, W., Nadolski, R., & Hummel, H. (2014). Serious gaming analytics—What students’ log files tell us about gaming and learning. International Journal of Serious Games, 1 (2), 35–50. https://doi.org/10.17083/ijsg.v1i2.9

Williams-Bell, F. M., Kapralos, B., Hogue, A., Murphy, B. M., & Weckman, E. J. (2015). Using serious games and virtual simulation for training in the fire service: A review. Fire Technology, 51 , 553–584. https://doi.org/10.1007/s10694-014-0398-1

Wools, S., Eggen, T., & Sanders, P. (2010). Evaluation of validity and validation by means of the argument-based approach. Cadmo . https://doi.org/10.3280/cad2010-001007

Wouters, P., van der Spek, E. D., & van Oostendorp, H. (2009). Current practices in serious game research: A review from a learning outcomes perspective. In T. Connolly, M. Stansfield, & L. Boyle (Eds.), Games-based learning advancements for multi-sensory human computer interfaces: Techniques and effective practices (pp. 232–250). IGI Global.

Young, M. F., Slota, S., Cutter, A. B., Jalette, G., Mullin, G., Lai, B., Simeoni, Z., Tran, M., & Yukhymenko, M. (2012). Our princess is in another castle: A review of trends in serious gaming for education. Review of Educational Research, 82 (1), 61–89. https://doi.org/10.3102/0034654312436980

Studies included in the systematic review

Adams, A., Hart, J., Iacovides, I., Beavers, S., Oliveira, M., & Magroudi, M. (2019). Co-created evaluation: Identifying how games support police learning. International Journal of Human-Computer Studies, 132 , 34–44. https://doi.org/10.1016/j.ijhcs.2019.03.009

Aksoy, E. (2019). Comparing the effects on learning outcomes of tablet-based and virtual reality–based serious gaming modules for basic life support training: Randomized trial. JMIR Serious Games, 7 (2), Article e13442. https://doi.org/10.2196/13442

Albert, A., Hallowell, M. R., Kleiner, B., Chen, A., & Golparvar-Fard, M. (2014). Enhancing construction hazard recognition with high-fidelity augmented virtuality. Journal of Construction Engineering and Management, 140 (7), Article 04014024. https://doi.org/10.1061/(ASCE)CO.1943-7862.0000860

Alyami, H., Alawami, M., Lyndon, M., Alyami, M., Coomarasamy, C., Henning, M., Hill, A., & Sundram, F. (2019). Impact of using a 3D visual metaphor serious game to teach history-taking content to medical students: Longitudinal mixed methods pilot study. JMIR Serious Games, 7 (3), Article e13748. https://doi.org/10.2196/13748

Ameerbakhsh, O., Maharaj, S., Hussain, A., & McAdam, B. (2019). A comparison of two methods of using a serious game for teaching marine ecology in a university setting. International Journal of Human-Computer Studies, 127 , 181–189. https://doi.org/10.1016/j.ijhcs.2018.07.004

Asadipour, A., Debattista, K., & Chalmers, A. (2017). Visuohaptic augmented feedback for enhancing motor skill acquisition. The Visual Computer, 33 (4), 401–411. https://doi.org/10.1007/s00371-016-1275-3

Barab, S. A., Scott, B., Siyahhan, S., Goldstone, R., Ingram-Goble, A., Zuiker, S. J., & Warren, S. (2009). Transformational play as a curriculur scaffold: Using videogames to support science education. Journal of Science Education and Technology, 18 (4), 305–320. https://doi.org/10.1007/s10956-009-9171-5

Benda, N. C., Kellogg, K. M., Hoffman, D. J., Fairbanks, R. J., & Auguste, T. (2020). Lessons learned from an evaluation of serious gaming as an alternative to mannequin-based simulation technology: Randomized controlled trial. JMIR Serious Games, 8 (3), Article e21123. https://doi.org/10.2196/21123

Bindoff, I., Ling, T., Bereznicki, L., Westbury, J., Chalmers, L., Peterson, G., & Ollington, R. (2014). A computer simulation of community pharmacy practice for educational use. American Journal of Pharmaceutical Education, 78 (9), Article 168. https://doi.org/10.5688/ajpe789168

Binsubaih, A., Maddock, S., & Romano, D. (2006). A serious game for traffic accident investigators. Interactive Technology and Smart Education, 3 (4), 329–346. https://doi.org/10.1108/17415650680000071

Blanié, A., Amorim, M. A., & Benhamou, D. (2020). Comparative value of a simulation by gaming and a traditional teaching method to improve clinical reasoning skills necessary to detect patient deterioration: A randomized study in nursing students. BMC Medical Education, 20 (1), Article 53. https://doi.org/10.1186/s12909-020-1939-6

Boada, I., Rodriguez-Benitez, A., Garcia-Gonzalez, J. M., Olivet, J., Carreras, V., & Sbert, M. (2015). Using a serious game to complement CPR instruction in a nurse faculty. Computer Methods and Programs in Biomedicine, 122 (2), 282–291. https://doi.org/10.1016/j.cmpb.2015.08.006

Brown, D. E., Moenning, A., Guerlain, S., Turnbull, B., Abel, D., & Meyer, C. (2018). Design and evaluation of an avatar-based cultural training system. The Journal of Defense Modeling and Simulation, 16 (2), 159–174. https://doi.org/10.1177/1548512918807593

Buttussi, F., Pellis, T., Cabas Vidani, A., Pausler, D., Carchietti, E., & Chittaro, L. (2013). Evaluation of a 3D serious game for advanced life support retraining. International Journal Medical Informatics, 82 (9), 798–809. https://doi.org/10.1016/j.ijmedinf.2013.05.007

Calderón, A., Ruiz, M., & O’Connor, R. V. (2018). A serious game to support the ISO 21500 standard education in the context of software project management. Computer Standards & Interfaces, 60 , 80–92. https://doi.org/10.1016/j.csi.2018.04.012

Chan, W. Y., Qin, J., Chui, Y. P., & Heng, P. A. (2012). A serious game for learning ultrasound-guided needle placement skills. IEEE Transactions on Information Technology in Biomedicine, 16 (6), 1032–1042. https://doi.org/10.1109/titb.2012.2204406

Chang, C., Kao, C., Hwang, G., & Lin, F. (2020). From experiencing to critical thinking: A contextual game-based learning approach to improving nursing students’ performance in electrocardiogram training. Educational Technology Research and Development, 68 (3), 1225–1245. https://doi.org/10.1007/s11423-019-09723-x

Chee, E. J. M., Prabhakaran, L., Neo, L. P., Carpio, G. A. C., Tan, A. J. Q., Lee, C. C. S., & Liaw, S. Y. (2019). Play and learn with patients—Designing and evaluating a serious game to enhance nurses’ inhaler teaching techniques: A randomized controlled trial. Games for Health Journal, 8 (3), 187–194. https://doi.org/10.1089/g4h.2018.0073

Chon, S., Timmermann, F., Dratsch, T., Schuelper, N., Plum, P., Berlth, F., Datta, R. R., Schramm, C., Haneder, S., Späth, M. R., Dübbers, M., Kleinert, J., Raupach, T., Bruns, C., & Kleinert, R. (2019). Serious games in surgical medical education: A virtual emergency department as a tool for teaching clinical reasoning to medical students. JMIR Serious Games, 7 (1), Article e13028. https://doi.org/10.2196/13028

Cook, N. F., McAloon, T., O’Neill, P., & Beggs, R. (2012). Impact of a web based interactive simulation game (PULSE) on nursing students’ experience and performance in life support training—A pilot study. Nurse Education Today, 32 (6), 714–720. https://doi.org/10.1016/j.nedt.2011.09.013

Cowley, B., Fantato, M., Jennett, C., Ruskov, M., & Ravaja, N. (2014). Learning when serious: Psychophysiological evaluation of a technology-enhanced learning game. Journal of Educational Technology & Society, 17 (1), 3–16.

Creutzfeldt, J., Hedman, L., & Felländer-Tsai, L. (2012). Effects of pre-training using serious game technology on CPR performance—An exploratory quasi-experimental transfer study. Scandinavian Journal of Trauma, Resuscitation and Emergency Medicine, 20 (1), Article 79. https://doi.org/10.1186/1757-7241-20-79

Creutzfeldt, J., Hedman, L., Medin, C., Heinrichs, W. L., & Felländer-Tsai, L. (2010). Exploring virtual worlds for scenario-based repeated team training of cardiopulmonary resuscitation in medical students. Journal of Medical Internet Research, 12 (3), Article e38. https://doi.org/10.2196/jmir.1426

Dankbaar, M. E. W., Alsma, J., Jansen, E. E. H., van Merrienboer, J. J. G., van Saase, J. L. C. M., & Schuit, S. C. E. (2016). An experimental study on the effects of a simulation game on students’ clinical cognitive skills and motivation. Advances in Health Sciences Education, 21 (3), 505–521. https://doi.org/10.1007/s10459-015-9641-x

Dankbaar, M. E. W., Bakhuys Roozeboom, M., Oprins, E. A. P. B., Rutten, F., van Merrienboer, J. J. G., van Saase, J. L. C. M., & Schuit, S. C. E. (2017a). Preparing residents effectively in emergency skills training with a serious game. Simulation in Healthcare, 12 (1), 9–16. https://doi.org/10.1097/sih.0000000000000194

Dankbaar, M. E. W., Richters, O., Kalkman, C. J., Prins, G., ten Cate, O. T. J., van Merrienboer, J. J. G., & Schuit, S. C. E. (2017). Comparative effectiveness of a serious game and an e-module to support patient safety knowledge and awareness. BMC Medical Education, 17 (1), Article 30. https://doi.org/10.1186/s12909-016-0836-5

de Sena, D. P., Fabrício, D. D., da Silva, V. D., Bodanese, L. C., & Franco, A. R. (2019). Comparative evaluation of video-based on-line course versus serious game for training medical students in cardiopulmonary resuscitation: A randomised trial. PLOS ONE, 14 (4), Article e0214722. https://doi.org/10.1371/journal.pone.0214722

Dib, H., & Adamo-Villani, N. (2014). Serious sustainability challenge game to promote teaching and learning of building sustainability. Journal of Computing in Civil Engineering, 28 (5), Article A4014007. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000357

Diehl, L. A., Souza, R. M., Gordan, P. A., Esteves, R. Z., & Coelho, I. C. M. (2017). InsuOnline, an electronic game for medical education on insulin therapy: A randomized controlled trial with primary care physicians. Journal of Medical Internet Research, 19 (3), Article e72. https://doi.org/10.2196/jmir.6944

Drummond, D., Delval, P., Abdenouri, S., Truchot, J., Ceccaldi, P., Plaisance, P., Hadchouel, A., & Tesnière, A. (2017). Serious game versus online course for pretraining medical students before a simulation-based mastery learning course on cardiopulmonary resuscitation: A randomised controlled study. European Journal of Anaesthesiology, 34 (12), 836–844. https://doi.org/10.1097/EJA.0000000000000675

Duque, G., Fung, S., Mallet, L., Posel, N., & Fleiszer, D. (2008). Learning while having fun: The use of video gaming to teach geriatric house calls to medical students. Journal of the American Geriatrics Society, 56 (7), 1328–1332. https://doi.org/10.1111/j.1532-5415.2008.01759.x

Fonteneau, T., Billion, E., Abdoul, C., Le, S., Hadchouel, A., & Drummond, D. (2020). Simulation game versus multiple choice questionnaire to assess the clinical competence of medical students: Prospective sequential trial. Journal of Medical Internet Research, 22 (12), Article e23254. https://doi.org/10.2196/23254

Gerard, J. M., Scalzo, A. J., Borgman, M. A., Watson, C. M., Byrnes, C. E., Chang, T. P., Auerbach, M., Kessler, D. O., Feldman, B. L., Payne, B. S., Nibras, S., Chokshi, R. K., & Lopreiato, J. O. (2018). Validity evidence for a serious game to assess performance on critical pediatric emergency medicine scenarios. Simulation in Healthcare, 13 (3), 168–180. https://doi.org/10.1097/SIH.0000000000000283

Graafland, M., Bemelman, W. A., & Schijven, M. P. (2014). Prospective cohort study on surgeons’ response to equipment failure in the laparoscopic environment. Surgical Endoscopy, 28 (9), 2695–2701. https://doi.org/10.1007/s00464-014-3530-x

Graafland, M., Bemelman, W. A., & Schijven, M. P. (2017). Game-based training improves the surgeon’s situational awareness in the operation room: A randomized controlled trial. Surgical Endoscopy, 31 (10), 4093–4101. https://doi.org/10.1007/s00464-017-5456-6

Hannig, A., Lemos, M., Spreckelsen, C., Ohnesorge-Radtke, U., & Rafai, N. (2013). Skills-O-Mat: Computer supported interactive motion- and game-based training in mixing alginate in dental education. Journal of Educational Computing Research, 48 (3), 315–343. https://doi.org/10.2190/EC.48.3.c

Hummel, H. G. K., van Houcke, J., Nadolski, R. J., van der Hiele, T., Kurvers, H., & Löhr, A. (2011). Scripted collaboration in serious gaming for complex learning: Effects of multiple perspectives when acquiring water management skills. British Journal of Educational Technology, 42 (6), 1029–1041. https://doi.org/10.1111/j.1467-8535.2010.01122.x

Jalink, M. B., Goris, J., Heineman, E., Pierie, J. P., & ten Cate Hoedemaker, H. O. (2014). Construct and concurrent validity of a Nintendo Wii video game made for training basic laparoscopic skills. Surgical Endoscopy, 28 (2), 537–542. https://doi.org/10.1007/s00464-013-3199-6

Katz, D., Zerillo, J., Kim, S., Hill, B., Wang, R., Goldberg, A., & DeMaria, S. (2017). Serious gaming for orthotopic liver transplant anesthesiology: A randomized control trial. Liver Transplantation, 23 (4), 430–439. https://doi.org/10.1002/lt.24732

Knight, J. F., Carley, S., Tregunna, B., Jarvis, S., Smithies, R., de Freitas, S., Dunwell, I., & Mackway-Jones, K. (2010). Serious gaming technology in major incident triage training: A pragmatic controlled trial. Resuscitation, 81 (9), 1175–1179. https://doi.org/10.1016/j.resuscitation.2010.03.042

LeFlore, J. L., Anderson, M., Zielke, M. A., Nelson, K. A., Thomas, P. E., Hardee, G., & John, L. D. (2012). Can a virtual patient trainer teach student nurses how to save lives—Teaching student nurses about pediatric respiratory diseases. Simulation in Healthcare, 7 (1), 10–17. https://doi.org/10.1097/SIH.0b013e31823652de

Li, K., Hall, M., Bermell-Garcia, P., Alcock, J., Tiwari, A., & González-Franco, M. (2017). Measuring the learning effectiveness of serious gaming for training of complex manufacturing tasks. Simulation & Gaming, 48 (6), 770–790. https://doi.org/10.1177/1046878117739929

Luu, C., Talbot, T. B., Fung, C. C., Ben-Isaac, E., Espinoza, J., Fischer, S., Cho, C. S., Sargsyan, M., Korand, S., & Chang, T. P. (2020). Development and performance assessment of a digital serious game to assess multi-patient care skills in a simulated pediatric emergency department. Simulation & Gaming, 51 (4), 550–570. https://doi.org/10.1177/1046878120904984

Middeke, A., Anders, S., Schuelper, M., Raupach, T., & Schuelper, N. (2018). Training of clinical reasoning with a serious game versus small-group problem-based learning: A prospective study. PLoS ONE, 13 (9), Article e0203851. https://doi.org/10.1371/journal.pone.0203851

Miller, C. H., Dunbar, N. E., Jensen, M. L., Massey, Z. B., Lee, Y., Nicholls, S. B., Anderson, C., Adams, A. S., Cecena, J. E., Thompson, W. M., & Wilson, S. N. (2019). Training law enforcement officers to identify reliable deception cues with a serious digital game. International Journal of Game-Based Learning, 9 (3), 1–22. https://doi.org/10.4018/IJGBL.2019070101

Mohan, D., Angus, D. C., Ricketts, D., Farris, C., Fischhoff, B., Rosengart, M. R., Yealy, D. M., & Barnato, A. E. (2014). Assessing the validity of using serious game technology to analyze physician decision making. PLOS ONE, 9 (8), Article e105445. https://doi.org/10.1371/journal.pone.0105445

Mohan, D., Farris, C., Fischhoff, B., Rosengart, M. R., Angus, D. C., Yealy, D. M., Wallace, D. J., & Barnato, A. E. (2017). Efficacy of educational video game versus traditional educational apps at improving physician decision making in trauma triage: Randomized controlled trial. BMJ, 359 , Article j5416. https://doi.org/10.1136/bmj.j5416

Mohan, D., Fischhoff, B., Angus, D. C., Rosengart, M. R., Wallace, D. J., Yealy, D. M., Farris, C., Chang, C. H., Kerti, S., & Barnato, A. E. (2018). Serious games may improve physician heuristics in trauma triage. Proceedings of the National Academy of Sciences, 115 (37), 9204–9209. https://doi.org/10.1073/pnas.1805450115

Moreno-Ger, P., Torrente, J., Bustamante, J., Fernandez-Galaz, C., Fernandez-Manjon, B., & Comas-Rengifo, M. D. (2010). Application of a low-cost web-based simulation to improve students’ practical skills in medical education. International Journal of Medical Informatics, 79 (6), 459–467. https://doi.org/10.1016/j.ijmedinf.2010.01.017

Perini, S., Luglietti, R., Margoudi, M., Oliveira, M., & Taisch, M. (2018). Learning and motivational effects of digital game-based learning (DGBL) for manufacturing education—The life cycle assessment (LCA) game. Computers in Industry, 102 , 40–49. https://doi.org/10.1016/j.compind.2018.08.005

Phungoen, P., Promto, S., Chanthawatthanarak, S., Maneepong, S., Apiratwarakul, K., Kotruchin, P., & Mitsungnern, T. (2020). Precourse preparation using a serious smartphone game on advanced life support knowledge and skills: Randomized controlled trial. Journal of Medical Internet Research, 22 (3), Article e16987. https://doi.org/10.2196/16987

Steinrücke, J., Veldkamp, B. P., & de Jong, T. (2020). Information literacy skills assessment in digital crisis management training for the safety domain: Developing an unobtrusive method. Frontiers in Education, 5 (140), Article 140. https://doi.org/10.3389/feduc.2020.00140

Su, C. (2016). The efects of students’ learning anxiety and motivation on the learning achievement in the activity theory based gamified learning environment. Eurasia Journal of Mathematics, Science and Technology Education, 13 , 1229–1258. https://doi.org/10.12973/eurasia.2017.00669a

Taillandier, F., & Adam, C. (2018). Games ready to use: A serious game for teaching natural risk management. Simulation & Gaming, 49 (4), 441–470. https://doi.org/10.1177/1046878118770217

Tan, A. J. Q., Lee, C. C. S., Lin, P. Y., Cooper, S., Lau, L. S. T., Chua, W. L., & Liaw, S. Y. (2017). Designing and evaluating the effectiveness of a serious game for safe administration of blood transfusion: A randomized controlled trial. Nurse Education Today, 55 , 38–44. https://doi.org/10.1016/j.nedt.2017.04.027

Zualkernan, I. A., Husseini, G. A., Loughlin, K. F., Mohebzada, J. G., & El Gami, M. (2013). Remote labs and game-based learning for process control. Chemical Engineering Education, 47 (3), 179–188.

Download references

Author information

Authors and affiliations.

eX:plain, Department of Applied Research, P.O. Box 1230, 3800 BE, Amersfoort, The Netherlands

Aranka Bijl

Faculty of Behavioural, Management and Social Sciences, Cognition, Data and Education, University of Twente, P.O. Box 217, 7500 AE, Enschede, The Netherlands

Aranka Bijl & Bernard P. Veldkamp

Cito, Department of Research and Innovation, P.O. Box 1034, 6801 MG, Arnhem, The Netherlands

Aranka Bijl, Saskia Wools & Sebastiaan de Klerk

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Aranka Bijl .

Ethics declarations

Conflict of interest.

We have no conflict of interest to disclose.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 42 kb)

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Bijl, A., Veldkamp, B.P., Wools, S. et al. Serious games in high-stakes assessment contexts: a systematic literature review into the game design principles for valid game-based performance assessment. Education Tech Research Dev (2024). https://doi.org/10.1007/s11423-024-10362-0

Download citation

Accepted : 24 February 2024

Published : 08 April 2024

DOI : https://doi.org/10.1007/s11423-024-10362-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Systematic literature review
• Serious games
• Professional competencies
• Performance assessment
• Game design principles
• Find a journal
• Publish with us
• Track your research