synthesis of Dept of Ed study

The Dept of Ed report is a meta- analysis

This type of study s is defined as a statistical technique for amalgamating, summarizing, and reviewing previous quantitative research. A meta-analysis provides a systematic overview of quantitative research which has examined a particular question. This type of study combines all the research on one topic into one large study with many participants. The danger is that in amalgamating a large set of different studies the construct definitions can become imprecise and the results difficult to interpret meaningfully.

“A meta-analysis is a two-stage process. The first stage is the extraction of data from each individual study and the calculation of a result for each individual study (the “point estimate” or “summary statistic”) with an estimate of the chance variation we would expect with studies like that (the “confidence interval”).

The second stage involves deciding whether it is appropriate to calculate a pooled average result across studies and, if so, calculating and presenting such a result. Part of this process is to give greater weight to the results from studies that give us more information, because these are likely to be closer to the truth.” (Singapore Med Journal, 2005, Vol 46(6) pp 270, Evidence Based Medicine and Healthcare, S Green)


The Department of Education study included the following:

1. A search of journals that returned 1132 studies
2. Resulted in a return of 176 online learning research studies that used an experimental quasi experimental design that objectively measured student learning outcomes.
3. Of these 99 had a face-to-face contrast component 9 of these involved K-12.
4. Of the 99-- 46 proved to have sufficient data for the study.


The 46 studies were:
1. Published in or after 2004.
2. Included 7 contrasts from five studies that included K-12 learners
3. The rest were evenly split between community college and graduates students or adults receiving professional training.
4. Demonstrated weakness in the studies were small sample sizes, failure to report retention rates, sample size is positive for undergraduate and older learners but not K-12.

The study sought to answer the following:


1. How does the effectiveness of online learning compare with that of face-to-face instruction?
2. Does supplementing face-to-face instruction with online instruction enhance learning?
3. What practices are associated with more effective online learning?
4. What conditions influence the effectiveness of online learning?


What is hidden in the study?

1. The study does not present that online is superior as a medium of delivery

2. The study does represent that the time spent on task, curriculum and pedagogy of online learning maybe critical variables.

3. The study does support the redesign of instruction to incorporate additional learning opportunities online.

4. The study found there was no significant effect from using different media in presenting instruction.


The study identified the following 10 practices, which are intervention implementation :


1. Pedagogy/learning experience
2. Computer mediated communication with instructor
3. Treatment duration
4. Media features
5. Time on task
6. One-way video or audio
7. Computer based instruction elements
8. Opportunity for face to face with the instructor
9. Opportunity for practice
10. Feedback provided

What conditions, which are status variables that are fairly impervious to external factors, that influence the effectiveness of online learning?

1. The year of the study
2. Learning type:
a. K-12 students
b. Undergraduate
c. Graduate student /other
3. Subject matter:
a. Medical/health care
b. Other



Further observations:


Blended Compared with Pure Online Learning

The studies reviewed did not provide a basis for determining the benefits of online versus blended learning modalities or conditions.

Media Elements


The fact that the majority of studies found no significant difference across media types is consistent with the theoretical position that the medium is simply a carrier of content and is unlikely to affect learning per se (Clark 1983, 1994).

....that the way in which a medium is used is more important than merely having access to it ...


“In summary, many researchers have hypothesized that the addition of images, graphics, audio, video or some combination would enhance student learning and positively affect achievement. However, the majority of studies to date have found that these media features do not affect learning outcomes significantly.”

Learning Experience Type

..the power of collaborative learning ...


…offered by the author is that students working in groups may scaffold each other’s work, hence reducing the benefit derived from externally provided questions…

Computer-Based Instruction

The study found the platform used to deliver quizzes and course content did have some impact on student success ...


The results of three studies exploring the effects of including different types of online simulations were modestly positive. Two of the studies indicated a positive effect from including an online simulation; however, one study found no significant difference ...this does have some implications for simulated learning environments in virtual worlds.



...online learning literature has also explored the effects of using computer-based instruction elements to individualize instruction so that the online learning module or platform responds dynamically to the participant’s questions, needs or performance ....


Supports for Learner Reflection ....


Overall, the available research evidence suggests that promoting self-reflection, self-regulation and self-monitoring leads to more positive online learning outcomes. Features such as prompts for reflection, self-explanation and self-monitoring strategies have shown promise for improving online learning outcomes


Moderating Online Groups ..

Organizations providing or promoting online learning generally recommend the use of instructors or other adults as online moderators, but research support for the effects of this practice on student learning is mixed.


Scripts for Online Interaction ....

..mechanisms such as scaffolds or scripts for student group interaction online have been found to influence the way students engage with each other and with the online material, but have not been found to improve learning…