Harvard and MIT release working papers on open online learning

Harvard University and the Massachusetts Institute of Technology (MIT) today released a series of working papers based on 17 online courses offered on the edX platform. Run in 2012 and 2013, the courses drew upon diverse topics — from ancient Greek poetry to electromagnetism — and an array of disciplines, including public health, engineering, and law.

The series features detailed reports about individual courses; these reports reveal differences and commonalities among massive open online courses (MOOCs). In the coming weeks, data sets and interactive visualization tools will also be made available.

Led by Andrew Ho of Harvard’s Graduate School of Education and Isaac Chuang of MIT’s electrical engineering and computer science and physics departments, the effort was in service of a mutual goal — “to research how students learn and how technologies can facilitate effective teaching both on-campus and online” — part of a mission statement established when MIT and Harvard joined to form edX, a nonprofit online learning platform, in May 2012.

The papers analyze an average of 20 gigabytes of data per course and draw on interviews with faculty and course teams as well as student metrics.

Key takeaways

Course completion rates, often seen as a bellwether for MOOCs, can be misleading and may at times be counterproductive indicators of the impact and potential of open online courses.

The researchers found evidence of large numbers of registrants who may not have completed a course but still accessed substantial amounts of course content. Across the 17 HarvardX and MITx courses covered in the reports, 43,196 registrants earned certificates of completion. Additionally, 35,937 registrants explored half or more of the units in a course without achieving certification.

“We found students in the courses who engaged with every single piece of the courseware, students who only read text or viewed videos, students who only took assessments or completed problem sets, and students representing nearly every possible combination of these behaviors,” said Chuang. “Experimentation is part of the learning process.”

An additional 469,702 registrants viewed less than half of the units in a course and another 292,852 registrants never engaged with the online content. In total, there were 841,687 registrations from 597,692 unique users.

The reports note that many faculty encouraged learners to participate in the course experiences in whatever ways they found useful, whether or not that involved trying to earn a certificate. For such faculty, the participation of engaged auditors was just as important as the participation of those who completed the assessments of the course.

“A fixation on completion rates limits our imagination of what might be possible with MOOCs. A better criterion for success might be for students to complete more of the course than they thought they would, or to learn more than they might have expected when they first clicked on a video or course forum,” said Ho.

Most MOOC attrition happened after students first registered for a course. On average, 50 percent of people left within a week or two of enrolling. After that window, attrition rates decreased substantially. The average probability of a student ceasing to engage in the second week of the course declined to 16 percent.

While the persistence rates in MOOCs look very different from those of conventional courses in higher education, they look very similar to how people interact with other Web-based media such as video or social network sites. Many registrants briefly browse a MOOC, while a smaller group persists and becomes more engaged over time.

The asynchronous nature of the HarvardX and MITx courses also turned out to be an important factor related to engagement. Some students registered for courses months in advance of the start date; others registered months after the course had launched.

In some circumstances, students joined courses in the week before the deadline for all graded materials, completing enough assessments to earn a certificate in the final seven days. In some courses, as many as one in five enrollments took place after the course certificate was no longer available.

Given the scale of some MOOCs, small percentages are often still large numbers of students — and signify a potentially large impact.

Demographic information about registrants can be misleading without context. The most typical registrant in these initial courses was a male with a bachelor’s degree, 26 or older. However, that profile describes less than one in three registrants (222,847, or 31 percent).

A total of 234,463 (33 percent) registrants reported a high school education or lower; 45,884 (6.3 percent) reported that they were 50 or older; and 20,745 (2.7 percent) had IP or mailing addresses from countries on the United Nations’ list of least-developed countries.

“While typical MOOC registrants have a college degree already, hundreds of thousands of our registrants do not. Many of our MOOC registrants are from the United States, but 72 percent are from abroad. These MOOCs are reaching many nontraditional and underserved communities of students, very different from typical students on campuses at traditional universities,” said Chuang.

One of the most interesting subpopulations: the group of people who enroll in and complete multiple courses. More than 4,000 registrants across HarvardX and MITx earned more than one certificate, including 1,912 who earned at least one certificate from each institution. Seventy-six registrants earned five or more certificates.

“Context is so important to understanding these results,” added Ho. “A graduate-level course in public health should attract different registrants, on average, than an introductory physics course, or a course on global poverty. In this light, it’s wonderful that so many students are engaging so deeply in courses across topics and institutions.”

Forward thinking

Looking ahead, Chuang and Ho view the publication of the working papers, and the open sharing of such data, as a key activity in a broad, positive dialogue happening in higher education.

Specifically, they emphasize that the rise of MOOCs has sparked and encouraged experimentation in teaching and in pedagogical research, benefiting both teachers and students. New tools, they say, give faculty more flexibility and offer novel opportunities to run experiments and gather data. Likewise, online-learning platforms put students in the driver’s seat, allowing them engage in a manner that suits individual needs.
“This isn’t just about MOOCs,” said Ho. “This is about the democratization of learning: Learners are in control. We are at the beginning of an exciting effort to understand how people learn and how to educate well and effectively at scale.”

Finally, the researchers expect the research to serve as a benchmark for future studies on MOOCs and support further work in the science of learning.

“The story hidden underneath this series of reports may be this: Institutions like ours are coming to appreciate how cross-institutional educational collaborations involving many students and many courses can open new routes to understand and improve student learning — making a difference around the world and back here on campus,” said Chuang.

The lead authors of the HarvardX and MITx course reports were Justin Reich, Richard L. Menschel HarvardX Research Fellow, and Daniel T. Seaton, postdoctoral research fellow at the Office of Digital Learning at MIT.

An event from 6 to 7:30 tonight at HGSE will feature discussion of the reports among faculty and researchers. It will be live-streamed.

To learn more:

• Course Reports
• Pilot Data Visualizations (created by Sergiy Nesterko, HarvardX Research Fellow)
• Courses on edX.org