Khan Academy Study by SRI Reviews Implementation and Not Outcomes

We are often asked: do online and blended learning work? As we have explained several times (e.g. in Keeping Pace 2011, pp 40-49), the question is more difficult than it might appear for at least two reasons.

First, defining “what works” is not easy. If “working” is defined as providing new opportunities to students, then the answer is undoubtedly yes. If it is defined as improving student outcomes in some cases, then again the answer is yes, as with students taking online Advanced Placement courses and in some programs exceeding national averages for AP exam scores. But if “working” is defined as raising student achievement across large scales, based on randomized control trials, the answer is that the research is limited and inconclusive.

The second reason that answering the “does it work” question is hard is because so many types of online and blended learning exist. The result is that comprehensive research is difficult and expensive, and applies to only a very small percentage of overall online and blended learning implementations.

The recent report from SRI looking at Khan Academy use in schools supports and amplifies the above points:

“SRI conducted an implementation study rather than an evaluation of Khan Academy’s impact on learning, given that implementation of Khan Academy varied across sites, Khan Academy tools and resources evolved over time, and schools, Khan Academy, and researchers collaborated regularly. Ongoing communications and tight relationships between the participating teachers, students, and school leaders with Khan Academy and SRI International resulted in both a more useful tool for teachers and students, and a better understanding of the various ways Khan Academy can be used in school settings to promote student learning.

…teachers and students were using Khan Academy tools and resources in considerably different ways across the nine study sites, and some of the sites also changed the ways they used it during the course of the two-year study. For these reasons, it was methodologically unsound to conduct a rigorous evaluation of Khan Academy’s impact on learning during the study period, including any use of randomized control trials, which would have required Khan Academy tools and resources to remain unchanged during the study and for teachers and students to use Khan Academy the same way. Moreover, at all but one of the sites, Khan Academy was principally used as a supplementary tool—not as the core primary curriculum—so the effects of Khan Academy cannot be separated from those contributed by other elements of the math curriculum.” (emphasis added)

While the above text may sound like the study did not yield useful results because it did not come to any conclusion about Khan’s impact on learning outcomes, the study does in fact contain practical information. We will review these findings in the next blog post.


Khan Academy and Implications of Teacher Control

Among the challenges in implementing blended learning into existing schools is that, in most cases, teachers view their classrooms as their domain. Keeping Pace research suggests that successful blended learning implementations are almost always at the school level (or higher, e.g. district level or consortium level). The school-level approach to blended learning, however, collides with the view that teachers own what occurs in their classrooms, because school-wide blended learning implementations often select content, technology platforms, end-user devices, instructional models, and other elements of instruction that teachers are often accustomed to controlling within the classrooms.

Larry Cuban’s recent blog post “Business as Usual in Corporations and Schools” makes a similar point, within a larger (and well worth reading) discussion of power, status, and hierarchy in companies and in schools. Cuban is not an uncritical advocate for technology in education, and often questions people who suggest that online and blended learning are already having a large-scale impact on teaching and learning. He is among the realists who see things as they are, and not as they wish them to be. Speaking of power and hierarchies in schools, he says:

“With all of the rules and hierarchical levels from classroom through the state superintendent of education, teachers have one thing in their power to do: close the classroom door. They are (and have been) gatekeepers for student content, skills, and attitudes…”

Although the end of the quote above may be a slight overstatement—because content is selected and skills determined by schools, districts, and states—it captures the view that teachers do expect a high level of control over their classrooms.

After a brief discussion about “the talk of moving to project-based learning and shifting the teacher’s role from the sage-on-the-state to the guide-on-the-side,” he concludes with this:

“On occasion, some of the reforms have stuck in some schools where teachers weathered criticism and supported one another through cascades of hype and criticism. In these scattered instances, teachers kept their doors open and built a stable school culture supporting…instructional reforms…[including] rich student-centered activities. But not most teachers who returned time and again to practices that worked better for them than the “new” reform simply because they could close their classroom doors.

These are (and have been) abiding features of public schools and companies that no amount of talk and hype about doing business differently has changed.”

Cuban’s comments came to mind when I read these sentences in the SRI report on Khan Academy implementation:

“Even though Khan Academy is primarily known for its video library and has been associated with the “flipped” classroom model (i.e., teachers assign students videos about new concepts to watch as homework, and use class time to extend the video lectures with discussion and interactive activities), teachers participating in the research were more focused on exploring how online, personalized practice opportunities for students could be incorporated into their existing instructional activities.” (emphasis added)

Khan Academy is widely used (10 million unique users per month as of February 2014, with about 65% of users from the United States, according to SRI), and it is undoubtedly helping many of those students. But whether any of the school-level implementations of Khan are game-changing has not yet been determined, as we will discuss in forthcoming posts. To the extent that usage in schools continues to be built around existing instructional activities, there is reason to doubt that outcomes will be significantly improved.

Michigan study looks at effectiveness of online learning

The Virtual Learning Research Institute, a division of the Michigan Virtual University, has released Michigan’s K-12 virtual learning effectiveness report. With this report Michigan joins the ranks of the few states (others include California and Colorado) that have released comprehensive reports about online and/or blended learning activity.

The key findings “include an apparent growth in the number of students and schools participating in virtual courses, with the majority of virtual enrollments coming in the core subject areas. Students taking virtual courses in a supplemental capacity appear to be more successful when they take only a few virtual enrollments a year. Developing practices to better support students who take higher amounts of virtual enrollments should be a priority.”

An important caveat is included that reflects the state of digital learning data: “Statistics shared in the report must be interpreted with care due to concerns about the accuracy of data reported to the state about virtual enrollments during the first three years of its collection.” Specifically, “MVU believes that the accuracy of the virtual delivery flag tied to student enrollments through the Michigan Student Data System Teacher-Student Data Link is less than optimal.”

Despite the data concerns, the information is valuable. MVU finds that 55,271 students took an online course in school year 2012-13. That number reflects an increase of 52% over the past two years, but with almost all of that increase taking place between school year 2010-11 and school year 2011-12. It is unclear if the greatly reduced growth in the past year is real or is a result of inaccurate data. In school year 2012-13 students took a total of 185,053 course enrollments, which more than doubled the number from two years prior. This number represents about 1% of all course enrollments in the state.

About “90% of the virtual enrollments each year came from students in grades 9-12,” and core subjects accounted for most of the enrollments. “Mathematics consistently accounted for the largest percentage of virtual enrollments with approximately 20% of virtual enrollments each of the past three years. Other core subjects like English Language and Literature, Social Sciences and History, and Life and Physical Sciences…equaled or exceeded 10% of the virtual enrollments for a school year.

The discussion of completion rates and impacts is too complex to easily summarize, but the conclusion is worth quoting:

“On the one hand, there is evidence within this report that may lead some to claim that K-12 virtual learning simply is not working in Michigan. Detractors could cite lower completion rates for virtual enrollments, or they could focus on the finding that the results for students in poverty are not on par with students who are not. On the other hand, there is evidence that virtual learning is clearly working. Proponents could cite that about 40% of Michigan schools had an 80% or higher completion rate for their virtual enrollments. Or that students who take one or two virtual courses a year have a completion rate of almost 70%. Both statistics seem like even more significant accomplishments given that the data indicate schools tend to limit virtual learning options for students, seeing it more as a credit recovery option than as an initial credit solution.

Regardless, the findings presented here are not intended to further polarize along the lines of virtual learning either working or not working, but rather to aid in understanding under what conditions virtual learning can work and in doing so, with an understanding of the current educational climate and educational demands of the 21st century, change the collective mindset from “if” to “how.” Existing research indicates, and the research team knows from experience, that virtual learning is far more likely to yield the desired results when the course content is high-quality, the virtual instructor is skilled at teaching online, and the student has wrap-around support including active local mentors and parents.”



Indiana leaves Common Core

We have previously discussed our support for Common Core and our concern about states considering dropping the content standards and/or the associated national assessments. Now comes the news that Indiana has pulled out of Common Core, and several other states are considering doing the same. As the Washington Post reports:

“Indiana Gov. Mike Pence (R)…signed new legislation that makes his state the first to opt out of Common Core standards, amid conservative outrage over the education benchmarks. It’s the first major victory for opponents of Common Core…”

Although this is a disappointing development, the Post article explores the fine line that it believes some governors are trying to walk:

“the new law doesn’t mean Indiana students will be learning anything dramatically different from kids in neighboring states. The legislation strikes references to Common Core and requires the state board of education to adopt what it calls “college and career readiness” standards that meet national and international benchmarks and comply with federal standards while maintaining Indiana’s sovereignty…Some critics of the board of education say the standards they are developing hew too closely to Common Core.

“That’s similar to the approach several other states are taking: Pass standards nearly identical to Common Core, but under a different name. An Oklahoma state Senate committee on Monday passed a version that would strip the Common Core name while leaving many or most of the same requirements intact.

“You’ve got these governors who understand the business argument for keeping Common Core,” said Michael Petrilli, executive vice president of the Thomas B. Fordham Institute, an education policy think tank. “But they’ve got this tea party base that’s really fired up about this issue, so they’re trying to find a way to walk this fine line by giving voice to the tea party concerns without backing away from higher standards.

“The Indiana bill, which began its life as a straight repeal of Common Core standards, was changed so much and left so many of the common requirements intact that the original author, state Sen. Scott Schneider (R), pulled his name and voted against the final version.”


“Education Secretary Arne Duncan, who has been supportive of the new rules, didn’t help the cause when he said in November that opposition comes from “white suburban moms who [find]— all of a sudden — their child isn’t as brilliant as they thought they were, and their school isn’t quite as good as they thought they were.” Duncan later apologized for his remarks.”

That last quote is a reminder that a gaffe in Washington DC is often defined as accidentally telling the truth. Although it was a mistake for Duncan to bring race into the equation, we agree that quite a bit of the opposition to the Common Core and national assessments is coming from educators and parents who are concerned that the new approach will show their students not performing as well as they thought.

Panel Discussion on Saturday April 5 at National School Boards Association Conference

If you are attending the National School Boards Association conference in New Orleans this weekend (April 5-7), please come join a panel discussion that I’ll be moderating at 3:45 Saturday afternoon titled “Planning for Quality in Your Online/Blended Learning Program.” Here’s the description:

These days it seems every school district wants its own online or blended learning program – if only to compete with charters and statewide schools. To do online/blended learning right, though, districts need to start with clear educational goals and then line up the specific curriculum, instructional, and training resources they need to achieve these goals. Learn how it’s done from a cross-country panel of district leaders.

I’ll be joined by Kelly Schwirzke of the Santa Cruz County Office of Education (CA), Kim Loomis of Clark County School District Virtual High School (Las Vegas, Nevada), and LaTanya White of the Pottstown School District (PA).

WordPress SEO fine-tune by Meta SEO Pack from Poradnik Webmastera