How to Evaluate a MOOC

February 27, 2013 | Blog

Since we’re already doggie-paddling around the deep-end of the MOOC phenomenon, it seems high time to begin thinking about how to assess MOOCs.  This is far from straightforward, since they are not typical courses and might best be conceived of as extension or continuing education activities.

Very few colleges and universities are revealing their assessment practices for MOOCs; Duke is one exception.  This report explains that selective university’s experiences with one course, and demonstrates how their administrators and faculty thought about measuring success. In my view, some of it is good, and some of it is exactly what not to do.

So here is my overview of an initial evaluative framework for thinking about whether or not a MOOC was a “success.” I’m posting this with the explicit intention of generating discussion to refine this framework and build it out, so please add your two cents.

The framework has three components: an assessment of costs expended to generate and implement the MOOC, an assessment of its intended impacts in multiple domains, and as assessment of unintended consequences.  Only by considering all three can one make an informed decision about whether a MOOC was worthwhile, yet reporting on MOOCs suggests that many institutions are focusing on only one or two of the three.

  1. COSTS
    1. How many total hours of effort were required to build the MOOC?
      1. How many of these were on the part of the faculty member?
      2. How many involved TA time?
      3. How many involved staff time?
    2. What technological costs were incurred?
      1. How much was spent to get the MOOC off the ground?
      2. What are the regular maintenance costs?
      3. What are the costs of upgrading the MOOC?
    3. How many hours of administrative time were devoted to meetings about the MOOC? 
      1. How much time and money is spent advertising the MOOC?
      2. How much time will be required to regulate it?
      3. How much time will be required to assess its outcomes?
  2. IMPACTS
    1. On Students.  
      1. How many students initially enrolled in the MOOC?
      2. How many of these were students were not already enrolled in the university?
      3. How many countries are represented among MOOC users?
        1. How many of these countries were previously not served by the university, or represent new relationships?
        2. How many of these countries are home to under-educated populations?
      4. How many of these students had no college experience prior to the MOOC?
      5. How many of these students already held a college degree of any kind?
      6. What percent of the total initial enrollment completed the first week of the course?
      7. What percent of the total initial enrollment completed the first assessment?
      8. What percent of the total initial enrollment completed the entire course?
    2. On Faculty.  
      1. What was the professor’s level of satisfaction with the course, as compared to satisfaction of teaching a usual university course?
      2. What new contacts for the professor resulted from the MOOC, related to his/her research agenda?
      3. How did teaching the MOOC inform the professor’s research agenda?
      4. How did teaching the MOOC change how the professor approaches his/her regular university courses?
      5. Is there any evidence that the professor and/or his/her research gained more recognition in the community where MOOC students came from? (e.g. media mentions, etc)
  3. SIDE EFFECTS
    1. How many course releases (from campus teaching) were required for the professor to teach the MOOC, and how many students on average are taught per course (typically)?
    2. How many press mentions occurred regarding the MOOCs?
      1. How many were positive?
      2. How many were negative?
    3. What did state legislators say about the MOOCs?
      1. How many felt the MOOC was a welcome activity?
      2. How many felt the MOOC was an unwelcome activity?
    4. What is the dollar value of the activities incurred by the MOOC contracted out to the provider rather than performed “in-house”?

As I said, this is just a rough start.  Write in, and help me flesh this out!

15 Comments

  1. Reply

    Sara Goldrick-Rab

    February 27, 2013

    Thanks for getting a conversation going on this. It looks like another one of these 'revolutionary developments' is on its way to picking up momentum.

    In addition to my classroom instruction, I have been teaching highly-motivated, well-prepared students at distance (some of them degree-seeking) using 'ancient' media like course notes, recorded video tapes, fax messaging, telephones, etc. for over 10 years now, about 10 to 20 students year. Certainly not massive. It takes time and effort, and can be personally rewarding.

    In my mind, any consideration towards learning effectiveness among the students seems to be missing from all the MOOC-talk. What does it even mean for 3000 students to take a course on the design of electronic amplifiers and receive a passing grade from a computerized assessment tool, as opposed to 30 students working with me personally? Do they all become proficient in the subject being studied? It is hard enough to have an authentic assessment in classical settings and ensure a reasonable amount of successful learning outcomes, particularly in terms of retention, and making a difference in student's personal and professional careers. We don't even know how to do this well in a live classroom, we all hide behind the screen of academic freedom, and rarely share any of our best practices among our colleagues.

    I think we have to take a hard look at putting into practice what we know about teaching and learning before we think about MOOC. From all we know about education, we do know for sure it takes time and effort. If productivity gains from MOOC indicate we can realize education without the necessary time and effort, I think we can be pretty sure that education is not taking place, it has only been replaced with some revenue-generating economic activity.

    So, let us put evaluation of learning outcomes as Item 0 before in your list... More later.

    Giri Venkataramanan, Professor, Electrical and Computer Engineering, UW-Madison

  2. Reply

    Sara Goldrick-Rab

    February 27, 2013

    Giri had trouble posting his comment so I pasted it in for him. If this happens to you, just let me know.

  3. Reply

    kurt

    February 28, 2013

    Hi! I posted a comment, but haven't seen it pop up yet...

  4. Reply

    Sara Goldrick-Rab

    March 1, 2013

    No idea why that's happening! Will try to find out.

  5. Reply

    Russell Cannon

    March 1, 2013

    The student impacts currently focus primarily on access; seems like you want to look at learning outcomes as well. You would probably be interested in student satisfaction as well as faculty satisfaction. Alumni and Prospective student outreach/engagements are often are often intended outcome of MOOCs (which may be massively open, but are created by institutions with massive interest in those two subgroups).

    Another potential outcome is leverage of the content on campus- are other courses able to use parts of a MOOC to substitute for an evening's readings? To keep students abroad engaged? In a campus system with branches- is there value-added to those branch institutions?

    • Reply

      Sara Goldrick-Rab

      March 3, 2013

      Hey Russ,

      I have a set focused on completion rates, but you're right, not learning outcomes. I figured, probably unfair to start with that since we don't do that with f2f courses (yet) right?

      I think student satisfaction with courses is not a great indicator of much, but suppose we could ask.

      How would you propose to measure the other things mentioned?

    • Reply

      John Nash

      March 4, 2013

      The student outcome question has been a perennial challenge as far back as I can remember. Barbara Means et al at SRI talked in 1993 about the challenges of what were referred to as "horse race" studies which attempted to compare conventional instruction to that which involved technology media (see http://www2.ed.gov/pubs/EdReformStudies/TechReforms/chap5a.html).

      I smile a little when I see that our collective reaction is to get back in that race (I still catch myself doing it).

      The problems then, as now, were:
      - holding things constant between the approaches
      - assuming (as you note, Sara) that learning outcomes are even set or measured in either setting
      - context of the learner goals

      It's the context of a learner's goal that I think deserves extra attention in MOOCs. How can we best understand whether a MOOC's learning outcomes are reached when there are, say, 3,000 students in a class all of whom are invested to differing degrees in the course. Because many MOOCs are delivered for free, students can easily "shop" for experiences that suit their aims, interests, and energies. To my mind that means one would have to study a myriad of sub-groups of learners to understand outcomes. In fact, it could be the case that the success of a MOOC may be measured less on what the instructor hopes to impart and more on what the student wanted to achieve and whether they felt they achieved it.

    • Reply

      Sara Goldrick-Rab

      March 8, 2013

      John,

      Interesting. Your comments remind me of the discussions we often have about evaluating community college outcomes. Those colleges often say, with so much variation in the reasons why students are attending community colleges, it's hard to know whether they are successful. Even those who say they want a degree often change their mind. Well, it's important to admit, we all do. The goal of a course, in my mind, should be to convince the person who initially indicated interest in learning the material (as revealed by enrolling) to persist until the course is finished. The total number of dropouts reflects departures at many stages, including after "first look" and I think it therefore a legit assessment.

      As for the assessment of learning objectives versus student stated goals, well, this is a slippery slope-- and I don't see why we'd apply standards in this arena that we'd laugh at in a regular higher education setting. Can you imagine a teacher whose students didn't learn to write be deemed successful because the students FELT they learned to write (but didn't know any better)? Either we should place MOOCs in the category of classes like yoga and not assess them other than whether they generate revenue-- and not allow them to compete with our core classes-- or we ensure they have to meet metrics that our core classes have to meet too-- after all, they are competition for resources!

  6. Reply

    Kevin Carey

    March 2, 2013

    It would be interesting to compare a MOOC evaluation as you describe to a similar course taught in a residential setting. Are there any examples of such courses being evaluated using this framework or something like it?

    • Reply

      Sara Goldrick-Rab

      March 3, 2013

      Kevin,

      By residential setting, do you mean face-to-face instruction? Of course there have been studies of that-- do you mean involving all of these outcomes, or a subset?

      Sara

    • Reply

      J.D. Walker

      March 4, 2013

      Have there actually been comparison-design studies involving MOOCs and comparable face-to-face courses? If so, I'd love to see references. There are many, many comparison studies of regular old online courses, of course, but I don't know of any that focused on MOOCs in particular.

    • Reply

      Sara Goldrick-Rab

      March 8, 2013

      No-- this is far too new a phenomenon to have been rigorously assessed like this, I think. Moreover, there's often no alternate F2F course to the MOOC. Moreover, creating a valid comparison group would be very difficult, since MOOC students are all over the world, in practice. I don't think random assignment has been tried anywhere...

    • Reply

      J.D. Walker

      April 3, 2013

      OK, that's what I thought. We (at U of Minnesota) might try to set up a quasi-experimental study that controls for at least certain learner characteristics (demographics and such) as well as for prior subject knowledge (through a pre-test). Not perfect, but better than nothing.

      In a way I don't think that sort of study is really necessary, though, because a raft of recent meta-analyses (by Means, Zhang, Bernard, etc) have reached conclusions similar to one another regarding online learning, e.g., that fully online courses have average learning outcomes inferior to blended courses; that fully online courses with less interaction have worse outcomes than those with more.

  7. Reply

    swalcher

    March 6, 2013

    I think this is very important discussion, and I'm so glad you've initiated it! I think the disruptive potential of MOOCs lies not only in their ability to change the ways we think about online learning in the 21st century, but in the ways they engage us to reevaluate practices that have come to seem commonplace. So at the risk of being a contrarian (a role I nonetheless relish), let me through out a couple of broader considerations.

    If we take a "connectivist" approach to assessing the success of a MOOC, we would want to consider the number of additional connections such a course supported and engendered. For example, I just finished a Coursera course that not only had a healthy discussion forum, but students went on to create a Facebook group, and one on LinkedIn...spaces that allowed students to share, argue, reflect, and provide additional resources and links with one another. So raw numbers of students who enroll in a MOOC versus those who "complete" the course (a typical measure in higher ed) seems to be missing much of where the action is with a MOOC.

    Is impact (in terms of numbers of forum comments, blog responses, social networking groups formed, links shared) a measure of success? Again, in the spirit of connectivism, I think it probably should be. And I don't think it would be impossible to measure a MOOC in these terms, but it would require thought.

    Related to this, I wonder if the success of a MOOC should not also be measured in part by the impact it has those involved in designing and facilitating it, and the effects this then has on teaching and research more broadly within an institution. This may be coming too close to my own particular theory of what makes a MOOC (or any other course, for that matter) great, but in what ways does offering a MOOC generate new knowledge/connections for the institution sponsoring it? I think I have been happiest and most successful as a teacher when I've used a course as an occasion for exploring an important theoretical idea or set of research questions, and have engaged students (as fellow-investigators) to try to understand and think through these issues with me. That is, the class becomes not a space for the transmission of knowledge, but an occasion for creating new knowledge/connections.

    So if I were in a position as an administrator to "green-light" a MOOC proposal at my university, I'd probably insist that the designers and facilitators of the course have a clear set of research questions that the course itself would attempt to help them answer. (Consider the data-collection possibilities as a researcher teaching a course with 160,000 students enrolled in it!)

    Again, in my experience, not only are courses like this more engaging for everyone involved in the process, but this introduce a whole new way of evaluating the success of the MOOC experience: "In what ways did this course help us achieve our institutional/research objectives outlined in our initial proposal for the course?"

    Probably more than two cents, but hopefully helpful.

    • Reply

      Sara Goldrick-Rab

      March 8, 2013

      First, I love the idea of insisting on using the course for research purposes, but I believe you will face significant FERPA and IRB issues.

      Second, I think that MOOCs may facilitate connections, as do face-to-face courses, but those are not the things we normally focus on as academic outcomes. Perhaps we should add them-- especially since the economic returns to degrees likely arise through the social connections made in college.

      Third, you raise a point about purposes of the MOOC that are interesting and have little to do with resource generation. I like that very much, but it raises a question about why we are devoting energy to THIS at a time when public universities need revenue more than anything else. I'd love to see MOOCs help create better teaching in higher education but am deeply skeptical that this will occur since it doesn't stop the current trend toward defunding faculty, ending tenure, etc.


Would you like to share your thoughts?

Would you like to share your thoughts?

Leave a Reply

© 2013 The EduOptimists. All Rights Reserved.