Tuesday, October 2, 2018

The Opportunity Myth Myth

When it comes to slick, pretty education "research," the folks at TNTP know their stuff. Reformsters have been milking the slick-but-hollow "Widget Effect" for years, and now TNTP has whipped up another sure-to-be-referenced-way-too-much "report" entitled "The Opportunity Myth."

If ed reform eats its own tail, what does it poop out?

Who are these folks? TNTP used to stand for The New Teacher Project; She Who Will Not Be Named created it as a spin-off of TFA, designed to put older career-changers into the classroom. At some point it changed into an advocacy group pushing a redesign of teaching (current slogan: reimagine teaching). TNTP is led by Daniel Weisberg, who started out as a lawyer and then served as a labor specialist under Joel Klein in NYC. The board is packed with entrepreneurs, PR specialists, and reform CEOs. You can hunt through the whole list of TNTP leaders and find that this organization devoted to teaching has no teachers in leadership positions (just a few TFA temps and other alternative paths to one or two resume-building years in the classroom).

So this report comes straight from the heart of reformdom.

The report is a slick piece of graphic-soaked digitized niftiness (Look! This part scrolls sideways! And here are more graphics!!) but it features the TNTP sleight of hand. The subheading is the first piece of misdirection-- "What students can show us about how school is letting them down-- and how to fix it." But despite a lot of dressing about Listening To The Students, at its heart, this report is making a familiar point.

The Remediation Myth

After opening with a story about an individual student, TNTP lays out a familiar picture of the problem.

While more students than ever before are enrolling in college, far fewer are succeeding once they get there. Nationwide, 40 percent of college students (including 66 percent of Black college students and 53 percent of Latinx college students) take at least one remedial course, where they spend time and money learning skills they were told they'd already mastered in high school. A recent study found that college remediation costs students and their families $1.5 billion annually...

There are several corners cut here on a subject that deserves attention. Were those students told they had mastered the skills? Because high school teachers have been part of the following conversation for the past several years:

College: This student is not ready to be here!
High school: Did you look at his transcript? Did you look at his grades? Did you look at which courses he took? Because we told you pretty clearly that he was not ready for college, and you took him anyway.
College: This student is not ready to be here!

By the way-- "learning skills"? Common Core and other reform movements may keep insisting it's all about skills, but higher learning also requires some content knowledge, too.

The rise of college remediation courses is a subject that deserves examination, because something's going on and we really need to know what. Possible explanations include:

1) Colleges desperate to fill seats accept underqualified students.

2) The college eligibility test, the one that determines who needs remediation, is not a good test.

3) Students need more remediation these days because more of the year is spent on test prep and testing instead of actual education.

4) Colleges are pushing maybe-not-necessary remediation because it makes them a whopping $1.5 billion each year.

I taught 11th graders at both ends of the academic spectrum. Here are two things I can tell you I've seen multiple times:

Student visits from college and says, "Yeah, they tried to push me into a remedial course, but I just didn't do it, and I've been fine."

Student in vocational prep class insisting they want to go to college; they just don't want to take the "hard" college prep course right now.

As I said, there's a huge conversation to be had here, because those remediation numbers are trying to tell us something important. But this report is not interested in that conversation-- it brings up the subject only so it can make the same old reform point---

Public schools are failing.

The We Did Research Myth

There are somewhere under 15,000 public, private, whatever school districts in the country. TNTP "partnered" with five of them. That is not an impressive sample size-- and they weren't even randomly selected. This is like deciding that, since you have a problem with geese running into jets at the airport, you will go study some penguins in the zoo.

They observed 1,000 lessons-- but that's 200 per system. They followed 4,000 students, but that's just 800 per school. Within each system, according to their technical appendix, they divided the district into elementary, middle and high grades. Then they split those subgroups between 1/2 above-average students and 1/2 below-average students. So the study literally does not include any average students. Ten teachers volunteered from each school, and each teacher picked two classes to have included in the study.

For the three urban districts, percentage of students involved in the study was 5% or lower. In the single rural district, 64% of the district's students were in a study classroom.

But wait-- there's less!! In each of the two classrooms, the participating teachers selected six students-- two way below level, two just below level, and two above level. So once we dig down, we find a tiny sampling of students that is skewed toward underachievers. Well-- relative underachievers, because these six were within one classroom. So presumably we've got the student in an AP classroom who is in over her head, and we've got the student who is taking the low-level class that is way below his ability.

From this tiny not-even-sort-of-random sampling, the authors are able to make all sorts of sweeping statements about how different sorts of students are being educated in this country.

The Grade Level Myth

The big sexy headline-generating takeaway from the report is that all these students are being failed by a system that doesn't require them to do work at their level. The lessons aren't challenging enough.

Students spend most of their time in school without access to four key resources: grade-appropriate assignments, strong instruction, deep engagement, and teachers who hold high expectations.

Because numbers are sexy, the report also tells us exactly how much time is wasted--  "Students spent more than 500 hours per school year on assignments that weren't appropriate for their grade."

The entire report rests on this assertion about the grade level of the instruction in these courses. But what does "grade level instruction" even mean? Once again we delve into the technical appendix (which, I should note, involves following a series of links to a pdf file written in about .2 font). Once again, the rigor is underwhelming.

Assignments were collected from the six special students (photographed by the teacher) and the assignment quality was rated by "raters," all of whom had "at least two hours of training." TNTP gave them a rubric to judge three domains--

1) Content. Does the assignment align with the expectations defined by grade-level standards.
2) Practice. Does the assignment provide meaningful practice opportunities for this content area and grade level?
3) Relevance: Does the assignment give students an authentic opportunity to connect academic standards to real-world issues and/or contexts?

By "standards" we mean the Common Core (NGSS for science). But if we're parsing Core standards by grade level, I foresee some problems. Here are two different years for the same standard (RL.1):

Cite textual evidence to support analysis of what the text says explicitly as well as inferences drawn from the text.

Quote accurately from a text when explaining what the text says explicitly and when drawing inferences from the text.

That's the fifth grade and sixth grade standard. Can you tell which is which? I know-- it's an unfair question because you haven't had your two hours of training. TNTP does site some subject-specific criteria for the rubric-- but the link is dead and the address leads to an error page.

The raters are also supposed to determine if the student completed the assignment successfully-- in other words, grade the assignment-- which is pretty impressive work for someone with two hours of training. They were also supposed to see if the student met the standards' requirement. It is not clear if these raters were hired through Craigslist or the TNTP intern program or some other agency.

The Strong Instruction Myth

This is good old fashioned tautological ouroborean research. First, you decide how to identify "strong instruction" based on what you consider identifying characteristics. Then you check the "strong instruction" to see if it shows signs of the characteristics you believe are important. Voila. It mostly does. In this case, "strong instruction" includes lessons that reflect "the demands of the standards."

And they watched two whole lessons by each volunteacher.

Moving on. Nothing to see here.

The Chicken Littling Myth

The biggest bit of misdirection in this report is the great amount of weight thrown behind the narrative that these students have dreams and ambitions, but they are being lied to, lied to by school systems that tell them that they are on a path to college and career readiness. There are plenty of charts and numbers to quantify the size and shape of student aspirations, and to suggest that teachers and school systems somehow don't know this about their students, as if the entire system is that guy sitting at the desk behind a newspaper that he never lowers to actually look at his students.

This is the oldest routine in the reformster playbook-- lean heavily on explaining just how bad the problem is. Lay out the problem in gut-wrenching detail. Make sure to define the problem in terms that fit your proposed solution. But while you have research and data and details about the problem, the part where you insist on your solution remains unsupported by anything except your assertions.

Reformster: Look at these test results. Look at these x-rays. You definitely have a brain tumor, and if  it's not fixed, you'll soon lose feeling in your limbs and your legs will stop working properly.
Patient: Oh my God! Save me!
Reformster: Certainly. I'm just going to use this chain saw to cut off your legs.
Patient: Wait! What? How will that help with my brain tumor?
Reformster: Look at these x-rays! Look at how big it is! Right there in your brain! This is terrible!
Patient: But how will hacking off my legs-
Reformster: X-rays! Brain! Terrrrrrrible!

Oh, and the writers also want you to know that socio-economics are no explanation or excuse for students' low performance.

The Recommendations Myth

The recommendations portion of this report starts with a reminder that we should be doing this For The Kids, and that the classroom experience of students should be the center of policy decisions. Then it calls for two commitments: First, give every student grade appropriate assignments, strong instruction, deep engagement, and teachers with high expectations regardless of any part of their identity. Second, every student and family should be "an authentic partner" and should have a serious role indecision-making-- a real choice, one might say.

This leads to five recommendations:

1) Listen to parents and students.

2) Grade-appropriate assignments for everybody. Regardless of whatever.

3) Give all students, especially the behind ones that is [insert rhetoric that gets you around actually saying "rigorous" here].

4) High expectations. Stop being racist.

5) Conduct an equity audit to make sure everyone is getting the good stuff.

The Deja Vu Myth

So, to sum up....

Schools are lying to families by telling them that their children are college and career ready when they actually aren't. Teachers should have higher expectations, which can be put into action by aligning class work with the very best college and career ready standards-- and poverty and racism, while bad, are no excuse for low performance on the school's part. Do it For The Kids- don't put adult concerns ahead of children's needs. The school system should be responsive first and foremost to parents, who should have choices available that suit their goals.

Does this all sound familiar? It should-- this entire report is the report that Reformsters (particularly Arne Duncan) wish they had come up with a decade ago when trying to pitch Common Core and charters.

We could spend even more time picking apart the research techniques here, though Matt Barnum at Chalkbeat hit the basic issues pretty well, but looking at the details is really beside the point. The details are window dressing. This is the same pitch we've been hearing for over a decade, adroitly stripped of the test-centric angle and the demand to evaluate teachers-- arguably the two most hackle-raising obstacles to reformster goals.

The authors have tried to ease up on teachers, even going so far as to write at one point "for those of us working in the school system..." which-- well, no. The folks at TNTP are not down there in the trenches with actual trained educators. But the repeated calls to listen to the students and teach them the right stuff make it clear that the premise here is that teachers currently do neither. It's an odd charge, given that teachers have been laboring under reform programs for years, but then the other curious feature of the report is that it does not acknowledge how NCLB, RttT, Common Core, and test-centered accountability have changed the landscape.

This report could have been written fifteen or twenty years ago. Again, I'll bet some Reformsters wish it had been.  But there is nothing new here.

Here is probably the report's biggest flaw. If you really thought the problem was that students aren't ready for college and career, as evidenced primarily by the rate of college remediation, why wouldn't you gather up students who needed remediation and those who didn't, and compare and contrast them. Why wouldn't you study them to see what one group got that the other lacked? You wouldn't do that because you have a point you want to make, and your research "design" is based not on a quest for answers, but on building a scaffold for the answers you already want to sell. This report was funded by the Joyce Foundation, the Chan Zuckerberg Initiative, the Nellie Mae Education Foundation, the Walton Family Foundation, the Overdeck Family Foundation, and the Barr Foundation.

The Unfortunate Truth

As Barnum notes:

The research of TNTP, previously known as the The New Teacher Project, has a track record of shaping policy, particularly with an influential 2009 report known as The Widget Effect, which focused on perceived flaws of teacher evaluation systems.

The report is old wine in a slick new skin. But it looks really, really cool, and it layers on the For The Children with a shovel and a backhoe, which is always compelling because we really should do it for the children-- but not if "it" means launch a highly profitable education-flavored industry based on pumping amateur hour standards out into a top-down imposed framework that stomps on actual education professionals.

I don't know that anyone will be "influenced" by the report, because it is old news. But people who are already invested in this reform agenda will have a slick new publication that they can wave around to bolster their position. The Widget Effect mattered because instead of just saying, "I think this should be policy because I like the idea," reformsters could call policy-makers and wave around an official-looking report-thing. The same will happen here. TNTP has provided the striking headline-- "Students are wasting 500 hours a year on work that's below their level"-- and a whole bunch of policy makers will never look any deeper than that.

So that's a bummer.

When someone waves this in your face, you can tell them that it's a report based on a tiny number of students in a tiny number of schools about which we know no pertinent details. You can tell them that the report may back up the notion that students have aspirations and sometimes have trouble achieving them, but that TNTP made up its own notion of what "grade level work" means, and someone else's notion would yield different results. You could point out that this is one more call for schools to buy material aligned to the Common Core standards, and if the Core was going to fix all this, wouldn't it have done so by now? You could ask them to tell you what, exactly, is a new idea included in this report?

Here's hoping that more people listen to you than are going to listen to this report.


  1. "it does not acknowledge how NCLB, RttT, Common Core, and test-centered accountability have changed the landscape." Very good point. The report is completely ahistorical. I'm curious: what reform or program of the past 20 years do you think has most helped increase rigor in the K-12 classroom?

    Lastly, If you want that subject-specific criteria for the rubric: http://lmgtfy.com/?q=achieve+the+core+%22assessment+evaluation+tool%22

    1. I believe that individual teachers that use Common Core only as a starting point make a difference. However, parents complain that the material is not in the approved curriculum and is too hard. Teachers are compelled to "dummy up" the material because of complaints from parents- the same parents who complain that their kids are not prepared for college. All students are measured by the same standards. I was on a state test writing committee. We were told to use Bloom's taxonomy, with most of the test or lessons at level 4-6. Those activities were rejected as too hard. 90% of the test became level 1 & 2. "Special Ed" students could not deal with higher levels, and they had to get at least 70% right. So special ed and Honors students were being judged by the same standard, and teachers of honors classes were discouraged from higher challenges, all due to pressure by parents.
      We will not improve education until actual learning & skills become more important than grades. Grades will get you into a college, but they won't keep you there.

  2. I get what you are saying about framing as a large-scale study, but I could do an action research project at one school and get the results that they did. It breaks my heart when I observe classrooms where teachers are teaching. They aren't moving students forward. They are doing activities that students can do and feel good about instead of activities that foster productive struggle and perseverance. I am hosting 4th graders on a college campus next month. Some of those kids are going to want to go to college. It will be up to the middle and high school to prepare them even those who live in home situations that work against that preparation.

  3. Had a PD session this morning, using three Opportunity Myth articles as our discussion points. During the session, I was not able to put into words how annoyed and confused I felt about so many points in the articles. I just kept my head down, and tried to be a good participant in all my different discussion groups. As soon as I was alone, I googled The Opportunity Myth, and found your fantastic blog. Thank you so much for this article. Thank you for the thoughtfulness of all of your points. Thank you for the clear language of what I was thinking and just couldn't say in the moment. Whew. I thought it was just me being a brat this morning.

  4. Many Years of ExperienceNovember 7, 2019 at 11:38 AM

    The Opportunity Myth is just another attempt at selling school districts a set of canned goods funded by expensive federal grants that only serve to take up the time of local teachers and administrators attending "trainings" which pull teachers away from their students by those with little on no classroom experience. This whole scam has now become a huge taxpayer-funded industry with no end in sight.

  5. Good to see a heterodox and skeptical view. I'm now a follower. As irritating as public relations techniques can be we should still consider whether the central argument has merit.
    The central argument in the Opportunity Myth is that more students are going to but not finishing college and that the reason is because they are poorly prepared.
    It is irksome that the OpMyth doesn't sample well is very glossy and uses lots of "anecdata" lift quotes to sell its solution, but is the problem real and is the solution viable?
    Yes and they've made a good case.
    The problem is actually a good one to have. As HS graduation rates have risen and college attendance rates have increased the NUMBER of low income and modestly well prepared students going to college has increased.
    Using Massachusetts college attendance data http://profiles.doe.mass.edu/nsc/gradsattendingcollege_dist.aspx?orgcode=00000000&fycode=2018&orgtypecode=0& (full disclosure I work for the state). Compare low income college attendance rates from 2004 with low income college attendance rates from 2108 and you will see 10,000 more heading to college but then as now (from other sources) only 1 in three completes a college degree (12% out of 37% then 18% out of 57% now)...
    I say it's a good problem to have because we converted high school dropouts into HS graduates and convinced larger numbers to go to college, that's like planting a lot more seeds. However our yield rate is still just 1/3. That means we have convinced many of the third and fourth quintile students to go to college but they aren't ready for many reasons... TNTP is suggesting that they want to go to college, but haven't been routinely challenged with college ready skills and lessons.
    Anyhow, your perspective helped me sort through the issue, thanks.

  6. Once again, reformers are shouting that teachers are "not good enough." Thank you for showing all of the flaws in this "study." We need to teach our students how to look at these reports critically, so they don't believe these irresponsibly repackaged ideas.

  7. Common Core was set as a goal, not a starting point. I taught HS honors science, and parents complained that I was expecting too much of the students- I was going beyond the standards. I was also on a state standards test writing committee, where we had to simplify the test so that special ed students could pass the same graduation test used to evaluate honors and college prep students.
    Parents today are demanding more influence in the classroom, not acknowledging that elected school boards are parental input. They are asking for anarchy. I taught my honors class at a higher level, and took flak from the administration because of parental complaints. I invited students to come back after their first year of college and tell me what I should change. The comment always said I didn't push them hard enough. I didn't challenge them to think. Precisely what I was told not to do by parents and administration.
    Until learning and study skills become more important than recorded grades, no programs will change this. Parents demand a Mercedes education for a Kia price tag.