NCTQ: It's So Easy

The National Council on Teacher Quality is one of the leaders in the production of education-related nonsense that is somehow taken seriously. The offices of NCTQ may not produce much of anything that provides real substance, but somewhere in that cushy suite there must be the best turd-polishing machine ever built.

NCTQ has published a new "report" that "seeks" to "answer" two questions:

Are teacher candidates graded too easily, misleading them about their readiness to teach? Are teacher preparation programs providing sufficiently rigorous training, or does the approach to training drive higher grades?

And when I say "seeks to answer," what I mean is "tries to cobble together support for the answer they've already settled on." There is no indication anywhere that NCTQ actually wondered what the answers to these questions might be. No, it appears that they set out to prove that teacher candidates are graded too easily as they meander through their rigorless teacher programs.


Who are these guys?

Does the NCTQ moniker seem familiar? That's because these are the guys who evaluated everyone's education program and ran it as a big story in US News (motto "When it comes to sales, lists are better than news"). That evaluation list caused a lot of stir. A lot.

Funny story. I interviewed a college president from a local school who was steamed about that list in part because it slammed them for the low quality of a teacher program that they don't even have. Turns out a lot of people had problems with NCTQ methodology, which involved not actually talking to anybody at the schools, but collecting information less detailed than what you can get from your high school guidance counselor. You can read a cranky critique here and a more scholarly one here. Bottom line-- they get great media penetration with a report that has less substance than my hair.

NCTQ has tried to blunt some of the criticism with moves like adding a Teacher Advisory Group composed of Real Live Teachers. Also, NCTQ uses very pretty graphics in soothing colors.

The new report-- Easy A's (they even italicize the title, like a book title, because it's more hefty and important than a mere "article title")-- is billed as "the latest latest installment of the National Council on Teacher Quality’s Teacher Prep Review, a decade-old initiative examining the quality of the preparation of new teachers in the United States." This is supposed to be part of their "growing body of work designed to ensure that teacher preparation programs live up to the awesome responsibility they assume." For the moment, let's look at the "findings" in this "report."

And the bottom line is...

They studied about 500 schools; these schools are collectively responsible for about half the teacher degrees granted. Two "findings" here.


First, they find the majority of institutions studied (58%) grading standards are lower than for students in other majors on the same campus.

Second, they find a strong link "between high grades and a lack of rigorous coursework, with the primary cause being assignments that fail to develop the critical skills and knowledge every new teacher needs."

Wow!!

I know! My mind boggles at the huge amount of research involved here. This must have required an extensive study of each of the 500 institutions studied. I mean, we're talking about comparing the rigor of assignments in both education and non-education courses, so researchers must have had to dig through tens of thousands of college course assignments in addition to an extensive study of the grading standards of thousands of professors to be able to make these comparisons.

And then to do all the research and number crunching needed to establish a correlation between the rigor of assignments and grades achieved-- this would be a more complicated model than VAM, to tease out all those data,

Also, it's worth noting that NCTQ knows what critical skills and knowledge all new teachers need, which must have been a huge research project all by itself. I do hope they publish that one soon, because if we had such a list, it would certainly revolutionize teacher evaluation and training. In fact, if they've done all this research and know all these answers, why aren't they just traveling from college to college and saying, "Here-- this is what our proven research shows you should be teaching teacher trainees."

Never mind.

Apparently the minds at NCTQ boggled at that research challenge as well. So let's look at what they actually did.

For the first point-- the idea that teacher grads are the recipients of departmental grade inflation-- NCTQ looked at commencement brochures. They checked commencement brochures to see a) who graduated from a teaching program and b) who had an honors grad designation based on GPA.

It is not clear how many years are spanned. There were 500-ish schools studied, and footnotes in an appendix indicate that a total of 436 commencement brochures were discarded for insufficient data. Yet the executive summary says that 44% of all teacher grads in all 509 schools earned honors, while only 30% of all graduating students did. And in 214 schools, there's no real difference, and in 62 schools, teachers had fewer honors grads. How did they get such precise numbers? That is not explained.

There's some more detailed breakdown of methodologies of teasing the data, but that's the data they accumulated from graduation brochures, and the whole argument boils down to "Barely more teachers graduate with honors than do other majors." Oddly enough, this does not lead NCTQ to conclude, "Good news, America! Teachers are actually smarter than the average college grad." I guess it comes down to how you interpret the data. That you collected from graduation brochures.

But what about that lack of rigor?

Having somehow concluded that teacher programs are hotbeds of easy grades, NCTQ turns to the question of who let the rigor out. Once again, their methodology is itself as rigorous as a bowl of noodles left to boil for twelve hours.

Multiple theories as to why students in education majors might appear to excel so often were also examined (e.g., clinical coursework that lends itself to high grades, too many arts and crafts assignments, too much group work, particularly egregious grade inflation, better quality instruction, more female students who tend to get higher grades, opportunities to revise work, and higher caliber students), but none appears to explain these findings as directly as the nature of the assignments.

First of all, interesting list of theories there. Second of all-- "none appears to explain"?? How did you judge that appearance, exactly. Did we just put each theory on a post-it note, stick it to the wall, stand back and hold up our thumbs and squint to see which one looked likely? Because usually the way you reject theories in research is with research.


The Big NCTQ Thumb came to rest on "criterion-referenced" and "criterion-deficient" assignments. We'll come back to that in a moment-- it deserves its own subheading. Just one more note about te methodology here.

NCTQ got their hands on syllabi for 1,161 courses, "not just on teacher education but across an array of majors." Except-- wait. We're looking at 509 schools, so 1,161 works out to a hair over two courses per school, including schools with multiple education programs. Oh, no-- never mind. Here it is in the appendix-- we're only going to do this in depth analysis for seven schools. Plus at thirty-three other schools, we will look at just the education programs. Oh, and on this chart it shows that of the seven in-depth schools, we'll look at only teacher programs in two. So, "wide array" means six other majors at five of the 509 schools.

And yes-- the data comes from course syllabi. At seven schools. Not the course, professors, students-- just the syllabi. So our first assumption will be that these syllabi with their lists of course assignments will tell us everything we need to know about how rigorous the coursework is.

Creating the right hatchet for the job

"Criterion-referenced" is a thing, and it basically means an objective test. "Criterion-deficient," on the other hand, will actually win you a game of googlewhacking because apparently nobody uses the term except NCTQ. "Criterion deficiency" is a real thing, used it seems mostly in the non-teaching world to describe a test that fails to assess an important criterion (e.g. you want a secretary who can word process, but the job evaluation doesn't check for word processing). I bring this up only because now I have a great fancy word for discussing high stakes standardized tests-- they suffer from citerion deficiency.

But back to our story.

NCTQ cross-reference course syllabi with grade records posted publicly by registrars and open records requests (Schools DO that?! Seven years ago I wasn't allowed to know the grades of my own children for whom I was paying bills because of privacy laws!) NCTQ "applauds the commitment to transparency" of those schools willing to complete ignore student privacy concerns.

NCTQ looked at 7,500 assignments and ranked them as either CR or CD (that's my abbreviation--if they can be lazy researchers, I can be a lazy typist). Here are the criteria used to tell the difference:

An assignment is considered criterion-referenced when it is focused on a clearly circumscribed body of knowledge and the assignment is limited so that the instructor can compare students’ work on the same assignment. 

Qualities that indicate an assignment is criterion-referenced include
* a limited scope
* evaluation based on objective criteria;
* students’ work products similar enough to allow comparison.

Qualities that indicate an assignment is criterion-deficient include
* an unlimited or very broad scope
* evaluation based on subjective criteria
* students’ work products that differ too much to be compared. 

NCTQ provides a sample of each. A CR lesson would be one in which the student is to apply a specific tool to critique videotaped (quaint) lessons. This is good because everyone uses the same tool for the same lessons so that the instructor knows exactly what is going on. A CD assignment would be to teach something-- anything-- to the class using information from the chapter. This is bad because everybody teaches something different, uses different specific parts of the chapter, and teaches in different ways. This would be bad because there would be too many differences the instructor would be unable to determine who is best at the teaching of stuff.


How is this distinction a useless one? Let me count the ways.

1) It assumes that the purpose of an assignment is to allow comparison of one student to another. This would be different from, say, using assignment as a way to develop and synthesize learning, or to mark the growth and development of the individual student.

2) It assumes that all good teachers look exactly the same and can be directly compared. It imagines that difference is a Bad Thing that never intrudes in a proper classroom. This is bananas.

3) It assumes that the best assignments are close-ended assignments that have only one possible outcome. Even reformsters steeped in Depth of Knowledge and Rigorology tout the value of open-ended response tasks with a variety of correct answers without demanding that the many correct responses be ranked in order of most correct.


4) It appears to place the highest value on objective testing. If that is true for my teacher training, is it not true for my practice? In other words, is NCTQ suggesting that the best assessments for me to use with my students are true-false and multiple choice tests rather than any sort of project-based assessment. Because, no.

5) It assumes that all students in the future teacher's future classroom will also be one size fits all. When I ask my students to prepare oral presentations, should I require that they all do reports on Abraham Lincoln so that they can be more easily and accurately compared?

6) It discounts the truth that part of being a professional is being ready and able to exercise subjective judgment in an objective manner. In other words, free from personal prejudice, but open to the ways that the individual student's personality, skills, and history play into the work at hand (without excusing crappy work).

7) They are trying to make this distinction based on assignments listed in syllabi.

There's more 

I did look for the appendix evaluating how well five weeks will prepare you for teaching if you come from a super-duper university, but that was one aspect of teacher training ease NCTQ did not address.

There are other appendices, examining ideas such as Why High Grades Are Bad (including, but not limited to, if grades are divorced from learning, would-be employers will find grades less useful). But I'm not going to plow through them because at the end of the day, this is a report in which some people collected some graduation brochures and course syllabi and close read their way to an indictment of all college teacher training programs. 

It is not that these questions are without merit. Particularly nowadays, as teacher programs are increasingly desperate to hold onto enough paying customers to keep the ivy covered lights on, teacher training programs are undoubtedly under increasing pressure to manufacture success for their students, one way or another. Nor is it unheard of for co-operating teachers to look at student teachers and think, "Who the hell let this hapless kid get this far, and who the hell is going to actually give him a teaching certificate?"

The question of how well training programs are preparing teachers for real jobs in the real world ought to be asked and discussed regularly (more colleges and universities might consider the radical notion of consulting actual teachers on the matter). And as more reformster foolishness infects college ed departments, the problem of Useless Training only becomes worse. So this is absolutely a matter that needs to be examined and discussed, but the method should be something more rigorous than collecting some commencement brochures and course syllabi and sitting in an office making ill-supported guesses about what these random slips of paper mean.

And yet I feel in my bones that soon enough I'll be reading main stream media accounts of the important "findings" of this significant "research." Talk about your easy A.