November 13, 2003

"Highly Reliable," "At Least Moderately Valid," and Deeply Flawed: Student Evaluations of Faculty

A couple of months ago, a reader sent me a link (via Tyler Cowen's Marginal Revolutions) to Michael Huemer's critical review of student evaluations. Another reader has since recommended this article by Mary Gray and Barbara R. Bergmann. Both raise troubling questions about the usefulness of student evaluations, though to somewhat different ends.

The main concern of Huemer's piece is that of finding or devising a way of accurately measuring teaching quality with a view to improving said quality. Interestingly enough, he reports that student evaluations are actually "highly reliable" in the sense that "students tend to agree with each other in their ratings of an instructor" and "at least moderately valid, in that student ratings of course quality correlate positively with other measures of teaching effectiveness." So if they're "highly reliable" and "at least moderately valid," what's the problem? Huemer identifies a number of problems, some of which have to do with what it is that students are measuring when they "agree with each other in their ratings of an instructor."

Perhaps the most serious is that of the "open secret" of grading bias: ie., given the "well-established correlation" between higher grades and higher evaluations, student evaluations "seem to be as much a measure of an instructor's leniency in grading as they are of teaching effectiveness." Other concerns have to do with the subtle but possibly significant effects of teaching to the evaluations, dumbing down course content and avoiding controversy for fear of negative student feedback.

One serious problem, however, is that, in Huemer's reading of the literature, peer review (which is frequently offered as an alternative to student review) turns out be be even less valid than student evaluations. He thus ends by proposing a number of practical changes to the design and structure of evaluations in an attempt to correct for the effects of bias.

The main concern of Gray and Bergmann's paper is that of the impact of evaluations on faculty morale and indeed on faculty career prospects. In their opening paragraph, they boldly assert that "what originated as a light-hearted dope sheet for the use of students has, at the hands of university and college administrators, turned into an instrument of unwarranted and unjust termination for large numbers of junior faculty and a source of humiliation for many of their senior colleagues." I guess I'm a little bit sceptical of the claim that "large numbers of junior faculty" have been fired on the basis of evaluations. But in addition to some of the concerns also treated by Huemer, I think they also raise some interesting points about the use of this instrument as a "shaming" device:

Jeffrey Stake, a law professor at Indiana University, argues that asking students their opinions undermines the trust and faith they need to place in the teacher. Instead of saying, 'Here is a great scholar and teacher; learn from her what you can,' the administration of evaluation forms says to students, 'We hired these teachers, but we are not sure they can teach or have taught you enough. Please tell us whether we guessed right.'

As an entire career can be terminated by not-good-enough evaluations, the procedure of administering the evaluation instruments and getting them turned in forces on the faculty member what Catholics call 'an occasion of sin.' The administration sets up a system that presents the faculty with a powerful temptation to cheat, and then has to invent demeaning procedures to prevent cheating. The teacher is explicitly forbidden to touch the evaluation sheets after they have been filled out. A student has to be designated to collect and take them to the appropriate office. This procedure tells the students that the teacher is more than likely to be a cheat and a sneak, who will cook the books if given a chance. Both students and teacher pretend not to notice the shaming involved, but it is palpable in such a situation.

Here again, I wonder if they are overstating their case: Does the procedure really tell students that their teacher is "more than likely to be a cheat and a sneak"?

In any case, Gray and Bergmann call for a complete abolition of this "inaccurate, misleading, and shaming procedure," and propose a number of alternative measures, most of which require peer review (which Huemer suggests is not necessarily any more valid or reliable than student evaluations).

Posted by Invisible Adjunct at November 13, 2003 12:00 PM

I remember student evaluations. They ran something like this.

"The amount of money we get from the University depends partly on the results of this survey. But the administrators only look at the numbers. So please please please give us good numbers, and tell us what you really think in the comments section, which is what we {the department} will be looking at."


I think that these days a 4.9 (where 5.0 is "exceptional") is considered a pretty poor score.

Posted by: polychrome at November 13, 2003 01:25 PM

I agree with the writers you cite who argue that evaluations, given what's become of them in the hands of administrators and some professors, should be junked. And I think they will be. The entire process has become corrupted - it's a remarkably easy way to intimidate and even get rid of faculty you don't like for various reasons -- among them, the fact that they give uninflated grades when your main concern is to make your students happy enough to stay and keep paying tuition. In general, evaluations have become part of the larger repressive atmosphere of universities, where what you say and how you act with students is monitored closely, and where you can be attacked very damagingly at any time. I'm thrilled to see more and more articles coming out confirming my sense that evaluations have become a nightmare for serious professors, a confirmation of their essential status as anxiously coddled consumers for students, and a great boon for cynical administrators.

Posted by: georgia at November 13, 2003 01:31 PM

I must say, what astonished me when I taught was how many students gave thoughtful and careful feedback in their evaluations. I was sure they would only be an outlet for anonymous griping about failed tests, honor court reports, poorly worded exam questions, and a thousand turned down requests for extensions, post-ponements, exceptions to course policy, etc. Instead, most students in my experience take the evaluation very seriously and take time to provide good comments.

Posted by: Matilde at November 13, 2003 02:28 PM

"Here again, I wonder if they are overstating their case: Does the procedure really tell students that their teacher is "more than likely to be a cheat and a sneak"?"

This has never crossed my mind until now. It's possible of course; I've always thought we did this to ensure students of the integrity of these evaluations, and that they are free to write what they feel.

Of course, like many in this business, I have my concerns about student evaluations. So many, myself included, take at least small measures to manipulate our students. There's grade inflation (not a small step), bringing cookies, bringing your pet to class (seen that done at least once), or whatever. I generally have evaluations done at the beginning of class, so that should a student or two hate my guts and want to retaliate for a bad grade, my teaching style, or my personal hygiene, he or she has to look at me for the rest of the period.

If it were up to me, all the various and stupid ratings systems would be dumped, and students would only be asked to provide written comments.

I have mixed feelings about student evaluations. We all know the problems and the abuses. Nevertheless things little bits of paper provide us an opportunity to receive constructive feedback from our students. Obviously this does not happen as often as it should.

Posted by: DM at November 13, 2003 03:41 PM

The bad teaching evaluations I've gotten have been caused by their utter unreliability, the good ones by my teaching excellence.

No one who gets high ones has the integrity to complain about them, at least not in my experience. After a substantial time out of undergrad, I now think quite differently about the teachers I had. There aren't any retroactive teacher evaluation forms in my alumni newsletter, however.

Posted by: Chun the Unavoidable at November 13, 2003 03:55 PM

The chair called me into his office about two thirds through my third semester in teaching. The meeting seemed to be prompted by a student who complained of a failing grade the previous semester. The meeting, however, consisted of a review of the previous semester's student evaluations, which were not overwhelmingly glowing, but generally reflected the kind of curve one finds with students. Out of 13 evaluations [there are always those 3-4 students who don't show up on this day and so are not included], I received 3 very good eval., 3 bad ones, and 7 in the middle somewhere. The general consensus was that I was demanding; having high expectations. For some this was a plus and for others, the opposite. Granted, I had recently come out of a very rigourous Research I university and where I was now teaching was quite less than that. I'd figure out how to adjust in time - if I chose to continue.
My 'defense', after spending nearly an hour going through all the methods that any other instructor at this institution would use to help stimulate students, motivate them, encourage them,etc., was that I was also trying to get them to think.
This didn't seem too foreign an idea in academia - the purpose being to help students to learn to think about the subject- not what to think - but how.
The chair immediately informed me that they were too young to think. I should be giving them more As and Bs; Bring them cookies to ..I don't know what for; That the work they produced didn't matter; It was only their effort that counted.
I do believe that effort is a fine thing to count toward good grades, as it is usually reflected in the kind of work that is produced. As is normally the case, effort expended and the resulting work ran the gamut of a bell curve. I wanted all of them to succeed, but consoled myself with the thought that it was normal to have a few really great students, a few unmotivated ones, and those in the middle.

This meeting was the first indication I had that something was amiss and after the chair and I had evaluated all the methods one could use - that I already did use, it was decided that it just must be my approach since everything else was in line with the school's teaching methods.
In the end, I was warned that I had 5 weeks to work on getting my student response evaluations to look better, because: "this institution doesn't fire anybody - they just don't rehire the next semester."

In the spirit of trying to improve my teaching skills, I did sit in on one of the department's tenured faculty's class. Honestly, I thought I was in an episode of Romper Room. These are young adults, aged anywhere from 21 - 33. Why weren't they offended? I was.

The tenured faculty had originally been invited to sit in on the meeting with the chair, but had a previous obligation [YOGA class] and could not attend my 'threatened to be fired' meeting. This instructor [ who was also my immediate supervisor] did, however, offer some sage advice during an apologetic phone call the following day: "Yes, grade inflation is a problem, but those who buck the trend, don't last long."

I didn't wait around to find out if I would be hired again. I took my pride and decided I would prefer to make an honest living instead.

To end, because of my specific experience, I would plead that the position may not be overstated. Junior faculty may not be getting fired, they just don't find themselves with a class to teach the next year unless, of course, they learn to toe the line.

Posted by: academic refugee at November 13, 2003 04:17 PM

One of the posters up there said people who get great evaluations lack the integrity to complain about the system. You'll just have to believe me that I get extremely good evaluations and all the stuff that goes with them - high enrollments, including a lot of alumni auditors, repeat customers, awards, etc. And I'm on record as furious about teaching evaluations. In fact in the last few years I've seldom handed them out. Sometimes I hand out a sheet of paper asking for written comments and I send that over to the department. And that's it. End of story. I never read my evaluations. I've been teaching for twenty years, by the way. Yes, of course, I'm tenured. But some negative consequences for me do flow from this behavior, since it makes my department and the administration pettish to have their evaluation party platters pooped on. It's worth it. It's time for the professoriate to see if it can recover a little dignity. Those of us who can afford the negative consequences of rebellion should assume them.

Posted by: wyatt at November 13, 2003 06:08 PM

Perhaps student evaluations of profs should only be done in classes where it is also possible to give an objective, department-wide test of what the students have learned. Maybe Linda gets much better evaluations that John from the Calculus 101 students...but how does the performance of the two groups compare on the test? Only this will let us know if Linda's evaluations were better because the class was easier..or if it was because she was actually a better teacher.

Posted by: David Foster at November 13, 2003 06:14 PM

Now that they have computerized the process, I think it would be possible to see all the teaching evaluations from a given student. This way faculty members and their evaluators could decide whether students were just whiners who didn't like any of their classes or had legitimate complaints.

While I'm sure some students are whiners, it is definitely the case that some of the tenured profs who are consistently getting bad recs need to stop being so defensive and listen to what their students have to say. There are some terrible teachers out there, and they need to be reminded at the end of every semester that they're not getting through to their students.

Joseph Epstein wrote an essay in the American Scholar 10-15 years ago on teaching evaluations, and, in describing his reaction to nitpicky comments about his teaching, said: "Such criticisms roll right off my back, like buckshot off of a duck's heart." Classic.

Posted by: alwaysfortunate at November 13, 2003 10:03 PM

As Bismarck said, those who like laws, sausages, and teaching ratings should not see them being made.

For three years I had stellar ratings. Then I gave students midterm grades before they filled in the questionnaires. (I didn't have midterms in previous years, just class project and final exam, grades to both given at the end of the semester.) The result: miserable ratings. We are talking seriously miserable. (Yes, I am a hard grader.)

I feel like the person whose name I substituted for mine in this message.

Posted by: Boethius at November 13, 2003 10:03 PM

My husband characterizes student evaluations of faculty as "letting the inmates run the asylum."
I think that sums up the whole charade quite well.

Posted by: Terminal_MA at November 13, 2003 10:39 PM

Ok ...

"he reports that student evaluations are actually "highly reliable""

Come on.

Either the entire concpet of grades is valid or it is not.

If it is, surely teachers can develope grading methods that work on them as well as on the students.

Surely the profs deserve to be graded just like anyone else.

Don't I wish in my non-academic job that I was free of any feedback.


At least any negative feedback.

Posted by: Visiting Blogger at November 13, 2003 10:55 PM

Remembering back to my undergraduate years, I always felt student evaluations were a waste of time. I never felt able to convey what was wrong, and sometimes didn't understand what was wrong until much later.

For example, one common problem was unengaging lectures. It was common to lecture from overhead foils, and hand out a copy of the foil set before class. This made the lecture much too passive; most students _need_ to take notes to keep their brains engaged. I, and probably most others, didn't see that as a problem at the time.

So I've come to the conclusion that student evaluations could be very helpful, if they could be administered, say, half a decade after students actually took the class. Of course, by then the adjunct faculty member would be gone...

Posted by: Karen at November 14, 2003 01:40 AM

Student evaluations haven't been helpful to me, especially because I am an admittedly hard grader. I'm a graduate student at a top-tier research university, and I've had undergraduates approach me after a lecture and moan about a low grade being undeserved because, "I was just taking this as an elective because I thought feminist theory would be easy!" I definitely got some miserable evals after that class.

Posted by: Icky Girl at November 14, 2003 02:07 AM

Student evaluations are an interesting exercise. As I have increased the academic rigor in my teaching during my career, I have seen a decline in my student evaluation ratings. Still, I read the written comments after each semester looking for the student who was confident enough to sign his or her name after writing comments. These "authentic" evaluations have always provided me confidence to maintain high expectations for student learning or to change my teaching to help students meet those high expectations.

It is difficult to accept criticism from young people after I have spent more than 30 years thinking about and working on my teaching. It takes about a week to get over the feeling of being insulted, but when I do I always see new avenues to improve my teaching. The process has helped me stay interested in students and their learning.

Posted by: Alan Frager at November 14, 2003 04:21 AM

The last posting sounds like Winston Smith after reprogramming.

Posted by: alma at November 14, 2003 09:00 AM

"The last posting sounds like Winston Smith after reprogramming."

To me, it sounds like a very honest and thoughtful comment from someone who cares about teaching.

Posted by: Invisible Adjunct at November 14, 2003 09:26 AM

My feeling is that the real problem with student ratings is that they transform the student into a customer. This is of course highly poroblematic since while students are there to learn, customers are there to be satisfied, entertained, etc..

This may be an overstatement but when students become customers the whole dynamic changes.

Can you imagine the outcry from high school teachers if their students were suddenly given the right to ananomously and publicly rate them?

Posted by: M at November 14, 2003 10:22 AM

Everyone rightly complains that faculty teaching is given less value than faculty research. Without student evaluations teaching will have even less impact on promotion and hiring decisions, with particular negative ramifications for adjuncts.

At the small liberal arts college where I now teach, my evaluations are thoughtful and careful. I found my evaluations as a graduate student at a big state school good and even helpful sometimes. I read my colleagues' evaluations regularly and I think I get a good picture of their strengths and weaknesses in the classroom--certainly better than any other method I can imagine.

Any evaluation method can be used or interpreted badly by administrators, etc. but I think the critics should offer some method of evaluating teaching that they think is more reliable.

Posted by: Lucy S. at November 14, 2003 11:05 AM

Student evaluations can be useful to an instructor IF the student writes in the comment section. The rest of the scale [1-5] is rather useless leaving a kind of ambiguous criticism that has one no better off than had there been no evaluation in the first place - except to inflict insult and pain.
Does the instructor motivate the student to learn? When the student marks the 1 or the 5 box, who cares? 5- [strongly agree] might mean that we have a Sidney Poitier in 'To Sir With Love' and 1 [stongly disagree] might mean that the instructor came and read a magazine largely ignoring the students. 2,3, or 4 ? What does it mean? In what way? Why or why not? How has the instructor tried but failed? Specifics, please. How can an instructor correct his behavior in the classroom if he doesn't know what is going wrong or why?

I taught a photography course and one student did leave a comment writing that it wasn't a good class because s/he had to learn by 'trial and error.'
How does one take this? At the very least, one can actually point to a misconception on the part of the student regarding the practice of taking pictures. It is NOT basic math.

Another student circled all the questions in the 5 area with one exception. They didn't feel that they learned anything. Was s/he being honest? Why did they feel that? Was it a reflection on the teacher or the student's own doing? How does that one negative pencil mark on an otherwise 'stellar' evaluation affect the instructor's standing with the administration?

I personally like the idea of writing a paragraph of comments about the class. It would provide useful feedback to the instructor, i.e., constructive criticism!

The section of the article that dealt with the particular way that evalutions are conducted, where the instructor leaves the room, is not to be trusted in handling the forms, is presumed to be prone to cheating, may not be overstated afterall. Consider how politiczed anonymous accusations have become. If the instructor is presumed to be prone to retribution and therefore must not know who is offended, who likes him, who doesn't feel motivated by him, etc., doesn't this create a politicized environment, not to mention one of increasing distrust and paranoia?

Administrations using these student evaluations to retain faculty is unconscionable. The same goes for the hiring committee looking for new instructors.

There's something here about categorizing human behavior into nice neat little boxes that flies in the face of what human behavior really is- namely, complex. But that's another topic, no?

Posted by: cesek at November 14, 2003 11:38 AM

"My feeling is that the real problem with student ratings is that they transform the student into a customer."

And you know what they say about the customer...

Posted by: shkspr at November 14, 2003 01:16 PM

Some here seem to agree that the process of administering teaching evaluations does tell students that we are more likely cheat and sneak. As long as we continue this process, what better alternatives are there? What kind of feeback will we get from our students if we are in class while they are writing the evals?

This story might interest some. Along with measuring our scores against others teaching the same course and/or something just as difficult, sometimes we can measure ourselves against ourselves. Back in grad school, my university used an annoying scale of 1-7, with seven being the highest. One Quarter my overall teaching effectiveness was 6.3 in one section, 6.0 in another, and 5.3 in the third. Same course, same workload, same TA etc... I believe the one that gave me a 5.3 met at noon.

Posted by: DM at November 14, 2003 01:55 PM

One Quarter my overall teaching effectiveness was 6.3 in one section, 6.0 in another, and 5.3 in the third. Same course, same workload, same TA etc... I believe the one that gave me a 5.3 met at noon.

Yes, my colleagues who do multiple sections of the same course have the same result--everyone loves them at one hour, hates them the next...

Very few of our students write anything in the comments, and only once or twice have I had anything helpful. Usually I just get adjectives ("brilliant!" "sucks!" etc.). I tried asking students at midsemester what else I could do in class to help them learn--changes in lecture style, etc.--and got straight "nothings" across the board. Arrgh.

Posted by: Miriam at November 14, 2003 02:24 PM

Hey, I know! Exams are the standard method of getting students to demonstrate what they've learnt, right? So, if it takes a 2 hour exam to measure how much someone learnt in a semester, it follows that a 2 hour essay covering how well they were instructed is also required.

Per subject, of course.

(only a bit serious)

Here's my biggest problem with student evaluations. How am I supposed to know how well a subject was taught? OK, good teachers and bad teachers can sometimes be easy to spot, but what about when the subject itself is hard? At the time, what am I supposed to say about how I feel a course in statistical mechanics, or chemical thermodynamics or whatever was taught?

Ask me a few years down the track, when the stuff we were taught has had to make itself useful on an extended basis. Then I can tell you how good the teaching was.

Posted by: polychrome at November 14, 2003 04:20 PM

Just anecdotal evidence, but I've known some colleagues who were genuinely bad instructors or at least making some very obvious mistakes in handling their classes (e.g. Queegish obsession with idea that students are cheating; routinely missing classes; derogatory remarks about women in science and engineering; failure or refusal to answer questions /provide examples, etc.) Almost invariably, they explained their poor evaluations by claiming that they were upholding standards and rigor, and those "other faculty" weren't.

Not to imply in any way that grading tough and stating that you are doing it really means you're a bad instructor. By the way, in our department grade distributions have been strongly *inversely* correlated with student evaluation ratings.

Posted by: hf at November 14, 2003 05:44 PM

Good point about knowing what one knows and the time required to know it.
That some students FEEL that they haven't learned anything is not an accurate measure of whether they actually learned anything at all.
Yet, this is one of the 'important' questions on the evaluations. Maybe they should be given the opportunity to evaluate an instructor after a semester or two has passed and they've gone on to the next section of a subject where the previously acquired knowledge is reinforced. Then those who didn't feel that they had learned anything will find themselves pleasantly surprised, as well as the opposite.

As far as consumer rights go: Real customers actually do have the chutzpah to come forth when they feel that they are not getting their money's worth. They return items, exchange them, negotiate, etc. I'm not too aware of how this is all done anonymously. Yes, higher education is more complex than that.

Students are quite passive in the classroom these days, or so it seems. Claiming fear of retribution when directly confronting an instructor...what is actually being taught to these students? Keep your mouth shut and complain to the adminstration later behind the instructor's back? Does that actually instigate real change in the classroom? The vast majority of teachers can't be in this because they want to silence 30 students 3 times a week and abuse the respect that students usually have [or have had] for someone who generally knows more. Yet judging from some posts, here and there across the net and the news, colleagues and such, one could get the impression that it is actually the teacher who is the enemy to be defeated. Their goals, their methods, and their ideologies are suspect, masking their real agenda. Students are the informers to the administration and witnesses to the transgression; all of it based on perception and rarely grounded in fact.

Sorry, I'm rather angry that any of this plays so well in the field of education. Seems to belong to some utopia where no one ever feels bad when challenged and learning is always easy. Oh - and really fun, too.

I think I'm off to cooking school or carpentry. Higher education may be what is actually overrated and after 30 years I'm honestly sick about it.

Posted by: cesek at November 14, 2003 06:15 PM

Good griefÖ From my perspective, the problem is not the evaluations themselves or their misuse, itís the complete *lack* of response to them, when warranted, from the administration. When I was a student, I really liked course evaluations and tried to fill them out as thoroughly and thoughtfully as possible. This might make many of you shudder, but at my undergrad school, the evaluations were summarized by a student committee and published in a book that came out before registration every year. These summaries were used (along with word of mouth) to clue us in about which profs should be flocked to and which needed to be avoided like the plague. And in my personal experience, in all but a very few cases they were pretty accurate. (In those few cases, when I liked somebody, most of my classmates did not for a variety of reasons.) Profs I thought were great got great evaluations. Lousy ones got lousy evaluations. There was little correlation between ďdifficultyĒ of the course and evaluation quality. In fact, some of the toughest profs teaching the toughest classes in the school got great evaluations because they were fantastic teachers.

As alwaysfortunate (post #9) pointed out, there are some truly horrific professors out there and students should at least know about them, even if the administration doesnít care. I had a number of real gems. One utterly ghastly prof had taught the same class for two years when my time came. She was dreadful. The lecture hall was always sparsely populated, but she probably didnít notice because she never looked up from the book she mumbled from. Her evaluations were routinely, and well-deservedly, horrid. Somebody the year before me suggested that to improve the course, they might flood the classroom for 40 days and 40 nights and start over, which sounded about right to me. My class uniformly panned her, too. Of course, the administration didnít give a hoot and she was brought back the following year. (I think that was the end of her rotation through that class, luckily for those two years behind me.) This is the real scandal here -- presumably the powers that be were thrilled enough to have somebody who was willing to teach a large intro survey class with lots of premeds (I wasnít one!) that they didnít care that that somebody was incompetent.

I have had a bit of TAing experience myself and appreciated the evaluations and feedback I got. Negative comments did sting briefly, but thatís life. Iíve gotten comments that were far more unfair directed my way in my job here in the non-academic world. And Iíve even gotten them on graded work from professors! If things were generally positive, I tried not to let them bother me. And I was even pleasantly surprised at times to see that students Iíd been tough with didnít hold that fact against me, but recognized that I had at least been fair.

Of course, maybe Iíve been lucky to have been at schools where the students were really interested in learning, and that at other places evaluations are used mainly to gripe about grades (or lack of cookies?!?!?). And perhaps in such cases they have less value. But then the concept is being tarred with an overly broad brush, IMHO.

A lot of readers here probably wonít like the following much, either. Student evaluations can also have legitimate cathartic value for students. I loved singing the praises of great profs, but I also enjoyed honestly evaluating the lousy performance of the handful of truly rotten profs I had. Some of them made our lives pretty miserable all term and it was nice to be able to at least have some outlet for expressing our frustration. Note that these were not ďhardĒ professors or tough graders. These were *bad* professors. People who couldnít teach and didnít try. People who had contempt for the class. People who had no way of communicating with undergrads. People who overtly *didnít care.* People who took their differences with their department/the university out on freshmen in lecture classes. To say that these folks shouldnít be panned is like saying that bad students shouldnít get bad grades. Note as well that I got *Aís* from some of these profs -- my grades from the profs had a weak correlation with my evaluations of the profs, as they were mostly about how much I learned from the classroom portion of the class. I learned a huge amount in a couple of classes I took from awful profs because the books we used were excellent (and, in most cases were the standard books used for the class and not ones specifically selected by the prof) and I was able study from them effectively alone or with classmates. Which made going to class sometimes pointless and sometimes counterproductive when complicated material was presented in a highly confusing way. Nevertheless, I had almost perfect attendance. Now I ask myself why I bothered. Of course, at my research-oriented school there were few if any repercussions for getting poor evaluations, so these profs didnít have the concerns that some of the posters here do.

Posted by: kater at November 14, 2003 06:29 PM

Cesek (post #26) wrote: ďStudents are quite passive in the classroom these days, or so it seems. Claiming fear of retribution when directly confronting an instructor...what is actually being taught to these students? Keep your mouth shut and complain to the adminstration later behind the instructor's back?Ē

Actually, in an introductory quantum mechanics class taught by one of the worst profs I had, some of the students did decide to start asking a lot of questions and challenging the prof in class (by, for example, asking him to slow down and explain a complicated point that he had glossed over to the confusion of at least a majority of the students). The profsís response after a few such questions was to ignore or blow off these students when they raised their hands. Though this guy was (or at least came across as) arrogant and a jerk, I think he actually wanted to do and thought he was doing a good job. In truth, he was in way over his head -- he was probably a very smart theoretician, but he wasnít able bring himself down to a level where he could communicate with undergrads who were getting their first real exposure to a very complicated and theoretical subject heíd probably spent 30+ years immersed in. Fortunately, we had a fabulous textbook that taught me all the quantum I ended up learning in undergrad. (Now, the author of that book is somebody Iíd really like to have a class from!)

Posted by: kater at November 14, 2003 06:42 PM

IA, it seems to me that your defence of the pallid, poorly written, and over-earnest appreciation of evaluations above is just the problem. One isn't allowed to be sharp - the sensitivity troops troop in and tell you how cruel you're being. That's the cause and the effect of teaching evaluation forms (and many others elements of the new university - speech codes come to mind) - they institutionalise inoffensive speech and bland amiability. Notice how many of your posters above talk of bringing milk and cookies to the kiddies at evaluation time. Nice to live in a rich country where everyone is nice to you and everything is just... well... nice.

Posted by: Hitch at November 14, 2003 06:46 PM

My defense of another commenter is "just the problem"? Hitch, since you appear to be new to this site, I'm going to let you in a little secret: when you visit the Invisible Adjunct weblog, you're not hobnobbing with the academic power structure.

Anyway, yes, the milk and cookies routine is a problem. Or at least, it points toward/is symptomatic of a larger problem. More on this later, but in brief: damned if I know what is the solution.

Posted by: Invisible Adjunct at November 14, 2003 07:23 PM

IA: I'm not new to the site - just too cowardly to use the same name all the time. And don't assume your site is a site of powerlessness. It isn't -- at all.

And after all, as you and your posters often say - the numbers in academia are increasingly on your side... to make yourself - yourselves - smaller than you are is a mistake, psychologically and institutionally.

Posted by: Hitch at November 14, 2003 09:39 PM

Okay: we rule!

But we need a new editor. I just noticed that the title of this entry refers to "student evalutions." I sincerely hope no reader has read this as "student evolutions."

Posted by: Invisible Adjunct at November 14, 2003 09:44 PM

I teach on a study abroad program, where we do a lot of experiential field based courses. I'm also trained as a sociologist, and have done a lot of consulting and research work with local communities and NGOs, especially in the field of participatory research--getting local communities to evaluate themselves.

We are not required to do quantitative evals--and I would refuse them if I was, knowing they cannot capture the complexity of what we do. Aside from which, they are usually the classic example of someone designing a survey (which is really what they are) with no pre-testing or training! (And yes, I realize how lucky I am.)

What we do--and could perhaps work in other contexts--is have the students evaluate themselves and then the course in short answer format. However, the KEY point and way to ask the question is that they have to not just criticize--but they must offer a solution to the problem they raise. We take seriously the idea of students as collaborators in the courses--not in an faux PC sort of way--but we are really interested in what students think.

We get the usualy mix of comments, but by and large, since the students know they are being listened to--AND they sign their names--we get great input. You still have to winnow out useful from not useful commentary, but it is one approach which I think can work. I'm not sure how widely, but does work for us.

(For those interested, Robert Chambers is a good place to start if you are interested, also do a search on PAR or PRA. Heaps of good ideas there both for teaching and evaluations which would make some admin types come unglued--but are VERY effective in the real world when helping communities struggling with HIV/AIDS, etc. If it can work with that--where the stakes are high--it can work in the classroom.)

Posted by: A teacher at November 15, 2003 11:09 AM

"I was just taking this as an elective because I thought feminist theory would be easy!"

evaluations for this type of class, or ones like the political economy of race, need to be done in more than one way and with aditional information accounted for. if the feminist theory course, for instance, is taught in more than one section and one section gets better evals than the other, the question must be asked, what sorts of students from what sorts of majors are concentrated in each section. is one section composed of mostly of male students looking for an easy elective while the other is composed mostly of students in the major because of the scheduling of other classes? this would affect the evals no matter how good or bad the teaching. concepts that get taught in courses that address race, gender, and class are harder to swallow for some students than others.

then there's the blood sugar effect. students are groggy after lunch. and crabbier and less sharp unless there's a coffee distribution point somewhere nearby. maybe there should be free chai for students program on all campuses to fix that problem across the board. if there were, maybe our students would bring us free chai from time to time, which would be nice because chai is so delicious yet expensive and we adjuncts get paid so poorly.

Posted by: meanregression at November 15, 2003 12:09 PM

I found student evaluations useful in my first few years of teaching. Though I tried to take their advice seriously, I didn't find it too useful; more useful was the evidence of students' confidence in my ability, which improved my confidence.

But evaluations have grown more useless over time. This is partially attributable to a natural improvement in my own teaching, but it's just as much because I already know from past evaluations what students think of me. And, as little attention as I pay them today, the college administration pays even less attention.

Evaluations are over-administered, and they consequently have less impact than they should, and students don't take them as seriously as they could. In most cases, they serve primarily to placate students by allowing them to vent any frustrations in a harmless direction. We should administer evaluations only when the instructor or the college is going to pay attention, and no more often.

Posted by: matins at November 15, 2003 03:34 PM

As much as I hate the whole ratings system, the numbers often can indicate when someone is retaliating for a bad grade. You'd see the overwhelming majority of marks on the higher end of the scale, and one-two on the lower. It's often obvious what those students are doing.

One Quarter I found myself in a TA assignment outside my field. The course was also changed after I had accepted the offer, and was turned into something completely unfamiliar to me. I worked very hard to learn the material and teach it to my students. My sections were back-back. The first time I met with my earlier section, I was awful, making too many mistakes and giving the students reason to question their TA's competence. I corrected these mistakes and led a very good section the following hour. Although I improved over the Quarter and regained the confidence of at least some of the students in the earlier section, I would periodically make mistakes that would be corrected for those in the following one. My overall TA rating for the earlier section was less than my usual, although far better than what I felt I deserved. My rating in the later one, in which the students and I had a good relationship, things generally went well, and I believe, I actually taught them something, was tops in that department. My point here is that grades can and do influence an instructor's ratings; quality also does. Students will reward someone who works hard, does a good job, and teaches them something.

Posted by: DM at November 15, 2003 05:56 PM

Possibly useful link, for those who take the feedback that their students might give seriously:

The Student Assessment of Learning Gains (SALG) puts the typical student evaluation questions framed more on "what did you learn, and did these parts of the course help you to do it?" It's also -extremely- customizable, and presents some templates that other people have created.

Posted by: Jeff Dougan at November 26, 2003 05:19 PM

If you look at education policy it is obvious that the student evaluation of teaching (SET) is being abused more and more by administrators. Some say the next step is removal of tenure.

I personally think the aim is to re-instate the apprentise-guild model of education. Just make sure your kid does not become labeled journeyman or journeyperson, sorry I slipped. Does that mean we need a law for "No Adult Left Behind"?

Some links I found on legalities:

Posted by: Donald at February 11, 2004 10:04 PM

I think student feedback for instructors is great: I take it very seriously, I read over my comments every semester. But the problem, honestly, has nothing to do with students' appreciation or callowness towards quality teaching. It is the use of this data by administrators (including tenured faculty on retention/promotion committees) which is problematic. At both institutions where I have worked, I have been explicitly told that the ONLY measure of successful teaching is evaluation scores ABOVE THE NORM. Not near the norm, but above the norm.

I could go on and on about statistical sophistication and "everybody above average" thinking. But it's just too painful and absurd.

Posted by: Jonathan Dresner at February 12, 2004 07:32 PM

The whole focus here is on the "excellence of teacher" score on the front page of the IDEA form. I even made a spreadsheet of my scores for my chair to show him how they improved each time I taught the class (in one class) and so the reduced score in another class seemed to be an anomaly. Faculty in my department seem to mostly get very high scores. Though my chair says he doesn't because he doesn't care about teaching. Others in other departments say they don't pay much attention to it since they got tenure.

Posted by: moom at February 13, 2004 10:24 AM