Recent Posts
|
|
First, Do No Harm
by Kavitha Mediratta on June 17 |
|
|
Five Top Challenges for New Leaders of Instructional Rounds
by Thomas Fowler-Finn on May 29 |
|
|
The Arts and UDL: What General Education Can Learn from the Margins
by Don Glass on April 12 |
|
|
Deeper Learning and the Common Core
by Robert Rothman on April 1 |
|
|
Universal Design for Learning and Improving Education for Incarcerated Youth
by Joanne Karger on March 20 |
Reputations (and More) at Risk: Using Value-Added Reports Responsibly
by Hilary Dauffenbach-Tabb on June 14, 2012
The insights gained from teacher value-added reports have the potential to benefit schools, students, and communities. However, because these reports are generated from complex statistical methods that rely on inaccurate or incomplete data and have wide margins of error, more responsible use of these reports is needed to reap their benefits—and minimize their risks. My own story is just one example of how this process can go terribly wrong and, in my view, provides a good entry point for reflection on ways to better align our practice of using these reports with the broader purpose of ensuring an excellent education for all students.
For the past three years, I have worked as a sixth- and seventh-grade math teacher in Brooklyn, N.Y. I have had two value-added scores published on the New York Times SchoolBook website which received the scores from the New York City Department of Education through a Freedom of Information Act request. One of these reports omitted all special education students that I taught in an inclusion classroom. The other contained forty-eight students I never taught. The New York City Department of Education was informed of these mistakes, but refused to generate a corrected report and knowingly released inaccurate data to the media. Unfortunately, I know there are many teachers with similar stories whose professional reputations may have been unduly harmed by the publication of this data.
This negative experience prompted me to think deeply about the potential negative consequences of publishing this data—on teachers and on the school system as a whole. I worry about whether or not I will be able to get a teaching job with this incorrect low-performing label attached to my name. I worry that teachers will avoid working in schools that are under-resourced and under-supported because this may limit their ability to receive high value-added scores. I worry that publicly reporting teachers’ effectiveness will be another reason among many why talented young people will avoid entering the teaching profession or leave just as they are becoming effective teachers.
As someone who is passionate about improving education, I believe we need to improve the quality of value-added reports. We should continue to support research and development so that the statistical models that are used are as reliable and accurate as possible. We should ensure that our data systems have the capacity to store and sort the information that is needed for these reports to be generated. We must allow teachers a chance to identify mistakes in the reports, and we must fix the errors before reports are used.
But we also must consider the ways data can be used responsibly to strengthen the school system. First and foremost, value-added scores must be used in conjunction with other indicators of a teacher’s performance, such as observations or student portfolios. One test, given on one day, should not alone determine a teacher’s or student’s ability level. A comprehensive judgment about a teacher’s ability (informed in part by value-added scores) could be used to improve educational outcomes in the following ways: identifying weak teachers and providing them with targeted professional development, identifying highly effective teachers to serve as mentors for novice or struggling teachers, placing students into classes in ways that ensure they do not have a less effective teacher for two years in a row, and relocating effective teachers to schools that do not have enough strong teachers.
All of these data-driven decisions can positively impact our schools, minimize the negative effects that individual teachers experience from flawed value-added scores, and do not require public release of teachers’ reports. As my story highlights, value-added data holds the potential to both enhance student learning and to hurt the very individuals and schools we are trying to improve.
This blog post continues the conversation from the Harvard Educational Review special symposium “By What Measure?: Mapping and Expanding the Teacher Effectiveness Debate.”
For the past three years, I have worked as a sixth- and seventh-grade math teacher in Brooklyn, N.Y. I have had two value-added scores published on the New York Times SchoolBook website which received the scores from the New York City Department of Education through a Freedom of Information Act request. One of these reports omitted all special education students that I taught in an inclusion classroom. The other contained forty-eight students I never taught. The New York City Department of Education was informed of these mistakes, but refused to generate a corrected report and knowingly released inaccurate data to the media. Unfortunately, I know there are many teachers with similar stories whose professional reputations may have been unduly harmed by the publication of this data.
This negative experience prompted me to think deeply about the potential negative consequences of publishing this data—on teachers and on the school system as a whole. I worry about whether or not I will be able to get a teaching job with this incorrect low-performing label attached to my name. I worry that teachers will avoid working in schools that are under-resourced and under-supported because this may limit their ability to receive high value-added scores. I worry that publicly reporting teachers’ effectiveness will be another reason among many why talented young people will avoid entering the teaching profession or leave just as they are becoming effective teachers.
As someone who is passionate about improving education, I believe we need to improve the quality of value-added reports. We should continue to support research and development so that the statistical models that are used are as reliable and accurate as possible. We should ensure that our data systems have the capacity to store and sort the information that is needed for these reports to be generated. We must allow teachers a chance to identify mistakes in the reports, and we must fix the errors before reports are used.
But we also must consider the ways data can be used responsibly to strengthen the school system. First and foremost, value-added scores must be used in conjunction with other indicators of a teacher’s performance, such as observations or student portfolios. One test, given on one day, should not alone determine a teacher’s or student’s ability level. A comprehensive judgment about a teacher’s ability (informed in part by value-added scores) could be used to improve educational outcomes in the following ways: identifying weak teachers and providing them with targeted professional development, identifying highly effective teachers to serve as mentors for novice or struggling teachers, placing students into classes in ways that ensure they do not have a less effective teacher for two years in a row, and relocating effective teachers to schools that do not have enough strong teachers.
All of these data-driven decisions can positively impact our schools, minimize the negative effects that individual teachers experience from flawed value-added scores, and do not require public release of teachers’ reports. As my story highlights, value-added data holds the potential to both enhance student learning and to hurt the very individuals and schools we are trying to improve.
This blog post continues the conversation from the Harvard Educational Review special symposium “By What Measure?: Mapping and Expanding the Teacher Effectiveness Debate.”
Comments:
| Jun 15, 2012 11:02 AM |
Hilary, this is an incredibly thoughtful and insightful piece about the dangers of using test scores for high-stakes decisions, including the public shaming of teachers through the publication of highly flawed value-added scores. If we want to codify teaching to the test (or worse, cheating as we've seen in Atlanta, DC, Penn, NY, MA, etc.), then using test scores, including VAMs, for high stakes decisions such as hiring, firing, rewarding, shaming, etc. is the best way to do it. Sadly, more and more states and districts are moving in this direction. While I agree with most of what you've written, I do respectfully disagree with your point that these VAM reports have the potential to benefit schools, students, and communities. What no one is talking about, sadly, including in a recently published HEPG by Professor Tom Kane arguing for the use of VAMs as one piece of teacher evaluation is that, because we don't (and can't and shouldn't) randomize kids to teachers, kids to schools, teachers to schools, schools to communities, etc., we can *never* make causal statements about a teacher's effectiveness based on their students' test scores. A high value-added score is just as likely to be due primarily to other factors, including out-of-school factors as well as chance, as it is to strong instruction. This seems counter-intuitive or hard to believe (e.g., it seems to make sense that stronger teachers would have higher VAM scores), but I've had professors at HGSE who are much smarter than I am at statistical analysis show me how this can be true. In fact, one professor showed me that the "value-added" change score for a year may have nothing to do with what a student has learned in terms of the curricular content (i.e., what's measured -- poorly -- on statewide, standardized tests) and have everything to do with the student's natural development, so to credit this change score to a single teacher's instruction is a fallacy and bad science. This professor also told me that VAMs -- which many are claiming to be new and exciting methodologies --were discredited in the 1980s when it was called "residual change" and again in the 70s when it was called something else, which I can't remember off the top of my head. To quote this professor: "If value-added is the best we've got, then we better get something else." And, of course, Professor Koretz shows in his book, Measuring Up, how teaching to the test can raise test scores without actually raising true student learning. So, given that no causal statements can be made about teachers' effectiveness from test scores, coupled with all of the unintended consequences that you point out in your piece, I don't think VAMs should ever be used to inform policy or instruction and I think they do far more harm than good. Anyway, I thought your piece is another great example of how teachers are affected on the ground by misguided policy and the misuse of standardized tests, and keep fighting the good fight in NYC.
– Chris Buttimer |
Submit your comment
The opinions expressed here do not reflect the opinions of the Harvard Education Publishing Group or the Harvard Graduate School of Education. Harvard Education Publishing Group is not responsible for the accuracy of any of the information supplied by guest bloggers.
