ASA Statement on Using Value-Added Models for Educational Assessment

American Statistical Association: Many states and school districts have adopted Value-Added Models (VAMs) as part of educational accountability systems. The goal of these models, which are also referred to as Value-Added Assessment (VAA) Models, is to estimate effects of individual teachers or schools on student achievement while accounting for differences in student background. VAMs are … Continue reading ASA Statement on Using Value-Added Models for Educational Assessment

About Value-Added And “Junk Science”

Matthew DiCarlo:

One can often hear opponents of value-added referring to these methods as “junk science.” The term is meant to express the argument that value-added is unreliable and/or invalid, and that its scientific “façade” is without merit.
Now, I personally am not opposed to using these estimates in evaluations and other personnel policies, but I certainly understand opponents’ skepticism. For one thing, there are some states and districts in which design and implementation has been somewhat careless, and, in these situations, I very much share the skepticism. Moreover, the common argument that evaluations, in order to be “meaningful,” must consist of value-added measures in a heavily-weighted role (e.g., 45-50 percent) is, in my view, unsupportable.
All that said, calling value-added “junk science” completely obscures the important issues. The real questions here are less about the merits of the models per se than how they’re being used.
If value-added is “junk science” regardless of how it’s employed, then a fairly large chunk of social scientific research is “junk science.” If that’s your opinion, then okay – you’re entitled to it – but it’s not very compelling, at least in my (admittedly biased) view.

What Value-Added Research Does And Does Not Show

Matthew DiCarlo:

Value-added and other types of growth models are probably the most controversial issue in education today. These methods, which use sophisticated statistical techniques to attempt to isolate a teacher’s effect on student test score growth, are rapidly assuming a central role in policy, particularly in the new teacher evaluation systems currently being designed and implemented. Proponents view them as a primary tool for differentiating teachers based on performance/effectiveness.
Opponents, on the other hand, including a great many teachers, argue that the models’ estimates are unstable over time, subject to bias and imprecision, and that they rely entirely on standardized test scores, which are, at best, an extremely partial measure of student performance. Many have come to view growth models as exemplifying all that’s wrong with the market-based approach to education policy.
It’s very easy to understand this frustration. But it’s also important to separate the research on value-added from the manner in which the estimates are being used. Virtually all of the contention pertains to the latter, not the former. Actually, you would be hard-pressed to find many solid findings in the value-added literature that wouldn’t ring true to most educators.

LAUSD won’t release teacher names with ‘value-added’ scores

Jason Song:

The Los Angeles Unified School District has declined to release to The Times the names of teachers and their scores indicating their effectiveness in raising student performance.
The nation’s second-largest school district calculated confidential “academic growth over time” ratings for about 12,000 math and English teachers last year. This fall, the district issued new ones to about 14,000 instructors that can also be viewed by their principals. The scores are based on an analysis of a student’s performance on several years of standardized tests and estimate a teacher’s role in raising or lowering student achievement.

Much more on value-added assessment, which, in Madison is based on the oft-criticized WKCE.

Value Added Report for the Madison School District

Full Report 1.1MB PDF

Value added is the use of statistical technique to isolate the contributions of schools to measured student knowledge from other influences such as prior student knowledge and demographics. In practice, value added focuses on the improvement of students from one year to the next on an annual state examination or other periodic assessment. The Value-Added Research Center (VARC) of the Wisconsin Center for Education Research produces value-added measures for schools in Madison using the Wisconsin Knowledge and Concepts Examination (WKCE) as an outcome. The model controls for prior-year WKCE scores, gender, ethnicity, disability, English language learner, low-income status, parent education, and full academic year enrollment to capture the effects of schools on student performance on the WKCE. This model yields measures of student growth in schools in Madison relative to each other. VARC also produces value-added measures using the entire state of Wisconsin as a data set, which yields measures of student growth in Madison Metropolitan School District (MMSD) relative to the rest of the state.
Some of the most notable results are:
1. Value added for the entire district of Madison relative to the rest of the state is generally positive, but it differs by subject and grade. In both 2008-09 and 2009-10, and in both math and reading, the value added of Madison Metropolitan School District was positive in more grades than it was negative, and the average value added across grades was positive in both subjects in both years. There are variations across grades and subjects, however. In grade 4, value-added is significantly positive in both years in reading and significantly negative in both years in math. In contrast, value-added in math is significantly positive–to a very substantial extent–in grade 7. Some of these variations may be the result of the extent to which instruction in those grades facilitate student learning on tested material relative to non-tested material. Overall, between November 2009 and November 2010, value-added for MMSD as a whole relative to the state was very slightly above average in math and substantially above average in reading. The section “Results from the Wisconsin Value-Added Model” present these results in detail.
2. The variance of value added across schools is generally smaller in Madison than in the state of Wisconsin as a whole, specifically in math. In other words, at least in terms of what is measured by value added, the extent to which schools differ from each other in Madison is smaller than the extent to which schools differ from each other elsewhere in Wisconsin. This appears to be more strongly the case in the middle school grades than in the elementary grades. Some of this result may be an artifact of schools in Madison being relatively large; when schools are large, they encompass more classrooms per grade, leading to more across-classroom variance being within-school rather than across-school. More of this result may be that while the variance across schools in Madison is entirely within one district, the variance across schools for the rest of the state is across many districts, and so differences in district policies will likely generate more variance across the entire state. The section “Results from the Wisconsin Value-Added Model” present results on the variance of value added from the statewide value-added model. This result is also evident in the charts in the “School Value-Added Charts from the MMSD Value-Added Model” section: one can see that the majority of schools’ confidence intervals cross (1) the district average, which means that we cannot reject the hypothesis that these schools’ values added are not different from the district average.
Even with a relatively small variance across schools in the district in general, several individual schools have values added that are statistically significantly greater or less than the district average. At the elementary level, both Lake View and Randall have values added in both reading and math that are significantly greater than the district average. In math, Marquette, Nuestro Mundo, Shorewood Hills, and Van Hise also have values added that are significantly greater than the district average. Values added are lower than the district average in math at Crestwood, Hawthorne, Kennedy, and Stephens, and in reading at Allis. At the middle school level, value added in reading is greater than the district average at Toki and lower than the district average at Black Hawk and Sennett. Value added in math is lower than the district average at Toki and Whitehorse.
3. Gaps in student improvement persist across subgroups of students. The value-added model measures gaps in student growth over time by race, gender, English language learner, and several other subgroups. The gaps are overall gaps, not gaps relative to the rest of the state. These gaps are especially informative because they are partial coefficients. These measure the black/white, ELL/non-ELL, or high-school/college-graduate-parent gaps, controlling for all variables available, including both demographic variables and schools attended. If one wanted to measure the combined effect of being both ELL and Hispanic relative to non-ELL and white, one would add the ELL/non-ELL gap to the Hispanic/white gap to find the combined effect. The gaps are within-school gaps, based on comparison of students in different subgroups who are in the same schools; consequently, these gaps do not include any effects of students of different subgroups sorting into different schools, and reflect within-school differences only. There does not appear to be an evident trend over time in gaps by race, low-income status, and parent education measured by the value-added model. The section “Coefficients from the MMSD Value-Added Model” present these results.
4. The gap in student improvement by English language learner, race, or low-income status usually does not differ substantively across schools; that between students with disabilities and students without disabilities sometimes does differ across schools. This can be seen in the subgroup value-added results across schools, which appear in the Appendix. There are some schools where value-added for students with disabilities differs substantively from overall value- added. Some of these differences may be due to differences in the composition of students with disabilities across schools, although the model already controls for overall differences between students with learning disabilities, students with speech disabilities, and students with all other disabilities. In contrast, value-added for black, Hispanic, ELL, or economically disadvantaged students is usually very close to overall value added.
Value added for students with disabilities is greater than the school’s overall value added in math at Falk and Whitehorse and in reading at Marquette; it is lower than the school’s overall value added in math at O’Keefe and Sennett and in reading at Allis, Schenk, and Thoreau. Value added in math for Hispanic students is lower than the school’s overall value added at Lincoln, and greater than the school’s overall value added at Nuestro Mundo. Value added in math is also higher for ELL and low-income students than it is for the school overall at Nuestro Mundo.

Much more on “value added assessment”, here.

‘Value-added’ teacher evaluations: Los Angeles Unified tackles a tough formula

Teresa Watanabe:

In Houston, school district officials introduced a test score-based evaluation system to determine teacher bonuses, then — in the face of massive protests — jettisoned the formula after one year to devise a better one.
In New York, teachers union officials are fighting the public release of ratings for more than 12,000 teachers, arguing that the estimates can be drastically wrong.
Despite such controversies, Los Angeles school district leaders are poised to plunge ahead with their own confidential “value-added” ratings this spring, saying the approach is far more objective and accurate than any other evaluation tool available.
“We are not questing for perfect,” said L.A. Unified’s incoming Supt. John Deasy. “We are questing for much better.”

Much more on “Value Added Assessment“, here.

Study backs ‘value-added’ analysis of teacher effectiveness

Classroom effectiveness can be reliably estimated by gauging students’ progress on standardized tests, Gates foundation study shows. Results come amid a national effort to reform teacher evaluations.
Teachers’ effectiveness can be reliably estimated by gauging their students’ progress on standardized tests, according to the preliminary findings of a large-scale study released Friday by leading education researchers.
The study, funded by the Bill and Melinda Gates Foundation, provides some of the strongest evidence to date of the validity of “value-added” analysis, whose accuracy has been hotly contested by teachers unions and some education experts who question the use of test scores to evaluate teachers.
The approach estimates a teacher’s effectiveness by comparing his or her students’ performance on standardized tests to their performance in previous years. It has been adopted around the country in cities including New York; Washington, D.C.; Houston; and soon, if local officials have their way, Los Angeles.
The $45-million Measures of Effective Teaching study is a groundbreaking effort to identify reliable gauges of teacher performance through an intensive look at 3,000 teachers in cities throughout the country. Ultimately, it will examine multiple approaches, including using sophisticated observation tools and teachers’ assessments of their own performance

Much more on value added assessment, here.

A Look at Madison’s Use of Value Added Assessment

Lynn Welch:

In the two years Madison has collected and shared value-added numbers, it has seen some patterns emerging in elementary school math learning. But when compared with other districts, such as Milwaukee, Kiefer says there’s much less variation in the value- added scores of schools within the Madison district.
“You don’t see the variation because we do a fairly good job at making sure all staff has the same professional development,” he says.
Proponents of the value-added approach agree the data would be more useful if the Wisconsin Department of Public Instruction were to establish a statewide value-added system. DPI is instead developing an assessment system to look at school-wide trends and improve instruction for individual students.
…..
But some question whether value-added data truly benefits all students, or is geared toward closing the gap between high- and low-performing students.
“Will the MMSD use new assessments…of students’ progress to match instruction levels with demonstrated learning levels?” asks Lorie Raihala, a Madison parent who is part of a group seeking better programming for high-achieving ninth- and 10th-graders at West High School. “So far the district has not done this.”
Others are leery of adding another measurement tool. David Wasserman, a teacher at Sennett Middle School and part of a planning group pushing to open Badger Rock Middle School, a green charter (see sidebar), made national news a few years ago when he refused to administer a mandatory statewide test. He still feels that a broad, student-centered evaluation model that takes multiple assessments into account gives the best picture.
“Assessment,” he says, “shouldn’t drive learning.”

Notes and links on “Value Added Assessment“, and the oft-criticized WKCE, on which it is based, here.

Adding Value to the Value-Added Debate

Liam Goldrick & Dr. Sara Goldrick-Rab

Seeing as I am not paid to blog as part of my daily job, it’s basically impossible for me to be even close to first out of the box on the issues of the day. Add to that being a parent of two small children (my most important job – right up there with being a husband) and that only adds to my sometimes frustration of not being able to weigh in on some of these issues quickly.
That said, here is my attempt to distill some key points and share my opinions — add value, if you will — to the debate that is raging as a result of the Los Angeles Times’s decision to publish the value-added scores of individual teachers in the L.A. Unified School District.
First of all, let me address the issue at hand. I believe that the LA Times’s decision to publish the value-added scores of individual teachers was irresponsible. Given what we know about the unreliability and variability in such scores and the likelihood that consumers of said scores will use them at face value without fully understanding all of the caveats, this was a dish that should have been sent back to the kitchen.
Although the LA Times is not a government or public entity, it does operate in the public sphere. And it has a responsibility as such an actor. Its decision to label LA teachers as ‘effective’ and ‘ineffective’ based on suspect value-added data alone is akin to an auditor secretly investigating a firm or agency without an engagement letter and publishing findings that may or may not hold water.
Frankly, I don’t care what positive benefits this decision by the LA Times might have engendered. Yes, the district and the teachers union have agreed to begin negotiations on a new evaluation system. Top district officials have said they want at least 30% of a teacher’s review to be based on value-added and have wisely said that the majority of the evaluations should depend on classroom observations. Such a development exonerates the LA Times, as some have argued. In my mind, any such benefits are purloined and come at the expense of sticking it — rightly in some cases, certainly wrongly in others — to individual teachers who mostly are trying their best.

Value Added Models& Student Information Systems

147K PDF via a Dan Dempsey email:

The following abstract and conclusion is taken from:
Volume 4, Issue 4 – Fall 2009 – Special Issue: Key Issues in Value-Added Modeling
Would Accountability Based on Teacher Value Added Be Smart Policy? An Examination of the Statistical Properties and Policy Alternatives
Douglas N. Harris of University of Wisconsin Madison
Education Finance and Policy Fall 2009, Vol. 4, No. 4: 319-350.
Available here:
http://www.mitpressjournals.org/doi/pdfplus/10.1162/edfp.2009.4.4.319
Abstract
Annual student testing may make it possible to measure the contributions to student achievement made by individual teachers. But would these “teacher value added” measures help to improve student achievement? I consider the statistical validity, purposes, and costs of teacher value-added policies. Many of the key assumptions of teacher value added are rejected by empirical evidence. However, the assumption violations may not be severe, and value-added measures still seem to contain useful information. I also compare teacher value-added accountability with three main policy alternatives: teacher credentials, school value-added accountability, and formative uses of test data. I argue that using teacher value-added measures is likely to increase student achievement more efficiently than a teacher credentials-only strategy but may not be the most cost-effective policy overall. Resolving this issue will require a new research and policy agenda that goes beyond analysis of assumptions and statistical properties and focuses on the effects of actual policy alternatives.
6. CONCLUSION
A great deal of attention has been paid recently to the statistical assumptions of VAMs, and many of the most important papers are contained in the present volume. The assumptions about the role of past achievement in affecting current achievement (Assumption No. 2) and the lack of variation in teacher effects across student types (Assumption No. 4) seem least problematic. However, unobserved differences are likely to be important, and it is unclear whether the student fixed effects models, or any other models, really account for them (Assumption No. 3). The test scale is also a problem and will likely remain so because the assumptions underlying the scales are untestable. There is relatively little evidence on how administration and teamwork affect teachers (Assumption No. 1).

Related: Value Added Assessment, Standards Based Report Cards and Los Angeles’s Value Added Teacher Data.
Many notes and links on the Madison School District’s student information system: Infinite Campus are here.

Who’s teaching L.A.’s kids? A Times “Value Added” analysis, using data largely ignored by LAUSD, looks at which educators help students learn, and which hold them back.

Jason Felch, Jason Song and Doug Smith

The fifth-graders at Broadous Elementary School come from the same world — the poorest corner of the San Fernando Valley, a Pacoima neighborhood framed by two freeways where some have lost friends to the stray bullets of rival gangs.
Many are the sons and daughters of Latino immigrants who never finished high school, hard-working parents who keep a respectful distance and trust educators to do what’s best.
The students study the same lessons. They are often on the same chapter of the same book.
Yet year after year, one fifth-grade class learns far more than the other down the hall. The difference has almost nothing to do with the size of the class, the students or their parents.
It’s their teachers.
With Miguel Aguilar, students consistently have made striking gains on state standardized tests, many of them vaulting from the bottom third of students in Los Angeles schools to well above average, according to a Times analysis. John Smith’s pupils next door have started out slightly ahead of Aguilar’s but by the end of the year have been far behind.

Much more on “Value Added Assessment” and teacher evaluations here. Locally, Madison’s Value Added Assessment evaluations are based on the oft criticized WKCE.

“Value Added Assessment” Madison School Board’s Performance & Achievement Committee Looks at “A Model to Measure Student Performance”

Video / 20MB Mp3 Audio Superintendent Art Rainwater gave a presentation on “Value Added Assessment” to the Madison School Board’s Performance & Achievement committee Monday evening. Art described VAA “as a method to track student growth longitudinally over time and to utilize that data to look at how successful we are at all levels of … Continue reading “Value Added Assessment” Madison School Board’s Performance & Achievement Committee Looks at “A Model to Measure Student Performance”

Florida Teachers’ Union Sues State on Data-Based Teacher Evaluations

Laura Waters:

Motoko Rich in the New York Times describes the federal lawsuit, initiated by seven Florida teachers with support from local NEA affiliates, which contends that the Florida DOE’s system of grading teachers based on student outcomes “violates teachers’ rights of due process and equal protection.”

Much more on “value added assessment, here“. Madison’s value added assessment scheme relies on the oft-criticized WKCE.

UW-Madison lab works with controversial data for Chicago schools

Todd Finkelmeyer:

Nearly 30,000 public school teachers and support staff went on strike in Chicago this past week in a move that left some 350,000 students without classes to attend.
And while this contentious battle between Chicago Public Schools and the Chicago Teachers Union blew up due to a range of issues — including compensation, health care benefits and job security concerns — one of the key sticking points reportedly was over the implementation of a new teacher evaluation system.
That’s noteworthy locally because researchers with UW-Madison’s Value Added Research Center (VARC) have been collaborating with the Chicago Public Schools for more than five years now in an effort to develop a comprehensive program to measure and evaluate the effectiveness of schools and teachers in that district.
But Rob Meyer, the director of VARC, says his center has stayed above the fray in this showdown between the Chicago teachers and the district, which appears close to being resolved.
“The controversy isn’t really about the merits of value-added and what we do,” says Meyer. “So we’ve simply tried to provide all the stakeholders in this discussion the best scientific information we can so everybody knows what they’re talking about.”

Much more on “value added assessment”, here.

I dare you to measure the “value” I add

Summer:

(When i wrote this, I had no idea just how deeply this would speak to people and how widely it would spread. So, I think a better title is I Dare You to Measure the Value WE Add, and I invite you to share below your value as you see it.)
Tell me how you determine the value I add to my class.
Tell me about the algorithms you applied when you took data from 16 students over a course of nearly five years of teaching and somehow used it to judge me as “below average” and “average”.
Tell me how you can examine my skills and talents and attribute worth to them without knowing me, my class, or my curriculum requirements.
Tell me how and I will tell you:

Much more on “value added assessment“, here.

New York Releases Teacher Rankings

Lisa Fleisher:

New York City on Friday released internal rankings of about 18,000 public schoolteachers who were measured over three years on their ability to affect student test scores.
The release of teacher’s job-performance data follows a yearlong legal battle with the United Federation of Teachers, which sued to block the release and protect teachers’ privacy. News organizations, including The Wall Street Journal, had requested the data in 2010 under the state Freedom of Information Law.
Friday’s release covers math and English teachers active between 2007 and 2010 in fourth- through eighth-grade classrooms. It does not include charter school teachers in those grades.
Schools Chancellor Dennis Walcott, who has pushed for accountability based on test scores, cautioned that the data were old and represented just one way to look at teacher performance.
“I don’t want our teachers disparaged in any way and I don’t want our teachers denigrated based on this information,” Mr. Walcott said Friday while briefing reporters on the Teacher Data Reports. “This is very rich data that has evolved over the years. … It’s old data and it’s just one piece of information.”

Related:

Time for Seattle Public Schools and teachers to partner in steps toward reform

Seattle Times:

SEATTLE Public Schools is right to push for a better, more honest way of evaluating teachers, even at the risk of a strike.
Tense contract negotiations between the district and the Seattle Education Association underscore the enormous opportunity at stake. Both sides agree the current system used to judge teachers is weak and unreliable. Ineffective teachers are ignored or shuffled to other schools to become other parents’ nightmare. Excellent teachers languish in a system that has no means to recognize or reward them.
The union leadership called for a few tweaks. But the district proposed a revamped system using student growth, as measured by test scores. Supporters of the status quo have tried to downplay the other forms of appraisal that would be used. They include student growth measurements selected by the teacher, principal observations of instruction and peer reviews. Also, student input at the high-school level.

Much more an value added assessment, here.

We can measure teacher effectiveness, objectively and transparently Read more: Richmond County Daily Journal – We can measure teacher effectiveness objectively and transparently

Terry Stoops

Is it possible to measure teacher effectiveness? For decades, public school principals have subjected teachers to a battery of observations and evaluations purportedly designed to assess the quality of classroom instruction. Rather than yield appreciable information, however, these kinds of teacher assessments merely served as one of the few formal requirements needed to attain lifetime job security, also known as tenure.
On the other hand, the “value-added” method of teacher evaluation continues to show promise as an objective and reliable assessment of teacher quality. Value-added analysis uses standardized tests to estimate teacher effectiveness. This powerful evaluation method employs advanced statistical techniques to project the future performance of individual students based on their past performance. The difference between the projected and actual performance of students determines the value added or subtracted by the teacher.
Value-added analysis has upended the conventional wisdom on teacher quality. For years, public school advocacy groups complained that the most talented teachers snub minority and low-income schools by migrating to less challenging and higher paying schools in culturally and economically homogeneous suburbs.

Terry Stoops is director of education studies at The John Locke Foundation.

The L.A. Times Flunks L.A. Schoolteachers The newspaper takes on the two L.A. sacred cows–teachers and unions–and lives to print again!

Jack Shafer

Nobody but a schoolteacher or a union acolyte could criticize the Los Angeles Times’ terrific package of stories–complete with searchable database–about teacher performance in the Los Angeles Unified School District.
Union leader A.J. Duffy of the United Teachers Los Angeles stupidly called for a boycott of the Times. Boycotts can be sensible things, but threatening to boycott a newspaper is like threatening to throw it into a briar patch. Hell, Duffy might as well have volunteered to sell Times subscriptions, door-to-door, as to threaten a boycott. Doesn’t he understand that the UTLA has no constituency outside its own members and lip service from members of other Los Angeles unions? Even they know the UTLA stands between them and a good education for their children.
Duffy further grouched that the Times was “leading people in a dangerous direction, making it seem like you can judge the quality of a teacher by … a test.” [Ellipsis in the original.] Gee, Mr. Duffy, aren’t students judged by test results?
American Federation of Teachers President Randi Weingarten also knocked the Times for publishing the database that measures the performance of 6,000 elementary-school teachers. Weingarten went on to denounce the database as “incomplete data masked as comprehensive evaluations.” Of course, had the Times analysis flattered teachers, Weingarten would be praising the results of the analysis.

Formula to Grade Teachers’ Skill Gains in Use, and Critics

Sam Dillon

How good is one teacher compared with another?
A growing number of school districts have adopted a system called value-added modeling to answer that question, provoking battles from Washington to Los Angeles — with some saying it is an effective method for increasing teacher accountability, and others arguing that it can give an inaccurate picture of teachers’ work.
The system calculates the value teachers add to their students’ achievement, based on changes in test scores from year to year and how the students perform compared with others in their grade.
People who analyze the data, making a few statistical assumptions, can produce a list ranking teachers from best to worst.
Use of value-added modeling is exploding nationwide. Hundreds of school systems, including those in Chicago, New York and Washington, are already using it to measure the performance of schools or teachers. Many more are expected to join them, partly because the Obama administration has prodded states and districts to develop more effective teacher-evaluation systems than traditional classroom observation by administrators.

New annual Wisconsin school testing system on hold

Amy Hetzner:

Nearly six months after the state announced it was scrapping its annual test for public school students, efforts to replace it with a new assessment are on hold and state officials now estimate it will take at least three years to make the switch.
The reason for the delay is tied to what is happening in the national education scene.
Wisconsin is among the 48 states that have signed onto the Common Core State Standards Initiative, which expects to complete work on grade-by-grade expectations for students in English and math by early spring. Once that is done, the anticipation is that the state will adopt the new standards, using them to help craft the new statewide test.
Wisconsin officials also are planning to compete for part of $350 million that the U.S. Education Department plans to award in the fall to state consortiums for test development.

The WKCE (Wisconsin Knowledge & Concepts Exam) has been criticized for its lack of rigor. The Madison School District is using the WKCE as the basis for its value added assessment initiative.