2006 MMSD WKCE Scores: A Closer Look

Test scores from the November 2006 Wisconsin Knowledge and Concepts Examination (WKCE) and companion Wisconsin Alternate Assessment (WAA) were released by the state Department of Public Instruction this week. The MMSD press release on Madison students' scores ("Despite changes and cuts, Madison students test well") reports the following "notable achievements":

1. that reading scores have remained steady and math scores have gone up;
2. that non-low income MMSD students score better than their non-low income peers statewide;
3. that a higher percentage of MMSD African-American students perform at the highest proficiency level than do other African-American students across the state as a whole; and
4. that a consistently higher percentage of MMSD students perform at the highest proficiency level than do students across the state as a whole.

Let's take a closer look at the PR and the data:

1: "Reading scores have remained steady and math scores have gone up."

[boxed text and charts excerpted from MMSD press release]

This chart is misleading. In 2002, the Wisconsin Department of Public Instruction revamped the scoring scale for the WKCE to lower the cut score (or threshold) for the "proficient" category (which is apparent in the jump between pre-2002 and post-2002 scores, as shown above). The DPI web site clearly states that "Proficiency data for November 2002 and later are not comparable to earlier years."

To provide an accurate basis for comparison, the chart should have looked like this:

The corrected chart shows that the percentage of MMSD students scoring at the advanced+proficient levels in reading declined from 2003 to 2006, and that the increase in the percentage of MMSD students scoring at the advanced+proficient level in math increased only slightly from 2003 to 2006. Although numeric percentages aren't specified, it's apparent that the percentage decline in reading exceeded the percentage increase in math. (So, if reading proficiency levels are being described as having remained "steady" because the decline wasn't statistically significant, the minimal change in math proficiency levels can't be touted as a noteworthy increase.)

2: "Non-low income MMSD students score better than their non-low income peers statewide."

There's no question about the data here. But what about the rest of the picture?

In reading and math, a greater percentage of MMSD low-income students scored at the basic+minimal levels (i.e. below grade level) than their peers statewide this year (scores below are for combined grades).

	MMSD	State
Reading	43.3%	32.7%
Math	49.2%	43.0%

Table 1: 2006 low-income, basic+minimal, combined grades

(This and all data are from the DPI web site, using WKCE+WAA scores.)

Looking at 4th grade scores, the percentages of low-income MMSD students performing below grade level in reading and math grew from 2003 to 2006, and grew at a faster rate than statewide peers.

	MMSD	State
2003	33.1%	29.6%
2006	43.2%	31.8%

Table 2: 2003 and 2006 reading, low-income, basic+minimal, 4th grade

	MMSD	State
2003	47.5%	41.4%
2006	48.7%	38.2%

Table 3: 2003 and 2006 math, low-income, basic+minimal, 4th grade

Looking at more 4th grade scores, a greater percentage of non-low income MMSD students score at the advanced level in reading and math than low income MMSD students, and this gap between high-performing non-low income and low income MMSD students grew from 2003 to 2006.

	Low income	Non-low income	Gap
2003	19.1%	56.7%	37.6%
2006	16.8%	64.5%	47.7%

Table 4: 2003 and 2006 reading, advanced, MMSD 4th grade

	Low income	Non-low income	Gap
2003	9.7%	43.4%	33.7%
2006	15.0%	57.8%	42.8%

Table 5: 2003 and 2006 math, advanced, MMSD 4th grade

This gap between low and non-low income performance at the advanced level exists across the state, but MMSD's gap grew at a faster rate.

	MMSD	State
2003	37.6%	27.8%
2006	47.7%	29.2%

Table 6: 2003 and 2006 reading, advanced, 4th grade, gap between low-income and non-low income

	MMSD	State
2003	33.7%	21.5%
2006	42.8%	25.3%

Table 7: 2003 and 2006 math, advanced, 4th grade, gap between low-income and non-low income

3: "A higher percentage of MMSD African American students perform at the highest proficiency level than do other African American students across the state as a whole."

The scale of the percentage range (y-axis) in this chart is magnified in a way that exaggerates this "achievement". (Even so, it's clear that grade by grade, black students don't outperform their state peers in grades 3, 4, or 10.)

	MMSD	State	Difference
Combined grades	17.1%	15.0%	2.1%
4th grade	16.3%	16.3%	0.0%

Table 8: 2006 reading, advanced, African-American

The scale for math is even more exaggerated, and the achievement somewhat less than "especially significant."

	MMSD	State	Difference
Combined grades	9.7%	7.9%	1.8%
4th grade	9.0%	10.5%	-1.5%

Table 9: 2006 math, advanced, African-American

What is especially significant, however, is the achievement gap between black and white students at the advanced level. MMSD's achievement gap exceeds that for the state, and has grown at a significantly faster rate.

	MMSD	State
2003	38.1%	31.7%
2006	48.9%	31.9%

Table 10: 2003 and 2006 reading, advanced, 4th grade, gap between white and black students

	MMSD	State
2003	36.5%	24.4%
2006	48.6%	30.3%

Table 11: 2003 and 2006 math, advanced, 4th grade, gap between white and black students

4: "A consistently higher percentage of MMSD students perform at the highest proficiency level than do students across the state as a whole."

However, MMSD's racial and economic achievement gaps at the advanced level exceed those for the state.

	MMSD	State
Reading	46.1%	34.8%
Math	41.4%	29.2%

Table 12: 2006, advanced, combined grades, gap between white and black students

	MMSD	State
Reading	45.0%	29.8%
Math	39.7%	24.5%

Table 13: 2006, advanced, combined grades, gap between non-low income and low income students

5: "A significant change in testing procedures resulted in a significantly increased percentage of students scoring in the lowest proficiency category."

Without more data (Exactly how much of the percentage increase in this category was attributable to this testing procedure change? How did this increase compare to other districts and the state as a whole, since they were also affected by this same testing procedure change?), this is not sufficient to explain away the increase in the below-grade level category.

Further thoughts:

1. Curriculum Effectiveness: It's reasonable to assume that students in outperforming categories (white and non-low income) are more likely to have extracurricular educational support and supplementation than other students, and are more likely to be able to overcome curriculum deficiencies they encounter in the MMSD. Any inference that the MMSD deserves all the credit for such students' achievements is misplaced. On the other hand, the success or failure of students at the bottom of the widening racial and economic achievement gaps (and who are likely to lack those extracurricular advantages) is highly dependent on the effectiveness of MMSD's curriculum and instructional choices. For example, we might look at the schools that would have qualified for the Reading First funding that was rejected by the MMSD in 2004, and the percentages of students in 4th grade that are reading below grade level:

   	2003	2006	Increase
   Orchard Ridge	18.4%	28.6%	10.2%
   Hawthorne	30.6%	30.6%	0.0%%
   Glendale	20.6%	47.4%	27.4%
   Lincoln	25.0%	50.0%	25.0%
   MMSD	18.4%	22.7%	4.3%
   State	17.4%	18.1%	0.7%

   Table 14: Reading, basic+minimal, 4th grade

2. Rigor of WKCE: NAEP scores for 2007 (when they're released) should be compared against WKCE scores as a reality check on whether WKCE testing and standards are consistent or are softening.

3. Hidden achievement gap: The WKCE scores in the "proficient" category should be examined on a disaggregated basis. If historically underperforming groups are clustered at the lower end of the category (yet are still being identified as "proficient" due to the lowering of the cut scores when the category standards were redefined in 2002), this is an achievement gap too, and shouldn't be ignored.

Posted by Chan Stroman at May 25, 2007 9:24 AM
Subscribe to this site via RSS/Atom: Newsletter signup | Send us your ideas