Madison School District Administration 2.8MB PDF:
I. Introduction
A. Title or topic – District Evaluation Protocol – The presentation is in response to the need to provide timely and prioritized information to the Board of Education around programs and interventions used within the District. The report describes a recommended approach to formalizing the program evaluation process within the District.
B. Presenters
Kurt Kiefer – Chief Information Office/Director of Research and Evaluation
Lisa Wachtel– Executive Director of Teaching & Learning
Steve Hartley – Chief of Staff
C. Background information – As part of the strategic plan it was determined that priority must be given to systematically collect data around programs and services provided within the district. The purposes for such information vary from determining program and intervention effectiveness for specific student outcomes, to customer satisfaction, to cost effectiveness analyses. In addition, at the December 2009 Board meeting the issue of conducting program evaluation in specific curricular areas was discussed. This report provides specific recommendations on how to coordinate such investigations and studies.
D. Action requested – The administration is requesting that the Board approve this protocol such that it becomes the model by which priority is established for conducting curricular, program, and intervention evaluations into the future.
II. Summary of Current Information
A. Synthesis of the topic· School districts are expected to continuously improve student achievement and ensure the effective use of resources. Evaluation is the means by which school systems determine the degree to which schools, programs, departments, and staff meet their goals as defined by their roles and responsibilities. It involves the collection of data that is then transformed into useful results to inform decisions. In particular, program evaluation is commonly defined as the systematic assessment of the operation and/or outcomes of a program, compared to a set of explicit or implicit standards as a means of contributing to the improvement of the program.
Program evaluation is a process. The first step to evaluating a program is to have a clear understanding of why the evaluation is being conducted in the first place. Focusing the evaluation helps an evaluator identify the most crucial questions and how those questions can be realistically answered given the context of the program and resources available. With a firm understanding of programs and/or activities that might be evaluated, evaluators consider who is affected by the program (stakeholders) and who might receive and or use information resulting from the evaluation (audiences). It is critical that the administration work with the
Evaluating the effectiveness of Madison School District expenditures on curriculum (such as math and reading recovery) along with professional development (adult to adult programs) has long been discussed by some Board and community members.
On page 2 of this 55-page document:
“All new programs and interventions should be selected based on the existing evidence of success. Secondary research should be conducted to determine the level of rigor of existing evidence. Quantitative effects on student achievement using randomized trials or quasi-experimental designs that involve treatment and control/comparison groups should be available. The research should be conducted by non-involved third parties, I.e. [sic], not vendors researching their own programs and interventions. The evidence should also examine whether or not the program or intervention has clearly defined methods for assuring high quality implementation, i.e .. fidelity.”
This is a strong standard for considering implementation for new programs and interventions. Unfortunately, I see very little evidence that MMSD has EVER considered scientific evidence for or against a new curriculum or “buy-in-to-school” program, much less in looking at experimentation design standards, and considering who has designed a “study”, and paid for it, relative to who benefits financially or professionally from it being implemented as quickly as possible. Our elementary and middle schools are given a lot of power to implement first, and ask questions about its value later. They pay “experts” to come in and train (or buy books about it!) for some new program that has been designed by the person selling it, with almost no investigation ahead of time into third party studies or reviews. This is not a one-school phenomenon either, though it does happen more in our elementary schools with micro-managing principals who desire to make a name for themselves quickly and be seen as a “rescuer” of a particular school that couldn’t possibly succeed without their current administrator. These experiments have been undertaken on our family’s children at several different schools with no warning, no evaluation of what was working before pitching a whole system that existed already, and very little real training or understanding for the teachers involved in implementing it daily: math curricula with little to no third-party data backing up their effectiveness; “culturally relevant” education that seems to be making core area test scores drop and not rise; different standards for different groups of people that allow parents to decide if it is “convenient” for them to get their kids to school on time for the majority of school days, with no repercussions at all for parents if they decide it is not worthwhile, though with huge repercussions for their children’s education, and the education of others in their class who are interrupted almost every single day by 10-20% of the class coming late each morning; encouraging kids to “explore their inner bling” and to find that if basketball (or music, football, etc.) makes more sense to them than math or reading, why that’s fine, that is the gift they should encourage at the expense of academic subjects, because that is what they can assume they will do the rest of their lives; the list goes on and on. And these are the attitudes and programs that our children are exposed to every single day, suffusing all areas, counter-attacking many families’ standards for accepting personal responsibility, striving in academics to learn and grow, accepting that others will have different views and backgrounds without having “the wrong” backgrounds and views, and so on.
This is an impressive-sounding document. The problem is, it only “recommends” that programs be implemented with evaluation in mind, that they be run by the School Board first for consideration in curriculum committees, etc. After the first year of implementation (often with no design for evaluation of effectiveness), they are often transported whole to a new school setting, without even waiting for any evaluation of effectiveness, or even real data collection. Because they “sound like they should work”.
Enough “should be” – what about need to and will be… What WILL be done?