However, this process yields sheer lunacy, mostly because of the ridiculous ineptitude of every single person involved. I remember specifically the first grant project I helped to evaluate. The local state government was offering up to $2,000,000 for grant proposals which would help the students in grades 6-8 who had failed their end-of-year standardized reading exam (a well-made test, in my opinion, in which failure basically means illiteracy). The specific project I was evaluating had only gotten $800,000 out of the maximum $2m. Its strategy was to purchase the male students iPod Touches, the female students makeovers, manicures, and pedicures at a local beauty parlor, and all students were offered an additional iPod Touch or Makeover, respectively, if they passed the exam at the end of the current year. The grant proposal had specifically listed these actions as being the goal of the proposal. If the iPods and makeovers were purchased, that constituted success.
When I asked the man who was in charge of the project if he really meant that these actions were the ‘strategy’, not the ‘goal’. He expressed confusion; he thought if the male students had iPod Touches, they obviously would get better at reading, and if the girls got makeovers, it would improve their self esteem and they would be more confident and get better at reading, so obviously isn’t the goal of the project to purchase iPods and Makeovers for the students? I explained to him that the goal was to make students, who had previously failed the exam, pass it on their next try. Success would, obviously, be measured in terms of how many students passed the exam. The strategy was whatever actions you took, whereas the goal was what you were trying to achieve. Now that the project was over, I told him that he had to go look at the reading scores and see if they improved. He couldn’t understand why he had needed to do this, and indeed, refused. I asked him how he had identified the students who he needed to give iPods to in the first place. Did he use their reading scores? Did he ask the school for a list of students who had failed the exam? No. He had asked the school for the free-lunch list, which determined which students came from low-income families, and for the bus route list, which determined which students came from low-income areas. He picked out any students who were on both of these lists who were also black. Since black students tend to have low reading scores, and low-income students tend to have low reading scores, those are the students who need the most help, and so are the students he targeted with his project. When I asked the school for the list of students who had failed the reading exam, it turns out that only 25% (14/56) of the students targeted by the program had failed the reading exam in the first place.
When I wrote up my evaluation, I described in rigorous detail everything the man had done wrong, put in a strong recommendation to not award him grant money in the future, and suggested that some sort of corruption investigation be conducted to see if he had committed any crimes (23 iPods + 23 Makeovers does not total to $800,000, after all). When I submitted this to my boss for approval, she was flabbergasted, and explained that the evaluators job was to collude with the grant proposal submitter, so that we got more evaluation jobs from them in the future. Over the next couple days, we had a long conversation, and in the end, she allowed my evaluation to go through.
The next project I evaluated was just as criminally neglectful as my first. And the next. And the next. In fact, for the first three years I worked at the firm, every single project I evaluated listed their ‘process’ and then said that their ‘goal’ was to enact the process. Every single project had used any subsidized lunch lists, bus route data, or demographic data they could get their hands on to decide which students to target; not a single project actually looked at test scores, for deciding either which students to target or figuring out if the project had even succeeded.
$Money first…….. Reading last, apparently