Katy Welter: Finding the "Fat Catchers" of Criminal Justice

Katy Welter: Finding the "Fat Catchers" of Criminal Justice

"Cook County criminal justice can learn some valuable lessons from the National Institutes of Health (NIH) and the Oakland A's. Namely, experts are often overconfident. They make mistakes because they see the world from a narrow, limited perspective, and they have inadequate, unsystematic information. Their incorrect diagnoses can be tremendously costly. Rather than rely on the personal observations and experience of experts to solve our problems, it may be better to make simple, incremental changes and then observe the results.

In 2002, the NIH abruptly terminated one of the largest medical studies ever conducted. The reason? The treatment provided to tens of thousands of menopausal women, hormone replacement therapy, turned out to increase the patients' risk of heart disease and cancer. This was particularly shocking because, for decades, the treatment had been widely thought to be relatively low risk. Doctors had not detected the complications.

At a recent conference in downtown Chicago, Jens Ludwig of the University of Chicago told a group of prosecutors (and a few observers) about the relevance of this to criminal justice policy. According to Ludwig, the initial hormone replacement studies made the treatment seem safe and effective because the women who signed up for the experimental treatment were likely to be health conscious--and, thus, were especially likely to be healthy. As a result, few of those women developed heart disease or cancers. If the women needing treatment had been randomly assigned to varying therapies, including hormone replacement, the risks of the hormones would have been seen. But the selection bias, as it's called, masked the negative effects, and it was subtle enough to escape the experts' notice, yet significant enough to devastate the NIH trial.

Similarly, innovative criminal justice reforms may produce misleading results if they are not implemented using systematic research methods. For example, a three percent decrease in recidivism is too small to be seen by decision makers active in the day-to-day administration of justice, but a change of that size could be detected in a well-designed experiment. And a 3% reduction in recidivism would save taxpayers and the victims of crime millions of dollars. It is something you would want to know about."