If you want to trust a prediction, you need to understand how all the computations work. For example, in health care, you need to know if the model even applies to your patient. And it’s really hard to troubleshoot models if you don’t know what’s in them. Sometimes models depend on variables in ways that you might not like if you knew what they were doing. For example, with the power company in New York, we gave them a model that depended on the number of neutral cables. They looked at it and said, “Neutral cables? That should not be in your model. There’s something wrong.” And of course there was a flaw in the database, and if we hadn’t been able to pinpoint it, we would have had a serious problem. So it’s really useful to be able to see into the model so you can troubleshoot it.
When did you first get concerned about non-transparent AI models in medicine?
My dad is a medical physicist. Several years ago, he was going to medical physics and radiology conferences. I remember calling him on my way to work, and he was saying, “You’re not going to believe this, but all the AI sessions are full. AI is taking over radiology.” Then my student Alina [Barnett] roped us into studying [AI models that examine] mammograms. Then I realized, OK, hold on. They’re not using interpretable models. They’re using just these black boxes; then they’re trying to explain their results. Maybe we should do something about this.
So we decided we would try to prove that you could construct interpretable models for mammography that did not lose accuracy over their black box counterparts. We just wanted to prove that it could be done.