The consequences of this oversight are pernicious. Women are far more likely to suffer the deleterious side effects of medication than men. Pregnant women get sick, but the consequences of taking many medications when pregnant are chronically understudied, or worse yet, unknown entirely. Women are far less likely to receive the correct treatment for heart attacks because their symptoms do not match “typical” (read: male) symptoms.
If evidence-based medicine is already far less evidence-based for anybody who is not a white male, how can the use of this unmodified data do anything other than unwittingly perpetuate this inequality? If we want to use AI to facilitate a more personalized medicine for all, it would help if we could first provide medicine that works for half the population.
The effects of this data can be even more insidious. AI systems often function as black boxes, which means technologists are unaware of how an AI came to its conclusion. This can make it particularly hard to identify any inequality, bias, or discrimination feeding into a particular decision. The inability to access the medical data upon which a system was trained—for reasons of protecting patients’ privacy or the data not being in the public domain—exacerbates this. Even if you had access to that data, the often proprietary nature of AI systems means interrogation would likely be impossible. By masking these sources of bias, an AI system could consolidate and deepen the already systemic inequalities in healthcare, all while making them harder to notice and challenge. Invariably, the result of this will be a system of medicine that is unfairly stacked against certain members of society.