Anticipating the Inevitable: Utilizing Machine Learning in Healthcare Infrastructure

Machine learning (ML) algorithms, once confined to defeating Grand Masters in chess and filling social media feeds with personalized content, are now making their way into clinical medicine. This shift, while promising, raises concerns about potential biases and unintended consequences, as highlighted in a recent editorial published in the American Journal of Bioethics.

The editorial, penned by Jonathan H. Chen MD, PhD and Abraham Verghese MD, MACP, discusses the amplifying effect of ML systems, which can make us better at doing whatever it is we are already doing, including perpetuating social biases if those biases exist.

A prime example of this is the case of Microsoft's artificially intelligent chatbot, Tay. Initially designed to engage in friendly banter, Tay picked up and magnified the sort of rhetoric that is common on Twitter, demonstrating the potential for ML to replicate and amplify biases. In response, her designers explicitly programmed her to be blind, deaf, and dumb to race, religion, gender, and any other politically sensitive subject in her next iteration, Zo.

Unfortunately, this isn't just a problem in the realm of social media. In the healthcare industry, ML applications have the potential to automatically diagnose medical images and drive medical decision making. However, there are concerns about the potential for these algorithms to exacerbate issues such as racial profiling when used in medical decisions.

A key study found that one insurer's ML algorithm was more likely to offer help to white patients over equally sick black patients, leading to state regulatory investigation for unethical conduct and bias. Similarly, the company Amazon had to scrap a machine learning algorithm designed to automatically screen job candidate resumes due to the algorithm learning to be overtly biased against women.

To address these concerns, Char et al offer a framework to address ethical concerns about ML in healthcare at every step, from the initial premise, the diversity of the team building the instrument, the validity of the data set used in the training algorithms, to their eventual deployment. The framework emphasizes the importance of explainability and auditability, particularly in the development and debugging phase of machine learning systems.

The study by Kumar et al on OrderRex clinical user testing is a randomized trial of recommender system decision support on simulated cases. The research underscores the potential benefits of ML in healthcare, but also emphasizes the need for careful consideration and rigorous testing to ensure that these systems do not inadvertently perpetuate biases.

The behavior of machine learning systems simply mirrors our behavior, going by what we actually do, and not by what we say we do, or think we do, or that we aspire to do. Condorcet's jury theorem, which posits that large groups make better decisions than individuals, is often cited as a reason for trusting these systems, particularly in healthcare. However, the "black box" problem makes it difficult to audit and examine how they did so, adding another layer of concern.

In conclusion, the increasing use of ML algorithms in healthcare deserves our attention. While these technologies have the potential to revolutionize the industry, it is crucial that we remain vigilant and proactive in addressing potential biases and unintended consequences. The acronym for these concerns, AI or ML in healthcare, should be heard as often as its better-known siblings, particularly when AI and ML are used to drive clinical decisions.