Can the Biases Built Into ChatGPT Be Fixed?

In this video, Leo Anthony Celi, MD, MPH, principal research scientist at the MIT Laboratory for Computational Physiology (LCP) and an intensive care unit physician at Beth Israel Deaconess Medical Center in Boston, discusses the biases that large language models like ChatGPT are built upon and outlines some solutions to make its use in medicine more equitable.

The following is a transcript of his remarks:

Recently there has been commotion when some investigators demonstrated that they were able to train ChatGPT to pass licensing exams. The first one was the U.S. Medical Licensing Exam. Since then it has passed almost every standardized test that is currently being used to evaluate students.

What we are concerned with is the fact that the data that is being used to train large language models is biased. The internet is dominated by content coming from well-funded institutions from rich countries, and that is particularly true for medical content. We are afraid that the output of large language models would reflect the bias, and that if you ask a question and you’re coming from Brazil or Uganda, the answer that you will get from large language models such as ChatGPT may not be applicable to you.

We are not discounting the power of these technologies. These technologies will for sure be a part of how we teach and how we learn, but I think we need to focus on improving the data input that we are training these large language models on. We have to make sure that there’s representation, especially from countries that are disproportionately burdened by disease.

Can we improve the data input of not just large language models, but also artificial intelligence in general?

One of the concrete recommendations that we have discussed in previous papers is diversifying the people who are sitting at the table, and that means we need more investigators coming from parts of the world that are disproportionately burdened by disease. We need more scientists from low- and middle-income countries because they understand health and disease better than the investigators who are from ivory tower institutions.

We are also recommending bringing in and recruiting more social scientists and working with the machine learning community. The social scientists are experts in understanding the disparities and the biases that exist and they may be able to help us address and guarantee that the algorithms we are developing will not perpetuate or even magnify the inequities that we’re seeing now.

I think those are two very explicit recommendations that we are giving to the community, but they are easier said than done. This would mean that we would have to overhaul the systems that create and validate knowledge, and there will be a lot of pushback about that.

We have to improve how we teach medicine or even other disciplines, because knowledge is being taught as something that is static, when in fact there is no such thing as permanent ground truth. The learners have to be aware that ground truth is continuously shifting. They also need to be aware of how the knowledge system has been put together. They have to be aware of the gaps and the uncertainties in the knowledge system and the impact of those gaps and uncertainties when applying the knowledge itself.

Unfortunately, that is not happening anywhere in any discipline. We are being fed to ingest and digest a body of knowledge, and we do not really understand how that body of knowledge came about. What are the limitations? What are the biases in that body of knowledge?

To us, there is a need to democratize the creation and the validation of knowledge. This should not be in the sole control of a few academics who may not be able to fully understand the context of health and disease from different lenses.

We think that this has been one of the contributors of why we’re seeing a lot of suboptimal outcomes, because there are very few people sitting on the table. By diversifying those people, by bringing in people who understand health and disease better, I think that we are finally going to move the needle.

I’ve had some pushback. People ask, how can you tell us that this will work? Because it’ll require a lot of energy and even cost to revamp the system. What are the guarantees that it’s going to work? Well, I don’t have any guarantees, but I also know that continuing the way we do things, continuing the current system, is not miraculously going to produce a different result. We keep saying that every system produces the results that it gets. And unless we change that system, we’re going to come up with the same problems that we’ve seen over the last century.