How should we proceed with image-analyzing AI?

Scientists are using AI-based models to scrutinize disease biology like never before, and each model is more revealing than the next. Deep learning algorithms can now find cancerous mutations,¹ estimate the mutational burdens of tumors,² and predict key gene expression³—all based on stained tissue images. However, because most of these models are “black boxes” and independently learn which features connect to disease, their reasoning can be difficult to decode.⁴ Does that matter and should it worry us? We asked two experts how they feel about the algorithms’ mysterious natures.

Liron Pantanowitz We should be excited but cautious. We are entering a new era of AI where some people worry that we are training algorithms to process data in a black box. Specifically, we are unsure if we are training them appropriately. However, despite their opacity, these models can be very helpful. For example, they can help us make new discoveries in spatial biology, like how tumors respond to immunotherapy in 3D. Deep learning algorithms can analyze spatial parameters and tell us things that would be inaccessible without them. We’ll be able to discover new diseases and responses to drugs that we’ve never been able to see before. That is super exciting.

Luigi Marchionni We need to be careful. Deep learning algorithms are intrinsically not very interpretable; they might give us the correct answer, but we don't know why. We cannot open the box and look inside. That is the perfect recipe for creating a biased model because it might not be trained on data that represents the population where it will eventually be used. It is like biomarker development, where we might overfit a model to noisy data, so it only works for that data set and not for the next one. But because AI is so powerful, it’s a much bigger problem. We should avoid the hype and spread awareness of the challenges.

These interviews have been condensed and edited for clarity.