Are Skin Lesion Analyzers Ready for Prime Time?

While the FDA is proposing to reclassify two class III computer-aided melanoma detection devices as class II devices, most members of its General and Plastic Surgery Devices Panel suggested that such a move would be premature during the Medical Devices Advisory Committee meeting on Friday.

According to the FDA, reclassification of the MelaFind and Nevisense devices should be considered because the risks to health associated with these skin lesion analyzers (SLAs) can be mitigated by a number of “special controls” that “provide a reasonable assurance of its safety and effectiveness.”

However, the general consensus among most panel members is that there currently isn’t enough available information to establish those special controls.

“My concern is that these devices need to show an improvement over what a dermatologist can do,” said Karla Ballman, PhD, of Weill Cornell Medicine in New York City. “Right now I don’t think we are comfortable setting what the sensitivity should be, and what the specificity should be. And even more so, what the relationship between the two should be. You need to have agreed upon performance standards in order to be a class II, and we don’t have that at this point. So, I don’t think we’re ready to define what the special controls should be.”

While these SLAs represent an area of great need, said Maral Skelsey, MD, of the Dermatologic Surgery Center of Washington in Chevy Chase, Maryland, “we’re just not ready for prime time, even with these special controls.”

Class III is the most stringent regulatory category for devices, and according to the FDA, class III devices are typically high-risk devices for which insufficient information exists to provide reasonable assurance of safety and effectiveness solely through general or special controls. In the case of a class II device, general controls would not be enough to provide assurance of safety and efficacy. However, there is enough information with these devices to establish special controls — such as performance standards or special labeling requirements — that would provide this assurance.

Thus, in the case of these computer-aided melanoma detection devices, mitigation measures can be taken to control for associated risks, which could include false-negative or false-positive results, user error or improper device use, or device failure or malfunction, said Scott Kominsky, PhD, of the FDA’s Office of Surgical and Infection Control Devices. “That is to say, when the device is used properly, the probable benefits t0 health outweigh the probable risks, there is not an unreasonable risk of illness or injury, and the device will provide clinically significant results in a significant portion of the target population.”

Murad Alam, MD, of Northwestern University in Chicago, noted that given the limited data available with only the two devices that have been FDA approved, “I’m struggling with the urgency to do this. It seems to me a little premature given the lack of clarity in this space.”

According to Neil Ogden, BME, chief of the FDA’s General Surgery Devices Branch, reclassifying devices is not an unusual occurrence. “We did this with hip and knee implants. We down-classified them from III to II,” he explained. “When we have enough information, and we think we can regulate these devices in a lower class, we move to do so.”

That eases the process for the FDA to get “high-quality devices that are safe and effective to the U.S. population in the least burdensome manner,” he added.

However, when addressing the question of the risks associated with SLAs, the panel highlighted the issue of false negatives and positives as an area of great concern.

For example, Skelsey pointed out that due to numerous false positives, “my real-world experience is I abandoned a device because it wasn’t functional in a dermatology office.”

“There needs to be more testing, especially around the issue of false negatives and false positives,” said Neil Farber, MD, of the University of California San Diego. “It’s such a risk and such a complex issue because you have an [artificial intelligence] device on the one hand and an interaction with dermatologists on the other. I think we need some real-world, real-use types of studies. And those would have to be prospective.”

While the majority of panel members indicated they were not in favor of reclassifying the devices, others disagreed.

Katalin Roth, MD, JD, of George Washington University in Washington, D.C., said that she believed down-classifying the devices would be appropriate as long as special controls were clear, the devices were used by dermatologists only with training, and with informed consent of patients.

SLAs in Development

While Friday’s meeting focused on the two devices that have already received FDA approval, on Thursday the panel discussed how the FDA should go about evaluating the benefits and risks of computer-aided SLAs that are currently in development.

Much of the discussion centered around the performance thresholds of devices intended for adjunctive use by dermatologists and non-dermatologists (i.e., primary care physicians), as well as performance thresholds for standalone devices (meaning the output would be relied upon at face value to guide management). Such a device could be used by lay-persons, as well as healthcare professionals.

In general, the panel resisted the idea of using preset thresholds to evaluate the performance of these devices. “No one seems to be comfortable with providing a preordained, across-the-board sensitivity/specificity threshold,” said Panel Chair Hobart W. Harris, MD, MPH, of the University of California San Francisco, in summarizing the panel’s view.

“I don’t think absolute sensitivity and specificity measures … are going to be sufficient for us to consider these devices,” said Veronica Rotemberg, MD, PhD, of Memorial Sloan Kettering Cancer Center in New York City. “The best type of test for these devices is going to be a prospective study in the intended-use setting rather than a threshold on retrospective data, and in that intended-use setting, improvement as defined by a value judgment over the standard of care in that setting.”

“I would agree that in the adjunctive setting you want to see if it is increasing accuracy — however you define that — over where it’s currently at,” said Alam. “I’m not so concerned about FDA regulation of devices that are for adjunctive use — it seems the FDA has a pretty good handle on that. The potential problems are these devices that are going to be in the lay public and are going to be marketed very aggressively.”

Once these devices are sanctioned by the FDA “no one will really think about the nuance of how they work or what they are intended for, they will just say they are FDA-approved,” he added. “So it’s a high bar for us that what gets out there is reasonably good. I would like it to be a sensitivity of at least what a dermatologist would do, and ideally, a little bit more.”