AI skin cancer test: a 2015 CNN beat GPT-6 and Fable 5 by 20 points

Short answer: No, today's frontier chatbots cannot reliably diagnose skin cancer from a photo. In a dermatologist's test of 200 biopsy-proven lesions, GPT-6 Astra and Claude Fable 5.1 named the right diagnosis 42% of the time, while a 2015 Google image network fine-tuned for about twenty minutes hit 62.5%. Our re-analysis of the raw replies found something worse than the score: GPT-6 Astra gave 56 of its 116 wrong answers at 80% confidence or higher.

TL;DR: Dr Magnus Lynch showed both images of 200 biopsied skin lesions to GPT-6 Astra, Claude Fable 5.1 and an Inception v3 CNN; the CNN got 125 right, the chatbots 84 and 85. We pulled his public data and scored confidence: Fable 5.1 was wrong at 80%+ confidence only twice, GPT-6 Astra 56 times, and in patient mode GPT-6 Astra raised skin cancer on 100 of 100 harmless lesions. Meanwhile OpenAI says over 230 million people ask ChatGPT health questions every week, and the FDA's list of 1,614 AI medical devices holds exactly 3 skin-lesion tools and zero tagged as LLMs, so the model that actually works is the one nobody is selling you.

Key numbers

What	Number	Source, checked Oct 25 2026
Correct diagnosis, 200 biopsied lesions: GPT-6 Astra / Fable 5.1 / 2015 CNN	84 / 85 / 125	Lynch Lab
Melanomas named as first choice (of 30)	8 / 11 / 13	Lynch Lab
Wrong answers given at 80%+ confidence: GPT-6 Astra / Fable 5.1	56 of 116 / 2 of 115	Lynch test data, our re-analysis
Harmless lesions where the patient reply raised skin cancer (of 100)	100 / 79	Lynch test data, our re-analysis
People asking ChatGPT health questions each week	over 230 million	OpenAI
AI-enabled devices on the FDA list / skin-lesion tools / tagged LLM	1,614 / 3 / 0	FDA AI device list, our count
Cost to fine-tune the CNN (about 20 min of one H100)	about $0.67 at today's cheapest H100 ($2.00/hr)	our GPU price tracker, our estimate

The test: 200 biopsied lesions, three models, one ugly gap

Dr Magnus Lynch is a dermatologist. He took 200 lesions at random from MILK10k, a public dataset where every lesion has a clinical close-up and a dermoscopic image (the lit magnifier dermatologists press against skin).

Half should never be sent home: 30 melanomas, 25 basal cell carcinomas, 25 squamous cell carcinomas and 20 actinic keratoses. Half are safe: 50 moles and 50 seborrhoeic keratoses, the harmless warty growths that look scarier than they are.

Every single one was biopsied, which makes the test hard on purpose. A mole someone cut out is a mole that scared a doctor.

The result, from his write-up published today: the CNN named the right diagnosis out of six for 125 lesions. GPT-6 Astra got 84. Claude Fable 5.1 got 85. Allow a second guess and it is 173 for the CNN against 134 and 138.

The CNN is Inception v3, an image network Google published in 2015. Lynch fine-tuned it on 68,011 public dermatology images from the ISIC Archive on a single H100 for about twenty minutes, with every photo of the 200 test lesions removed first. At today's cheapest H100 on our GPU tracker ($2.00 an hour on Vast.ai), that training run costs about 67 cents. Our estimate, not his invoice.

It runs entirely in a phone browser over WebGPU. Nothing gets uploaded. Lowkey the most important line in the whole post.

Where the chatbots broke: the melanoma on the ear

Lynch opens with one picture: an invasive melanoma on an ear, under 0.8 mm thick. Both frontier models called it a seborrhoeic keratosis and said the patient could be discharged. The CNN called it a melanoma.

That is not a cherry-picked fluke. Across all 30 melanomas, GPT-6 Astra named 8, Fable 5.1 named 11 and the CNN 13. Each chatbot also had a favourite wrong answer.

• GPT-6 Astra called 75 lesions basal cell carcinoma when 25 were, including 14 of the 30 melanomas.
• Claude Fable 5.1 called 53 lesions actinic keratosis when 20 were.
• The CNN was strongest on the boring stuff: 37 of 50 moles and 26 of 50 seborrhoeic keratoses, against 16 and 13 for GPT-6 Astra and 33 and 6 for Fable 5.1.

The LLMs did one thing well: they almost never let a dangerous lesion go. GPT-6 Astra kept 97 of the 100 unsafe lesions and Fable 5.1 kept 96. They did it by refusing to discharge most of the safe ones too, sending home only 32 and 22 of 100. That is not diagnosis. That is a smoke alarm that goes off every time you make toast.

Our pull: GPT-6 Astra is wrong and sure of it

Lynch published every prompt and every model reply. So we scored the part his headline skipped: how confident each model was when it was wrong.

The clinician prompt asked each model for a confidence from 0 to 100. Here is what the raw replies show.

• GPT-6 Astra's average confidence was 82.5 when right and 79.5 when wrong. It basically cannot tell the difference.
• When GPT-6 Astra said it was 75 to 89% sure, it was right 52 times out of 135. That is 39%.
• Fable 5.1 averaged 64.6 when right and 56.5 when wrong, and when it said 75 to 89%, it was right 17 of 20 times.
• Three of the melanomas GPT-6 Astra waved through came with 82, 83 and 89% confidence and a tidy one-line explanation of the features it saw.

Same score, opposite personalities. Fable 5.1 is a mediocre dermatologist that knows it is mediocre. GPT-6 Astra is a mediocre dermatologist with the swagger of a senior consultant. For a tool people use alone at 2 AM, the second one is the dangerous one, and nobody's benchmark leaderboard measures it.

Patient mode: every mole is cancer now

Lynch also asked the way a real person would: "I've got this mark on my skin and I'm a bit worried about it... do I need to get it seen by a doctor?" The replies were coded blind to the diagnosis, and we counted them.

• With both photos, GPT-6 Astra raised skin cancer on all 100 harmless lesions and told 97 of those 100 people to see a doctor promptly or urgently.
• Fable 5.1 raised skin cancer on 79 of the 100 harmless lesions.
• On a further 100 melanomas with only a phone-style photo, GPT-6 Astra told all 100 to get seen promptly or urgently. Fable 5.1 told 85 that, 14 to book something routine, and 1 to leave it alone.

Read that as the business model, not a bug. Patient mode is a liability shield: say "see a doctor" to everyone, and no lawyer can ever pin a missed melanoma on you. It costs the company nothing. It costs the patient a panic and a dermatology wait list that every worried-well chatbot user now joins.

Follow the money: 230 million questions a week, 0 cleared LLMs

OpenAI says over 230 million people ask ChatGPT health and wellness questions every week, and it built ChatGPT Health on top of that. That traffic is the prize. Skin photos are the easiest health question to ask a phone.

Now look at the other side of the counter. We downloaded the FDA's own list of AI-enabled medical devices and counted it.

• 1,614 AI devices are on the list, the latest decided June 29 2026.
• 1,230 of them, 76%, are radiology. The top filers are scanner giants: Siemens (56 under one entity), Canon (42), Aidoc (30), United Imaging (29).
• Exactly 3 are skin-lesion tools in 15 years: MelaFind (2011), Nevisense (2017) and DermaSensor (January 2024).
• Zero are tagged as LLM-based. The FDA's page says it "will explore methods to identify and tag" devices that use large language models. Translation: it does not even have the label yet.

So here is the setup. A tiny specialist model that beat both frontier chatbots would need a clearance path that has let 3 skin tools through since 2011 before anyone could sell it as a diagnostic. A general chatbot that is wrong more often answers hundreds of millions of health questions a week with no clearance at all, because it is "wellness", not a device.

Who benefits from that line? Not the 20-minute CNN, and not you. Regulation as written protects the incumbents twice: the scanner giants who can afford the filings, and the chat apps that never have to file.

The real lesson: specialist beats generalist, and it fits in your pocket

This is not an "AI is dangerous" story. The winner here is AI too. It is just the cheap, narrow, boring kind that the frontier labs do not put on stage.

The fight in AI right now is one giant model for everything versus small models trained for one job. On skin, the small one won by 20 points while running offline in a browser tab. It is the same pattern as running models in the browser over WebGPU and small open models matching Fable 5 on code: the frontier premium buys you range, not accuracy on the job in front of you.

Lynch's caveats are fair and we will repeat them straight. 200 lesions and 30 melanomas leave wide error bars. Each chatbot case ran once. Inception v3 is a baseline, and he notes the regulated skin products built on newer ensembles do "considerably better". The MILK10k images have been public since 2025, so if anything the chatbots had a head start.

None of that rescues the chatbots. It makes the gap more embarrassing. A network from 2015, given twenty minutes of homework, beat two of the most expensive models on Earth.

Our bet: a narrow skin model that runs on the phone gets an FDA clearance before any chatbot gets one for diagnosis, and the labs respond by fine-tuning specialist heads and calling them "health mode". Nobody has proven that yet. The pattern says it is where this goes. The labs selling one model for everything are lowkey cooked on any task where a specialist can be trained for under a dollar.

What this means for you

If you are worried about a spot: do not use a chatbot as the decider. Its "see a doctor" is free advice it gives to everyone, and its "looks harmless, 89% sure" is the line that can actually hurt you. Get it looked at, and push for dermoscopy.

If you build AI products: pick the model for the job, not the brand. Run a confidence check like ours on your own data before you ship: bucket the model's stated confidence and see how often it is right in each bucket. If 80% means 39%, the number is decoration. More on the money side of this in our piece on why people say AI is bad for society, and the cancer side in the mRNA melanoma vaccine data. Everything else lives in AI industry and on the LLM price hub.

If you want your own AI builds to do what you meant, on whichever model fits the job, including a small local one, the $29 AI Builder Kit is five offline apps for exactly that. If it does not get your AI building what you meant, we keep working with you at no extra cost until it does.

Not medical advice. We are a software studio reading public data; see a clinician about anything on your skin.

Sources

• Dr Magnus Lynch, Lynch Lab, "A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis", Oct 25 2026
• Lynch Lab test data: cases, prompts, every model reply, scoring code, downloaded Oct 25 2026 (our re-analysis in the data download below)
• MILK10k dataset, MILK study team, ISIC, checked Oct 25 2026
• US FDA, Artificial Intelligence-Enabled Medical Devices list (CSV), page modified Sep 4 2026, downloaded Oct 25 2026
• OpenAI, Introducing ChatGPT Health, checked Oct 25 2026
• OpenRouter models API: GPT-6 Astra and Claude Fable 5.1 list prices, $10 / $50 per million tokens, Oct 25 2026
• Tensorust GPU price tracker: H100 hourly prices, Oct 25 2026
• Hacker News discussion (289 points, 145 comments), Oct 25 2026