In our last newsletter we looked at the privacy policies of the big AI companies, what they hold, and whether the things you type get used to train their models. This week is the follow-up question: if your data did go into the training, how easy is it for someone else to get at it?

A new study in Nature answers it. The clearest way to see how starts with an example close to home. At UAB in Birmingham, radiologists run an AI-powered clinic that scans thousands of radiology reports to catch adrenal gland masses that turn up by accident—the kind found when someone gets a CT after a car wreck and the scan happens to show something on an organ nobody was looking at. In one year the UAB team flagged 899 patients, saw 382 of them in the clinic, and sixteen ended up having surgery to remove the growth. That’s AI doing real good, catching things that would otherwise slip through. It also means a model somewhere learned from a pile of real Alabamians’ medical records. The Nature study is about what a stranger can figure out from a model like that.

The method, in plain terms

The researchers ran what they call one of the first patient-level privacy audits of medical AI. They tested seven real clinical datasets: chest X-rays, mammograms, ECGs, dermatology photos, eye scans, and emergency-room records. The method they used—a way of testing whether one specific person’s data was in a model’s training set—has a dull name, “membership inference,” but the idea is simple. You show the model a specific person’s scan and watch how confident it acts. Models tend to be a hair more sure of themselves on data they were trained on, like a student who’s a little too quick to answer the one exam question they happened to study. That small tell is enough to guess whether your record was in the training set.

Two things make this worth paying attention to. You only have to ask the model one question. And the whole thing runs on an ordinary computer, no fancy hardware required. This isn’t a government‑lab operation. It’s something a motivated person with a laptop could try.

Why does it matter whether your data trained a model? Because sometimes the sensitive fact is simply that you were in the group at all. If a model was trained only on people with a certain cancer, and someone proves your record was in it, they’ve proven your diagnosis. Being in that group was the private part.

What the average hides

This is what should change how you read any “our AI is private” claim. Measured across everyone in the dataset, the method barely beats a coin toss. The usual privacy score—an AUC where 0.5 is random guessing—came in around 0.61 to 0.65. If you only looked at that number, you’d shrug and move on.

But that average is doing what averages often do: hiding the extremes. Underneath it, a small group of patients were nearly perfectly identifiable. The typical person was mostly safe. A handful were wide open. And the standard way the industry reports privacy risk smears those two groups into one comfortable‑looking number that misses the people actually in danger.

Bigger models, more exposure

The trend gets worse as models get bigger, which is awkward, because bigger is exactly where the whole field is headed. On the dermatology dataset, the researchers scaled the model up in steps and watched the share of near‑perfectly exposed patients climb. It went from none, to one in ten thousand, to one in a thousand, to one in ten as the model grew. Same data, more capacity, and suddenly a tenth of patients are sitting ducks. The more powerful the model, the more it seems to memorize the unusual cases instead of just learning the general pattern.

That’s the quiet tension in medical AI. The models that diagnose best are often the ones that remember hardest.

Who gets hit

The uneven exposure is the part that matters most here. When the researchers looked at who landed in the most‑vulnerable slice, they found underrepresented groups showed up far more than their share. In the emergency‑room dataset, records from Black patients appeared 31% more than expected, cancer patients 18% more, and Medicaid patients 126% more.

The reason is mechanical, and that’s what makes it stubborn. Models memorize the rare and the unusual. If a group is a small slice of the training data, its members look unusual to the model, so they get memorized, so they get exposed more easily. The people already underrepresented in medical datasets end up carrying more of the privacy risk. Worse care from being left out, and now a bigger leak on top of it.

The part that isn’t doom

The fix exists, and it works. It’s called differential privacy, and the plain‑English version is that you add a controlled amount of statistical noise while the model trains. Enough noise and no single patient’s data leaves a strong enough fingerprint to trace. The researchers confirmed that stronger privacy protection lowered the odds of a correct guess for everyone, and they were optimistic it can be done without wrecking how well the model diagnoses.

The catch is that fully protecting everyone takes a stronger dose of this protection than what’s commonly used today, applied per patient rather than per record. So it’s available, it’s provable, and it’s mostly not switched on yet. The tools are on the shelf. Getting them into the systems your hospital actually buys is the work that’s left.

Back to your chatbot

The bridge back to last week needs care, because it’s easy to overstate. This study tested diagnostic models built by hospitals and researchers. It did not test ChatGPT or Claude, and nobody proved your therapy vent to a chatbot is sitting in a model waiting to be extracted.

But the researchers were direct about the reach of it. The same method applies to generative models—the large language models and image generators—with little or no modification, and related work has already reconstructed training data out of production language models. So the principle from last week holds and gets sharper. What you put in can, under the right conditions, come back out. That’s the whole reason the consumer‑versus‑business line from part one matters. The business tiers that keep your data out of training aren’t just protecting you from a human reading your chats. They’re keeping your words out of the one place this kind of deduction can reach.

Type into the free box and your words might train the model. This week’s finding is what that sentence can cost, and that the cost doesn’t land evenly. It falls hardest on whoever the model finds unusual enough to remember.

This wraps the two‑part privacy series. Part one was what goes in. This was what comes out. The through‑line is the same the whole way down. Your data doesn’t stop being yours the moment you hand it over, and knowing where it goes is the difference between using these tools and being used by them.

LA-AI Foley

// community highlights

Know someone who’d enjoy this?

Don’t miss the next issue.