Skip to main content

Mask Your Logs, Not Your Prompts

· 7 min read
Rafael Fernandes
Rafael Fernandes
NLP Engineer & Tech Writer at WiLine
Share:
Privacy · AI News

Mask your logs, not your prompts

Redact before the modelRedact before the logs

Almost every "secure your LLM app" guide gives the same advice: before a prompt reaches the model, strip the personal data out of it — swap names, emails, and account numbers for [REDACTED] or <PERSON>, then call the model.

It sounds obviously right. I assumed it was, too. Then I went and read the research on what masking actually does to a model, and the papers point the other way. The short version: mask your logs, not your prompts.

First, what are we protecting?​

PII is personally identifiable information — a name, an email, a phone number, an account or government ID. Data that points at a specific person.

When people reach for pre-call masking, they're usually blending two different worries into one:

  1. "I don't want to send sensitive data to whoever runs the model."
  2. "I don't want to store sensitive data in my logs."

Pre-call masking is aimed at #1. Hold on to that — it turns out to matter.

Problem 1: a masked prompt is a confused model​

Here's the simplest failure. Put three people in a prompt and redact all of them to [REDACTED], and the model can no longer tell them apart. Who signed the contract? Who was cc'd? The words that carried those relationships are gone, so the answer degrades.

This isn't just intuition. A 2026 benchmark called RedacBench measured the trade-off directly, across 514 texts and 187 policies. Even when a capable model does the redacting, turning the security dial up to ~81% of sensitive content removed leaves you keeping only 37.6% of the text's non-sensitive meaning. You throw away roughly 60% of what made the prompt useful to buy that privacy.

"But I use smart tokenization" — it still bites​

The obvious fix is to stop using dumb placeholders. Deterministic tokenization maps the same value to the same token every time — "John Smith" always becomes PERSON_42, "Jane Doe" always PERSON_17 — so the model can still track who did what. That's genuinely better.

But it isn't free either. One engineer ran 109 masking tests across healthcare, legal, financial, and developer workflows and wrote up where it broke. (Full disclosure: he also sells a tokenization tool, so take his framing with a grain of salt — but the failures he logged are concrete and easy to reproduce.)

  • Context-phrase refusals. Tokenize an SSN into GOV_ID_8x3m, but leave the words "social security number" sitting next to it, and the model's safety filter can refuse the whole request — it sees a sensitive label beside an opaque token and flags it.
  • False positives. The word "Will" in "this will update the record" got caught by the name detector.
  • Misses. A short name like "Li" in a table row, or an SSN buried in code comments, slid past the detector when there wasn't enough surrounding context.
  • Streaming corruption. A name or SSN can be split across two or three streaming chunks; process them one at a time and you mangle the entity.

His overall detection came out to 89% — and his sharper point was that the missing 11% is where it hurts: you're forced to either block a legitimate request or leak. There's no comfortable default.

Why none of this is surprising (my read, not a proof)​

Step back and it fits a pattern the research keeps finding: models are fragile to how a prompt is worded.

  • "On the Worst Prompt Performance of LLMs" took the same question, reworded it in semantically identical ways, and watched one model's accuracy swing by 45 points (worst case, 9.38%).
  • The DETAIL framework found that more specific prompts reason better, especially on smaller models and step-by-step tasks.

Neither of those papers tested PII masking — so this next step is my inference, not their claim — but masking is a prompt edit. It makes the prompt less specific and changes its wording, which is exactly the lever these papers show models are sensitive to. You're rolling dice you don't need to roll.

The hidden bill​

Masking also costs money in a way that's easy to miss, because it changes the prompt on every call. That quietly breaks prompt caching — the discount you get when a prompt's opening is identical to a previous one.

Both major providers cache by prefix, and both say a change up front invalidates it:

  • OpenAI: "Cache hits are only possible for exact prefix matches… a change before the breakpoint will prevent a cache hit."
  • Anthropic: "Changes at each level invalidate that level and all subsequent levels."

A cache read costs about 10% of the normal input-token price. So every masked prompt that misses the cache pays close to ten times more on those tokens, and gives up the faster first token too. On top of that, when a masked placeholder like PERSON_42 leaks into a tool call, the tool rejects it and the agent retries — and a study of coding agents found that small prompt-wording changes can multiply token use 2.4–7.4× with no gain in success. (Again: not masking specifically, but the same mechanism.)

The reframe: the model was never the risk​

Here's the part I had backwards. The problem was never that the model sees the data. A model reading your prompt to answer it is just doing its job — that inference pass isn't where data leaks. The real question is retention: what gets stored, and where.

Once you see it that way, the fix is obvious. Put the masking where storage actually happens and where you're in control: your logs.

Mask your logs, not your prompts​

The pattern — often called logging-only — is simple:

  • The raw prompt reaches the model, so reasoning stays intact, the cache still hits, and nothing gets refused.
  • Your PII masking runs only on the path to storage — the request/response logs and traces your gateway writes.

Think of it as three rungs, worst to best:

  1. Dumb masking ([REDACTED]) — destroys entity relationships.
  2. Tokenization (PERSON_42) — keeps identities, but still triggers refusals and misses.
  3. Logging-only — the model reads the real text; masking happens where the data rests.

One honest caveat: whatever provider you send prompts to, it's worth knowing its retention policy — that's a separate question from what the model reads, and it's the right place to put that worry.

If you run your own gateway​

The nice part: this is a configuration, not a rewrite. If you've already deployed a gateway on a WEC Instance, the guardrail runs in log-only mode — clean prompt out to the model, masked copy into the logs. A follow-up tutorial will wire it up end to end.

Masking a prompt before the model buys you a compliance checkbox and quietly sells your model's intelligence. Put the privacy work where the data actually rests — in the logs — and let the model read the real thing.

Sources​

Comments & questions

Hit an error, spotted a typo, or have a question? Leave a note below.