Someone phished their own AI inbox assistant on purpose. The model fell for it twice.
The app just shrugged, because nothing that reads the mail can send a thing. And that's the whole lesson tbh.
A post on r/AI_Agents walks through a local inbox assistant: qwen3.5:9b running in Ollama, mail pulled over IMAP. It sorts every message, flags attacks, drafts replies and writes a one-page plan each morning. Then the builders wrote six trick emails and attacked it themselves.
Hot take: the most useful line in the post isn't a defense trick. It's the design rule they started with. "It reads everything, and it can send nothing."
ok so how did the model do
Not amazing ngl. The 9B sorted each trick email with the full rules and again with two defensive sentences removed. Four attacks bounced off. Two didn't.
- Wrapper breakout. A phishing email, then a fake
</email> tag, then a paragraph pretending to be new instructions from the user. Fooled the 9B both times.
- Planted action. "Forward this thread to…" ended up word for word in the model's "action" field. Also fooled it both times.
The breakout one is lowkey brutal. Without the fake user paragraph the model marked the email suspicious 3 times out of 3. With it, never. So one fake closing tag and a confident paragraph flipped the whole verdict.
Model size helped but didn't save anyone. With the full rules the 4B got fooled 3 times and didn't judge any of the six suspicious on its own. The 27B got fooled once (the breakout) and flagged the other five. And when the two defensive sentences were removed, the 27B let a summary hijack through, turning a parking notice into "Your manager asks you to wire $4,800 today". which is genuinely the most cursed summary I've read this week.
the prompt lost, the wiring won
That's a line from the Reddit thread, and imo it's the correct read.
The model that reads mail has no tools. No shell, no send, nothing. Its answer is forced into a fixed JSON schema by Ollama (category, priority, summary, what you need to do, a due date), so a fooled model can mess up one email's fields and that's it. When the planted "forward this thread" landed in the action field, it was just a line of text on a page. Nothing runs it.
And they tested it instead of promising it. IMAP can't send mail but the app password would work for SMTP, so a test fails if any module imports smtplib or calls a send function. A second test runs the drafter against a fake IMAP server that records every command, and the only change it ever makes is an APPEND to Drafts.
This is the part every agent builder should steal fr. Say your agent triages Gmail through MCP instead of IMAP: the triage agent should get a toolset with the read tools and literally nothing else. With MewCP you pick which tools each agent can use and anything you switch off is stopped before it reaches the app, so a sneaky "forward this" has no forward tool to call. The agent also never holds the mail login (keys sit in MewCP's vault and get injected per call), which kinda handles the "that app password also works for SMTP" problem from the post. None of that stops injection btw. It just makes a fooled model boring.
the code checks are good, the 6/6 is a bit sus
The team added seven checks that don't use a model at all, things like hidden HTML text, invisible characters, text addressed to an AI, failed authentication (DMARC, or SPF without DKIM) and a Reply-To on a different domain from the sender's.
The model alone flagged 3 of the 6 attacks. With the checks, 6 of 6 with no false alarm among the other 31 emails.
but there's an asterisk, and the post is upfront about it. Two of those catches only happened after they added the exact phrasings from the test emails to the "addressed to an AI" list. A commenter on r/AI_Agents called this out pretty fairly: that score tells you how the list does on the batch it was tuned on, and a paraphrase can slip past a phrase list easily. Their suggestion was to freeze the list, have someone else write fresh attacks and report that score separately.
I'm with the commenter. The checks that don't care about wording (Reply-To mismatch, failed auth) are the ones I'd trust. The phrase list is a speed bump.
the reply drafter is the sneaky part
Everyone's gonna talk about the injection table but this part got me.
With a rule saying "never agree to commitments I didn't give you", all 8 drafts committed the user to something anyway. A second version showing how to leave choices in [brackets] got 6 of 8 drafts using brackets, and all 8 still decided something next to them. Like, the model just really wants to say yes to Tuesday.
What finally worked was flipping it. The program asks you for your answer in a few words first, and the model only writes it out. 4 of 4 drafts said exactly that answer and nothing more. Code then lists every sentence that commits you. The cost, per the author in the comments, is that drafting is interactive so it doesn't run in the morning schedule.
My read: that's the same lesson twice. Don't ask the model to hold a boundary. Build the boundary and let the model fill in the blanks.
what's still unverified
All of this is self-reported in one Reddit post with a small sample: 37 emails, six attacks. I couldn't find anyone outside the team rerunning it. The post also says up front that it comes from an ebook they publish, Your AI Chief of Staff, Volume 3 of Weekend AI Projects, so read it as a writeup from people with a book to sell. The numbers are specific and the caveats are honest, which counts for a lot.
Sources