Migrating decades of personal archived email from Thunderbird mbox format into Proton Mail via the Bridge IMAP interface. On Linux. From the command line. Because GUI add-ons kept crashing.
The Context
I’ve been online since the BBS days on a Commodore 64. Watched email evolve from FidoNet netmail through SMTP, POP3, IMAP, webmail, and now end-to-end encrypted providers. My personal archive contains messages from Outlook Web Access, Lotus Notes, Gmail, Yahoo, and half a dozen defunct providers. Each era wrote headers differently. Each client had its own interpretation of “RFC compliant.” The result is an archive that represents 20+ years of real-world email, which is to say, a complete and utter mess by any strict standard.
These are personal emails. Correspondence with friends, family, colleagues over the years. Calendar invites, project discussions, photos attached to messages from people I’ve lost touch with. The kind of thing you don’t think about until you realise how much of your life is stored in it.
I’ve lost so many of these archives over the years. Hard drives that died. Email providers that shut down without warning. Accounts I couldn’t migrate because the export tools didn’t exist or the format was proprietary. Each loss was a small erasure of personal history, just gone. This migration was about making sure that doesn’t happen again. Proton Mail, with end-to-end encryption and a company that takes data sovereignty seriously, felt like the right place to land. But I don’t trust a single provider implicitly. I encrypt my own emails with PGP keys before sending, treating Proton as the offsite destination in my 3-2-1 backup strategy. Two copies locally, one encrypted offsite. Redundancy without compromise.
The Problem
Proton Bridge’s RFC 5322 parser.
Bridge uses Go’s go-message library, which enforces strict RFC 5322 compliance. Laudable in principle. In practice, it means any email written before approximately 2015 by Outlook, Exchange, Lotus Notes, or various legacy systems gets rejected with cryptic parse errors.
The parser does not forgive. It does not interpret. It does not “best-effort” anything. It sees a malformed header and it stops.
When you’ve been collecting email since the days when a 2400 baud modem was fast, you accumulate a lot of messages that predate strict standards compliance. Back then, the miracle was that the message arrived at all. Nobody cared whether the From: header had properly escaped quotes.
The Toolkit
- Proton Mail Bridge - local IMAP server at 127.0.0.1:1143
- imap-upload - Python-based tool for pushing mbox files to IMAP
- A custom sanitiser script - the thing I had to build
What Happened, In Order
Phase 1: Authentication
Proton Bridge doesn’t use your email address as the IMAP username. It generates unique credentials accessible through Bridge’s internal info tools. First hurdle: figuring out that your standard email address doesn’t work and retrieving the real credentials.
Phase 2: The “From header: expected quoted string end” error
Every single message in the first archive, 669 out of 1,059, rejected with: plaintext failed to parse From header: [Error offset=164]: expected "for quoted string end"
Cause: Thunderbird stored From: headers with display names containing unescaped quotes and special characters. Bridge’s parser encounters an unclosed quoted string and aborts.
This is the kind of thing that would have been unthinkable in the BBS era. Messages were plain text, ASCII, no MIME, no quoted-string encoding. But as email grew up and added display names, quoted-printable encoding, and internationalised headers, the formats diverged. Every client did it slightly differently. Nobody enforced the standard. Until now.
Phase 3: The “unhandled charset windows-1252” error
3,843 messages in the largest archive used charset=windows-1252, the default Windows encoding for over a decade. Bridge’s parser doesn’t handle it.
Fix: Convert all legacy charsets (windows-1252, cp1252, iso-8859-1, latin-1, etc.) to UTF-8 before upload.
In the Commodore 64 days, we didn’t have charsets. We had PETSCII, and we liked it. Then the PC era brought code pages, then Windows-1252, then UTF-8. Each transition left behind messages encoded in the old standard. Bridge only speaks UTF-8.
Phase 4: The disappearing envelope lines
The first sanitiser version used Python’s email.message_from_bytes() to parse and rebuild each message. This function strips the From mbox envelope line because it’s not part of the RFC 5322 message format. The sanitised output had zero envelope lines. imap-upload saw one giant blob, found zero valid mbox messages, and exited with OK: 0.
Fix: Abandon email.message_from_bytes() entirely. Parse the raw file line-by-line in binary mode. Capture and preserve From envelope lines manually.
This was the most insidious bug. A silent failure. No errors, no warnings, no crash. Just OK: 0, which looks like success until you check the target folder and find it empty. In the BBS days, when something went wrong, you got a carrier signal drop or a garbage screen full of PETSCII noise. You knew immediately. Modern tooling fails silently, reports success, and leaves you staring at an empty inbox wondering where 14,000 messages went.
Phase 5: The “expected ‘:’ for group start” error
After fixing the envelope lines, Bridge rejected messages with: failed to parse From header: [Error offset=37]: expected ':' for group start
Cause: The archived messages had From: headers with bare names and no email addresses. From: MAILER-DAEMON, From: a.friend. No @, no angle brackets. Bridge’s parser sees MAILER-DAEMON and interprets it as an RFC 5322 group address name, expecting a colon to follow.
Fix: Extract valid email addresses from each header. If none exists, substitute unknown@localhost. Always wrap in angle brackets to eliminate group-address ambiguity.
Phase 6: The same error, but on To: headers
Once From: was fixed, Bridge started rejecting To: headers with the same group-address error. Headers like To: J.Smith 1337h4x0r. No angle brackets, no email address.
Fix: Apply the same sanitisation logic to all address headers: From, To, Cc, Bcc, Reply-To, Return-Path, Sender. Extract valid addresses, wrap in angle brackets, fall back to unknown@localhost.
Phase 7: Folders not appearing in the web UI
Folders created via IMAP didn’t show up in Proton’s web interface. Cause: Bridge doesn’t auto-subscribe to created folders. The web UI only displays subscribed folders.
Fix: Explicitly subscribe to each created folder via IMAP.subscribe().
The Final Pipeline
| |
The Results
Lessons Learned
- Proton Bridge’s parser is intentionally strict and Proton has stated they will not relax it. If you’re migrating legacy email, you must sanitise before upload. There is no “lenient mode.”
- Python’s email module strips mbox envelope lines. If you need to preserve mbox format, parse raw bytes line-by-line instead of using message_from_bytes().
- Bash variables are not visible to Python subprocesses unless exported. This sounds obvious. It is not obvious when you’re three hours into debugging a KeyError.
- imaplib has a hardcoded 1 MB line-length limit. For folders with hundreds of thousands of messages, you must override _MAXLINE or every SEARCH command crashes.
- Folders created via IMAP must be explicitly subscribed or they won’t appear in Proton’s web UI.
- The imap-upload tool reports OK: 0 when it can’t parse the mbox file. Not as an error, but as “zero messages uploaded successfully.” Always check this number, not just the error count.
The Trade-Off
The sanitiser strips display names from address headers. Messages that originally showed From: “Alex” alex@example.com now show From: alex@example.com. The email address, subject, body, and attachments are all preserved. The display name is lost.
Given the alternative, zero messages imported, that’s an acceptable trade.
Final Thoughts
I’ve been online since BBSes on a Commodore 64. Migrated email across more platforms than I can count. POP3 to IMAP, Outlook to Thunderbird, Exchange to Gmail, Gmail to self-hosted, self-hosted to Proton. Each migration has its own quirks, its own parser incompatibilities, its own silent failures.
I’ve lost so many archives over the years. Hard drives that died. Providers that vanished. Formats that became unreadable. Each loss is a small erasure of personal history. Messages from people who are gone, conversations you can’t recreate, moments that only exist in that one copy of that one mbox file on that one drive that spun down for the last time.
This migration was about not losing any more. Getting everything into one place, encrypted, backed up, under my control. The technical challenge was real. Seven distinct failure modes, each one hiding behind the last. But the motivation was simpler: these are my emails. My correspondence. My life, in message form. I wasn’t going to let a strict parser take any more of it.
If you’re sitting on 20 years of personal archived email from a dozen different clients and servers, strict parsing means you’re going to spend an afternoon writing a sanitiser. Hopefully this saves someone else theirs, and maybe saves a few archives that would otherwise be lost.
The code is below. Adapt it, improve it, and may your OK count always match your message count.