[Users] encoded addresses causing problems in reply

[Derek B. Noonburg]

On Wed, 7 Sep 2022 22:17:42 +0200
Milan Obuch <claws-mail-users at dino.sk> wrote:

> On Wed, 7 Sep 2022 12:17:37 -0700
> "Derek B. Noonburg" <derekn at foolabs.com> wrote:
> 
> > If I do a reply-all to an email with ISO-2022-JP encoded addresses,
> > the to/cc headers in the resulting message end up mangled.
> > 
> > I'm not sure exactly where this is happening -- it may or may not
> > have anything to do with claws -- but I'm hoping to get some advice
> > on how to debug the problem.
> > 
> > I unfortunately can't forward the email, because it's from a
> > customer. I understand this makes it harder to debug, but I'm
> > hoping maybe someone has encountered something similar, or has some
> > other helpful suggestions anyway.  
> 
> I did an analysis of some mail header problem recently, in my case, it
> was in no relation to Claws Mail. As you can't forward the offending
> mail, it would be nice to find some way to replicate the issue.
> 
> > Looking at the original email (to which I'm replying) with "view
> > source", the from/cc headers look like this:
> > 
> > From: =?iso-2022-jp?B?xxxxxxxx==?=
> > 	<aaaa at example.com>
> > CC: =?iso-2022-jp?B?xxxxxxxx?=
> > 	<bbbb at example.com>,
> > 	=?iso-2022-jp?B?xxxxxxx==?=
> > 	<cccc at example.com>
> > 	=?iso-2022-jp?B?xxxxxxx==?=  
> 
> When looking to source, I'd be interested in X-Mailer of similar
> header if any. If it's not there, maybe you can ask your customer
> which MUA is used. If your issue is address encoding problem, and
> what your description suggests, exact address is not that important,
> it's presence of base64 encoded Japanese name.

No X-Mailer header, but lots of X-MS-Exchange-blahblah headers.

> If anybody 
> > When I hit reply-all, the headers in the claws compose window look
> > perfect.  (I don't read Japanese, but as far as I can tell,
> > everything looks good.)  I'm seeing:
> > 
> > To: JJJ / xxxx, xxxx <aaaa at example.com>
> > Cc: JJJ / xxxx, xxxx <bbbb at example.com>
> > Cc: JJJ / xxxx, xxxx <cccc at example.com>
> > 
> > where "J" are Japanese characters.
> > 
> > One suspicious thing here is that there are spaces, slashes, and
> > commas, and no quoting.  But I'm not sure what the rules are for
> > encoding email addresses (the "=?iso-222-jp...." stuff).  In the
> > original email headers, the spaces and punctuation are "inside" the 
> > encoded blobs.  
> 
> I did some simple experiments in Claws Mail - if I put a comma into
> Name field of my account, there is quoting in From header. If there is
> no comma, just space and/or slash, no quoting. I have no exact
> knowledge on encoding rules, this is just an observation, how Claws
> Mail handles some cases.
> 
> > If I bcc the reply to myself and view source on that, the headers
> > are mangled pretty badly:
> > 
> > To: =?iso-2022-jp?Q?=1B$B at foolabs.com>,
> > 	#0f6Q=1B at foolabs.com (B?= / =?iso-2022-jp?Q?xxxxx?=
> >  =?iso-2022-jp?Q?xxxx?= <aaaa at example.com>
> > Cc: "=?iso-2022-jp?Q?=xxxx"@foolabs.com;, <R=1B at foolabs.com>
> > 
> > where "foolabs.com" is my sending domain.  The one original "to"
> > address (aaaa at example.com) shows up correctly amidst the gibberish,
> > but everything else is gone.  
> 
> Did this mail go through your mail server? If so, this may be caused
> by some rewrite rules in server. Commonly used one is adding domain to
> local part assuming it is a name only. If you are sending this mail
> from Claws Mail, you should have saved copy in Sent folder (or another
> folder if you configure it so). It would be interesting to see whether
> there is any difference in headers in you locally saved mail (before
> sending to server) and mail processed by server (delivered to bcc
> address).

Yes, the additions of "@foolabs.com" are certainly coming from my mail
server -- which makes me suspect there's a problem in the headers
inserted by claws before it gets to my server.

Good point on the sent/outgoing folder.  I took a look, and then also
did a little research on the MIME Encoded-Word format.

It looks like the original email uses base64 (=?iso-2022-jp?B?...?=),
while the reply (generated by claws) uses quoted-printable
(=?iso-2022-jp?Q?...?=).  That's resulting in some invalid (I think)
characters in the to/cc headers, e.g., there's a ">" character that's
pretty clearly causing a problem.

Is it possible that claws is generating to/cc headers using
quoted-printable, but not checking for various cahracters that need to
be escaped (",", ">", not sure what else)?

> > I can't really send a bunch of email to the customer to test this,
> > and I can't forward email due to confidentiality (I know this makes
> > debugging difficult) -- but if anyone has suggestions for how I can
> > debug this, I'd really appreciate it.  
> 
> I remember we did someone on this mailing list using Japanese
> characters in his mails, I don't know whether he is still active.
> Mybe he can help more...
> 
> As I am writing this, I realized in past I had problems with subjects
> containing some accented letters. In body of the message the problem
> did not appear. After some experiments with various setting I found
> simple solution - setting LANG environment variable to C.UTF-8
> (standard value was C.ASCII I think). I am using FreeBSD, so this
> could be different for you, but it is simple to check and try,
> eventually.

If I manually set the encoding to UTF-8 (compose window -> options ->
character encoding), that seems to help.  But I think it's just luck:
that is, I suspect the ISO-2022-JP encoding happens to contain ">" and
"," while the UTF-8 encoding doesn't.

- Derek