Bug 13245 - Send Cart via Email corrupts some Unicode characters
Summary: Send Cart via Email corrupts some Unicode characters
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: OPAC (show other bugs)
Version: unspecified
Hardware: All All
: P3 minor (vote)
Assignee: Owen Leonard
QA Contact: Testopia
URL:
Keywords:
: 18845 (view as bug list)
Depends on:
Blocks:
 
Reported: 2014-11-13 11:25 UTC by Pongtawat
Modified: 2019-10-14 19:55 UTC (History)
4 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Thai character that has problem (42 bytes, text/plain)
2014-11-17 03:59 UTC, Pongtawat
Details
Screenshot of Sent Cart page, showing how the comment should read (7.98 KB, image/png)
2014-11-17 04:00 UTC, Pongtawat
Details
Screenshot of email received (34.45 KB, image/png)
2014-11-17 04:01 UTC, Pongtawat
Details
Screenshot of cart email form (313.37 KB, image/jpeg)
2016-10-25 09:16 UTC, Filippos Kolovos
Details
Screenshot of cart email received (134.67 KB, image/jpeg)
2016-10-25 09:17 UTC, Filippos Kolovos
Details
Screenshot of list email form (323.58 KB, image/jpeg)
2016-10-25 09:18 UTC, Filippos Kolovos
Details
Screenshot of list email received (136.71 KB, image/jpeg)
2016-10-25 09:19 UTC, Filippos Kolovos
Details
Patch for correcting the cart email encoding (941 bytes, patch)
2016-10-25 09:21 UTC, Filippos Kolovos
Details | Diff | Splinter Review
Patch for correcting the list email encoding (438 bytes, patch)
2016-10-25 09:22 UTC, Filippos Kolovos
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Pongtawat 2014-11-13 11:25:42 UTC
In OPAC -> Cart -> Send (via Email), if the comment text or the text in item list being send contains some specific Unicode character, the email received will have that character corrupted.

Step to reproduce:
- Go to OPAC, search for something
- Add some item to cart
- View your cart
- Click "Send"
- Enter your email address and enter "ออ" as the comment.
- View your email, you will get something like "�¸ �" as comment.

This is a known bug from HTML-Format used by html2text TT filter which was used to format email message:

https://rt.cpan.org/Public/Bug/Display.html?id=9700

It can be easily fixed there by editing just one line as shown in the bug report above. However, it still not fixed as of HTML-Format v2.11.

We may also fix this problem on Koha side as well by not using html2text and turn the template into plain text (or turn the email into HTML).

Tested on 3.12, but this part of code is the same in master so it should be affected as well.
Comment 1 Frédéric Demians 2014-11-14 11:08:22 UTC
> - Enter your email address and enter "ออ" as the comment.
> - View your email, you will get something like "�¸ �" as comment.
> 
> This is a known bug from HTML-Format used by html2text TT filter which was
> used to format email message:
> 
> https://rt.cpan.org/Public/Bug/Display.html?id=9700

It works on master with French accented characters. Could your provide the culprit characters in an attached text file, and in a screenshot?

> We may also fix this problem on Koha side as well by not using html2text and
> turn the template into plain text (or turn the email into HTML).

It's not an option. The template is in HTML in order to be translatable: see bug 8062.
Comment 2 Pongtawat 2014-11-17 03:59:21 UTC
Created attachment 33605 [details]
Thai character that has problem
Comment 3 Pongtawat 2014-11-17 04:00:11 UTC
Created attachment 33606 [details]
Screenshot of Sent Cart page, showing how the comment should read
Comment 4 Pongtawat 2014-11-17 04:01:41 UTC
Created attachment 33607 [details]
Screenshot of email received

Screenshot of email received, notice the broken letter. Some browser will just remove this character, while some other will switch to non-UTF8 encoding (which is worse)
Comment 5 Pongtawat 2014-11-17 04:14:22 UTC
Thank you for make it clear why we need HTML there. 

I have attached an example and screenshots. The letter we have problem with is Thai character "อ" which is encoded in UTF8 as E0 B8 AD. 

The following code in HTML-Format will simply remove A0 and AD:

$text =~ tr/\xA0\xAD/ /d;

Hence the comment in the received email source in the example above is as follow:

--------------------------------------------------------------------
=E0=B8=E0=B8=B2 - =E0=B9=80=E0=B8 - =E0=B9=82=E0=B8 - =E0=B8=E0=B8
--------------------------------------------------------------------

while the correct one should be:

--------------------------------------------------------------------
=E0=B8=AD=E0=B8=B2 - =E0=B9=80=E0=B8=AD - =E0=B9=82=E0=B8=AD - =E0=B8=AD=E0=B8=AD
--------------------------------------------------------------------

In my installation, I have switched Send Cart to HTML email, avoiding html2text, since HTML emails should be the norm now. IMO, it would be great if send cart could use easily-to-change template just like notice and slip.
Comment 6 Owen Leonard 2016-06-21 18:51:48 UTC
Is this bug still valid? I appears to be working correctly in master, but I'm not entirely sure I'm testing correctly.
Comment 7 Jonathan Druart 2016-08-16 14:30:15 UTC
(In reply to Owen Leonard from comment #6)
> Is this bug still valid? I appears to be working correctly in master, but
> I'm not entirely sure I'm testing correctly.

Yes I guess it is still an issue as we use HTML-Format 2.11.
I'd opt to close this bug as it's not a bug from Koha.
Comment 8 Filippos Kolovos 2016-10-25 09:10:03 UTC
Goodmorning to all,

There is also another problem concerning the cart (and lists) when they are sent via email nad contain Greek characters at the subject and at the name of the sender. 

The subject and the sender's name arrive unintelligible to the end user, but only the greek characters. However, the message's content arrives correctly. 

Steps to Reproduce for the cart:
- Go to the OPAC environment and login (preferably the full name of the logged in user should contain non-english characters as well)
- Change to greek language, or a language that will contain unicode characters at the subject line
- Add some items to your cart
- View your cart
- Click "Send"
- Enter the email address of the recipient and a comment (the comment is optional)
- Click "Send"

Steps to reproduce for the List:
- Go to the OPAC environment and login (preferably the full name of the logged in user should contain non-english characters as well)
- Change to greek language, or a language that will contain unicode characters at the subject line
- Select a list from your lists, or create one giving it a name with non-english characters
- Click "Send"
- Enter the email address of the recipient and a comment (the comment is optional)
- Click "Send"

The subjects of the emails received for each cart / lists are unintelligible. The are supposed to read "Your Cart" and "Your List: name of list" in Greek.
Also, the sender's name, if it is Greek, or I suppose non-english, it is also unintelligible.

I attach:

- Four screenshots concerning the send form for each case and the email received for each case.
- Two proposed patches, one for each case (opac-sendbasket.pl and opac-sendshelf.pl). It seems that the subject and sender's encoding has to be in MIME-Q and not in UTF-8, as they have to be quoted-printable "encoded".
Comment 9 Filippos Kolovos 2016-10-25 09:16:18 UTC
Created attachment 56836 [details]
Screenshot of cart email form
Comment 10 Filippos Kolovos 2016-10-25 09:17:43 UTC
Created attachment 56837 [details]
Screenshot of cart email received

In here you can see that the subject and the sender's name are unintelligible. This happens to any mailer I have tried.
Comment 11 Filippos Kolovos 2016-10-25 09:18:31 UTC
Created attachment 56838 [details]
Screenshot of list email form

This is a test list, with a greek name that is ready to be sent via the form in  opac
Comment 12 Filippos Kolovos 2016-10-25 09:19:21 UTC
Created attachment 56839 [details]
Screenshot of list email received

This is the email received for the list sent, with the subject line unintelligible as well.
Comment 13 Filippos Kolovos 2016-10-25 09:21:32 UTC
Created attachment 56840 [details] [review]
Patch for correcting the cart email encoding

This proposed patch corrects the encoding of the user's name and the subject of the email.
Comment 14 Filippos Kolovos 2016-10-25 09:22:09 UTC
Created attachment 56841 [details] [review]
Patch for correcting the list email encoding

This proposed patch corrects the encoding of the email's subject.
Comment 15 Jonathan Druart 2016-10-25 09:47:24 UTC
Hi Filippos,
Thanks for you patch but have a look at our wiki page (https://wiki.koha-community.org/wiki/Submitting_A_Patch) to know how format it correctly.
When it's done you can change the status of this bug report to "Needs Signoff".
Comment 16 Filippos Kolovos 2016-10-25 11:44:37 UTC
Dear Jonathan goodafternoon!

Thank you for your comment! I am a first-timer and I apologize if I do some miss-steps. 

Only one question, how will I be able to change the published comments and/or patches, since it does not let me edit my comments and patches?

Thank again you very much for your help!
Comment 17 Jonathan Druart 2016-10-25 12:12:08 UTC
(In reply to Filippos Kolovos from comment #16)
> Only one question, how will I be able to change the published comments
> and/or patches, since it does not let me edit my comments and patches?

You cannot, just attach the new patches and obsolete the old ones (details > edit details > tick "obsolete" > submit)
Comment 18 Pongtawat 2017-08-15 15:00:41 UTC
(In reply to Owen Leonard from comment #6)

I just revisit this bug and my original problem was already gone (tested with 16.11). This is probably due to unicode clean up sometimes ago.

However, I leave this bug open since Filippos add another problem with send cart here.
Comment 19 Katrin Fischer 2017-08-18 10:33:10 UTC
*** Bug 18845 has been marked as a duplicate of this bug. ***
Comment 20 Katrin Fischer 2017-08-18 10:44:05 UTC
We've also had reports about this problem in 16.11 and 3.22 - Filippos, will you be able to work on this again?
Comment 21 Owen Leonard 2018-06-29 14:18:00 UTC
(In reply to Pongtawat from comment #18)

> However, I leave this bug open since Filippos add another problem with send
> cart here.

I don't know what problem this refers to. If anyone knows, please open a new bug for it.