Bug 26948

Summary: Some Koha Emails are double encoded (HOLD, ODUE, ...)
Product: Koha Reporter: Joonas Kylmälä <joonas.kylmala>
Component: System AdministrationAssignee: Joonas Kylmälä <joonas.kylmala>
Status: CLOSED FIXED QA Contact: Martin Renvoize <martin.renvoize>
Severity: critical    
Priority: P5 - low CC: fridolin.somers, gmcharlt, jonathan.druart, lucas, martin.renvoize
Version: Main   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
20.11.00
Bug Depends on: 22343    
Bug Blocks:    
Attachments: Bug 22343: (QA follow-up) Remove double encoding from sent emails
Bug 22343: Remove double encoding from sent emails
Bug 26948: Remove double encoding from sent emails
Bug 26948: Regression tests
Bug 26948: Remove double encoding from sent emails
Bug 26948: Remove double encoding from sent emails
Bug 26948: Unit tests
Bug 26948: Remove double encoding from sent emails
Bug 26948: Regression tests
Bug 26948: Remove double encoding from sent emails
Bug 26948: Regression tests

Description Joonas Kylmälä 2020-11-06 12:08:45 UTC
After the refactoring in bug 22343 some emails sent from Koha are double encoded, meaning they show us garbled for the email recipient.
Comment 1 Joonas Kylmälä 2020-11-06 12:09:36 UTC Comment hidden (obsolete)
Comment 2 Joonas Kylmälä 2020-11-06 12:10:27 UTC Comment hidden (obsolete)
Comment 3 Joonas Kylmälä 2020-11-06 12:14:18 UTC
Created attachment 113244 [details] [review]
Bug 26948: Remove double encoding from sent emails

The sent emails from Koha were garbled / double encoded because
Koha::Email now after the refactoring and moving to use Email::Stuffer
handles the encoding itself. Previously the Koha::Email callee had to
handle the encoding. This patch removes the double encoding in the
email subject and content.

To test:
 1) Put some UTF-8 characters like ä in the HOLD template subject and body
 2) Create hold and check-in an item to make it Waiting and cause the
 email to be send
 3) Notice the email comes garbled / double encoded
 4) Apply patch
 5) Repeat step 2) and notice the email comes without any issues in
    encoding and look OK.
 6) Make sure "prove t/Koha/Email.t" and "prove
    t/db_dependent/Letters.t" and passes
Comment 4 Tomás Cohen Arazi 2020-11-06 12:30:30 UTC
Created attachment 113245 [details] [review]
Bug 26948: Regression tests

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 5 Joonas Kylmälä 2020-11-06 13:02:33 UTC
(In reply to Tomás Cohen Arazi from comment #4)
> Created attachment 113245 [details] [review] [review]
> Bug 26948: Regression tests
> 
> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

As discussed on #koha@freenode IRC channel, the test didn't test this particular issue. This still would benefit from a regression test.
Comment 6 Jonathan Druart 2020-11-09 09:37:34 UTC
Upping severity here.
Comment 7 Jonathan Druart 2020-11-12 10:15:35 UTC
Created attachment 113545 [details] [review]
Bug 26948: Remove double encoding from sent emails

The sent emails from Koha were garbled / double encoded because
Koha::Email now after the refactoring and moving to use Email::Stuffer
handles the encoding itself. Previously the Koha::Email callee had to
handle the encoding. This patch removes the double encoding in the
email subject and content.

To test:
 1) Put some UTF-8 characters like ä in the HOLD template subject and body
 2) Create hold and check-in an item to make it Waiting and cause the
 email to be send
 3) Notice the email comes garbled / double encoded
 4) Apply patch
 5) Repeat step 2) and notice the email comes without any issues in
    encoding and look OK.
 6) Make sure "prove t/Koha/Email.t" and "prove
    t/db_dependent/Letters.t" and passes

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 8 Kyle M Hall 2020-11-12 13:05:40 UTC
Created attachment 113553 [details] [review]
Bug 26948: Remove double encoding from sent emails

The sent emails from Koha were garbled / double encoded because
Koha::Email now after the refactoring and moving to use Email::Stuffer
handles the encoding itself. Previously the Koha::Email callee had to
handle the encoding. This patch removes the double encoding in the
email subject and content.

To test:
 1) Put some UTF-8 characters like ä in the HOLD template subject and body
 2) Create hold and check-in an item to make it Waiting and cause the
 email to be send
 3) Notice the email comes garbled / double encoded
 4) Apply patch
 5) Repeat step 2) and notice the email comes without any issues in
    encoding and look OK.
 6) Make sure "prove t/Koha/Email.t" and "prove
    t/db_dependent/Letters.t" and passes

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 9 Martin Renvoize 2020-11-12 13:36:41 UTC
Created attachment 113555 [details] [review]
Bug 26948: Unit tests
Comment 10 Martin Renvoize 2020-11-12 13:49:13 UTC
Created attachment 113558 [details] [review]
Bug 26948: Remove double encoding from sent emails

The sent emails from Koha were garbled / double encoded because
Koha::Email now after the refactoring and moving to use Email::Stuffer
handles the encoding itself. Previously the Koha::Email callee had to
handle the encoding. This patch removes the double encoding in the
email subject and content.

To test:
 1) Put some UTF-8 characters like ä in the HOLD template subject and body
 2) Create hold and check-in an item to make it Waiting and cause the
 email to be send
 3) Notice the email comes garbled / double encoded
 4) Apply patch
 5) Repeat step 2) and notice the email comes without any issues in
    encoding and look OK.
 6) Make sure "prove t/Koha/Email.t" and "prove
    t/db_dependent/Letters.t" and passes

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 11 Martin Renvoize 2020-11-12 13:49:16 UTC
Created attachment 113559 [details] [review]
Bug 26948: Regression tests

We already mock the send_or_die method from Email::Stuffer; This patch
updates that mock to extract the internal representation of the email
encoded message and looks for a known double encoded representation of a
character in the sample body.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 12 Martin Renvoize 2020-11-12 13:50:47 UTC
Created attachment 113560 [details] [review]
Bug 26948: Remove double encoding from sent emails

The sent emails from Koha were garbled / double encoded because
Koha::Email now after the refactoring and moving to use Email::Stuffer
handles the encoding itself. Previously the Koha::Email callee had to
handle the encoding. This patch removes the double encoding in the
email subject and content.

To test:
 1) Put some UTF-8 characters like ä in the HOLD template subject and body
 2) Create hold and check-in an item to make it Waiting and cause the
 email to be send
 3) Notice the email comes garbled / double encoded
 4) Apply patch
 5) Repeat step 2) and notice the email comes without any issues in
    encoding and look OK.
 6) Make sure "prove t/Koha/Email.t" and "prove
    t/db_dependent/Letters.t" and passes

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 13 Martin Renvoize 2020-11-12 13:50:51 UTC
Created attachment 113561 [details] [review]
Bug 26948: Regression tests

We already mock the send_or_die method from Email::Stuffer; This patch
updates that mock to extract the internal representation of the email
encoded message and looks for a known double encoded representation of a
character in the sample body.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 14 Martin Renvoize 2020-11-12 13:52:09 UTC
Ooops, collision.. I've restored your second SO/QA Kyle.


So, this isn't my neatest work, but it does at least add a regression test for the content double encoding.. the subject could likely be covered in a similar way but my brain was failing me to see how to check that.

Unit tests++
Comment 15 Jonathan Druart 2020-11-12 16:28:57 UTC
Pushed to master for 20.11, thanks to everybody involved!
Comment 16 Lucas Gass 2020-11-16 22:47:44 UTC
missing dependencies for 20.05.x, no backport