Bug 26948 - Some Koha Emails are double encoded (HOLD, ODUE, ...)
Summary: Some Koha Emails are double encoded (HOLD, ODUE, ...)
Status: RESOLVED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: System Administration (show other bugs)
Version: master
Hardware: All All
: P5 - low critical (vote)
Assignee: Joonas Kylmälä
QA Contact: Martin Renvoize
URL:
Keywords:
Depends on: 22343
Blocks:
  Show dependency treegraph
 
Reported: 2020-11-06 12:08 UTC by Joonas Kylmälä
Modified: 2021-01-05 08:26 UTC (History)
6 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Text to go in the release notes:
Version(s) released in:
20.11.00


Attachments
Bug 22343: (QA follow-up) Remove double encoding from sent emails (1.82 KB, patch)
2020-11-06 12:09 UTC, Joonas Kylmälä
Details | Diff | Splinter Review
Bug 22343: Remove double encoding from sent emails (1.75 KB, patch)
2020-11-06 12:10 UTC, Joonas Kylmälä
Details | Diff | Splinter Review
Bug 26948: Remove double encoding from sent emails (1.69 KB, patch)
2020-11-06 12:14 UTC, Joonas Kylmälä
Details | Diff | Splinter Review
Bug 26948: Regression tests (874 bytes, patch)
2020-11-06 12:30 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 26948: Remove double encoding from sent emails (1.77 KB, patch)
2020-11-12 10:15 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 26948: Remove double encoding from sent emails (1.84 KB, patch)
2020-11-12 13:05 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 26948: Unit tests (1.36 KB, patch)
2020-11-12 13:36 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 26948: Remove double encoding from sent emails (1.83 KB, patch)
2020-11-12 13:49 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 26948: Regression tests (2.39 KB, patch)
2020-11-12 13:49 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 26948: Remove double encoding from sent emails (1.88 KB, patch)
2020-11-12 13:50 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 26948: Regression tests (2.39 KB, patch)
2020-11-12 13:50 UTC, Martin Renvoize
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Joonas Kylmälä 2020-11-06 12:08:45 UTC
After the refactoring in bug 22343 some emails sent from Koha are double encoded, meaning they show us garbled for the email recipient.
Comment 1 Joonas Kylmälä 2020-11-06 12:09:36 UTC Comment hidden (obsolete)
Comment 2 Joonas Kylmälä 2020-11-06 12:10:27 UTC Comment hidden (obsolete)
Comment 3 Joonas Kylmälä 2020-11-06 12:14:18 UTC
Created attachment 113244 [details] [review]
Bug 26948: Remove double encoding from sent emails

The sent emails from Koha were garbled / double encoded because
Koha::Email now after the refactoring and moving to use Email::Stuffer
handles the encoding itself. Previously the Koha::Email callee had to
handle the encoding. This patch removes the double encoding in the
email subject and content.

To test:
 1) Put some UTF-8 characters like ä in the HOLD template subject and body
 2) Create hold and check-in an item to make it Waiting and cause the
 email to be send
 3) Notice the email comes garbled / double encoded
 4) Apply patch
 5) Repeat step 2) and notice the email comes without any issues in
    encoding and look OK.
 6) Make sure "prove t/Koha/Email.t" and "prove
    t/db_dependent/Letters.t" and passes
Comment 4 Tomás Cohen Arazi 2020-11-06 12:30:30 UTC
Created attachment 113245 [details] [review]
Bug 26948: Regression tests

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 5 Joonas Kylmälä 2020-11-06 13:02:33 UTC
(In reply to Tomás Cohen Arazi from comment #4)
> Created attachment 113245 [details] [review] [review]
> Bug 26948: Regression tests
> 
> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

As discussed on #koha@freenode IRC channel, the test didn't test this particular issue. This still would benefit from a regression test.
Comment 6 Jonathan Druart 2020-11-09 09:37:34 UTC
Upping severity here.
Comment 7 Jonathan Druart 2020-11-12 10:15:35 UTC
Created attachment 113545 [details] [review]
Bug 26948: Remove double encoding from sent emails

The sent emails from Koha were garbled / double encoded because
Koha::Email now after the refactoring and moving to use Email::Stuffer
handles the encoding itself. Previously the Koha::Email callee had to
handle the encoding. This patch removes the double encoding in the
email subject and content.

To test:
 1) Put some UTF-8 characters like ä in the HOLD template subject and body
 2) Create hold and check-in an item to make it Waiting and cause the
 email to be send
 3) Notice the email comes garbled / double encoded
 4) Apply patch
 5) Repeat step 2) and notice the email comes without any issues in
    encoding and look OK.
 6) Make sure "prove t/Koha/Email.t" and "prove
    t/db_dependent/Letters.t" and passes

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 8 Kyle M Hall 2020-11-12 13:05:40 UTC
Created attachment 113553 [details] [review]
Bug 26948: Remove double encoding from sent emails

The sent emails from Koha were garbled / double encoded because
Koha::Email now after the refactoring and moving to use Email::Stuffer
handles the encoding itself. Previously the Koha::Email callee had to
handle the encoding. This patch removes the double encoding in the
email subject and content.

To test:
 1) Put some UTF-8 characters like ä in the HOLD template subject and body
 2) Create hold and check-in an item to make it Waiting and cause the
 email to be send
 3) Notice the email comes garbled / double encoded
 4) Apply patch
 5) Repeat step 2) and notice the email comes without any issues in
    encoding and look OK.
 6) Make sure "prove t/Koha/Email.t" and "prove
    t/db_dependent/Letters.t" and passes

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 9 Martin Renvoize 2020-11-12 13:36:41 UTC
Created attachment 113555 [details] [review]
Bug 26948: Unit tests
Comment 10 Martin Renvoize 2020-11-12 13:49:13 UTC
Created attachment 113558 [details] [review]
Bug 26948: Remove double encoding from sent emails

The sent emails from Koha were garbled / double encoded because
Koha::Email now after the refactoring and moving to use Email::Stuffer
handles the encoding itself. Previously the Koha::Email callee had to
handle the encoding. This patch removes the double encoding in the
email subject and content.

To test:
 1) Put some UTF-8 characters like ä in the HOLD template subject and body
 2) Create hold and check-in an item to make it Waiting and cause the
 email to be send
 3) Notice the email comes garbled / double encoded
 4) Apply patch
 5) Repeat step 2) and notice the email comes without any issues in
    encoding and look OK.
 6) Make sure "prove t/Koha/Email.t" and "prove
    t/db_dependent/Letters.t" and passes

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 11 Martin Renvoize 2020-11-12 13:49:16 UTC
Created attachment 113559 [details] [review]
Bug 26948: Regression tests

We already mock the send_or_die method from Email::Stuffer; This patch
updates that mock to extract the internal representation of the email
encoded message and looks for a known double encoded representation of a
character in the sample body.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 12 Martin Renvoize 2020-11-12 13:50:47 UTC
Created attachment 113560 [details] [review]
Bug 26948: Remove double encoding from sent emails

The sent emails from Koha were garbled / double encoded because
Koha::Email now after the refactoring and moving to use Email::Stuffer
handles the encoding itself. Previously the Koha::Email callee had to
handle the encoding. This patch removes the double encoding in the
email subject and content.

To test:
 1) Put some UTF-8 characters like ä in the HOLD template subject and body
 2) Create hold and check-in an item to make it Waiting and cause the
 email to be send
 3) Notice the email comes garbled / double encoded
 4) Apply patch
 5) Repeat step 2) and notice the email comes without any issues in
    encoding and look OK.
 6) Make sure "prove t/Koha/Email.t" and "prove
    t/db_dependent/Letters.t" and passes

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 13 Martin Renvoize 2020-11-12 13:50:51 UTC
Created attachment 113561 [details] [review]
Bug 26948: Regression tests

We already mock the send_or_die method from Email::Stuffer; This patch
updates that mock to extract the internal representation of the email
encoded message and looks for a known double encoded representation of a
character in the sample body.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 14 Martin Renvoize 2020-11-12 13:52:09 UTC
Ooops, collision.. I've restored your second SO/QA Kyle.


So, this isn't my neatest work, but it does at least add a regression test for the content double encoding.. the subject could likely be covered in a similar way but my brain was failing me to see how to check that.

Unit tests++
Comment 15 Jonathan Druart 2020-11-12 16:28:57 UTC
Pushed to master for 20.11, thanks to everybody involved!
Comment 16 Lucas Gass 2020-11-16 22:47:44 UTC
missing dependencies for 20.05.x, no backport