Bug 34870 - Unrecognized special characters when writing off an invoice with a note
Summary: Unrecognized special characters when writing off an invoice with a note
Status: Pushed to oldstable
Alias: None
Product: Koha
Classification: Unclassified
Component: I18N/L10N (show other bugs)
Version: Main
Hardware: All All
: P5 - low minor (vote)
Assignee: Emily-Rose Francoeur
QA Contact: Marcel de Rooy
URL:
Keywords:
Depends on:
Blocks: 34956
  Show dependency treegraph
 
Reported: 2023-09-21 18:00 UTC by Emily-Rose Francoeur
Modified: 2023-10-17 17:24 UTC (History)
7 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
This fixes the display of UTF-8 characters for write off notes under a patron's accounting section. Previously, if you added a note when writing off multiple charges ([Patron] > Accounting > Make a payment > Payment note column > + Add note), a note with special characters (for example, éçö) did not display correctly.
Version(s) released in:
23.11.00,23.05.05,22.11.11


Attachments
Bug 34870: Unrecognized special characters (1.64 KB, patch)
2023-09-21 19:11 UTC, Emily-Rose Francoeur
Details | Diff | Splinter Review
Bug 34870: Unrecognized special characters (1.71 KB, text/plain)
2023-09-21 20:33 UTC, Laura Escamilla
Details
Bug 34870:Perform UTF8 encoding before redirection (2.29 KB, patch)
2023-09-22 18:00 UTC, Emily-Rose Francoeur
Details | Diff | Splinter Review
Bug 34870: Perform UTF8 encoding before redirection (2.34 KB, patch)
2023-09-28 20:58 UTC, David Nind
Details | Diff | Splinter Review
Bug 34870: Perform UTF8 encoding before redirection (2.45 KB, patch)
2023-09-29 07:35 UTC, Marcel de Rooy
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Emily-Rose Francoeur 2023-09-21 18:00:34 UTC
When we add a note with special characters to an invoice and then click on “Write off selected” the note field will display “�” symbols where the special characters are supposed to be.

This is how you can reproduce the bug :
1- Go to any patron profile > Accounting > Create manual invoice
2- Fill the fields and click the “Save” button
3- Click “Make a payment”
4- Select the created invoice and click on “+ Add note”
5- Enter special characters (e.g., éçö)
6- Click the “Write off selected” button
7- In the note field, you should see “�” symbols where the special characters are supposed to be
Comment 1 Emily-Rose Francoeur 2023-09-21 19:11:26 UTC
Created attachment 156005 [details] [review]
Bug 34870: Unrecognized special characters

The informations entered in the note are encoded and sent in the 'notes' parameter of the URL. The issue arises because paycollect.pl get the information using param(), which does not support encoded text. By using param_fetch(), we can retrieve the informations without any issues.

TEST PLAN
1) Go to any patron profile > Accounting > Create manual invoice
2) Fill the fields and click the “Save” button
3) Click “Make a payment”
4) Select the created invoice and click on “+ Add note”
5) Enter special characters (e.g., éçö)
6) Click the “Write off selected” button
7) In the note field, you should see “�” symbols where the special characters are supposed to be
8) Apply the patch
9) Refresh the page; now the special characters display correctly
Comment 2 Laura Escamilla 2023-09-21 20:33:35 UTC
Created attachment 156006 [details]
Bug 34870: Unrecognized special characters

The informations entered in the note are encoded and sent in the 'notes' parameter of the URL. The issue arises because paycollect.pl get the information using param(), which does not support encoded text. By using param_fetch(), we can retrieve the informations without any issues.

TEST PLAN
1) Go to any patron profile > Accounting > Create manual invoice
2) Fill the fields and click the “Save” button
3) Click “Make a payment”
4) Select the created invoice and click on “+ Add note”
5) Enter special characters (e.g., éçö)
6) Click the “Write off selected” button
7) In the note field, you should see “�” symbols where the special characters are supposed to be
8) Apply the patch
9) Refresh the page; now the special characters display correctly

Signed-off-by: Laura Escamilla <laura.escamilla@bywatersolutions.com>
Comment 3 Katrin Fischer 2023-09-21 22:04:47 UTC
Don't forget to set assignee :)
Comment 4 Jonathan Druart 2023-09-22 13:41:46 UTC
No, you need to URI escape it, or other characters won't pass (;).
Also there is something very wrong here. If we select several lines and add different notes, only the last one is kept?

Looks like it needs a bit more of thinking overall.

Also the commit title can be improved to tell more what the patch does.
Comment 5 Emily-Rose Francoeur 2023-09-22 18:00:24 UTC
Created attachment 156120 [details] [review]
Bug 34870:Perform UTF8 encoding before redirection

Display special characters correctly when writing off an invoice.

The issue arises because pay.pl does not perform UTF-8 encoding on the “notes” parameter before redirecting the page. By using uri_escape_utf8, characters with a code above 255 are also UTF-8 encoded. Then, paycollect.pl can collect the information without any trouble.

This patch work with “;” and “:” characters.

TEST PLAN
1) Go to any patron profile > Accounting
2) Click “Create manual invoice”
3) Fill the fields and click the “Save” button
4) Repeat steps 2 and 3
5) Click “Make a payment”
6) Add a note with special characters for each one (e.g., éçö)
7) Select the created invoices and click the “Write off selected” button
8) In the note field, you should see “�” symbols where the special characters are supposed to be
9) Click the “Cancel” button
10) Now add a note with “:;” for one of the invoices
11) Select the invoices and click the “Write off selected” button
12) You should see a page with an error 500
13) Go back to the “Make a payment” page
14) Apply the patch
15) Add a note with special characters for each one (e.g., éçö)
16) Add “:;” to one of the note fields
17) Select the invoices and click the “Write off selected” button
18) Now the special characters are displayed correctly
Comment 6 Emily-Rose Francoeur 2023-09-22 18:23:13 UTC
This is an updated test plan, because there is no 500 error page displayed after entering “:;” in the note field.

TEST PLAN
1) Go to any patron’s profile > Accounting
2) Click “Create manual invoice”
3) Complete the required fields and click the “Save” button
4) Repeat steps 2 and 3
5) Click “Make a payment”
6) Add a note with special characters for each one (e.g., éçö)
7) Add “:;” to the note field for one of the invoices
8) Select the created invoices and click the “Write off selected” button
9) In the note field, you should see “�” symbols where the special characters are supposed to be, and there would be nothing for “:;” characters.
10) Click the “Cancel” button
11) Apply the patch
12) Repeat steps 6 to 8
13) Verify that the special characters are now displayed correctly
Comment 7 David Nind 2023-09-26 23:16:37 UTC
Hi Émily-Rose.

I had a go at testing, and can confirm the issue.

However, with the patch applied it doesn't work for me as expected. 

I get this after clicking 'Write off selected':

:;
éçö

(where :; entered into the first note and éçö for the second note)

I tested using Firefox (browser cache cleared after applying the patch)


David


Testing notes (using koha-testing-docker):

1. The note field mentioned is in the column 'Payment note' on the 'Make a payment' tab. Click the '+ Add note' to add a payment note (it took me a while to find this!).
Comment 8 Emily-Rose Francoeur 2023-09-28 12:56:04 UTC
I forgot last time to mark one of the attachments as obsolete. Now, it should work. I've also updated the test plan to make it clearer.

TEST PLAN
1) Go to any patron’s profile > Accounting
2) Click on “Create manual invoice”
3) Complete the required fields and click the “Save” button
4) Repeat steps 2 and 3
5) Click on “Make a payment”
6) Click on “+ Add note” and add a note with special characters (e.g., éçö) for each invoice you create
7) Add “:;” to the note field for one of the invoices
8) Select the created invoices and click the “Write off selected” button
9) In the note field, you should see “�” symbols where the special characters are supposed to be, and there should be nothing for “:;” characters.
10) Click the “Cancel” button
11) Apply the patch
12) Repeat steps 6 to 8
13) Verify that the special characters are now displayed correctly
Comment 9 David Nind 2023-09-28 20:58:13 UTC
Created attachment 156357 [details] [review]
Bug 34870: Perform UTF8 encoding before redirection

Display special characters correctly when writing off an invoice.

The issue arises because pay.pl does not perform UTF-8 encoding on the “notes” parameter before redirecting the page. By using uri_escape_utf8, characters with a code above 255 are also UTF-8 encoded. Then, paycollect.pl can collect the information without any trouble.

This patch work with “;” and “:” characters.

TEST PLAN
1) Go to any patron profile > Accounting
2) Click “Create manual invoice”
3) Fill the fields and click the “Save” button
4) Repeat steps 2 and 3
5) Click “Make a payment”
6) Add a note with special characters for each one (e.g., éçö)
7) Select the created invoices and click the “Write off selected” button
8) In the note field, you should see “�” symbols where the special characters are supposed to be
9) Click the “Cancel” button
10) Now add a note with “:;” for one of the invoices
11) Select the invoices and click the “Write off selected” button
12) You should see a page with an error 500
13) Go back to the “Make a payment” page
14) Apply the patch
15) Add a note with special characters for each one (e.g., éçö)
16) Add “:;” to one of the note fields
17) Select the invoices and click the “Write off selected” button
18) Now the special characters are displayed correctly

Signed-off-by: David Nind <david@davidnind.com>
Comment 10 David Nind 2023-09-28 20:59:02 UTC
(In reply to Émily-Rose Francoeur from comment #8)
> I forgot last time to mark one of the attachments as obsolete. Now, it
> should work. I've also updated the test plan to make it clearer.

Thanks Émily-Rose! Now signed off.

David
Comment 11 Marcel de Rooy 2023-09-29 07:35:29 UTC
Created attachment 156363 [details] [review]
Bug 34870: Perform UTF8 encoding before redirection

Display special characters correctly when writing off an invoice.

The issue arises because pay.pl does not perform UTF-8 encoding on the “notes” parameter before redirecting the page. By using uri_escape_utf8, characters with a code above 255 are also UTF-8 encoded. Then, paycollect.pl can collect the information without any trouble.

This patch work with “;” and “:” characters.

TEST PLAN
1) Go to any patron profile > Accounting
2) Click “Create manual invoice”
3) Fill the fields and click the “Save” button
4) Repeat steps 2 and 3
5) Click “Make a payment”
6) Add a note with special characters for each one (e.g., éçö)
7) Select the created invoices and click the “Write off selected” button
8) In the note field, you should see “�” symbols where the special characters are supposed to be
9) Click the “Cancel” button
10) Now add a note with “:;” for one of the invoices
11) Select the invoices and click the “Write off selected” button
12) You should see a page with an error 500
13) Go back to the “Make a payment” page
14) Apply the patch
15) Add a note with special characters for each one (e.g., éçö)
16) Add “:;” to one of the note fields
17) Select the invoices and click the “Write off selected” button
18) Now the special characters are displayed correctly

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
[EDIT] Added parentheses around construct for clarity.
Comment 12 Marcel de Rooy 2023-09-29 07:56:29 UTC
(In reply to Jonathan Druart from comment #4)
> No, you need to URI escape it, or other characters won't pass (;).
> Also there is something very wrong here. If we select several lines and add
> different notes, only the last one is kept?
> 
> Looks like it needs a bit more of thinking overall.
> 
> Also the commit title can be improved to tell more what the patch does.

Will open a new report for the multiple notes thing.
Comment 13 Tomás Cohen Arazi 2023-10-04 15:09:42 UTC
Pushed to master for 23.11.

Nice work everyone, thanks!
Comment 14 Fridolin Somers 2023-10-06 00:36:19 UTC
Pushed to 23.05.x for 23.05.05
Comment 15 Matt Blenkinsop 2023-10-17 17:24:44 UTC
Nice work everyone!

Pushed to oldstable for 22.11.x