Bug 18461

Summary: Internal server error while approving OPAC-edited patron attributes containing umlauts (äöü)
Product: Koha Reporter: Marc Véron <veron>
Component: PatronsAssignee: Jonathan Druart <jonathan.druart>
Status: CLOSED FIXED QA Contact: Testopia <testopia>
Severity: critical    
Priority: P5 - low CC: alexbuckley, gmcharlt, jonathan.druart, katrin.fischer, kyle.m.hall, kyle, nick, tomascohen
Version: Main   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18339
Change sponsored?: --- Patch complexity: Trivial patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Bug Depends on: 13757    
Bug Blocks:    
Attachments: Bug 18461: Internal server error while approving OPAC-edited patron attributes containing umlauts (äöü)
Bug 18461: Internal server error while approving OPAC-edited patron attributes containing umlauts (äöü)
Bug 18461: (bug 13757 follow-up) Do not mix decode_json and to_json
Bug 18461: Make tests break on wrong UTF8 data handilng
Bug 18461: (bug 13757 follow-up) Do not mix decode_json and to_json
Bug 18461: Make tests break on wrong UTF8 data handilng
Bug 18461: (bug 13757 follow-up) Do not mix decode_json and to_json

Description Marc Véron 2017-04-19 20:02:40 UTC
If a OPAC user enters text containing umlauts (äöü) in a patron atribute and saves it, the Staff client will get an inernal server error in members-update.pl (approving).

E.g. a text like 111ä222 leads to the following in plack error log:

malformed UTF-8 character in JSON string, at character offset 14 (before "\x{fffd}2","code":"T...") at (...)/Koha/Patron/Modifications.pm line 100.
Comment 2 Marc Véron 2017-04-21 16:26:26 UTC
Created attachment 62538 [details] [review]
Bug 18461: Internal server error while approving OPAC-edited patron attributes containing umlauts (äöü)

If a OPAC user enters text containing umlauts (äöü) in a patron atribute and
saves it, the Staff client will get an inernal server error in
members-update.pl (approving).

E.g. a text like 111ä222 leads to the following in plack error log:

malformed UTF-8 character in JSON string, at character offset 14 (before
"\x{fffd}2","code":"T...") at (...)/Koha/Patron/Modifications.pm line 100.

To test:
- Reproduce problem above
- Apply patch
- Verify that Home > Patron > Update records displays without error
  and 'New value' displays properly
Comment 3 Marc Véron 2017-04-21 16:29:13 UTC
Setting to 'major' because of internal server error in staff client and,
if bug is triggered, you can no longer access Home > Patron > Update.
(You have to send a new, umlaut-less update request from OPAC)
Comment 4 Alex Buckley 2017-04-22 12:07:08 UTC
Created attachment 62569 [details] [review]
Bug 18461: Internal server error while approving OPAC-edited patron attributes containing umlauts (äöü)

If a OPAC user enters text containing umlauts (äöü) in a patron atribute and
saves it, the Staff client will get an inernal server error in
members-update.pl (approving).

E.g. a text like 111ä222 leads to the following in plack error log:

malformed UTF-8 character in JSON string, at character offset 14 (before
"\x{fffd}2","code":"T...") at (...)/Koha/Patron/Modifications.pm line 100.

To test:
- Reproduce problem above
- Apply patch
- Verify that Home > Patron > Update records displays without error
  and 'New value' displays properly

Followed test plan and it works as intended

Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Comment 5 Jonathan Druart 2017-04-24 16:51:36 UTC
It does not work for me, I get "The passed extended_attributes is not valid JSON"
Comment 6 Jonathan Druart 2017-04-24 16:58:13 UTC
Created attachment 62617 [details] [review]
Bug 18461: (bug 13757 follow-up) Do not mix decode_json and to_json

Internal server error while approving OPAC-edited patron attributes
containing umlauts (äöü)

If a OPAC user enters text containing umlauts (äöü) in a patron atribute
and saves it, the Staff client will get an inernal server error in
members-update.pl (approving).

E.g. a text like 111ä222 leads to the following in plack error log:

malformed UTF-8 character in JSON string, at character offset 14

To test:
- Reproduce problem above
- Apply patch
- Verify that Home > Patron > Update records displays without error
and 'New value' displays properly
- Approve!
Comment 7 Tomás Cohen Arazi 2017-04-24 18:50:45 UTC
Created attachment 62631 [details] [review]
Bug 18461: Make tests break on wrong UTF8 data handilng

This patch makes the tests fail if extended attributes handling fails due to
wrong UTF-8 data handling.

To test:
- Run:
  $ sudo koha-shell kohadev
 k$ cd kohaclone
 k$ prove t/db_dependent/Koha/Patron/Modifications.t
=> FAIL: Tests explode

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 8 Tomás Cohen Arazi 2017-04-24 18:50:50 UTC
Created attachment 62632 [details] [review]
Bug 18461: (bug 13757 follow-up) Do not mix decode_json and to_json

Internal server error while approving OPAC-edited patron attributes
containing umlauts (äöü)

If a OPAC user enters text containing umlauts (äöü) in a patron atribute
and saves it, the Staff client will get an inernal server error in
members-update.pl (approving).

E.g. a text like 111ä222 leads to the following in plack error log:

malformed UTF-8 character in JSON string, at character offset 14

To test:
- Reproduce problem above
- Apply patch
- Verify that Home > Patron > Update records displays without error
and 'New value' displays properly
- Approve!

Also, run the tests:
- Run:
  $ sudo koha-shell kohadev
 k$ cd kohaclone
 k$ prove t/db_dependent/Koha/Patron/Modifications.t
=> FAIL: Tests explode due to encoding problems in JSON handling
- Apply the patch
- Run the tests again:
 k$ prove t/db_dependent/Koha/Patron/Modifications.t
=> SUCCESS: Tests pass!

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 9 Nick Clemens 2017-04-25 10:02:26 UTC
Passing QA here but cant attach patches, ironically, because of the umlauts in the commit messages:
UnicodeEncodeError: 'ascii' codec can't encode characters in position 116-118: ordinal not in range(128)
Comment 10 Jonathan Druart 2017-04-25 12:33:12 UTC
(In reply to Nick Clemens from comment #9)
> Passing QA here but cant attach patches, ironically, because of the umlauts
> in the commit messages:
> UnicodeEncodeError: 'ascii' codec can't encode characters in position
> 116-118: ordinal not in range(128)

Should work without the -e option of git-bz
Comment 11 Nick Clemens 2017-04-25 13:10:48 UTC
Created attachment 62666 [details] [review]
Bug 18461: Make tests break on wrong UTF8 data handilng

This patch makes the tests fail if extended attributes handling fails due to
wrong UTF-8 data handling.

To test:
- Run:
  $ sudo koha-shell kohadev
 k$ cd kohaclone
 k$ prove t/db_dependent/Koha/Patron/Modifications.t
=> FAIL: Tests explode

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 12 Nick Clemens 2017-04-25 13:10:52 UTC
Created attachment 62667 [details] [review]
Bug 18461: (bug 13757 follow-up) Do not mix decode_json and to_json

Internal server error while approving OPAC-edited patron attributes
containing umlauts (äöü)

If a OPAC user enters text containing umlauts (äöü) in a patron atribute
and saves it, the Staff client will get an inernal server error in
members-update.pl (approving).

E.g. a text like 111ä222 leads to the following in plack error log:

malformed UTF-8 character in JSON string, at character offset 14

To test:
- Reproduce problem above
- Apply patch
- Verify that Home > Patron > Update records displays without error
and 'New value' displays properly
- Approve!

Also, run the tests:
- Run:
  $ sudo koha-shell kohadev
 k$ cd kohaclone
 k$ prove t/db_dependent/Koha/Patron/Modifications.t
=> FAIL: Tests explode due to encoding problems in JSON handling
- Apply the patch
- Run the tests again:
 k$ prove t/db_dependent/Koha/Patron/Modifications.t
=> SUCCESS: Tests pass!

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 13 Kyle M Hall 2017-04-28 10:25:28 UTC
Pushed to master for 17.05, thanks Tomas, Jonathan!
Comment 14 Katrin Fischer 2017-05-13 11:56:19 UTC
Depends on bug 13757 which is not part of 16.11.x.