Bug 15684

Summary: Fix encoding issues with quote upload
Product: Koha Reporter: Nicole C. Engard <nengard>
Component: ToolsAssignee: Marc Véron <veron>
Status: CLOSED FIXED QA Contact: Testopia <testopia>
Severity: major    
Priority: P5 - low CC: brendan, f.demians, jonathan.druart, julian.maurice, veron
Version: Main   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8963
Change sponsored?: --- Patch complexity: Trivial patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Attachments: sample quotes
Goethe quotes without umlaut
Goethe with umlaut
Bug 15684: Fix encoding issues with quote upload
Bug 15684: Fix encoding issues with quote upload
Bug 15684: Fix encoding issues with quote upload

Description Nicole C. Engard 2016-01-27 19:26:12 UTC
Created attachment 47358 [details]
sample quotes

We can't seem to import quotes anymore.  I just did on 3.20, but in 3.22 when the file loads up clicking the 'save' button does nothing.  This happens on a few sites.
Comment 1 Marc Véron 2016-01-28 20:01:25 UTC
Created attachment 47404 [details]
Goethe quotes without umlaut

This file does not contain ä, ö, ü etc. It works.
(Encoding is UTF8)
Comment 2 Marc Véron 2016-01-28 20:02:50 UTC
Created attachment 47405 [details]
Goethe with umlaut

This file contains ä, ö, ü.
It can not be saved.
Comment 3 Marc Véron 2016-01-28 20:06:28 UTC
Goethe with umlaut gives the following warning:
quotes-upload_ajax.pl: malformed UTF-8 character in JSON string, at character offset 52 (before "\\x{4b34}."],[2,"Goet...") at /usr/share/kohaclone/tools/quotes/quotes-upload_ajax.pl line 44., referer: http://10.0.0.62:8080/cgi-bin/koha/tools/quotes-upload.pl

I fixed Nicole's example (not all quotes were put in quotes "" and tried to upload.
This is the warning I get:
quotes-upload_ajax.pl: Wide character in subroutine entry at /usr/share/kohaclone/tools/quotes/quotes-upload_ajax.pl line 44., referer: http://10.0.0.62:8080/cgi-bin/koha/tools/quotes-upload.pl
Comment 4 Marc Véron 2016-01-28 21:13:32 UTC Comment hidden (obsolete)
Comment 5 Mirko Tietgen 2016-01-28 22:00:42 UTC Comment hidden (obsolete)
Comment 6 Jonathan Druart 2016-01-29 11:09:19 UTC
The patch looks good bug I don't see the quotes in the table after selecting the file.
Note that I also get a "Uploads limited to csv. Incorrect filetype: plain/text", I have bypassed forcing the filetype in the tt (+fileType = 'csv';).
Comment 7 Marc Véron 2016-01-29 11:30:54 UTC
(In reply to Jonathan Druart from comment #6)
> The patch looks good bug I don't see the quotes in the table after selecting
> the file.
> Note that I also get a "Uploads limited to csv. Incorrect filetype:
> plain/text", I have bypassed forcing the filetype in the tt (+fileType =
> 'csv';).

The instructions say: 
The quote uploader accepts standard csv files with two columns: "source","text"

This bug does not aim to change this behaviour. I think changing the accepted file type extension and/or the behaviour regarding the quotes should be covered by an other bug.
Comment 8 Kyle M Hall 2016-01-29 12:12:23 UTC
Created attachment 47463 [details] [review]
Bug 15684: Fix encoding issues with quote upload

To verify:
- Download attachment 'Goethe with umlaut' from Bug 15684
- Go to Tools > Quote Editor
- Click "Import quotes" and select the file
- The quotes display in a table. Click 'Sav quotes'
- Result: Nothing happens

To test:
- Apply patch
- Repeat steps above
- Verify that you get a message '2 quotes saved' and that the quotes
  are added in to the quotes table
- Test with attachment 'sample quotes' as well

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 9 Brendan Gallagher 2016-02-04 00:15:34 UTC
Pushed to Master - Should be in the May 2016 release.  Thanks
Comment 10 Julian Maurice 2016-02-04 11:41:11 UTC
Patch pushed to 3.22.x, will be in 3.22.3
Comment 11 Frédéric Demians 2016-02-20 08:52:10 UTC
This patch has been pushed to 3.20.x, will be in 3.20.9.