Bug 15684 - Fix encoding issues with quote upload
Summary: Fix encoding issues with quote upload
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Tools (show other bugs)
Version: master
Hardware: All All
: P5 - low major (vote)
Assignee: Marc Véron
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-01-27 19:26 UTC by Nicole C. Engard
Modified: 2019-06-27 09:24 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
sample quotes (624 bytes, text/csv)
2016-01-27 19:26 UTC, Nicole C. Engard
Details
Goethe quotes without umlaut (148 bytes, application/csv)
2016-01-28 20:01 UTC, Marc Véron
Details
Goethe with umlaut (178 bytes, application/csv)
2016-01-28 20:02 UTC, Marc Véron
Details
Bug 15684: Fix encoding issues with quote upload (2.31 KB, patch)
2016-01-28 21:13 UTC, Marc Véron
Details | Diff | Splinter Review
Bug 15684: Fix encoding issues with quote upload (2.36 KB, patch)
2016-01-28 22:00 UTC, Mirko Tietgen
Details | Diff | Splinter Review
Bug 15684: Fix encoding issues with quote upload (2.41 KB, patch)
2016-01-29 12:12 UTC, Kyle M Hall
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Nicole C. Engard 2016-01-27 19:26:12 UTC
Created attachment 47358 [details]
sample quotes

We can't seem to import quotes anymore.  I just did on 3.20, but in 3.22 when the file loads up clicking the 'save' button does nothing.  This happens on a few sites.
Comment 1 Marc Véron 2016-01-28 20:01:25 UTC
Created attachment 47404 [details]
Goethe quotes without umlaut

This file does not contain ä, ö, ü etc. It works.
(Encoding is UTF8)
Comment 2 Marc Véron 2016-01-28 20:02:50 UTC
Created attachment 47405 [details]
Goethe with umlaut

This file contains ä, ö, ü.
It can not be saved.
Comment 3 Marc Véron 2016-01-28 20:06:28 UTC
Goethe with umlaut gives the following warning:
quotes-upload_ajax.pl: malformed UTF-8 character in JSON string, at character offset 52 (before "\\x{4b34}."],[2,"Goet...") at /usr/share/kohaclone/tools/quotes/quotes-upload_ajax.pl line 44., referer: http://10.0.0.62:8080/cgi-bin/koha/tools/quotes-upload.pl

I fixed Nicole's example (not all quotes were put in quotes "" and tried to upload.
This is the warning I get:
quotes-upload_ajax.pl: Wide character in subroutine entry at /usr/share/kohaclone/tools/quotes/quotes-upload_ajax.pl line 44., referer: http://10.0.0.62:8080/cgi-bin/koha/tools/quotes-upload.pl
Comment 4 Marc Véron 2016-01-28 21:13:32 UTC Comment hidden (obsolete)
Comment 5 Mirko Tietgen 2016-01-28 22:00:42 UTC Comment hidden (obsolete)
Comment 6 Jonathan Druart 2016-01-29 11:09:19 UTC
The patch looks good bug I don't see the quotes in the table after selecting the file.
Note that I also get a "Uploads limited to csv. Incorrect filetype: plain/text", I have bypassed forcing the filetype in the tt (+fileType = 'csv';).
Comment 7 Marc Véron 2016-01-29 11:30:54 UTC
(In reply to Jonathan Druart from comment #6)
> The patch looks good bug I don't see the quotes in the table after selecting
> the file.
> Note that I also get a "Uploads limited to csv. Incorrect filetype:
> plain/text", I have bypassed forcing the filetype in the tt (+fileType =
> 'csv';).

The instructions say: 
The quote uploader accepts standard csv files with two columns: "source","text"

This bug does not aim to change this behaviour. I think changing the accepted file type extension and/or the behaviour regarding the quotes should be covered by an other bug.
Comment 8 Kyle M Hall 2016-01-29 12:12:23 UTC
Created attachment 47463 [details] [review]
Bug 15684: Fix encoding issues with quote upload

To verify:
- Download attachment 'Goethe with umlaut' from Bug 15684
- Go to Tools > Quote Editor
- Click "Import quotes" and select the file
- The quotes display in a table. Click 'Sav quotes'
- Result: Nothing happens

To test:
- Apply patch
- Repeat steps above
- Verify that you get a message '2 quotes saved' and that the quotes
  are added in to the quotes table
- Test with attachment 'sample quotes' as well

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 9 Brendan Gallagher 2016-02-04 00:15:34 UTC
Pushed to Master - Should be in the May 2016 release.  Thanks
Comment 10 Julian Maurice 2016-02-04 11:41:11 UTC
Patch pushed to 3.22.x, will be in 3.22.3
Comment 11 Frédéric Demians 2016-02-20 08:52:10 UTC
This patch has been pushed to 3.20.x, will be in 3.20.9.