Bug 12174 - Zotero connection with Koha: problem with UTF-8
Summary: Zotero connection with Koha: problem with UTF-8
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Architecture, internals, and plumbing (show other bugs)
Version: Main
Hardware: All All
: P5 - low minor (vote)
Assignee: Zeno Tajoli
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-05-02 09:51 UTC by Zeno Tajoli
Modified: 2016-06-21 21:37 UTC (History)
3 users (show)

See Also:
Change sponsored?: Sponsored
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
file opac-export.pl is patched (1.58 KB, patch)
2014-05-03 09:27 UTC, Zeno Tajoli
Details | Diff | Splinter Review
file opac-export.pl is patched (1.59 KB, patch)
2014-05-03 09:57 UTC, Zeno Tajoli
Details | Diff | Splinter Review
a follow-up to fix a missing print (598 bytes, patch)
2014-05-03 09:58 UTC, Zeno Tajoli
Details | Diff | Splinter Review
[SIGNED-OFF] Bug 12174 - Zotero connection with Koha: problem with UTF-8 (2.05 KB, patch)
2014-06-22 03:06 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
[SIGNED-OFF] Bug 12174 follow-up of the patch (a missing print statement) (709 bytes, patch)
2014-06-22 03:06 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
[PASSED QA] Bug 12174 - Zotero connection with Koha: problem with UTF-8 (2.23 KB, patch)
2014-07-30 21:52 UTC, Katrin Fischer
Details | Diff | Splinter Review
[PASSED QA] Bug 12174 follow-up of the patch (a missing print statement) (765 bytes, patch)
2014-07-30 21:53 UTC, Katrin Fischer
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Zeno Tajoli 2014-05-02 09:51:23 UTC
If i donwload Koha records with Zotero from server with Koha 3.12/3.14,
I have problems with UTF-8. I don't had this problem with an old 3.2 in the past.
Comment 1 Zeno Tajoli 2014-05-02 10:11:00 UTC
I find the problem.
It is a mix of apache / new Zotero working.
1)The bug is not present in installation that use Ngix instead of apache.
2)Zotero doesn't use only COinS from Koha but if use also the record export
(../cgi-bin/koha/opac-export.pl)

3)The cgi opac-export.pl it is ok to write a file on this but not if you try to use it like a stream in Zotero.
Try:
curl -i 'http://pust.urbe.it/cgi-bin/koha/opac-export.pl?format=utf8&op=export&bib=71000&save=Go'
The result:

HTTP/1.1 200 OK
Date: Fri, 02 May 2014 09:41:22 GMT
Server: Apache/2.2.22 (Debian)
Content-Disposition: attachment; filename="bib-71000.utf8"
Transfer-Encoding: chunked
Content-Type: application/octet-stream; charset=ISO-8859-1

01403nam a2200265 i 4500001000600000003000900006005001700015 [the iso2709 output]

4)The header "charset=ISO-8859-1" is the default on Apache (Ngix doesn't inser anything instead).

5)Zotero read the header and it thinks that the stream in iso2709 format is written with ISO-8859-1 charset.


So the fix is to be esplict on charset when we print the result.

From
94:     print $query->header(
95:      -type => 'application/octet-stream',
96:      -attachment=>"bib-$biblionumber.$format");
To
94:     print $query->header(
95:      -type => 'application/octet-stream',
96:      -charset=>'utf-8',
97:      -attachment=>"bib-$biblionumber.$format");

I will write and send the patch soon
Comment 2 Zeno Tajoli 2014-05-03 09:27:29 UTC Comment hidden (obsolete)
Comment 3 Zeno Tajoli 2014-05-03 09:57:47 UTC Comment hidden (obsolete)
Comment 4 Zeno Tajoli 2014-05-03 09:58:31 UTC Comment hidden (obsolete)
Comment 5 Zeno Tajoli 2014-05-03 09:59:56 UTC
To test the bug:
-- without the patch use Zotero on a recent Koha install with Apache.
You can use PTFS sandboxes. You can't use Biblibre sandboxes, they use Ngix.
-- Test a record with special chars.
-- Install the patch
-- Catch (using Zotero) the same record.
-- Test all directly download from Opac.
-- To check MARC-8, use a tool that support it. For example MarcEdit.
Comment 6 Bernardo Gonzalez Kriegel 2014-06-22 03:06:33 UTC Comment hidden (obsolete)
Comment 7 Bernardo Gonzalez Kriegel 2014-06-22 03:06:42 UTC Comment hidden (obsolete)
Comment 8 Katrin Fischer 2014-07-30 21:52:50 UTC
Created attachment 30359 [details] [review]
[PASSED QA] Bug 12174 - Zotero connection with Koha: problem with UTF-8

This patch solves the bug for UTF-8 output. MARC-8 works only if you save it as a file. In fact the standard Internet stack (browser, web server, etc.) doesn't know anything about MARC-8 or it's ISO base (ISO 2022).

To test the bug:
-- without the patch use Zotero on a recent Koha install with Apache.
You can use PTFS sandboxes. You can't use Biblibre sandboxes, they use Ngix.
-- Test a record with special chars.
-- Install the patch
-- Catch (using Zotero) the same record.
-- Test all directly download from Opac.
-- To check MARC-8, use a tool that support it. For example MarcEdit.

http://bugs.koha-community.org/show_bug.cgi?id=12174
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>

Work as described
Tested with zotero add-on for firefox.
Before patch bad diacritics, after patch ok

Record exported as MARC-8, transformed using
yaz-iconv -f marc8 -t utf-8 bib.marc8 > bib.utf8
yaz-marcdump -i marc -o line bib.utf8, checking accents,
also comparing with direct utf-8 export

No koha-qa errors

Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Tested with Zotero in Firefox and with yaz-iconv as suggested by
Bernardo, no problems found.
Passes tests and QA script.
Comment 9 Katrin Fischer 2014-07-30 21:53:16 UTC
Created attachment 30360 [details] [review]
[PASSED QA] Bug 12174 follow-up of the patch (a missing print statement)

http://bugs.koha-community.org/show_bug.cgi?id=12174
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Comment 10 Tomás Cohen Arazi 2014-07-31 15:04:49 UTC
Patches pushed to master.

Thanks Zeno!