Bug 12174

Summary: Zotero connection with Koha: problem with UTF-8
Product: Koha Reporter: Zeno Tajoli <ztajoli>
Component: Architecture, internals, and plumbingAssignee: Zeno Tajoli <ztajoli>
Status: CLOSED FIXED QA Contact: Testopia <testopia>
Severity: minor    
Priority: P5 - low CC: bgkriegel, marjorie.barry-vila, tomascohen
Version: Main   
Hardware: All   
OS: All   
Change sponsored?: Sponsored Patch complexity: Small patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:
Attachments: file opac-export.pl is patched
file opac-export.pl is patched
a follow-up to fix a missing print
[SIGNED-OFF] Bug 12174 - Zotero connection with Koha: problem with UTF-8
[SIGNED-OFF] Bug 12174 follow-up of the patch (a missing print statement)
[PASSED QA] Bug 12174 - Zotero connection with Koha: problem with UTF-8
[PASSED QA] Bug 12174 follow-up of the patch (a missing print statement)

Description Zeno Tajoli 2014-05-02 09:51:23 UTC
If i donwload Koha records with Zotero from server with Koha 3.12/3.14,
I have problems with UTF-8. I don't had this problem with an old 3.2 in the past.
Comment 1 Zeno Tajoli 2014-05-02 10:11:00 UTC
I find the problem.
It is a mix of apache / new Zotero working.
1)The bug is not present in installation that use Ngix instead of apache.
2)Zotero doesn't use only COinS from Koha but if use also the record export
(../cgi-bin/koha/opac-export.pl)

3)The cgi opac-export.pl it is ok to write a file on this but not if you try to use it like a stream in Zotero.
Try:
curl -i 'http://pust.urbe.it/cgi-bin/koha/opac-export.pl?format=utf8&op=export&bib=71000&save=Go'
The result:

HTTP/1.1 200 OK
Date: Fri, 02 May 2014 09:41:22 GMT
Server: Apache/2.2.22 (Debian)
Content-Disposition: attachment; filename="bib-71000.utf8"
Transfer-Encoding: chunked
Content-Type: application/octet-stream; charset=ISO-8859-1

01403nam a2200265 i 4500001000600000003000900006005001700015 [the iso2709 output]

4)The header "charset=ISO-8859-1" is the default on Apache (Ngix doesn't inser anything instead).

5)Zotero read the header and it thinks that the stream in iso2709 format is written with ISO-8859-1 charset.


So the fix is to be esplict on charset when we print the result.

From
94:     print $query->header(
95:      -type => 'application/octet-stream',
96:      -attachment=>"bib-$biblionumber.$format");
To
94:     print $query->header(
95:      -type => 'application/octet-stream',
96:      -charset=>'utf-8',
97:      -attachment=>"bib-$biblionumber.$format");

I will write and send the patch soon
Comment 2 Zeno Tajoli 2014-05-03 09:27:29 UTC Comment hidden (obsolete)
Comment 3 Zeno Tajoli 2014-05-03 09:57:47 UTC Comment hidden (obsolete)
Comment 4 Zeno Tajoli 2014-05-03 09:58:31 UTC Comment hidden (obsolete)
Comment 5 Zeno Tajoli 2014-05-03 09:59:56 UTC
To test the bug:
-- without the patch use Zotero on a recent Koha install with Apache.
You can use PTFS sandboxes. You can't use Biblibre sandboxes, they use Ngix.
-- Test a record with special chars.
-- Install the patch
-- Catch (using Zotero) the same record.
-- Test all directly download from Opac.
-- To check MARC-8, use a tool that support it. For example MarcEdit.
Comment 6 Bernardo Gonzalez Kriegel 2014-06-22 03:06:33 UTC Comment hidden (obsolete)
Comment 7 Bernardo Gonzalez Kriegel 2014-06-22 03:06:42 UTC Comment hidden (obsolete)
Comment 8 Katrin Fischer 2014-07-30 21:52:50 UTC
Created attachment 30359 [details] [review]
[PASSED QA] Bug 12174 - Zotero connection with Koha: problem with UTF-8

This patch solves the bug for UTF-8 output. MARC-8 works only if you save it as a file. In fact the standard Internet stack (browser, web server, etc.) doesn't know anything about MARC-8 or it's ISO base (ISO 2022).

To test the bug:
-- without the patch use Zotero on a recent Koha install with Apache.
You can use PTFS sandboxes. You can't use Biblibre sandboxes, they use Ngix.
-- Test a record with special chars.
-- Install the patch
-- Catch (using Zotero) the same record.
-- Test all directly download from Opac.
-- To check MARC-8, use a tool that support it. For example MarcEdit.

http://bugs.koha-community.org/show_bug.cgi?id=12174
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>

Work as described
Tested with zotero add-on for firefox.
Before patch bad diacritics, after patch ok

Record exported as MARC-8, transformed using
yaz-iconv -f marc8 -t utf-8 bib.marc8 > bib.utf8
yaz-marcdump -i marc -o line bib.utf8, checking accents,
also comparing with direct utf-8 export

No koha-qa errors

Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Tested with Zotero in Firefox and with yaz-iconv as suggested by
Bernardo, no problems found.
Passes tests and QA script.
Comment 9 Katrin Fischer 2014-07-30 21:53:16 UTC
Created attachment 30360 [details] [review]
[PASSED QA] Bug 12174 follow-up of the patch (a missing print statement)

http://bugs.koha-community.org/show_bug.cgi?id=12174
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Comment 10 Tomás Cohen Arazi (tcohen) 2014-07-31 15:04:49 UTC
Patches pushed to master.

Thanks Zeno!