Bug 18051

Summary: Advanced Editor - Rancor - encoding issues with some sources
Product: Koha Reporter: Nick Clemens <nick>
Component: CatalogingAssignee: Nick Clemens <nick>
Status: CLOSED FIXED QA Contact: Testopia <testopia>
Severity: major    
Priority: P5 - low CC: aleisha, barbara.johnson, bugzilla, lucas, m.de.rooy, victor
Version: master   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: Trivial patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
20.11.00, 20.05.06, 19.11.12, 19.05.17
Attachments: Garbled diacritics in Z39.50 results list
Garbled diacritics in MARC record
Bug 18051: Set UTF8 Flag for record searches/imports from rancor
Bug 18051: Set UTF8 Flag for record searches/imports from rancor
Bug 18051: Set UTF8 Flag for record searches/imports from rancor

Description Nick Clemens 2017-02-03 15:05:40 UTC
This source for example:
http://www.webclarity.info/registry/display?fileID=69

All displays fine from basic editor - rancor however causes garbled characters
Comment 1 Barbara Johnson 2020-10-26 18:17:13 UTC
Created attachment 112526 [details]
Garbled diacritics in Z39.50 results list
Comment 2 Barbara Johnson 2020-10-26 18:18:25 UTC
Created attachment 112527 [details]
Garbled diacritics in MARC record
Comment 3 Barbara Johnson 2020-10-26 18:18:41 UTC
When performing a Z39.50 search from within the Advanced Cataloging Editor we often get results that display garbled diacritics.  Sometimes changing the encoding from utf8 to MARC8 or vice versa can help, but often the diacritics don't display correctly with either encoding.  We have to spend extra time editing these records to make sure we correct everything.  It would improve accuracy and efficiency to have the diacritics display correctly.

This isn't a problem when using Z39.50 to add a new record - it would be great if it worked the same way from within the Advanced Editor.
Comment 4 Katrin Fischer 2020-10-26 19:37:27 UTC
Hi Barbara, do you have an example target where the error occurs maybe?
Comment 5 Barbara Johnson 2020-10-26 19:58:06 UTC
Yes, it happens with Milwaukee, CARL, Marmot, Ohioink, Sacramento.  In fact, it happens with most of them.  There are only a few that display the diacritics correctly, such as Seattle or King County.
Comment 6 Katrin Fischer 2020-10-26 21:26:15 UTC
Hm, failing to find the Z39.50 data for these targets - and the ones I can find I am not sure are current and/or are missing information on the encoding.
Comment 7 Barbara Johnson 2020-10-26 23:20:50 UTC
I’ve listed several below.  I realized that almost all of the ones that give bad diacritics are III catalogs, but I did find a non-III (Vancouver) that does the same thing.  Seattle is an example of the correct diacritics.

Milw​​​​aukee
Host name: countycat.mcfls.org
Port: 210
Database: INNOPAC
Syntax: MARC21/USMARC
Encoding: utf8

Ohiolink
Host name: olc1.ohiolink.edu
Port: 210
Database: INNOPAC
Syntax: MARC21/USMARC
Encoding: utf8

Sacramento
Host name: www.saclibrarycatalog.org
Port: 210
Database: INNOPAC
Syntax: MARC21/USMARC
Encoding: utf8

Vancouver
Host name: z3950.vpl.ca
Port: 210
Database: HORIZON
Syntax: MARC21/USMARC
Encoding: utf8

Seattle
Host name: zserver.spl.org
Port: 210
Database: HORIZON
Syntax: MARC21/USMARC
Encoding: MARC-8
Comment 8 Nick Clemens 2020-10-29 12:22:42 UTC
Created attachment 112663 [details] [review]
Bug 18051: Set UTF8 Flag for record searches/imports from rancor

The advanced cataloging editor uses Koha::MetaSearcher to find and import records
from Z3950.

The records are correctly converted to UTF8, however, we need to set the flag in the
record to ensure they are parsed correctly later

To test:
 1 - Add OHIOLINK as a z39 source as described in earlier comments
 2 - Browse to Cataloging->Advanced editor
 3 - Click Search->Advanced
 4 - Select OHIOLINK
 5 - Search for author 'capek karel'
 6 - Note records look bad
 7 - Apply patch
 8 - Restart all the things
 9 - Repeat
10 - Success!
Comment 9 ByWater Sandboxes 2020-10-29 13:55:04 UTC
Created attachment 112664 [details] [review]
Bug 18051: Set UTF8 Flag for record searches/imports from rancor

The advanced cataloging editor uses Koha::MetaSearcher to find and import records
from Z3950.

The records are correctly converted to UTF8, however, we need to set the flag in the
record to ensure they are parsed correctly later

To test:
 1 - Add OHIOLINK as a z39 source as described in earlier comments
 2 - Browse to Cataloging->Advanced editor
 3 - Click Search->Advanced
 4 - Select OHIOLINK
 5 - Search for author 'capek karel'
 6 - Note records look bad
 7 - Apply patch
 8 - Restart all the things
 9 - Repeat
10 - Success!

Signed-off-by: Barbara Johnson <barbara.johnson@bedfordtx.gov>
Comment 10 Barbara Johnson 2020-10-29 14:00:06 UTC
Success!!  I tested records in multiple languages on Ohiolink as well as other Z39.50 servers and diacritics displayed correctly.  I also tested Seattle which uses MARC-8 and had already been displaying the records correctly and they still do.
Comment 11 Katrin Fischer 2020-11-01 13:22:24 UTC
Thx Nick for taking this on and to Barbara for providing the extra information!
Comment 12 Katrin Fischer 2020-11-01 13:22:50 UTC
Created attachment 112804 [details] [review]
Bug 18051: Set UTF8 Flag for record searches/imports from rancor

The advanced cataloging editor uses Koha::MetaSearcher to find and import records
from Z3950.

The records are correctly converted to UTF8, however, we need to set the flag in the
record to ensure they are parsed correctly later

To test:
 1 - Add OHIOLINK as a z39 source as described in earlier comments
 2 - Browse to Cataloging->Advanced editor
 3 - Click Search->Advanced
 4 - Select OHIOLINK
 5 - Search for author 'capek karel'
 6 - Note records look bad
 7 - Apply patch
 8 - Restart all the things
 9 - Repeat
10 - Success!

Signed-off-by: Barbara Johnson <barbara.johnson@bedfordtx.gov>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 13 Jonathan Druart 2020-11-02 10:05:18 UTC
Pushed to master for 20.11, thanks to everybody involved!
Comment 14 Lucas Gass 2020-11-16 18:09:44 UTC
backported to 20.05.x for 20.05.06
Comment 15 Aleisha Amohia 2020-11-17 05:13:07 UTC
backported to 19.11.x for 19.11.12
Comment 16 Victor Grousset/tuxayo 2020-11-17 13:04:52 UTC
Backported to 19.05.x branch for 19.05.17