Bug 11997 - Download / export lists of records coding and format errors
Summary: Download / export lists of records coding and format errors
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: Lists (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: Galen Charlton
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-03-25 17:18 UTC by Pablo AB
Modified: 2023-07-09 10:49 UTC (History)
2 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Exported list to buggy ISO2709 (~MARC file) (8.70 KB, text/plain)
2015-09-14 15:00 UTC, Pablo AB
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Pablo AB 2014-03-25 17:18:42 UTC
Either from OPAC and Intranet, while exporting lists with formats "MARC" (OPAC) / "Iso2709" (on Intranet; different name to same thing) the downloaded file named "shelf.iso2709" is on latin1 instead of utf8, never reporting this, and also it seems very buggy:

$ yaz-marcdump -f latin1 -t utf8 -i marc -o line  "shelf.iso2709" | head -n30
 [...]
245 00 $a Introducción general a la informática :  $b periféricos y redes 
locales /  $c Mario Carlos Ginzburg; ilustrado por Rafael Ginzburg
(Separator but not at end of field length=134)
250 a4 $  ed.

(Separator but not at end of field length=11)

260 aB $e nos Aires :  $b Mario Carlos Ginzburg,  $c 2006

(Separator but not at end of field length=51)
Comment 1 Pablo AB 2014-03-25 18:29:21 UTC
Selecting 8859-1 (latin1) while staging records for import doesn't fix the issue. A workaround was to:

yaz-iconv -f latin1 -t utf8 shelf.iso2709 > shelf.iso2709-utf

This "fix" the file and then yaz-marcdump shelf.iso2709-utf gives the expected output.
Comment 2 Jonathan Druart 2015-04-28 09:07:47 UTC
Pablo, could you please test it again against the master branch?
Some recent fixes related to the export and encoding have been pushed.
Comment 3 Pablo AB 2015-04-28 12:58:18 UTC
Still there, Koha 3.18.06.000:
 $ yaz-marcdump shelf.iso2709
  [ugly thing with � and «(Separator but not at end of field...»]
 $ yaz-iconv -f latin1 -t utf8 shelf.iso2709 > shelf.iso2709-utf
 $ yaz-marcdump shelf.iso2709-utf 
  [Quod erat demonstrandum :P Expected result]
Comment 4 Jonathan Druart 2015-04-29 08:02:27 UTC
Pablo,

Bug 11944 has been pushed to master and will be in Koha 3.20.

I can confirm you this bug is fixed on the master branch (tested right now).

I don't plan to provide patches for encoding related issues on the branches without patches from bug 11944.
Comment 5 Pablo AB 2015-09-04 13:17:13 UTC
Hi Jonathan!
The bug is still there in master. Now I'm using 3.20.03.000 and the I can see the record looks good on Koha, but exporting the problem is still there. You can look at it with the procedure of comment 3.
Comment 6 Jonathan Druart 2015-09-04 13:36:03 UTC
I don't understand how you export your records.
I don't manage to get a file with a .iso2709 extension.
Comment 7 Pablo AB 2015-09-04 13:47:52 UTC
- Search for books on intranet.
- Clic on Add to and save them on a list.
- Go to Home › Lists › Your lists 
- Clic on the list you just create
- Clic on Download list and select "iso2709"
- Save the "shelf.iso2709" file.
- On terminal, follow steps on comment 3

NOTE: An ISO2709 is *almost* the same as a "MARC file". A mrc is an iso2709 that also have some other properties, like always subfields on tags>010. To avoid confusions maybe here template should say "MARC file" and filename should be: «list-<valid_name_of_list_name_for_filename>.mrc».
Comment 8 Pablo AB 2015-09-04 13:59:34 UTC
Just created bug 14786 about this terminological issues.
Comment 9 Jonathan Druart 2015-09-04 14:28:13 UTC
Looks good here:

200 1  $a Une Fêteກ ຂ ຄ ງ ຈ ຊ ຍ monstre <test2> test3 $f Stéphanie Laslett $g ill. Nigel MacMullen $b LIVR $é éééééééééééééééééééééééééé $ñ ññññññññññññññññññññññññññ
Comment 10 Jonathan Druart 2015-09-04 14:29:36 UTC
% yaz-iconv -f latin1 -t utf8 shelf.iso2709 > shelf.iso2709-utf
% yaz-marcdump shelf.iso2709-utf | grep 200
200 1  $a Une Fêteຠຠຠຠຠຠຠmonstre <test2> test3 $f Stéphanie Laslett $g ill. Nigel MacMullen $b LIVR $à ©Ã©Ã©Ã©Ã©Ã©Ã©Ã©Ã©Ã©Ã©Ã©Ã©Ã©Ã©Ã©Ã©Ã©Ã©Ã©Ã©
Comment 11 Jonathan Druart 2015-09-04 14:36:49 UTC
tested on master and 3.20.02
Comment 12 Pablo AB 2015-09-04 18:57:21 UTC
Well, not in my case. I tried on different Kohas, all using MARC 21. Maybe (I don't think so) have something to do with the fact that my records have newlines in it. Bug 11388.
More tests: First on MySQL directly:
 <datafield tag="245" ind1="0" ind2="0">
  <subfield code="a">Química : </subfield>

$ yaz-marcdump bib-2087.utf8 bib-13459.utf8 | grep "^245"
245 00 $a Química :  $b una ciencia experimental /  [...]
245 00 $a Estudio experimental de dispersión de [...]

$ yaz-marcdump shelf.iso2709 | grep "^245"
245 00 $a Qu�mica :  $b una ciencia experimental /  [...]

[Notice here second record is missing. If my list have only one record above command throw no output.]

$ yaz-iconv -f latin1 -t utf8 shelf.iso2709 > shelf.iso2709-utf

$ yaz-marcdump shelf.iso2709-utf | grep "^245"
245 00 $a Química :  $b una ciencia experimental /  [...]
245 00 $a Estudio experimental de dispersión de [...]
Comment 13 Jonathan Druart 2015-09-14 09:56:09 UTC
Could you attach the file here?
Comment 14 Pablo AB 2015-09-14 15:00:27 UTC
Created attachment 42528 [details]
Exported list to buggy ISO2709 (~MARC file)
Comment 15 Jonathan Druart 2015-09-14 15:27:11 UTC
I got several
  (Separator but not at end of field length=96)
Not sure it's normal.


% yaz-marcdump shelf.iso | grep "^245"
245 00 $a Física /  $c Paul A. Tipler $n [volumen 2]

Where does the record come from? Manually cataloged into Koha? z3950 import?

How does look the record using the MARC preview (detail page, link MARC Preview: Show)?
Comment 16 Katrin Fischer 2023-07-09 10:49:40 UTC
This sounds a bit like bug 17842 - I realize we only fixed that for the cart there and no mention of lists. If the bug is still present on lists, it might provide some insight.