Related to bug 17754 MARC::Record and MARC::File::* modules sometimes use the position 09 of the leader to detect encoding. A blank character means 'MARC-8' while an 'a' means 'UTF-8'. In a UNIMARC authority this position is used to store the authority type (see https://www.transition-bibliographique.fr/wp-content/uploads/2021/02/AIntroLabel-2004.pdf [FR]). In this case, 'a' means 'Personal Name'. The result is that the import will succeed for a Personal Name authority, but it will fail for all other authority types. Steps to reproduce: 0. Be sure to have a Koha UNIMARC instance. 1. Download the MARCXML for "Honoré de Balzac" curl -o balzac.marcxml https://www.idref.fr/02670305X.xml 2. Verify that it's encoded in UTF-8 file balzac.marcxml (should output "balzac.marcxml: XML 1.0 document, UTF-8 Unicode text") 3. Go to Tools » Stage MARC for import and import balzac.marcxml with the following settings: Record type: Authority Character encoding: UTF-8 Format: MARCXML Do not touch the other settings 4. Once imported, go to the staged MARC management tool and find your batch. Click on the authority title "Balzac Honoré de 1799-1850" to show the MARC inside a modal window. There should be no encoding issue. 5. Write down the imported record id (the number in column '#') and go to the MARC authority editor. Replace all URL parameters by 'breedingid=THE_ID_YOU_WROTE_DOWN' The URL should look like this: /cgi-bin/koha/authorities/authorities.pl?breedingid=198 You should see no encoding issues. Do not save the record. 6. Import the batch into the catalog. Verify that the authority record has no encoding issue. 7. Now download the MARCXML for "Athènes (Grèce)" curl -o athènes.marcxml https://www.idref.fr/027290530.xml 8. Repeat steps 2 to 6 using athènes.marcxml file. At steps 4 and 5 you should see encoding issues and that the position 9 of the leader was rewritten from 'c' to 'a'. Strangely, importing this batch fix the encoding issue, but we still lose the information in position 09 of the leader
Related to bug 18533 too
Created attachment 126983 [details] [review] Bug 29333: Fix encoding of imported UNIMARC authorities MARC::Record and MARC::File::* modules sometimes use the position 09 of the leader to detect encoding. A blank character means 'MARC-8' while an 'a' means 'UTF-8'. In a UNIMARC authority this position is used to store the authority type (see https://www.transition-bibliographique.fr/wp-content/uploads/2021/02/AIntroLabel-2004.pdf [FR]). In this case, 'a' means 'Personal Name'. The result is that the import will succeed for a Personal Name authority, but it will fail for all other authority types. Steps to reproduce: 0. Be sure to have a Koha UNIMARC instance. 1. Download the MARCXML for "Honoré de Balzac" curl -o balzac.marcxml https://www.idref.fr/02670305X.xml 2. Verify that it's encoded in UTF-8 file balzac.marcxml (should output "balzac.marcxml: XML 1.0 document, UTF-8 Unicode text") 3. Go to Tools » Stage MARC for import and import balzac.marcxml with the following settings: Record type: Authority Character encoding: UTF-8 Format: MARCXML Do not touch the other settings 4. Once imported, go to the staged MARC management tool and find your batch. Click on the authority title "Balzac Honoré de 1799-1850" to show the MARC inside a modal window. There should be no encoding issue. 5. Write down the imported record id (the number in column '#') and go to the MARC authority editor. Replace all URL parameters by 'breedingid=THE_ID_YOU_WROTE_DOWN' The URL should look like this: /cgi-bin/koha/authorities/authorities.pl?breedingid=198 You should see no encoding issues. Do not save the record. 6. Import the batch into the catalog. Verify that the authority record has no encoding issue. 7. Now download the MARCXML for "Athènes (Grèce)" curl -o athènes.marcxml https://www.idref.fr/027290530.xml 8. Repeat steps 2 to 6 using athènes.marcxml file. At steps 4 and 5 you should see encoding issues and that the position 9 of the leader was rewritten from 'c' to 'a'. Strangely, importing this batch fix the encoding issue, but we still lose the information in position 09 of the leader This patch makes use of the MARCXML representation of the record instead of the ISO2709 representation, because, unlike MARC::Record::new_from_usmarc, MARC::Record::new_from_xml allows us to pass directly the encoding and the format, which prevents data to be double encoded when position 09 of the leader is different that 'a' Test plan: - Follow the "steps to reproduce" above and verify that you have no encoding issues.
Hi Julian. Thanks for the great test plan! Just to check that this patch should fix both: - the encoding issues during the staging and import stages, and - the c being changed to an a on import. For me, after applying the patch and working through the test plan for athènes.marcxml, the c in the leader is still changed to an a after import - it remains as a c in steps 4 and 5. The encoding issues are fixed. (UNIMARC instance, have cleared caches (Koha and browser), restarted, etc). David
Mmmm diff --git a/catalogue/showmarc.pl b/catalogue/showmarc.pl index 8292bfe746..ab6c29487f 100755 --- a/catalogue/showmarc.pl +++ b/catalogue/showmarc.pl @@ -61,7 +61,9 @@ if(!ref $record) { } if($view eq 'card' || $view eq 'html') { - my $xml = $importid ? $record->as_xml(): GetXmlBiblio($biblionumber); + my $marcflavour = C4::Context->preference('marcflavour'); + my $format = $marcflavour eq 'UNIMARC' ? 'UNIMARCAUTH' : 'USMARC'; + my $xml = $importid ? $record->as_xml($format): GetXmlBiblio($biblionumber); This should be only for authorities no ? showmarc.pl is also used to display biblio records
Created attachment 135230 [details] [review] Bug 29333: Fix encoding of imported UNIMARC authorities MARC::Record and MARC::File::* modules sometimes use the position 09 of the leader to detect encoding. A blank character means 'MARC-8' while an 'a' means 'UTF-8'. In a UNIMARC authority this position is used to store the authority type (see https://www.transition-bibliographique.fr/wp-content/uploads/2021/02/AIntroLabel-2004.pdf [FR]). In this case, 'a' means 'Personal Name'. The result is that the import will succeed for a Personal Name authority, but it will fail for all other authority types. Steps to reproduce: 0. Be sure to have a Koha UNIMARC instance. 1. Download the MARCXML for "Honoré de Balzac" curl -o balzac.marcxml https://www.idref.fr/02670305X.xml 2. Verify that it's encoded in UTF-8 file balzac.marcxml (should output "balzac.marcxml: XML 1.0 document, UTF-8 Unicode text") 3. Go to Tools » Stage MARC for import and import balzac.marcxml with the following settings: Record type: Authority Character encoding: UTF-8 Format: MARCXML Do not touch the other settings 4. Once imported, go to the staged MARC management tool and find your batch. Click on the authority title "Balzac Honoré de 1799-1850" to show the MARC inside a modal window. There should be no encoding issue. 5. Write down the imported record id (the number in column '#') and go to the MARC authority editor. Replace all URL parameters by 'breedingid=THE_ID_YOU_WROTE_DOWN' The URL should look like this: /cgi-bin/koha/authorities/authorities.pl?breedingid=198 You should see no encoding issues. Do not save the record. 6. Import the batch into the catalog. Verify that the authority record has no encoding issue. 7. Now download the MARCXML for "Athènes (Grèce)" curl -o athènes.marcxml https://www.idref.fr/027290530.xml 8. Repeat steps 2 to 6 using athènes.marcxml file. At steps 4 and 5 you should see encoding issues and that the position 9 of the leader was rewritten from 'c' to 'a'. Strangely, importing this batch fix the encoding issue, but we still lose the information in position 09 of the leader This patch makes use of the MARCXML representation of the record instead of the ISO2709 representation, because, unlike MARC::Record::new_from_usmarc, MARC::Record::new_from_xml allows us to pass directly the encoding and the format, which prevents data to be double encoded when position 09 of the leader is different that 'a' Test plan: - Follow the "steps to reproduce" above and verify that you have no encoding issues.
(In reply to Fridolin Somers from comment #4) > This should be only for authorities no ? > showmarc.pl is also used to display biblio records Fixed in the latest patch (In reply to David Nind from comment #3) > Hi Julian. > > Thanks for the great test plan! > > Just to check that this patch should fix both: > - the encoding issues during the staging and import stages, and > - the c being changed to an a on import. > > For me, after applying the patch and working through the test plan for > athènes.marcxml, the c in the leader is still changed to an a after import - > it remains as a c in steps 4 and 5. The encoding issues are fixed. (UNIMARC > instance, have cleared caches (Koha and browser), restarted, etc). This patch is only about fixing encoding issues. The 'c' being changed to an 'a' can be addressed in a followup bug I think.
Created attachment 135252 [details] [review] Bug 29333: Fix encoding of imported UNIMARC authorities MARC::Record and MARC::File::* modules sometimes use the position 09 of the leader to detect encoding. A blank character means 'MARC-8' while an 'a' means 'UTF-8'. In a UNIMARC authority this position is used to store the authority type (see https://www.transition-bibliographique.fr/wp-content/uploads/2021/02/AIntroLabel-2004.pdf [FR]). In this case, 'a' means 'Personal Name'. The result is that the import will succeed for a Personal Name authority, but it will fail for all other authority types. Steps to reproduce: 0. Be sure to have a Koha UNIMARC instance. 1. Download the MARCXML for "Honoré de Balzac" curl -o balzac.marcxml https://www.idref.fr/02670305X.xml 2. Verify that it's encoded in UTF-8 file balzac.marcxml (should output "balzac.marcxml: XML 1.0 document, UTF-8 Unicode text") 3. Go to Tools » Stage MARC for import and import balzac.marcxml with the following settings: Record type: Authority Character encoding: UTF-8 Format: MARCXML Do not touch the other settings 4. Once imported, go to the staged MARC management tool and find your batch. Click on the authority title "Balzac Honoré de 1799-1850" to show the MARC inside a modal window. There should be no encoding issue. 5. Write down the imported record id (the number in column '#') and go to the MARC authority editor. Replace all URL parameters by 'breedingid=THE_ID_YOU_WROTE_DOWN' The URL should look like this: /cgi-bin/koha/authorities/authorities.pl?breedingid=198 You should see no encoding issues. Do not save the record. 6. Import the batch into the catalog. Verify that the authority record has no encoding issue. 7. Now download the MARCXML for "Athènes (Grèce)" curl -o athènes.marcxml https://www.idref.fr/027290530.xml 8. Repeat steps 2 to 6 using athènes.marcxml file. At steps 4 and 5 you should see encoding issues and that the position 9 of the leader was rewritten from 'c' to 'a'. Strangely, importing this batch fix the encoding issue, but we still lose the information in position 09 of the leader This patch makes use of the MARCXML representation of the record instead of the ISO2709 representation, because, unlike MARC::Record::new_from_usmarc, MARC::Record::new_from_xml allows us to pass directly the encoding and the format, which prevents data to be double encoded when position 09 of the leader is different that 'a' Test plan: - Follow the "steps to reproduce" above and verify that you have no encoding issues. Signed-off-by: David Nind <david@davidnind.com>
Hi Julian, could this relate to bug 30946 with MARC21 somehow?
Hm, I've tried a reset_all_unimarc on ktd in order to get ready for testing this, but it doesn't look right: No record passed at /kohadevbox/koha/Koha/MetadataIterator.pm line 71. Something went wrong reading record for biblio Koha::Schema::Result::Biblio=HASH(0x560da1706fb0)->biblionumber: Can't bless non-reference value at /kohadevbox/koha/Koha/BiblioUtils.pm line 69. And after it finishes when I search for anything: Unsupported UNIMARC character encoding [] for XML output for UNIMARC; 100$a -> at /usr/share/perl5/MARC/File/XML.pm line 570 I guess I need some help here, but maybe someone wants to finish QA here meanwhile.
(In reply to Katrin Fischer from comment #9) > Hm, I've tried a reset_all_unimarc on ktd in order to get ready for testing > this, but it doesn't look right: I've found I need to change KOHA_MARC_FLAVOUR=unimarc in the .env file then start up KTD. I've had no luck when starting with a MARC21 system, then using reset_all_unimarc when things are running to get a unimarc system.
(In reply to Katrin Fischer from comment #8) > Hi Julian, could this relate to bug 30946 with MARC21 somehow? Hi Katrin, bug 30946 is about a database update that fail on a borrowernumber column. I don't see how these 2 bugs can relate. Did you mistype the bug number ?
(In reply to Julian Maurice from comment #11) > (In reply to Katrin Fischer from comment #8) > > Hi Julian, could this relate to bug 30946 with MARC21 somehow? > > Hi Katrin, bug 30946 is about a database update that fail on a > borrowernumber column. I don't see how these 2 bugs can relate. Did you > mistype the bug number ? Of course I did... :( It's 17754 that we just ran into on 20.11.
(In reply to Katrin Fischer from comment #12) > Of course I did... :( It's 17754 that we just ran into on 20.11. Indeed they look very similar. Will you try to apply this patch to see if it fixes 17754 too ?
Created attachment 137209 [details] [review] Bug 29333: Fix encoding of imported UNIMARC authorities MARC::Record and MARC::File::* modules sometimes use the position 09 of the leader to detect encoding. A blank character means 'MARC-8' while an 'a' means 'UTF-8'. In a UNIMARC authority this position is used to store the authority type (see https://www.transition-bibliographique.fr/wp-content/uploads/2021/02/AIntroLabel-2004.pdf [FR]). In this case, 'a' means 'Personal Name'. The result is that the import will succeed for a Personal Name authority, but it will fail for all other authority types. Steps to reproduce: 0. Be sure to have a Koha UNIMARC instance. 1. Download the MARCXML for "Honoré de Balzac" curl -o balzac.marcxml https://www.idref.fr/02670305X.xml 2. Verify that it's encoded in UTF-8 file balzac.marcxml (should output "balzac.marcxml: XML 1.0 document, UTF-8 Unicode text") 3. Go to Tools » Stage MARC for import and import balzac.marcxml with the following settings: Record type: Authority Character encoding: UTF-8 Format: MARCXML Do not touch the other settings 4. Once imported, go to the staged MARC management tool and find your batch. Click on the authority title "Balzac Honoré de 1799-1850" to show the MARC inside a modal window. There should be no encoding issue. 5. Write down the imported record id (the number in column '#') and go to the MARC authority editor. Replace all URL parameters by 'breedingid=THE_ID_YOU_WROTE_DOWN' The URL should look like this: /cgi-bin/koha/authorities/authorities.pl?breedingid=198 You should see no encoding issues. Do not save the record. 6. Import the batch into the catalog. Verify that the authority record has no encoding issue. 7. Now download the MARCXML for "Athènes (Grèce)" curl -o athènes.marcxml https://www.idref.fr/027290530.xml 8. Repeat steps 2 to 6 using athènes.marcxml file. At steps 4 and 5 you should see encoding issues and that the position 9 of the leader was rewritten from 'c' to 'a'. Strangely, importing this batch fix the encoding issue, but we still lose the information in position 09 of the leader This patch makes use of the MARCXML representation of the record instead of the ISO2709 representation, because, unlike MARC::Record::new_from_usmarc, MARC::Record::new_from_xml allows us to pass directly the encoding and the format, which prevents data to be double encoded when position 09 of the leader is different that 'a' Test plan: - Follow the "steps to reproduce" above and verify that you have no encoding issues. Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Nice bit of cleaning up along with the fix.. works well and I'm not seeing any regressions.. Unit tests included and passing, QA scripts happy. Passing QA
Pushed to master for 22.11. Nice work everyone, thanks!
(In reply to Julian Maurice from comment #13) > (In reply to Katrin Fischer from comment #12) > > Of course I did... :( It's 17754 that we just ran into on 20.11. > Indeed they look very similar. Will you try to apply this patch to see if it > fixes 17754 too ? I finally got to test with this patch and it didn't fix it. But I found another clue: the encoding in the MARC record was not set (for MARC21 LDR. pos. 9 = a). Setting it manually and then reimporting fixed my issue.
Backported to 22.05.x for 22.05.05
thx, pushed to 21.11.x for 21.11.12
Not backported to oldoldstable (21.05.x). Feel free to ask if it's needed. Nothing to document, marking resolved.