Bug 29333 - Importing UNIMARC authorities in MARCXML UTF-8 breaks the encoding
Summary: Importing UNIMARC authorities in MARCXML UTF-8 breaks the encoding
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: MARC Authority data support (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal (vote)
Assignee: Julian Maurice
QA Contact: Martin Renvoize
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-10-27 11:48 UTC by Julian Maurice
Modified: 2023-08-10 18:49 UTC (History)
6 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Medium patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
22.11.00, 22.05.05, 21.11.12


Attachments
Bug 29333: Fix encoding of imported UNIMARC authorities (6.32 KB, patch)
2021-10-27 12:15 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 29333: Fix encoding of imported UNIMARC authorities (19.77 KB, patch)
2022-05-20 09:31 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 29333: Fix encoding of imported UNIMARC authorities (19.82 KB, patch)
2022-05-20 19:47 UTC, David Nind
Details | Diff | Splinter Review
Bug 29333: Fix encoding of imported UNIMARC authorities (19.88 KB, patch)
2022-07-06 13:32 UTC, Martin Renvoize
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Julian Maurice 2021-10-27 11:48:08 UTC
Related to bug 17754

MARC::Record and MARC::File::* modules sometimes use the position 09 of the leader to detect encoding. A blank character means 'MARC-8' while an 'a' means 'UTF-8'.

In a UNIMARC authority this position is used to store the authority type (see https://www.transition-bibliographique.fr/wp-content/uploads/2021/02/AIntroLabel-2004.pdf [FR]).
In this case, 'a' means 'Personal Name'.

The result is that the import will succeed for a Personal Name authority, but it will fail for all other authority types.

Steps to reproduce:
0. Be sure to have a Koha UNIMARC instance. 
1. Download the MARCXML for "Honoré de Balzac"
   curl -o balzac.marcxml https://www.idref.fr/02670305X.xml
2. Verify that it's encoded in UTF-8
   file balzac.marcxml
   (should output "balzac.marcxml: XML 1.0 document, UTF-8 Unicode text")
3. Go to Tools » Stage MARC for import and import balzac.marcxml with the following settings:
   Record type: Authority
   Character encoding: UTF-8
   Format: MARCXML
   Do not touch the other settings
4. Once imported, go to the staged MARC management tool and find your batch. Click on the authority title "Balzac Honoré de 1799-1850" to show the MARC inside a modal window. There should be no encoding issue.
5. Write down the imported record id (the number in column '#') and go to the MARC authority editor. Replace all URL parameters by 'breedingid=THE_ID_YOU_WROTE_DOWN'
   The URL should look like this: /cgi-bin/koha/authorities/authorities.pl?breedingid=198
   You should see no encoding issues. Do not save the record.
6. Import the batch into the catalog. Verify that the authority record has no encoding issue.
7. Now download the MARCXML for "Athènes (Grèce)"
   curl -o athènes.marcxml https://www.idref.fr/027290530.xml
8. Repeat steps 2 to 6 using athènes.marcxml file. At steps 4 and 5 you should see encoding issues and that the position 9 of the leader was rewritten from 'c' to 'a'. Strangely, importing this batch fix the encoding issue, but we still lose the information in position 09 of the leader
Comment 1 Julian Maurice 2021-10-27 11:49:46 UTC
Related to bug 18533 too
Comment 2 Julian Maurice 2021-10-27 12:15:46 UTC
Created attachment 126983 [details] [review]
Bug 29333: Fix encoding of imported UNIMARC authorities

MARC::Record and MARC::File::* modules sometimes use the position 09 of
the leader to detect encoding. A blank character means 'MARC-8' while an
'a' means 'UTF-8'.

In a UNIMARC authority this position is used to store the authority type
(see https://www.transition-bibliographique.fr/wp-content/uploads/2021/02/AIntroLabel-2004.pdf [FR]).
In this case, 'a' means 'Personal Name'.

The result is that the import will succeed for a Personal Name
authority, but it will fail for all other authority types.

Steps to reproduce:
0. Be sure to have a Koha UNIMARC instance.
1. Download the MARCXML for "Honoré de Balzac"
   curl -o balzac.marcxml https://www.idref.fr/02670305X.xml
2. Verify that it's encoded in UTF-8
   file balzac.marcxml
   (should output "balzac.marcxml: XML 1.0 document, UTF-8 Unicode
   text")
3. Go to Tools » Stage MARC for import and import balzac.marcxml with
   the following settings:
   Record type: Authority
   Character encoding: UTF-8
   Format: MARCXML
   Do not touch the other settings
4. Once imported, go to the staged MARC management tool and find your
   batch. Click on the authority title "Balzac Honoré de 1799-1850" to
   show the MARC inside a modal window. There should be no encoding
   issue.
5. Write down the imported record id (the number in column '#') and go
   to the MARC authority editor. Replace all URL parameters by
   'breedingid=THE_ID_YOU_WROTE_DOWN'
   The URL should look like this:
   /cgi-bin/koha/authorities/authorities.pl?breedingid=198
   You should see no encoding issues. Do not save the record.
6. Import the batch into the catalog. Verify that the authority record
   has no encoding issue.
7. Now download the MARCXML for "Athènes (Grèce)"
   curl -o athènes.marcxml https://www.idref.fr/027290530.xml
8. Repeat steps 2 to 6 using athènes.marcxml file. At steps 4 and 5 you
   should see encoding issues and that the position 9 of the leader was
   rewritten from 'c' to 'a'. Strangely, importing this batch fix the
   encoding issue, but we still lose the information in position 09 of
   the leader

This patch makes use of the MARCXML representation of the record instead
of the ISO2709 representation, because, unlike
MARC::Record::new_from_usmarc, MARC::Record::new_from_xml allows us to
pass directly the encoding and the format, which prevents data to be
double encoded when position 09 of the leader is different that 'a'

Test plan:
- Follow the "steps to reproduce" above and verify that you have no
  encoding issues.
Comment 3 David Nind 2022-05-01 23:47:14 UTC
Hi Julian.

Thanks for the great test plan!

Just to check that this patch should fix both:
- the encoding issues during the staging and import stages, and 
- the c being changed to an a on import.

For me, after applying the patch and working through the test plan for athènes.marcxml, the c in the leader is still changed to an a after import - it remains as a c in steps 4 and 5. The encoding issues are fixed. (UNIMARC instance, have cleared caches (Koha and browser), restarted, etc).

David
Comment 4 Fridolin Somers 2022-05-20 04:12:14 UTC
Mmmm

diff --git a/catalogue/showmarc.pl b/catalogue/showmarc.pl
index 8292bfe746..ab6c29487f 100755
--- a/catalogue/showmarc.pl
+++ b/catalogue/showmarc.pl
@@ -61,7 +61,9 @@ if(!ref $record) {
 }
 
 if($view eq 'card' || $view eq 'html') {
-    my $xml = $importid ? $record->as_xml(): GetXmlBiblio($biblionumber);
+    my $marcflavour = C4::Context->preference('marcflavour');
+    my $format = $marcflavour eq 'UNIMARC' ? 'UNIMARCAUTH' : 'USMARC';
+    my $xml = $importid ? $record->as_xml($format): GetXmlBiblio($biblionumber);

This should be only for authorities no ?
showmarc.pl is also used to display biblio records
Comment 5 Julian Maurice 2022-05-20 09:31:35 UTC
Created attachment 135230 [details] [review]
Bug 29333: Fix encoding of imported UNIMARC authorities

MARC::Record and MARC::File::* modules sometimes use the position 09 of
the leader to detect encoding. A blank character means 'MARC-8' while an
'a' means 'UTF-8'.

In a UNIMARC authority this position is used to store the authority type
(see https://www.transition-bibliographique.fr/wp-content/uploads/2021/02/AIntroLabel-2004.pdf [FR]).
In this case, 'a' means 'Personal Name'.

The result is that the import will succeed for a Personal Name
authority, but it will fail for all other authority types.

Steps to reproduce:
0. Be sure to have a Koha UNIMARC instance.
1. Download the MARCXML for "Honoré de Balzac"
   curl -o balzac.marcxml https://www.idref.fr/02670305X.xml
2. Verify that it's encoded in UTF-8
   file balzac.marcxml
   (should output "balzac.marcxml: XML 1.0 document, UTF-8 Unicode
   text")
3. Go to Tools » Stage MARC for import and import balzac.marcxml with
   the following settings:
   Record type: Authority
   Character encoding: UTF-8
   Format: MARCXML
   Do not touch the other settings
4. Once imported, go to the staged MARC management tool and find your
   batch. Click on the authority title "Balzac Honoré de 1799-1850" to
   show the MARC inside a modal window. There should be no encoding
   issue.
5. Write down the imported record id (the number in column '#') and go
   to the MARC authority editor. Replace all URL parameters by
   'breedingid=THE_ID_YOU_WROTE_DOWN'
   The URL should look like this:
   /cgi-bin/koha/authorities/authorities.pl?breedingid=198
   You should see no encoding issues. Do not save the record.
6. Import the batch into the catalog. Verify that the authority record
   has no encoding issue.
7. Now download the MARCXML for "Athènes (Grèce)"
   curl -o athènes.marcxml https://www.idref.fr/027290530.xml
8. Repeat steps 2 to 6 using athènes.marcxml file. At steps 4 and 5 you
   should see encoding issues and that the position 9 of the leader was
   rewritten from 'c' to 'a'. Strangely, importing this batch fix the
   encoding issue, but we still lose the information in position 09 of
   the leader

This patch makes use of the MARCXML representation of the record instead
of the ISO2709 representation, because, unlike
MARC::Record::new_from_usmarc, MARC::Record::new_from_xml allows us to
pass directly the encoding and the format, which prevents data to be
double encoded when position 09 of the leader is different that 'a'

Test plan:
- Follow the "steps to reproduce" above and verify that you have no
  encoding issues.
Comment 6 Julian Maurice 2022-05-20 09:34:15 UTC
(In reply to Fridolin Somers from comment #4)
> This should be only for authorities no ?
> showmarc.pl is also used to display biblio records

Fixed in the latest patch

(In reply to David Nind from comment #3)
> Hi Julian.
> 
> Thanks for the great test plan!
> 
> Just to check that this patch should fix both:
> - the encoding issues during the staging and import stages, and 
> - the c being changed to an a on import.
> 
> For me, after applying the patch and working through the test plan for
> athènes.marcxml, the c in the leader is still changed to an a after import -
> it remains as a c in steps 4 and 5. The encoding issues are fixed. (UNIMARC
> instance, have cleared caches (Koha and browser), restarted, etc).

This patch is only about fixing encoding issues. The 'c' being changed to an 'a' can be addressed in a followup bug I think.
Comment 7 David Nind 2022-05-20 19:47:43 UTC
Created attachment 135252 [details] [review]
Bug 29333: Fix encoding of imported UNIMARC authorities

MARC::Record and MARC::File::* modules sometimes use the position 09 of
the leader to detect encoding. A blank character means 'MARC-8' while an
'a' means 'UTF-8'.

In a UNIMARC authority this position is used to store the authority type
(see https://www.transition-bibliographique.fr/wp-content/uploads/2021/02/AIntroLabel-2004.pdf [FR]).
In this case, 'a' means 'Personal Name'.

The result is that the import will succeed for a Personal Name
authority, but it will fail for all other authority types.

Steps to reproduce:
0. Be sure to have a Koha UNIMARC instance.
1. Download the MARCXML for "Honoré de Balzac"
   curl -o balzac.marcxml https://www.idref.fr/02670305X.xml
2. Verify that it's encoded in UTF-8
   file balzac.marcxml
   (should output "balzac.marcxml: XML 1.0 document, UTF-8 Unicode
   text")
3. Go to Tools » Stage MARC for import and import balzac.marcxml with
   the following settings:
   Record type: Authority
   Character encoding: UTF-8
   Format: MARCXML
   Do not touch the other settings
4. Once imported, go to the staged MARC management tool and find your
   batch. Click on the authority title "Balzac Honoré de 1799-1850" to
   show the MARC inside a modal window. There should be no encoding
   issue.
5. Write down the imported record id (the number in column '#') and go
   to the MARC authority editor. Replace all URL parameters by
   'breedingid=THE_ID_YOU_WROTE_DOWN'
   The URL should look like this:
   /cgi-bin/koha/authorities/authorities.pl?breedingid=198
   You should see no encoding issues. Do not save the record.
6. Import the batch into the catalog. Verify that the authority record
   has no encoding issue.
7. Now download the MARCXML for "Athènes (Grèce)"
   curl -o athènes.marcxml https://www.idref.fr/027290530.xml
8. Repeat steps 2 to 6 using athènes.marcxml file. At steps 4 and 5 you
   should see encoding issues and that the position 9 of the leader was
   rewritten from 'c' to 'a'. Strangely, importing this batch fix the
   encoding issue, but we still lose the information in position 09 of
   the leader

This patch makes use of the MARCXML representation of the record instead
of the ISO2709 representation, because, unlike
MARC::Record::new_from_usmarc, MARC::Record::new_from_xml allows us to
pass directly the encoding and the format, which prevents data to be
double encoded when position 09 of the leader is different that 'a'

Test plan:
- Follow the "steps to reproduce" above and verify that you have no
  encoding issues.

Signed-off-by: David Nind <david@davidnind.com>
Comment 8 Katrin Fischer 2022-06-12 11:29:27 UTC
Hi Julian, could this relate to bug 30946 with MARC21 somehow?
Comment 9 Katrin Fischer 2022-06-12 11:33:23 UTC
Hm, I've tried a reset_all_unimarc on ktd in order to get ready for testing this, but it doesn't look right:

No record passed at /kohadevbox/koha/Koha/MetadataIterator.pm line 71.
Something went wrong reading record for biblio Koha::Schema::Result::Biblio=HASH(0x560da1706fb0)->biblionumber: Can't bless non-reference value at /kohadevbox/koha/Koha/BiblioUtils.pm line 69.

And after it finishes when I search for anything:

Unsupported UNIMARC character encoding [] for XML output for UNIMARC; 100$a ->  at /usr/share/perl5/MARC/File/XML.pm line 570

I guess I need some help here, but maybe someone wants to finish QA here meanwhile.
Comment 10 David Nind 2022-06-12 21:53:14 UTC
(In reply to Katrin Fischer from comment #9)
> Hm, I've tried a reset_all_unimarc on ktd in order to get ready for testing
> this, but it doesn't look right:

I've found I need to change KOHA_MARC_FLAVOUR=unimarc in the .env file then start up KTD.

I've had no luck when starting with a MARC21 system, then using reset_all_unimarc when things are running to get a unimarc system.
Comment 11 Julian Maurice 2022-06-13 06:47:07 UTC
(In reply to Katrin Fischer from comment #8)
> Hi Julian, could this relate to bug 30946 with MARC21 somehow?

Hi Katrin, bug 30946 is about a database update that fail on a borrowernumber column. I don't see how these 2 bugs can relate. Did you mistype the bug number ?
Comment 12 Katrin Fischer 2022-06-13 07:50:55 UTC
(In reply to Julian Maurice from comment #11)
> (In reply to Katrin Fischer from comment #8)
> > Hi Julian, could this relate to bug 30946 with MARC21 somehow?
> 
> Hi Katrin, bug 30946 is about a database update that fail on a
> borrowernumber column. I don't see how these 2 bugs can relate. Did you
> mistype the bug number ?

Of course I did... :( It's 17754 that we just ran into on 20.11.
Comment 13 Julian Maurice 2022-06-13 08:02:17 UTC
(In reply to Katrin Fischer from comment #12)
> Of course I did... :( It's 17754 that we just ran into on 20.11.
Indeed they look very similar. Will you try to apply this patch to see if it fixes 17754 too ?
Comment 14 Martin Renvoize 2022-07-06 13:32:10 UTC
Created attachment 137209 [details] [review]
Bug 29333: Fix encoding of imported UNIMARC authorities

MARC::Record and MARC::File::* modules sometimes use the position 09 of
the leader to detect encoding. A blank character means 'MARC-8' while an
'a' means 'UTF-8'.

In a UNIMARC authority this position is used to store the authority type
(see https://www.transition-bibliographique.fr/wp-content/uploads/2021/02/AIntroLabel-2004.pdf [FR]).
In this case, 'a' means 'Personal Name'.

The result is that the import will succeed for a Personal Name
authority, but it will fail for all other authority types.

Steps to reproduce:
0. Be sure to have a Koha UNIMARC instance.
1. Download the MARCXML for "Honoré de Balzac"
   curl -o balzac.marcxml https://www.idref.fr/02670305X.xml
2. Verify that it's encoded in UTF-8
   file balzac.marcxml
   (should output "balzac.marcxml: XML 1.0 document, UTF-8 Unicode
   text")
3. Go to Tools » Stage MARC for import and import balzac.marcxml with
   the following settings:
   Record type: Authority
   Character encoding: UTF-8
   Format: MARCXML
   Do not touch the other settings
4. Once imported, go to the staged MARC management tool and find your
   batch. Click on the authority title "Balzac Honoré de 1799-1850" to
   show the MARC inside a modal window. There should be no encoding
   issue.
5. Write down the imported record id (the number in column '#') and go
   to the MARC authority editor. Replace all URL parameters by
   'breedingid=THE_ID_YOU_WROTE_DOWN'
   The URL should look like this:
   /cgi-bin/koha/authorities/authorities.pl?breedingid=198
   You should see no encoding issues. Do not save the record.
6. Import the batch into the catalog. Verify that the authority record
   has no encoding issue.
7. Now download the MARCXML for "Athènes (Grèce)"
   curl -o athènes.marcxml https://www.idref.fr/027290530.xml
8. Repeat steps 2 to 6 using athènes.marcxml file. At steps 4 and 5 you
   should see encoding issues and that the position 9 of the leader was
   rewritten from 'c' to 'a'. Strangely, importing this batch fix the
   encoding issue, but we still lose the information in position 09 of
   the leader

This patch makes use of the MARCXML representation of the record instead
of the ISO2709 representation, because, unlike
MARC::Record::new_from_usmarc, MARC::Record::new_from_xml allows us to
pass directly the encoding and the format, which prevents data to be
double encoded when position 09 of the leader is different that 'a'

Test plan:
- Follow the "steps to reproduce" above and verify that you have no
  encoding issues.

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 15 Martin Renvoize 2022-07-06 13:32:59 UTC
Nice bit of cleaning up along with the fix.. works well and I'm not seeing any regressions.. 

Unit tests included and passing, QA scripts happy.

Passing QA
Comment 16 Tomás Cohen Arazi 2022-07-08 18:50:14 UTC
Pushed to master for 22.11.

Nice work everyone, thanks!
Comment 17 Katrin Fischer 2022-07-15 15:52:30 UTC
(In reply to Julian Maurice from comment #13)
> (In reply to Katrin Fischer from comment #12)
> > Of course I did... :( It's 17754 that we just ran into on 20.11.
> Indeed they look very similar. Will you try to apply this patch to see if it
> fixes 17754 too ?

I finally got to test with this patch and it didn't fix it. But I found another clue: the encoding in the MARC record was not set (for MARC21 LDR. pos. 9 = a). Setting it manually and then reimporting fixed my issue.
Comment 18 Lucas Gass 2022-08-23 15:32:44 UTC
Backported to 22.05.x for 22.05.05
Comment 19 Arthur Suzuki 2022-08-30 09:05:32 UTC
thx, pushed to 21.11.x for 21.11.12
Comment 20 Victor Grousset/tuxayo 2022-09-09 19:57:17 UTC
Not backported to oldoldstable (21.05.x). Feel free to ask if it's needed.

Nothing to document, marking resolved.