Bug 34467 - OAI GetRecord bad encoding for UNIMARC
Summary: OAI GetRecord bad encoding for UNIMARC
Status: Pushed to oldstable
Alias: None
Product: Koha
Classification: Unclassified
Component: Web services (show other bugs)
Version: Main
Hardware: All All
: P5 - low minor (vote)
Assignee: Frédéric Demians
QA Contact: Testopia
URL:
Keywords:
: 33994 34515 (view as bug list)
Depends on:
Blocks: 36335
  Show dependency treegraph
 
Reported: 2023-08-03 09:07 UTC by Frédéric Demians
Modified: 2024-03-15 14:50 UTC (History)
3 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
23.11.00,23.05.05,22.11.11


Attachments
Bug 34467: OAI GetRecord bad encoding for Unimarc (1.35 KB, patch)
2023-08-03 09:12 UTC, Frédéric Demians
Details | Diff | Splinter Review
T2211: OAI GetRecord bad encoding for Unimarc (1.40 KB, patch)
2023-10-05 19:50 UTC, David Nind
Details | Diff | Splinter Review
Bug 34467: OAI GetRecord bad encoding for Unimarc (1.46 KB, patch)
2023-10-05 21:39 UTC, Katrin Fischer
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Frédéric Demians 2023-08-03 09:07:14 UTC

    
Comment 1 Frédéric Demians 2023-08-03 09:12:40 UTC
Created attachment 154215 [details] [review]
Bug 34467: OAI GetRecord bad encoding for Unimarc

For some Unimarc biblio records, the OAI Server returns a bad XML
document, ie a document with extended characters, such as accents, not
encoded in UTF8. It seems to come from somewhere in MARC::Record library
trying to deduce biblio record endoding from some fields. It may work
for MARC21 records. It doesn't for Unimarc records. So it's necessary to
send to the method transforming record into XML the marcflavour. This
way there is no encoding guessing from Unimarc biblio records content.
Comment 2 Frédéric Demians 2023-08-03 12:41:39 UTC
The issue comes from leader position 9. In MARC21, if pos9 doesn't contain a 'a', MARC::File::XML transcode data from marc8 to utf8. In Unimarc, pos9 is not used (then often empty) and marc8 is never used.
Comment 3 David Nind 2023-10-01 19:57:53 UTC
Are you able to add a test plan?
Comment 4 Frédéric Demians 2023-10-02 06:55:40 UTC
(In reply to David Nind from comment #3)
> Are you able to add a test plan?

It seems possible, but, for me, now, the test t/db_dependent/OAI/Server.t fails on master. Is it the case for you also?
Comment 5 David Nind 2023-10-02 20:58:50 UTC
(In reply to Frédéric Demians from comment #4)

> It seems possible, but, for me, now, the test t/db_dependent/OAI/Server.t
> fails on master. Is it the case for you also?

I can confirm that the tests fail for me, both before and after.

I was more after a test plan, so I could replicate the issue, apply the patch, and make sure it fixes things.


Test results - before
~~~~~~~~~~~~~~~~~~~~~

prove t/db_dependent/OAI/Server.t
t/db_dependent/OAI/Server.t .. 13/34     # No tests run!
t/db_dependent/OAI/Server.t .. 30/34 
#   Failed test 'No tests run for subtest "Bug 19725: OAI-PMH ListRecords and ListIdentifiers should use biblio_metadata.timestamp"'
#   at t/db_dependent/OAI/Server.t line 413.
Can't call method "timestamp" on an undefined value at t/db_dependent/OAI/Server.t line 400.
# Looks like your test exited with 2 just after 30.
t/db_dependent/OAI/Server.t .. Dubious, test returned 2 (wstat 512, 0x200)
Failed 5/34 subtests 

Test Summary Report
-------------------
t/db_dependent/OAI/Server.t (Wstat: 512 Tests: 30 Failed: 1)
  Failed test:  30
  Non-zero exit status: 2
  Parse errors: Bad plan.  You planned 34 tests but ran 30.
Files=1, Tests=30,  3 wallclock secs ( 0.01 usr  0.00 sys +  1.12 cusr  0.13 csys =  1.26 CPU)
Result: FAILU)
Result: PASS


Test results - after
~~~~~~~~~~~~~~~~~~~~

There was a lot more output, and I haven't included it here (just the start and end).

prove t/db_dependent/OAI/Server.t
t/db_dependent/OAI/Server.t .. 19/34 
#   Failed test 'ListRecords marcxml'
#   at t/db_dependent/OAI/Server.t line 184.
# Compared $data->{"ListRecords"}{"record"}[0]{"metadata"}{"record"}{"leader"}
#    got : '00183     2200097   4500'
# expect : '00183    a2200097   4500'
# PARAM:---
# metadataPrefix: marcxml
# verb: ListRecords

.....


# request: http://localhost
# xmlns: http://www.openarchives.org/OAI/2.0/
# xmlns:xsi: http://www.w3.org/2001/XMLSchema-instance
# xsi:schemaLocation: http://www.openarchives.org/OAI/2.0/ http://www.openarchives.org/OAI/2.0/OAI-PMH.xsd
    # No tests run!
t/db_dependent/OAI/Server.t .. 30/34 
#   Failed test 'No tests run for subtest "Bug 19725: OAI-PMH ListRecords and ListIdentifiers should use biblio_metadata.timestamp"'
#   at t/db_dependent/OAI/Server.t line 413.
Can't call method "timestamp" on an undefined value at t/db_dependent/OAI/Server.t line 400.
# Looks like your test exited with 2 just after 30.
t/db_dependent/OAI/Server.t .. Dubious, test returned 2 (wstat 512, 0x200)
Failed 10/34 subtests 

Test Summary Report
-------------------
t/db_dependent/OAI/Server.t (Wstat: 512 Tests: 30 Failed: 6)
  Failed tests:  21-24, 29-30
  Non-zero exit status: 2
  Parse errors: Bad plan.  You planned 34 tests but ran 30.
Files=1, Tests=30,  3 wallclock secs ( 0.02 usr  0.00 sys +  1.17 cusr  0.11 csys =  1.30 CPU)
Result: FAIL
Comment 6 Frédéric Demians 2023-10-04 16:38:27 UTC
(In reply to David Nind from comment #5)
> I was more after a test plan, so I could replicate the issue, apply the
> patch, and make sure it fixes things.

On an UNIMARC Koha installation, modify let say biblio record 1234:

- on leader position 9, enter a blank
- on 200$a, add a è

Get this biblio record from OAI server with GetRecord verb. Something like that:

/cgi-bin/koha/oai.pl?verb=GetRecord&metadataPrefix=marcxml&identifier=PREFIX:1234

Replace PREFIX with syspref OAI-PMH:archiveID.

You will (should) get a 200$a without a è.

Apply the patch, and repeat.
Comment 7 David Nind 2023-10-05 19:50:32 UTC
Created attachment 156583 [details] [review]
T2211: OAI GetRecord bad encoding for Unimarc

For some Unimarc biblio records, the OAI Server returns a bad XML
document, ie a document with extended characters, such as accents, not
encoded in UTF8. It seems to come from somewhere in MARC::Record library
trying to deduce biblio record endoding from some fields. It may work
for MARC21 records. It doesn't for Unimarc records. So it's necessary to
send to the method transforming record into XML the marcflavour. This
way there is no encoding guessing from Unimarc biblio records content.

https://bugs.koha-community.org/show_bug.cgi?id=34467
Signed-off-by: David Nind <david@davidnind.com>
Comment 8 David Nind 2023-10-05 19:59:56 UTC
Thanks Frédéric!

The information on how to test helped, and I have now signed it off.

David

Testing notes (using KTD):

1. Edited .env and change KOHA_MARC_FLAVOUR=unimarc

2. Enabled the OAI-PMH system preference.

3. Modified record 1234:
   - on leader position 9, enter a blank
   - on 200$a, add a è
   - Here is what is in the leader: 01552nam  2200829   4500
   - Had to add all the required subfields
   - Edited the item for the record so that the item has the correct entries from item dropdown list options and the item type

4. Ran flush_memcached and restart_all and use a refreshed browser/

5. Used this URL: http://127.0.0.1:8080/cgi-bin/koha/oai.pl?verb=GetRecord&metadataPrefix=marcxml&identifier=KOHA-OAI-TEST:1234

6. Note that the 200$a does not have an è.

7. Applied the patch, run flush_memcached and restart_all

8. The OAI response now includes an è in the 200$a
Comment 9 Katrin Fischer 2023-10-05 21:39:18 UTC
Created attachment 156588 [details] [review]
Bug 34467: OAI GetRecord bad encoding for Unimarc

For some Unimarc biblio records, the OAI Server returns a bad XML
document, ie a document with extended characters, such as accents, not
encoded in UTF8. It seems to come from somewhere in MARC::Record library
trying to deduce biblio record endoding from some fields. It may work
for MARC21 records. It doesn't for Unimarc records. So it's necessary to
send to the method transforming record into XML the marcflavour. This
way there is no encoding guessing from Unimarc biblio records content.

https://bugs.koha-community.org/show_bug.cgi?id=34467
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 10 Tomás Cohen Arazi 2023-10-09 19:42:13 UTC
Pushed to master for 23.11.

Nice work everyone, thanks!
Comment 11 Fridolin Somers 2023-10-10 06:32:15 UTC
*** Bug 34515 has been marked as a duplicate of this bug. ***
Comment 12 Fridolin Somers 2023-10-10 06:33:42 UTC
Great this is a duplicate of my Bug 34515

Maybe for another bug report, we should use $metadata->schema like did Bug 32336 :
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=148628
Comment 13 Fridolin Somers 2023-10-10 06:34:09 UTC
And please help me test Bug 34514
Comment 14 Fridolin Somers 2023-10-10 06:40:18 UTC
Pushed to 23.05.x for 23.05.05
Comment 15 Fridolin Somers 2023-10-10 19:06:18 UTC
*** Bug 33994 has been marked as a duplicate of this bug. ***
Comment 16 Matt Blenkinsop 2023-10-17 17:35:59 UTC
Nice work everyone!

Pushed to oldstable for 22.11.x