Bug 34467

Created attachment 154215 [details] [review]
Bug 34467: OAI GetRecord bad encoding for Unimarc

For some Unimarc biblio records, the OAI Server returns a bad XML
document, ie a document with extended characters, such as accents, not
encoded in UTF8. It seems to come from somewhere in MARC::Record library
trying to deduce biblio record endoding from some fields. It may work
for MARC21 records. It doesn't for Unimarc records. So it's necessary to
send to the method transforming record into XML the marcflavour. This
way there is no encoding guessing from Unimarc biblio records content.

The issue comes from leader position 9. In MARC21, if pos9 doesn't contain a 'a', MARC::File::XML transcode data from marc8 to utf8. In Unimarc, pos9 is not used (then often empty) and marc8 is never used.

Are you able to add a test plan?

(In reply to David Nind from comment #3)
> Are you able to add a test plan?

It seems possible, but, for me, now, the test t/db_dependent/OAI/Server.t fails on master. Is it the case for you also?

(In reply to Frédéric Demians from comment #4)

> It seems possible, but, for me, now, the test t/db_dependent/OAI/Server.t
> fails on master. Is it the case for you also?

I can confirm that the tests fail for me, both before and after.

I was more after a test plan, so I could replicate the issue, apply the patch, and make sure it fixes things.


Test results - before
~~~~~~~~~~~~~~~~~~~~~

prove t/db_dependent/OAI/Server.t
t/db_dependent/OAI/Server.t .. 13/34     # No tests run!
t/db_dependent/OAI/Server.t .. 30/34 
#   Failed test 'No tests run for subtest "Bug 19725: OAI-PMH ListRecords and ListIdentifiers should use biblio_metadata.timestamp"'
#   at t/db_dependent/OAI/Server.t line 413.
Can't call method "timestamp" on an undefined value at t/db_dependent/OAI/Server.t line 400.
# Looks like your test exited with 2 just after 30.
t/db_dependent/OAI/Server.t .. Dubious, test returned 2 (wstat 512, 0x200)
Failed 5/34 subtests 

Test Summary Report
-------------------
t/db_dependent/OAI/Server.t (Wstat: 512 Tests: 30 Failed: 1)
  Failed test:  30
  Non-zero exit status: 2
  Parse errors: Bad plan.  You planned 34 tests but ran 30.
Files=1, Tests=30,  3 wallclock secs ( 0.01 usr  0.00 sys +  1.12 cusr  0.13 csys =  1.26 CPU)
Result: FAILU)
Result: PASS


Test results - after
~~~~~~~~~~~~~~~~~~~~

There was a lot more output, and I haven't included it here (just the start and end).

prove t/db_dependent/OAI/Server.t
t/db_dependent/OAI/Server.t .. 19/34 
#   Failed test 'ListRecords marcxml'
#   at t/db_dependent/OAI/Server.t line 184.
# Compared $data->{"ListRecords"}{"record"}[0]{"metadata"}{"record"}{"leader"}
#    got : '00183     2200097   4500'
# expect : '00183    a2200097   4500'
# PARAM:---
# metadataPrefix: marcxml
# verb: ListRecords

.....


# request: http://localhost
# xmlns: http://www.openarchives.org/OAI/2.0/
# xmlns:xsi: http://www.w3.org/2001/XMLSchema-instance
# xsi:schemaLocation: http://www.openarchives.org/OAI/2.0/ http://www.openarchives.org/OAI/2.0/OAI-PMH.xsd
    # No tests run!
t/db_dependent/OAI/Server.t .. 30/34 
#   Failed test 'No tests run for subtest "Bug 19725: OAI-PMH ListRecords and ListIdentifiers should use biblio_metadata.timestamp"'
#   at t/db_dependent/OAI/Server.t line 413.
Can't call method "timestamp" on an undefined value at t/db_dependent/OAI/Server.t line 400.
# Looks like your test exited with 2 just after 30.
t/db_dependent/OAI/Server.t .. Dubious, test returned 2 (wstat 512, 0x200)
Failed 10/34 subtests 

Test Summary Report
-------------------
t/db_dependent/OAI/Server.t (Wstat: 512 Tests: 30 Failed: 6)
  Failed tests:  21-24, 29-30
  Non-zero exit status: 2
  Parse errors: Bad plan.  You planned 34 tests but ran 30.
Files=1, Tests=30,  3 wallclock secs ( 0.02 usr  0.00 sys +  1.17 cusr  0.11 csys =  1.30 CPU)
Result: FAIL

(In reply to David Nind from comment #5)
> I was more after a test plan, so I could replicate the issue, apply the
> patch, and make sure it fixes things.

On an UNIMARC Koha installation, modify let say biblio record 1234:

- on leader position 9, enter a blank
- on 200$a, add a è

Get this biblio record from OAI server with GetRecord verb. Something like that:

/cgi-bin/koha/oai.pl?verb=GetRecord&metadataPrefix=marcxml&identifier=PREFIX:1234

Replace PREFIX with syspref OAI-PMH:archiveID.

You will (should) get a 200$a without a è.

Apply the patch, and repeat.

Created attachment 156583 [details] [review]
T2211: OAI GetRecord bad encoding for Unimarc

For some Unimarc biblio records, the OAI Server returns a bad XML
document, ie a document with extended characters, such as accents, not
encoded in UTF8. It seems to come from somewhere in MARC::Record library
trying to deduce biblio record endoding from some fields. It may work
for MARC21 records. It doesn't for Unimarc records. So it's necessary to
send to the method transforming record into XML the marcflavour. This
way there is no encoding guessing from Unimarc biblio records content.

https://bugs.koha-community.org/show_bug.cgi?id=34467
Signed-off-by: David Nind <david@davidnind.com>

Thanks Frédéric!

The information on how to test helped, and I have now signed it off.

David

Testing notes (using KTD):

1. Edited .env and change KOHA_MARC_FLAVOUR=unimarc

2. Enabled the OAI-PMH system preference.

3. Modified record 1234:
   - on leader position 9, enter a blank
   - on 200$a, add a è
   - Here is what is in the leader: 01552nam  2200829   4500
   - Had to add all the required subfields
   - Edited the item for the record so that the item has the correct entries from item dropdown list options and the item type

4. Ran flush_memcached and restart_all and use a refreshed browser/

5. Used this URL: http://127.0.0.1:8080/cgi-bin/koha/oai.pl?verb=GetRecord&metadataPrefix=marcxml&identifier=KOHA-OAI-TEST:1234

6. Note that the 200$a does not have an è.

7. Applied the patch, run flush_memcached and restart_all

8. The OAI response now includes an è in the 200$a

Created attachment 156588 [details] [review]
Bug 34467: OAI GetRecord bad encoding for Unimarc

For some Unimarc biblio records, the OAI Server returns a bad XML
document, ie a document with extended characters, such as accents, not
encoded in UTF8. It seems to come from somewhere in MARC::Record library
trying to deduce biblio record endoding from some fields. It may work
for MARC21 records. It doesn't for Unimarc records. So it's necessary to
send to the method transforming record into XML the marcflavour. This
way there is no encoding guessing from Unimarc biblio records content.

https://bugs.koha-community.org/show_bug.cgi?id=34467
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Pushed to master for 23.11.

Nice work everyone, thanks!

*** Bug 34515 has been marked as a duplicate of this bug. ***

Great this is a duplicate of my Bug 34515

Maybe for another bug report, we should use $metadata->schema like did Bug 32336 :
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=148628

And please help me test Bug 34514

Pushed to 23.05.x for 23.05.05

*** Bug 33994 has been marked as a duplicate of this bug. ***

Nice work everyone!

Pushed to oldstable for 22.11.x

This fixes a bug. No changes required to the manual.