Trying to launch a UNIMARC Koha with ktd with "--es7" and "--es8" options, I have this error. Not sure if it is a ktd bug or a Koha bug... Can't locate object method "type" via package "Unsupported UNIMARC character encoding [] for XML output for unimarc; 100$a -> 20091001afrey50 ba0 at /usr/share/perl5/MARC/File/XML.pm line 570. koha-1 | [1219] Indexing biblios koha-1 | [1219] 1000 records processed koha-1 | [1219] 2000 records processed koha-1 | [1219] 3000 records processed koha-1 | [1219] 4000 records processed koha-1 | [1219] Committing final records... es-1 | {"type": "server", "timestamp": "2025-01-16T14:54:41,756Z", "level": "INFO", "component": "o.e.c.m.MetadataMappingService", "cluster.name": "docker-cluster", "node.name": "37065e538e28", "message": "[koha_kohadev_biblios/1QeJV2h3R-SPTVkcuxG8TQ] update_mapping [_doc]", "cluster.uuid": "0PUajrC7QTuY5ta--TrqVw", "node.id": "Q2nG1GyeSqmrNhMk5n4nrQ" } koha-1 | [1219] Total 4849 records indexed koha-1 | [1219] Indexing authorities koha-1 | [1219] 1000 records processed koha-1 | [1219] 2000 records processed koha-1 | [1219] 3000 records processed koha-1 | [1219] 4000 records processed koha-1 | [1219] 5000 records processed koha-1 | [1219] Committing 5000 records... koha-1 | Warnings encountered while roundtripping a MARC record to/from USMARC. Failing over to MARCXML. at /kohadevbox/koha/misc/search_tools/rebuild_elasticsearch.pl line 317. koha-1 | substr outside of string at /usr/share/perl5/MARC/File/XML.pm line 563. koha-1 | Use of uninitialized value $enc in string eq at /usr/share/perl5/MARC/File/XML.pm line 565. koha-1 | Use of uninitialized value $enc in string eq at /usr/share/perl5/MARC/File/XML.pm line 565. koha-1 | Use of uninitialized value $enc in string eq at /usr/share/perl5/MARC/File/XML.pm line 567. koha-1 | Use of uninitialized value $enc in concatenation (.) or string at /usr/share/perl5/MARC/File/XML.pm line 570. koha-1 | Can't locate object method "type" via package "Unsupported UNIMARC character encoding [] for XML output for unimarc; 100$a -> 20091001afrey50 ba0 at /usr/share/perl5/MARC/File/XML.pm line 570. koha-1 | " (perhaps you forgot to load "Unsupported UNIMARC character encoding [] for XML output for unimarc; 100$a -> 20091001afrey50 ba0 at /usr/share/perl5/MARC/File/XML.pm line 570. koha-1 | "?) at /kohadevbox/koha/misc/search_tools/rebuild_elasticsearch.pl line 321. db-1 | 2025-01-16 14:54:46 12 [Warning] Aborted connection 12 to db: 'koha_kohadev' user: 'koha_kohadev' host: '172.23.0.5' (Got an error reading communication packets) db-1 | 2025-01-16 14:54:46 13 [Warning] Aborted connection 13 to db: 'koha_kohadev' user: 'koha_kohadev' host: '172.23.0.5' (Got an error reading communication packets) db-1 | 2025-01-16 14:54:46 14 [Warning] Aborted connection 14 to db: 'koha_kohadev' user: 'koha_kohadev' host: '172.23.0.5' (Got an error reading communication packets)
I can confirm that I get the same result with ES7 and ES8.
I also tried checking out Koha 24.11.00 (git checkout tags/v24.11.00), instead of just main, and got the same error as well.
I also got the same error with both ES7 and ES8 with 24.05.x.
Thank you for testing. Are you sure it is a ktd issue and not a Koha issue ?
(In reply to Mathieu Saby from comment #4) > Thank you for testing. Are you sure it is a ktd issue and not a Koha issue ? I am wondering if this could relate to bug 39013. Could you try leaving off es and start with Zebra? Does this work? If it's a ktd issue, we need to move the bug to the ktd issue tracker later.
It works well with Zebra Maybe linked with https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38416 ?
(In reply to Katrin Fischer from comment #5) > (In reply to Mathieu Saby from comment #4) > > Thank you for testing. Are you sure it is a ktd issue and not a Koha issue ? > > I am wondering if this could relate to bug 39013. Could you try leaving off > es and start with Zebra? Does this work? > > If it's a ktd issue, we need to move the bug to the ktd issue tracker later. Sorry typo in bug number: bug 38913
I tried to apply bug 38913 with no effect But in a Zebra Unimarc KTD I run a report and found 10 records where the 100$a field seems to have 35 bytes and not 36. Biblionumbers 1651, 336, 3116, 4001, 3950, 3685, 4141, 3012, 3700, 212 Could this be the cause of the issue?
(In reply to Mathieu Saby from comment #8) > I tried to apply bug 38913 with no effect > > But in a Zebra Unimarc KTD I run a report and found 10 records where the > 100$a field seems to have 35 bytes and not 36. Biblionumbers 1651, 336, > 3116, 4001, 3950, 3685, 4141, 3012, 3700, 212 > > Could this be the cause of the issue? I wonder why it suddenly fails now, but having the wrong length seems a data issue.
I open this issue in gitlab : https://gitlab.com/koha-community/koha-misc4dev/-/issues/84 Does the current bug need to be closed, or maybe there is patch to add to avoid an indexing failure in the case of bad length in that field ?
Following the work done in https://gitlab.com/koha-community/koha-misc4dev/-/merge_requests/74 by Mathieu Saby to fix subfield 100$a for certain biblionumbers, we were still unable to launch a UNIMARC-based Koha instance using the latest koha-testing-docker container image (the kohadev-koha-1 container would exit during the authorities ES indexing process). Eventually, I managed to make it work by checking out a branch on my local kohaclone at a specific commit (1c3ed6e4db) prior to running the `ktd --es7 up` command. That particular commit was chosen because it's immediately before commit 0d862343dd (Bug 38416: Failover to MARCXML if cannot roundtrip USMARC during indexing). What lead us to this commit/bug report was the "Warnings encountered while roundtripping a MARC record to/from USMARC" line in the container logs at the end of the authorities ES indexing process. Obviously this is just a workaround and not a real fix, but at least we can now launch `ktd --shell` to access the Koha container and investigate further.
Apparently, the majority of UNIMARC sample authority records from koha-misc4dev have an extra space at the end of the leader value (in both their MARCXML and the MARC/ISO2709 representation). If you remove the extra space (making the leader length equal to 24 characters), the ES indexing completes without any warnings/errors: $ ktd --shell k$ cd /kohadevbox/misc4dev/data/sql/unimarc/1812/after_22155 k$ sed -i 's/ 4500 <\/leader>/ 4500<\/leader>/g' auth_header.sql k$ sed -E -i 's/([0-9]{5})([cdn])([xyz]) ([a-m])22([0-9]{5}) 4500 /\1\2\3 \422\5 4500/g' auth_header.sql k$ reset_all [...] Running [sudo koha-shell kohadev -p -c 'PERL5LIB=/kohadevbox/koha:/kohadevbox/koha/lib:/kohadevbox/qa-test-tools perl /kohadevbox/koha/misc/search_tools/rebuild_elasticsearch.pl -v']... [343545] Checking state of biblios index [343545] Checking state of authorities index [343545] Indexing biblios [343545] 1000 records processed [343545] 2000 records processed [343545] 3000 records processed [343545] 4000 records processed [343545] Committing final records... [343545] Total 4849 records indexed [343545] Indexing authorities [343545] 1000 records processed [343545] 2000 records processed [343545] 3000 records processed [343545] 4000 records processed [343545] 5000 records processed [343545] Committing 5000 records... [343545] Commit complete [343545] 6000 records processed [343545] Committing final records... [343545] Total 6635 records indexed [...]
Andreas, could you provide a merge request for https://gitlab.com/koha-community/koha-misc4dev ?
(In reply to Jonathan Druart from comment #13) > Andreas, could you provide a merge request for > https://gitlab.com/koha-community/koha-misc4dev ? I sure can, Jonathan! Please see: https://gitlab.com/koha-community/koha-misc4dev/-/merge_requests/75 This was my first ever merge request on GitLab, I hope I got everything right...
UNIMARC KTD instances are now working again ;-) Tested with `ktd --es7 up` and `ktd --es8 up` after a `ktd pull`. Marking this Bug as 'RESOLVED'. Thanks to all!