Bug 38908 - KTD - UNIMARC Koha start up does not work for Elasticsearch
Summary: KTD - UNIMARC Koha start up does not work for Elasticsearch
Status: RESOLVED MOVED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal
Assignee: Bugs List
QA Contact: Testopia
URL: https://gitlab.com/koha-community/koh...
Keywords:
Depends on:
Blocks:
 
Reported: 2025-01-16 15:27 UTC by Mathieu Saby
Modified: 2025-02-13 13:14 UTC (History)
4 users (show)

See Also:
GIT URL:
Initiative type: ---
Sponsorship status: ---
Comma delimited list of Sponsors:
Crowdfunding goal: 0
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Mathieu Saby 2025-01-16 15:27:05 UTC
Trying to launch a UNIMARC Koha with ktd with "--es7" and "--es8" options, I have this error. Not sure if it is a ktd bug or a Koha bug...

Can't locate object method "type" via package "Unsupported UNIMARC character encoding [] for XML output for unimarc; 100$a -> 20091001afrey50      ba0 at /usr/share/perl5/MARC/File/XML.pm line 570.


koha-1       | [1219] Indexing biblios
koha-1       | [1219] 1000 records processed
koha-1       | [1219] 2000 records processed
koha-1       | [1219] 3000 records processed
koha-1       | [1219] 4000 records processed
koha-1       | [1219] Committing final records...
es-1         | {"type": "server", "timestamp": "2025-01-16T14:54:41,756Z", "level": "INFO", "component": "o.e.c.m.MetadataMappingService", "cluster.name": "docker-cluster", "node.name": "37065e538e28", "message": "[koha_kohadev_biblios/1QeJV2h3R-SPTVkcuxG8TQ] update_mapping [_doc]", "cluster.uuid": "0PUajrC7QTuY5ta--TrqVw", "node.id": "Q2nG1GyeSqmrNhMk5n4nrQ"  }
koha-1       | [1219] Total 4849 records indexed
koha-1       | [1219] Indexing authorities
koha-1       | [1219] 1000 records processed
koha-1       | [1219] 2000 records processed
koha-1       | [1219] 3000 records processed
koha-1       | [1219] 4000 records processed
koha-1       | [1219] 5000 records processed
koha-1       | [1219] Committing 5000 records...
koha-1       | Warnings encountered while roundtripping a MARC record to/from USMARC. Failing over to MARCXML. at /kohadevbox/koha/misc/search_tools/rebuild_elasticsearch.pl line 317.
koha-1       | substr outside of string at /usr/share/perl5/MARC/File/XML.pm line 563.
koha-1       | Use of uninitialized value $enc in string eq at /usr/share/perl5/MARC/File/XML.pm line 565.
koha-1       | Use of uninitialized value $enc in string eq at /usr/share/perl5/MARC/File/XML.pm line 565.
koha-1       | Use of uninitialized value $enc in string eq at /usr/share/perl5/MARC/File/XML.pm line 567.
koha-1       | Use of uninitialized value $enc in concatenation (.) or string at /usr/share/perl5/MARC/File/XML.pm line 570.
koha-1       | Can't locate object method "type" via package "Unsupported UNIMARC character encoding [] for XML output for unimarc; 100$a -> 20091001afrey50      ba0 at /usr/share/perl5/MARC/File/XML.pm line 570.
koha-1       | " (perhaps you forgot to load "Unsupported UNIMARC character encoding [] for XML output for unimarc; 100$a -> 20091001afrey50      ba0 at /usr/share/perl5/MARC/File/XML.pm line 570.
koha-1       | "?) at /kohadevbox/koha/misc/search_tools/rebuild_elasticsearch.pl line 321.
db-1         | 2025-01-16 14:54:46 12 [Warning] Aborted connection 12 to db: 'koha_kohadev' user: 'koha_kohadev' host: '172.23.0.5' (Got an error reading communication packets)
db-1         | 2025-01-16 14:54:46 13 [Warning] Aborted connection 13 to db: 'koha_kohadev' user: 'koha_kohadev' host: '172.23.0.5' (Got an error reading communication packets)
db-1         | 2025-01-16 14:54:46 14 [Warning] Aborted connection 14 to db: 'koha_kohadev' user: 'koha_kohadev' host: '172.23.0.5' (Got an error reading communication packets)
Comment 1 David Nind 2025-01-16 20:44:31 UTC
I can confirm that I get the same result with ES7 and ES8.
Comment 2 David Nind 2025-01-16 20:50:26 UTC
I also tried checking out Koha 24.11.00 (git checkout tags/v24.11.00), instead of just main, and got the same error as well.
Comment 3 David Nind 2025-01-16 21:43:24 UTC
I also got the same error with both ES7 and ES8 with 24.05.x.
Comment 4 Mathieu Saby 2025-01-17 07:48:49 UTC
Thank you for testing. Are you sure it is a ktd issue and not a Koha issue ?
Comment 5 Katrin Fischer 2025-01-17 09:37:11 UTC
(In reply to Mathieu Saby from comment #4)
> Thank you for testing. Are you sure it is a ktd issue and not a Koha issue ?

I am wondering if this could relate to bug 39013. Could you try leaving off es and start with Zebra? Does this work?

If it's a ktd issue, we need to move the bug to the ktd issue tracker later.
Comment 6 Mathieu Saby 2025-01-17 09:48:59 UTC
It works well with Zebra

Maybe linked with https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38416 ?
Comment 7 Katrin Fischer 2025-01-17 09:51:57 UTC
(In reply to Katrin Fischer from comment #5)
> (In reply to Mathieu Saby from comment #4)
> > Thank you for testing. Are you sure it is a ktd issue and not a Koha issue ?
> 
> I am wondering if this could relate to bug 39013. Could you try leaving off
> es and start with Zebra? Does this work?
> 
> If it's a ktd issue, we need to move the bug to the ktd issue tracker later.

Sorry typo in bug number: bug 38913
Comment 8 Mathieu Saby 2025-01-17 14:16:50 UTC
I tried to apply bug 38913 with no effect

But in a Zebra Unimarc KTD I run a report and found 10 records where the 100$a field seems to have 35 bytes and not 36. Biblionumbers 1651, 336, 3116, 4001, 3950, 3685, 4141, 3012, 3700, 212

Could this be the cause of the issue?
Comment 9 Katrin Fischer 2025-01-20 08:09:54 UTC
(In reply to Mathieu Saby from comment #8)
> I tried to apply bug 38913 with no effect
> 
> But in a Zebra Unimarc KTD I run a report and found 10 records where the
> 100$a field seems to have 35 bytes and not 36. Biblionumbers 1651, 336,
> 3116, 4001, 3950, 3685, 4141, 3012, 3700, 212
> 
> Could this be the cause of the issue?

I wonder why it suddenly fails now, but having the wrong length seems a data issue.
Comment 10 Mathieu Saby 2025-01-20 11:52:47 UTC
I open this issue in gitlab : https://gitlab.com/koha-community/koha-misc4dev/-/issues/84

Does the current bug need to be closed, or maybe there is patch to add to avoid an indexing failure in the case of bad length in that field ?
Comment 11 Andreas Roussos 2025-02-13 10:00:12 UTC
Following the work done in https://gitlab.com/koha-community/koha-misc4dev/-/merge_requests/74 by Mathieu Saby to fix subfield 100$a for certain biblionumbers, we were still unable to launch a UNIMARC-based Koha instance using the latest koha-testing-docker container image (the kohadev-koha-1 container would exit during the authorities ES indexing process).

Eventually, I managed to make it work by checking out a branch on my local kohaclone at a specific commit (1c3ed6e4db) prior to running the `ktd --es7 up` command.

That particular commit was chosen because it's immediately before commit 0d862343dd (Bug 38416: Failover to MARCXML if cannot roundtrip USMARC during indexing).

What lead us to this commit/bug report was the "Warnings encountered while roundtripping a MARC record to/from USMARC" line in the container logs at the end of the authorities ES indexing process.

Obviously this is just a workaround and not a real fix, but at least we can now launch `ktd --shell` to access the Koha container and investigate further.
Comment 12 Andreas Roussos 2025-02-13 11:14:13 UTC
Apparently, the majority of UNIMARC sample authority records from koha-misc4dev have an extra space at the end of the leader value (in both their MARCXML and the MARC/ISO2709 representation).

If you remove the extra space (making the leader length equal to 24 characters), the ES indexing completes without any warnings/errors:

 $ ktd --shell
k$ cd /kohadevbox/misc4dev/data/sql/unimarc/1812/after_22155
k$ sed -i 's/   4500 <\/leader>/   4500<\/leader>/g' auth_header.sql
k$ sed -E -i 's/([0-9]{5})([cdn])([xyz])  ([a-m])22([0-9]{5})   4500 /\1\2\3  \422\5   4500/g' auth_header.sql
k$ reset_all
[...]
Running [sudo koha-shell kohadev -p -c 'PERL5LIB=/kohadevbox/koha:/kohadevbox/koha/lib:/kohadevbox/qa-test-tools perl /kohadevbox/koha/misc/search_tools/rebuild_elasticsearch.pl -v']...
[343545] Checking state of biblios index
[343545] Checking state of authorities index
[343545] Indexing biblios
[343545] 1000 records processed
[343545] 2000 records processed
[343545] 3000 records processed
[343545] 4000 records processed
[343545] Committing final records...
[343545] Total 4849 records indexed
[343545] Indexing authorities
[343545] 1000 records processed
[343545] 2000 records processed
[343545] 3000 records processed
[343545] 4000 records processed
[343545] 5000 records processed
[343545] Committing 5000 records...
[343545] Commit complete
[343545] 6000 records processed
[343545] Committing final records...
[343545] Total 6635 records indexed
[...]
Comment 13 Jonathan Druart 2025-02-13 11:22:30 UTC
Andreas, could you provide a merge request for https://gitlab.com/koha-community/koha-misc4dev ?
Comment 14 Andreas Roussos 2025-02-13 11:52:30 UTC
(In reply to Jonathan Druart from comment #13)
> Andreas, could you provide a merge request for
> https://gitlab.com/koha-community/koha-misc4dev ?
I sure can, Jonathan!

Please see:
https://gitlab.com/koha-community/koha-misc4dev/-/merge_requests/75

This was my first ever merge request on GitLab, I hope I got everything right...
Comment 15 Andreas Roussos 2025-02-13 13:14:28 UTC
UNIMARC KTD instances are now working again ;-)

Tested with `ktd --es7 up` and `ktd --es8 up` after a `ktd pull`.

Marking this Bug as 'RESOLVED'.

Thanks to all!