In Koha/SearchEngine/Elasticsearch.pm, we already failover to MARCXML if the MARC record is over 99999 characters long. There are other situations where USMARC cannot be generated correctly. For instance, when 1 field has over 9999 bytes. If we try roundtripping the USMARC and we get warnings, we should failover to MARCXML. We may get false positives, but it will prevent worse problems.
Created attachment 174318 [details] [review] Bug 38416: Failover to MARCXML if cannot roundtrip USMARC during indexing This change failsover to MARCXML from USMARC if there are any warnings generated by MARC::File::USMARC::decode when trying to roundtrip the record. Test plan: 0. Apply the patch 1. Setup your koha-testing-docker to use Elasticsearch 2. Create a new record with 15,000 characters in the 500$a field 3. Index that record (e.g. perl misc/search_tools/rebuild_elasticsearch.pl --biblios -v -v) 4. Note that a warning saying the following appears: "Warnings encountered while roundtripping a MARC record to/from USMARC. Failing over to MARCXML" 5. View the "Elasticsearch record" on the detail page and note that the marc_format is MARCXML 6. Perform a search for the record (the keyword should be something that brings up other results too) 7. Note that the record appears correctly in the search results
Created attachment 174319 [details] [review] Bug 38416: Add unit tests
If we did push bug 38270, it would be tempting to failover to MARCXML_COMPRESSED actually...
Note: When rebuilding the koha-testing-docker Elasticsearch indexes, I noticed zero impact on indexing time by adding the roundtripping step.
Created attachment 174389 [details] [review] Bug 38416: Tidy
Created attachment 174459 [details] [review] Bug 38416: Failover to MARCXML if cannot roundtrip USMARC during indexing This change failsover to MARCXML from USMARC if there are any warnings generated by MARC::File::USMARC::decode when trying to roundtrip the record. Test plan: 0. Apply the patch 1. Setup your koha-testing-docker to use Elasticsearch 2. Create a new record with 15,000 characters in the 500$a field 3. Index that record (e.g. perl misc/search_tools/rebuild_elasticsearch.pl --biblios -v -v) 4. Note that a warning saying the following appears: "Warnings encountered while roundtripping a MARC record to/from USMARC. Failing over to MARCXML" 5. View the "Elasticsearch record" on the detail page and note that the marc_format is MARCXML 6. Perform a search for the record (the keyword should be something that brings up other results too) 7. Note that the record appears correctly in the search results Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Created attachment 174460 [details] [review] Bug 38416: Add unit tests Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Created attachment 174461 [details] [review] Bug 38416: Tidy Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Created attachment 174542 [details] [review] Bug 38416: Failover to MARCXML if cannot roundtrip USMARC during indexing This change failsover to MARCXML from USMARC if there are any warnings generated by MARC::File::USMARC::decode when trying to roundtrip the record. Test plan: 0. Apply the patch 1. Setup your koha-testing-docker to use Elasticsearch 2. Create a new record with 15,000 characters in the 500$a field 3. Index that record (e.g. perl misc/search_tools/rebuild_elasticsearch.pl --biblios -v -v) 4. Note that a warning saying the following appears: "Warnings encountered while roundtripping a MARC record to/from USMARC. Failing over to MARCXML" 5. View the "Elasticsearch record" on the detail page and note that the marc_format is MARCXML 6. Perform a search for the record (the keyword should be something that brings up other results too) 7. Note that the record appears correctly in the search results Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Created attachment 174543 [details] [review] Bug 38416: Add unit tests Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Created attachment 174544 [details] [review] Bug 38416: Tidy Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Pushed for 24.11! Well done everyone, thank you!