Bug 39541 - Do we need to store the whole MARC record in Elasticsearch?
Summary: Do we need to store the whole MARC record in Elasticsearch?
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement
Assignee: Bugs List
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2025-04-03 09:25 UTC by Thomas Klausner
Modified: 2025-04-06 23:36 UTC (History)
2 users (show)

See Also:
GIT URL:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Thomas Klausner 2025-04-03 09:25:07 UTC
Currently the whole MARC Record is stored in the ElasticSearch Record. At the Koha Hackfest 2025 I learned that one reason for this is to be able to search the whole record (Bug 22258). Another reason might be to avoid hitting the DB when displaying search results.

But:

* We sometime encounter problems with very large records (due to limits with UNIMARC)
* The code to store the whole record is rather convoluted (involes encoding/decoding the record, checking for errors, falling back to other encoding methods..)
* It bloats the ES index (if it's not needed)

So I will investigate:

* would displaying search results work without the whole MARC record?
* aren't we hitting the DB anyway during displaying search results
  * in OPAC
  * in Staff
  * in API

If it seems easy to optionally not add the whole record to the index, I would suggest

* Adding a new Option "Don't store" to syspref ElasticsearchMARCFormat
* If it is set, skip adding the whole MARC
Comment 1 David Cook 2025-04-06 23:35:57 UTC
I find this idea interesting as well. 

Another benefit is that it would help us to separate away from MARC in the future. We can index anything into our search engine, do a consistent search, and then do display either using the search engine result or using whatever makes sense on a per record basis.
Comment 2 David Cook 2025-04-06 23:36:32 UTC
This isn't a topic that I would urgently pursue, but if someone (ie you Thomas) plan to work on it, I'd be happy to do test/QA on it.