Bug 39541

Summary: Do we need to store the whole MARC record in Elasticsearch?
Product: Koha Reporter: Thomas Klausner <domm>
Component: Searching - ElasticsearchAssignee: Bugs List <koha-bugs>
Status: NEW --- QA Contact: Testopia <testopia>
Severity: enhancement    
Priority: P5 - low CC: dcook, domm
Version: Main   
Hardware: All   
OS: All   
GIT URL: Change sponsored?: ---
Patch complexity: --- Documentation contact:
Documentation submission: Text to go in the release notes:
Version(s) released in:
Circulation function:

Description Thomas Klausner 2025-04-03 09:25:07 UTC
Currently the whole MARC Record is stored in the ElasticSearch Record. At the Koha Hackfest 2025 I learned that one reason for this is to be able to search the whole record (Bug 22258). Another reason might be to avoid hitting the DB when displaying search results.

But:

* We sometime encounter problems with very large records (due to limits with UNIMARC)
* The code to store the whole record is rather convoluted (involes encoding/decoding the record, checking for errors, falling back to other encoding methods..)
* It bloats the ES index (if it's not needed)

So I will investigate:

* would displaying search results work without the whole MARC record?
* aren't we hitting the DB anyway during displaying search results
  * in OPAC
  * in Staff
  * in API

If it seems easy to optionally not add the whole record to the index, I would suggest

* Adding a new Option "Don't store" to syspref ElasticsearchMARCFormat
* If it is set, skip adding the whole MARC
Comment 1 David Cook 2025-04-06 23:35:57 UTC
I find this idea interesting as well. 

Another benefit is that it would help us to separate away from MARC in the future. We can index anything into our search engine, do a consistent search, and then do display either using the search engine result or using whatever makes sense on a per record basis.
Comment 2 David Cook 2025-04-06 23:36:32 UTC
This isn't a topic that I would urgently pursue, but if someone (ie you Thomas) plan to work on it, I'd be happy to do test/QA on it.