Bug 30836 - Elasticsearch mappings.yaml is out of sync with upgraded databases
Summary: Elasticsearch mappings.yaml is out of sync with upgraded databases
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: Architecture, internals, and plumbing (show other bugs)
Version: unspecified
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Bugs List
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-05-24 08:46 UTC by Martin Renvoize
Modified: 2022-08-09 06:15 UTC (History)
8 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Martin Renvoize 2022-05-24 08:46:31 UTC
We discovered on IRC today that the mappings.yaml default mappings for ES indexing has fallen out of sync with upgraded databases..

We need to create a DB update for each mappings change and we haven't been doing so.
Comment 1 Martin Renvoize 2022-05-24 08:47:30 UTC
This also needs a guideline for QA people to follow so we don't fall out of sync again in the future.
Comment 2 Martin Renvoize 2022-05-24 08:48:22 UTC
http://irc.koha-community.org/koha/2022-05-24#i_2426285 for irc reference
Comment 3 Nick Clemens 2022-05-24 12:30:11 UTC
Out of the box it does seem to work fine adding a mapping to the db directly, however, it relies on dynamic mapping being enabled:
https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-mapping.html

Otherwise, we will need to update the indexes in ES as well
Comment 4 Tomás Cohen Arazi 2022-07-13 21:42:04 UTC
We ceratinly need to define a good way to patch things in production for the regular user, leaving room so we don't break more advanced users setups.

If this is not doable, then we need to document better how to reset the indexes, or provide a tool to let them know they are out of sync.

This is a blocker for several enhancements.
Comment 5 Ere Maijala 2022-08-02 07:00:54 UTC
I'm not sure if I can come up with a good solution. Automatically updating current mappings can create problems when the defaults have been customized.

What we might be able to do is to check, field by field, if the mappings have been customized, and apply any changes only on top of old defaults. I'm not sure if that's feasible in real life, but worth a thought. At least it should be possible to check that any mappings required by Koha are available regardless of their configuration, and add any that are missing completely. Perhaps a warning about updated mappings during upgrade would be useful as well.

I think that in many cases requiring resetting to defaults would be too destructive. Having to redo all customizations could be pretty a lot of work unless you manage them in a customized mappings.yaml or something. It would be useful to have a tool that would show the differences between current and default mappings. This would make it easier to determine any manual changes needed or useful.

All in all, I'm not sure I like having the database part of the configuration. Sure, the UI makes it easier to start changing the mappings, but if you need to do more, it's cumbersome. Also changing the settings can have consequences that you need to be aware of, and sometimes I feel it should not be that easy to change them.

Regardless of how the mappings update is managed, any changes will need to trigger a prominent notification that reindexing is required.

Sorry for this being a collection of random thoughts...
Comment 6 David Cook 2022-08-02 07:29:40 UTC
(In reply to Ere Maijala from comment #5)
> I'm not sure if I can come up with a good solution. Automatically updating
> current mappings can create problems when the defaults have been customized.

This reminds me that we have this problem in other parts of Koha as well, and we've never found a good solution. 

For example, MARC Bibliographic Frameworks. Also, "Koha to MARC mapping". 

(In reply to Ere Maijala from comment #5)
> What we might be able to do is to check, field by field, if the mappings
> have been customized, and apply any changes only on top of old defaults. I'm
> not sure if that's feasible in real life, but worth a thought. At least it
> should be possible to check that any mappings required by Koha are available
> regardless of their configuration, and add any that are missing completely.
> Perhaps a warning about updated mappings during upgrade would be useful as
> well.

Actually... I think that this is probably the way to go. 

I'm not familiar with the Elasticsearch mappings setup, but in theory couldn't you compare default values against user-generated values? And then update everything but those? 

And then maybe have some way of telling users which values are default and which values aren't in a visual way?  I imagine that might be easier for Elasticsearch mappings than it would for say "MARC Bibliographic Frameworks".
Comment 7 Stefan Berndtsson 2022-08-02 07:37:25 UTC
Isn't this what Bug 19707 is supposed to handle? I know it has failed QA for now though, so some fixing would be necessary, but if I remember correctly it adds sync/reset options for mappings.
Comment 8 Ere Maijala 2022-08-04 08:28:25 UTC
Looks like Bug 19707 would indeed provide a lot of the functionality. What remains would be automatic updating of non-customized fields and prominent message during upgrade that prompts to merge any changes manually.