Bug 41668

Summary: Inconsistency between mappings.yaml and Elasticsearch web UI causes data loss
Product: Koha Reporter: David Cook <dcook>
Component: Searching - ElasticsearchAssignee: Bugs List <koha-bugs>
Status: In Discussion --- QA Contact: Testopia <testopia>
Severity: normal    
Priority: P5 - low CC: domm, e.betemps, tomascohen
Version: Main   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35138
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18235
GIT URL: Initiative type: ---
Sponsorship status: --- Comma delimited list of Sponsors:
Crowdfunding goal: 0 Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:
Bug Depends on: 35138    
Bug Blocks:    

Description David Cook 2026-01-20 23:47:54 UTC
When resetting the Elasticsearch settings using mappings.yaml, it appears that the "Facetable" setting isn't being updated correctly.

For instance, the default mappings.yaml says "publisher" is facetable, but after resetting the Elasticsearch from file, it says it's not facetable.
Comment 1 David Cook 2026-01-21 00:00:39 UTC
Ok in main... I've reset and it is setting in the database but the UI still says "No"  for Facetable for publisher, local-classification, copydate.

I re-indexed with the default conf, and I see "publisher__facet" and "copydate__facet" keys in the ES document despite Facetable saying "no" for publisher  and copydate, and those facets not being defined in "Facet Order".
Comment 2 David Cook 2026-01-21 00:05:08 UTC
Ahhh if I save the mappings via the UI it sets "facet" to 0 for publisher, local-classification, and copydate.

That would explain why they weren't showing up in the database when I looked previously...
Comment 3 David Cook 2026-01-21 00:28:41 UTC
Ahhh it's because "is_facetable" is based off Koha::SearchEngine::Elasticsearch::get_facet_fields which uses

Koha::SearchFields->search( { facet_order => { '!=' => undef } }, { order_by => ['facet_order'] } )->as_list;

Which has a certain logic to it but it's a problem.
Comment 4 David Cook 2026-01-21 00:49:44 UTC
Hmm it's not enough to have that "Yes" show up in the UI for publisher, local-classification, and copydate... there's something with the save overall that is blitzing it. Jeez...
Comment 5 David Cook 2026-01-21 00:56:28 UTC
That said... maybe it makes sense to only set "facet" if the mapping has a facet_order as well. 

Because if it doesn't, it's never going to show up anyway, and then we're creating indexed values that will never be used. 

But if that's the case maybe we should update the indexing to only index facet names that appear in Koha::SearchEngine::Elasticsearch->get_facet_fields() and to remove "facet: 1" from the default mappings.yaml for indexes that don't have a facet_order out of the box.
Comment 6 David Cook 2026-01-21 01:02:30 UTC
What should we do here? 

Option 1:
- Change mappings.yaml to set facet: 0 for publisher, local-classification, copydate
- This will mean that mappings.yaml and the UI are more in sync. That is, people won't look at mappings.yaml and wonder why the UI says "facetable: no" when the YAML says "facet: 1". That said, they could do that to their custom YAML and be confused still. 

Option 2a:
- Change mappings.pl to show facet: 1 for fields like publisher, local-classification, copydate even though they don't have facet_order set. 
- This will mean these fields are indexed into "*__facet" fields even though those facets won't be shown. That seems suboptimal

Option 2b:
- Do option 2a PLUS update the indexing so that "*__facet" fields are only created if there is a facet_order for that facet, so we don't create indexes that aren't used

Option 3:
- Something else I haven't thought of?
Comment 7 David Cook 2026-01-21 01:07:25 UTC
Note a manifestation of this bug... if you have a default/vanilla Elasticsearch mappings, you can add the "publisher" facet immediately and have it work out of the box, because "publisher__facet" is already indexed out of the box. 

I had 2 similar Koha instances and one had a working publisher facet and one didn't and it's because one had an original ES config and one must've had a slightly modified one where the publisher facet had been blitzed by clicking "Save" for an unrelated configuration change.

It was crazy-making for sure.
Comment 8 Tomás Cohen Arazi (tcohen) 2026-01-21 14:31:35 UTC
I recall domm mentioned last hackfest that saving the mappings to a file with `export_elasticsearch_mappings.pl` and then applying the file using the `--reset` flag for `rebuild_elasticsearch.pl` yielded inconsistent results in the configured mappings. I bet both are related. I think there was a bug for that too. Not facet-specific but related.
Comment 9 David Cook 2026-01-21 22:07:01 UTC
(In reply to Tomás Cohen Arazi (tcohen) from comment #8)
> I recall domm mentioned last hackfest that saving the mappings to a file
> with `export_elasticsearch_mappings.pl` and then applying the file using the
> `--reset` flag for `rebuild_elasticsearch.pl` yielded inconsistent results
> in the configured mappings. I bet both are related. I think there was a bug
> for that too. Not facet-specific but related.

I raised a bug for that too yesterday hehe: bug 41670

I don't know that it's related though. This facetable stuff specifically relates to changes made in bug 35138. I think bug 41670 is partially a code bug and partially the default file just being inaccurate.

But with this one... I don't think the answer is quite straightforward.