Bug 25325

Summary: ElasticSearch mapping export lacks staff_client/opac fields
Product: Koha Reporter: Frédéric Demians <f.demians>
Component: Searching - ElasticsearchAssignee: Frédéric Demians <f.demians>
Status: CLOSED FIXED QA Contact:
Severity: normal    
Priority: P5 - low CC: 1joynelson, fridolin.somers, jonathan.druart, katrin.fischer, martin.renvoize, victor
Version: master   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
20.05.00
Bug Depends on: 23204    
Bug Blocks: 28279    
Attachments: Bug 25325 ElasticSearch mapping export lacks staff_client/opac fields
Bug 25325 ElasticSearch mapping export lacks staff_client/opac fields
Bug 25325: Test ES lack of staff_client/opac fields
Bug 25325: ElasticSearch mapping export lacks staff_client/opac fields
Bug 25325: Test ES lack of staff_client/opac fields
Bug 25325: ElasticSearch mapping export lacks staff_client/opac fields
Bug 25325: Test ES lack of staff_client/opac fields
Bug 25325: ElasticSearch mapping export lacks staff_client/opac fields

Description Frédéric Demians 2020-04-30 05:49:44 UTC
export_elasticsearch_mappings.pl doesn't export staff_client and opac fields. Its means that when reimporting (Reset mapping in Search engine configuration) indexing rules, the searchability (staff/opac) of fields will be lost.
Comment 1 Frédéric Demians 2020-04-30 05:59:15 UTC
Created attachment 104004 [details] [review]
Bug 25325 ElasticSearch mapping export lacks staff_client/opac fields

Add to export_elasticsearch_mappings.pl two missing fields: staff_client
and opac.

Test plan:

1. Run export_elasticsearch_mappings.pl script. Write the result in
   mappings.yaml (the one defined in koha-conf.xml).
2. In Admin > Search engine configuration, click on Reset Mappings
3. Confirm that in Search fields tab, the Searchable column hasn't
   changed.
Comment 2 Victor Grousset/tuxayo 2020-04-30 22:20:21 UTC
Created attachment 104063 [details] [review]
Bug 25325 ElasticSearch mapping export lacks staff_client/opac fields

Add to export_elasticsearch_mappings.pl two missing fields: staff_client
and opac.

Test plan:

1. Run export_elasticsearch_mappings.pl script. Write the result in
   mappings.yaml (the one defined in koha-conf.xml).
2. In Admin > Search engine configuration, click on Reset Mappings
3. Confirm that in Search fields tab, the Searchable column hasn't
   changed.

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 3 Victor Grousset/tuxayo 2020-04-30 22:20:44 UTC
Hi, I tried to test and improvised a lot because Koha ES is new for me.

First, I switch SearchEngine syspref to ES but I don't know what else to do. (I have ES running as part of koha-testing-docker)

> 3. Confirm that in Search fields tab, the Searchable column hasn't
>   changed.

Without the patch, the Searchable column hasn't changed. So couldn't reproduce the issue.
I suspect that it's because my test data (from koha-testing-docker) has everything to "yes". And that "yes" might be the default.

So in the intranet I change few a columns to "no" and save.

>  An error occurred when updating Elasticsearch index mappings:
> Unable to update mappings for index "koha_kohadev_biblios".
> Reason was: "no such index". Index needs to be recreated and reindexed. 

>  An error occurred when updating Elasticsearch index mappings:
> Unable to update mappings for index "koha_kohadev_authorities".
> Reason was: "no such index". Index needs to be recreated and reindexed. 

>  Index 'koha_kohadev_biblios' needs to be recreated. 

>  Index 'koha_kohadev_authorities' needs to be recreated. 


Ok that doesn't look good. I must be missing some basic ES config.

At least I can confirm that with the patch there is a diff on the output of export_elasticsearch_mappings.pl

It's precisely on the changes I made which is encouraging. Let's try anyway.

So my test plan completed was:

- SearchEngine syspref to ES (don't know if that's useful)
- set few fields to not searchable
- scary errors...
- misc/search_tools/export_elasticsearch_mappings.pl
- save to mappings.yaml
  - for me it's: /kohadevbox/koha/admin/searchengine/elasticsearch/mappings.yaml
- restart_all (don't know if that's useful)
- reset mappings
- changes to the fields are lost
- apply patch
- restart_all
- set few fields to not searchable
- scary errors...
- misc/search_tools/export_elasticsearch_mappings.pl
- save to mappings.yaml
- restart_all (don't know if that's useful)
- reset mappings
- changes to the fields are still there! :D

So in spite of the scary errors and my dubious setup. It really look like it's working :D
Comment 4 Katrin Fischer 2020-04-30 22:44:08 UTC
*** Bug 25337 has been marked as a duplicate of this bug. ***
Comment 5 Frédéric Demians 2020-05-01 04:16:12 UTC
Victor, your correct 'yes' is default value for 'staff_client' and 'opac' fields. It means you have to set some ES fields to 'false' to see the problem. You can also see the reverse function of exporting ES config which is:

Koha::SearchEngine::ElasticSearch.pm::reset_elasticsearch_mappings

http://git.koha-community.org/gitweb/?p=koha.git;a=blob;f=Koha/SearchEngine/Elasticsearch.pm;h=66166713259f4a175295384fb30486ee2d8b6431;hb=HEAD#l360

Thks for testing/signing.
Comment 6 Jonathan Druart 2020-05-05 10:45:17 UTC
Salut Frédéric, could you add tests for that?
Comment 7 Frédéric Demians 2020-05-05 17:17:51 UTC
(In reply to Jonathan Druart from comment #6)
> Salut Frédéric, could you add tests for that?

Yes, sorry.
Comment 8 Frédéric Demians 2020-05-05 17:25:59 UTC
Created attachment 104376 [details] [review]
Bug 25325: Test ES lack of staff_client/opac fields

Run this test to see that export_elasticsearch_mappings.pl doesn't export
staff_client/opac fields.
Comment 9 Frédéric Demians 2020-05-05 17:26:28 UTC
Created attachment 104377 [details] [review]
Bug 25325: ElasticSearch mapping export lacks staff_client/opac fields

Run test:
prove t/db_dependent/Koha/SearchEngine/Elasticsearch/ExportConfig.t

Or:

Run export_elasticsearch_mappings.pl script. See that opac/staff_client fields
are exorted.
Comment 10 Victor Grousset/tuxayo 2020-05-06 18:24:05 UTC
Should the first patch be manually tested again?
Comment 11 Victor Grousset/tuxayo 2020-05-06 18:30:17 UTC
Wait, it's all automated tests now. I got also confused that the implementation changed from the one on comment 1
Comment 12 Victor Grousset/tuxayo 2020-05-06 18:44:43 UTC
Created attachment 104451 [details] [review]
Bug 25325: Test ES lack of staff_client/opac fields

Run this test to see that export_elasticsearch_mappings.pl doesn't export
staff_client/opac fields.

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 13 Victor Grousset/tuxayo 2020-05-06 18:44:47 UTC
Created attachment 104452 [details] [review]
Bug 25325: ElasticSearch mapping export lacks staff_client/opac fields

Run test:
prove t/db_dependent/Koha/SearchEngine/Elasticsearch/ExportConfig.t

Or:

Run export_elasticsearch_mappings.pl script. See that opac/staff_client fields
are exorted.

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 14 Jonathan Druart 2020-05-07 09:03:44 UTC
Created attachment 104482 [details] [review]
Bug 25325: Test ES lack of staff_client/opac fields

Run this test to see that export_elasticsearch_mappings.pl doesn't export
staff_client/opac fields.

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 15 Jonathan Druart 2020-05-07 09:03:47 UTC
Created attachment 104483 [details] [review]
Bug 25325: ElasticSearch mapping export lacks staff_client/opac fields

Run test:
prove t/db_dependent/Koha/SearchEngine/Elasticsearch/ExportConfig.t

Or:

Run export_elasticsearch_mappings.pl script. See that opac/staff_client fields
are exorted.

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 16 Martin Renvoize 2020-05-11 09:01:29 UTC
Nice work everyone!

Pushed to master for 20.05
Comment 17 Joy Nelson 2020-05-11 21:42:25 UTC
not backported to 19.11.x
Comment 18 Jonathan Druart 2021-04-28 08:58:56 UTC
On bug 23204 we made sure that the mappings.yaml file included in the codebase was the same as the output of misc/search_tools/export_elasticsearch_mappings.pl
However since this change (pushed only two months later) they differ.

Is that a big deal? Should we resync them?
Comment 19 Fridolin Somers 2021-05-03 10:52:33 UTC
I say we should update mapping.yaml, it is a reference when starting to manage Elasticsearch with files.