Bug 25325 - ElasticSearch mapping export lacks staff_client/opac fields
Summary: ElasticSearch mapping export lacks staff_client/opac fields
Status: Pushed to master
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: Frédéric Demians
QA Contact:
URL:
Keywords:
: 25337 (view as bug list)
Depends on: 23204
Blocks:
  Show dependency treegraph
 
Reported: 2020-04-30 05:49 UTC by Frédéric Demians
Modified: 2020-05-18 09:49 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:
20.05.00


Attachments
Bug 25325 ElasticSearch mapping export lacks staff_client/opac fields (2.76 KB, patch)
2020-04-30 05:59 UTC, Frédéric Demians
Details | Diff | Splinter Review
Bug 25325 ElasticSearch mapping export lacks staff_client/opac fields (2.82 KB, patch)
2020-04-30 22:20 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review
Bug 25325: Test ES lack of staff_client/opac fields (1.78 KB, patch)
2020-05-05 17:25 UTC, Frédéric Demians
Details | Diff | Splinter Review
Bug 25325: ElasticSearch mapping export lacks staff_client/opac fields (1.55 KB, patch)
2020-05-05 17:26 UTC, Frédéric Demians
Details | Diff | Splinter Review
Bug 25325: Test ES lack of staff_client/opac fields (1.83 KB, patch)
2020-05-06 18:44 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review
Bug 25325: ElasticSearch mapping export lacks staff_client/opac fields (1.61 KB, patch)
2020-05-06 18:44 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review
Bug 25325: Test ES lack of staff_client/opac fields (1.91 KB, patch)
2020-05-07 09:03 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 25325: ElasticSearch mapping export lacks staff_client/opac fields (1.68 KB, patch)
2020-05-07 09:03 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Frédéric Demians 2020-04-30 05:49:44 UTC
export_elasticsearch_mappings.pl doesn't export staff_client and opac fields. Its means that when reimporting (Reset mapping in Search engine configuration) indexing rules, the searchability (staff/opac) of fields will be lost.
Comment 1 Frédéric Demians 2020-04-30 05:59:15 UTC
Created attachment 104004 [details] [review]
Bug 25325 ElasticSearch mapping export lacks staff_client/opac fields

Add to export_elasticsearch_mappings.pl two missing fields: staff_client
and opac.

Test plan:

1. Run export_elasticsearch_mappings.pl script. Write the result in
   mappings.yaml (the one defined in koha-conf.xml).
2. In Admin > Search engine configuration, click on Reset Mappings
3. Confirm that in Search fields tab, the Searchable column hasn't
   changed.
Comment 2 Victor Grousset/tuxayo 2020-04-30 22:20:21 UTC
Created attachment 104063 [details] [review]
Bug 25325 ElasticSearch mapping export lacks staff_client/opac fields

Add to export_elasticsearch_mappings.pl two missing fields: staff_client
and opac.

Test plan:

1. Run export_elasticsearch_mappings.pl script. Write the result in
   mappings.yaml (the one defined in koha-conf.xml).
2. In Admin > Search engine configuration, click on Reset Mappings
3. Confirm that in Search fields tab, the Searchable column hasn't
   changed.

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 3 Victor Grousset/tuxayo 2020-04-30 22:20:44 UTC
Hi, I tried to test and improvised a lot because Koha ES is new for me.

First, I switch SearchEngine syspref to ES but I don't know what else to do. (I have ES running as part of koha-testing-docker)

> 3. Confirm that in Search fields tab, the Searchable column hasn't
>   changed.

Without the patch, the Searchable column hasn't changed. So couldn't reproduce the issue.
I suspect that it's because my test data (from koha-testing-docker) has everything to "yes". And that "yes" might be the default.

So in the intranet I change few a columns to "no" and save.

>  An error occurred when updating Elasticsearch index mappings:
> Unable to update mappings for index "koha_kohadev_biblios".
> Reason was: "no such index". Index needs to be recreated and reindexed. 

>  An error occurred when updating Elasticsearch index mappings:
> Unable to update mappings for index "koha_kohadev_authorities".
> Reason was: "no such index". Index needs to be recreated and reindexed. 

>  Index 'koha_kohadev_biblios' needs to be recreated. 

>  Index 'koha_kohadev_authorities' needs to be recreated. 


Ok that doesn't look good. I must be missing some basic ES config.

At least I can confirm that with the patch there is a diff on the output of export_elasticsearch_mappings.pl

It's precisely on the changes I made which is encouraging. Let's try anyway.

So my test plan completed was:

- SearchEngine syspref to ES (don't know if that's useful)
- set few fields to not searchable
- scary errors...
- misc/search_tools/export_elasticsearch_mappings.pl
- save to mappings.yaml
  - for me it's: /kohadevbox/koha/admin/searchengine/elasticsearch/mappings.yaml
- restart_all (don't know if that's useful)
- reset mappings
- changes to the fields are lost
- apply patch
- restart_all
- set few fields to not searchable
- scary errors...
- misc/search_tools/export_elasticsearch_mappings.pl
- save to mappings.yaml
- restart_all (don't know if that's useful)
- reset mappings
- changes to the fields are still there! :D

So in spite of the scary errors and my dubious setup. It really look like it's working :D
Comment 4 Katrin Fischer 2020-04-30 22:44:08 UTC
*** Bug 25337 has been marked as a duplicate of this bug. ***
Comment 5 Frédéric Demians 2020-05-01 04:16:12 UTC
Victor, your correct 'yes' is default value for 'staff_client' and 'opac' fields. It means you have to set some ES fields to 'false' to see the problem. You can also see the reverse function of exporting ES config which is:

Koha::SearchEngine::ElasticSearch.pm::reset_elasticsearch_mappings

http://git.koha-community.org/gitweb/?p=koha.git;a=blob;f=Koha/SearchEngine/Elasticsearch.pm;h=66166713259f4a175295384fb30486ee2d8b6431;hb=HEAD#l360

Thks for testing/signing.
Comment 6 Jonathan Druart 2020-05-05 10:45:17 UTC
Salut Frédéric, could you add tests for that?
Comment 7 Frédéric Demians 2020-05-05 17:17:51 UTC
(In reply to Jonathan Druart from comment #6)
> Salut Frédéric, could you add tests for that?

Yes, sorry.
Comment 8 Frédéric Demians 2020-05-05 17:25:59 UTC
Created attachment 104376 [details] [review]
Bug 25325: Test ES lack of staff_client/opac fields

Run this test to see that export_elasticsearch_mappings.pl doesn't export
staff_client/opac fields.
Comment 9 Frédéric Demians 2020-05-05 17:26:28 UTC
Created attachment 104377 [details] [review]
Bug 25325: ElasticSearch mapping export lacks staff_client/opac fields

Run test:
prove t/db_dependent/Koha/SearchEngine/Elasticsearch/ExportConfig.t

Or:

Run export_elasticsearch_mappings.pl script. See that opac/staff_client fields
are exorted.
Comment 10 Victor Grousset/tuxayo 2020-05-06 18:24:05 UTC
Should the first patch be manually tested again?
Comment 11 Victor Grousset/tuxayo 2020-05-06 18:30:17 UTC
Wait, it's all automated tests now. I got also confused that the implementation changed from the one on comment 1
Comment 12 Victor Grousset/tuxayo 2020-05-06 18:44:43 UTC
Created attachment 104451 [details] [review]
Bug 25325: Test ES lack of staff_client/opac fields

Run this test to see that export_elasticsearch_mappings.pl doesn't export
staff_client/opac fields.

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 13 Victor Grousset/tuxayo 2020-05-06 18:44:47 UTC
Created attachment 104452 [details] [review]
Bug 25325: ElasticSearch mapping export lacks staff_client/opac fields

Run test:
prove t/db_dependent/Koha/SearchEngine/Elasticsearch/ExportConfig.t

Or:

Run export_elasticsearch_mappings.pl script. See that opac/staff_client fields
are exorted.

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 14 Jonathan Druart 2020-05-07 09:03:44 UTC
Created attachment 104482 [details] [review]
Bug 25325: Test ES lack of staff_client/opac fields

Run this test to see that export_elasticsearch_mappings.pl doesn't export
staff_client/opac fields.

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 15 Jonathan Druart 2020-05-07 09:03:47 UTC
Created attachment 104483 [details] [review]
Bug 25325: ElasticSearch mapping export lacks staff_client/opac fields

Run test:
prove t/db_dependent/Koha/SearchEngine/Elasticsearch/ExportConfig.t

Or:

Run export_elasticsearch_mappings.pl script. See that opac/staff_client fields
are exorted.

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 16 Martin Renvoize 2020-05-11 09:01:29 UTC
Nice work everyone!

Pushed to master for 20.05
Comment 17 Joy Nelson 2020-05-11 21:42:25 UTC
not backported to 19.11.x