Bug 23204 - Add script for exporting Elasticsearch mappings to YAML
Summary: Add script for exporting Elasticsearch mappings to YAML
Status: Pushed to master
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Alex Arnaud
QA Contact:
URL:
Keywords:
Depends on:
Blocks: 19482 24997 25325 25337
  Show dependency treegraph
 
Reported: 2019-06-25 13:39 UTC by Alex Arnaud
Modified: 2020-06-22 21:51 UTC (History)
12 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:
20.05.00


Attachments
Bug 23204 - script for exporting Elasticsearch mappings (3.95 KB, patch)
2019-06-25 14:15 UTC, Alex Arnaud
Details | Diff | Splinter Review
Bug 23204 - script for exporting Elasticsearch mappings (4.01 KB, patch)
2019-07-05 18:01 UTC, Bouzid Fergani
Details | Diff | Splinter Review
Bug 23204 - script for exporting Elasticsearch mappings (4.01 KB, patch)
2019-08-06 08:04 UTC, Alex Arnaud
Details | Diff | Splinter Review
Bug 23204: move code in a unit tested sub (8.21 KB, patch)
2019-08-06 08:04 UTC, Alex Arnaud
Details | Diff | Splinter Review
Bug 23204: export field weights (1.07 KB, patch)
2019-08-06 08:04 UTC, Alex Arnaud
Details | Diff | Splinter Review
Bug 23204: add exec permission on the script (566 bytes, patch)
2019-08-13 08:41 UTC, Alex Arnaud
Details | Diff | Splinter Review
Bug 23204: sort search fields by name (849 bytes, patch)
2019-10-03 07:25 UTC, Alex Arnaud
Details | Diff | Splinter Review
Bug 23204: (follow-up) also sort by marc field (1.46 KB, patch)
2019-12-04 08:39 UTC, Fridolin SOMERS
Details | Diff | Splinter Review
Bug 23204: script for exporting Elasticsearch mappings (4.09 KB, patch)
2020-03-18 14:32 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 23204: move code in a unit tested sub (8.28 KB, patch)
2020-03-18 14:32 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 23204: export field weights (1.14 KB, patch)
2020-03-18 14:32 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 23204: add exec permission on the script (630 bytes, patch)
2020-03-18 14:32 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 23204: sort search fields by name (919 bytes, patch)
2020-03-18 14:32 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 23204: (follow-up) also sort by marc field (1.54 KB, patch)
2020-03-18 14:32 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 23204: Fix ordering of fields in exported file (2.68 KB, patch)
2020-03-25 17:11 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 23204: script for exporting Elasticsearch mappings (4.16 KB, patch)
2020-03-26 14:16 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 23204: move code in a unit tested sub (8.36 KB, patch)
2020-03-26 14:16 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 23204: export field weights (1.21 KB, patch)
2020-03-26 14:16 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 23204: add exec permission on the script (704 bytes, patch)
2020-03-26 14:16 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 23204: sort search fields by name (993 bytes, patch)
2020-03-26 14:16 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 23204: (follow-up) also sort by marc field (1.62 KB, patch)
2020-03-26 14:16 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 23204: Fix ordering of fields in exported file (2.74 KB, patch)
2020-03-26 14:16 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 23204: Update the mappings.yaml file with the output generated (78.49 KB, patch)
2020-03-26 14:16 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 23204: Adjust copyright and license (2.17 KB, patch)
2020-03-26 14:16 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 23204: Fix tests swapping fields (they are not ordered) (2.52 KB, patch)
2020-03-27 15:37 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Alex Arnaud 2019-06-25 13:39:45 UTC
Sometimes we need to report librarian's mapping changes in DB to the mappings.yaml. This can be long and tedious. So, here is a script that do the job.
Comment 1 Alex Arnaud 2019-06-25 14:15:54 UTC Comment hidden (obsolete)
Comment 2 Bouzid Fergani 2019-07-05 18:01:49 UTC Comment hidden (obsolete)
Comment 3 Nick Clemens 2019-07-26 16:10:03 UTC
This is great, few things:

1 - This should be a subroutine that can be unit tested (also then we can easily integrate into staff client and allow users to export mappings)
2 - This should take preserves weights specified in the client
Comment 4 Alex Arnaud 2019-08-06 08:04:34 UTC
Created attachment 92022 [details] [review]
Bug 23204 - script for exporting Elasticsearch mappings

Test plan:

  - launch perl misc/search_tools/export_elasticsearch_mappings.pl >
    /path/to/my_mappings.yaml
  - set koha-conf.elasticsearch_index_mappings to
    /path/to/my_mappings.yaml,
  - go to admin -> Search engine configuration,
  - click on "Reset mappins",
  - check that your search fields and mappings are as expected

Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Comment 5 Alex Arnaud 2019-08-06 08:04:43 UTC
Created attachment 92023 [details] [review]
Bug 23204: move code in a unit tested sub
Comment 6 Alex Arnaud 2019-08-06 08:04:49 UTC
Created attachment 92024 [details] [review]
Bug 23204: export field weights
Comment 7 Fridolin SOMERS 2019-08-09 09:33:45 UTC
misc/search_tools/export_elasticsearch_mappings.pl is missing exec permissions : 
chmod a+x
Comment 8 Alex Arnaud 2019-08-13 08:41:32 UTC
Created attachment 92174 [details] [review]
Bug 23204: add exec permission on the script
Comment 9 Fridolin SOMERS 2019-09-10 13:58:02 UTC
Really cool script.

Just, the generated YAML does not ensure sorting order.
Fields must be sorted by name and mappings by marc type (at least).
Comment 10 Alex Arnaud 2019-10-03 07:25:13 UTC
Created attachment 93524 [details] [review]
Bug 23204: sort search fields by name
Comment 11 Fridolin SOMERS 2019-12-04 08:39:45 UTC
Created attachment 95986 [details] [review]
Bug 23204: (follow-up) also sort by marc field

Test plan :
Launching script several times and see that marc field is alway ordered increasing
Comment 12 Nick Clemens 2019-12-11 20:08:45 UTC
Question: Should we only load/export the mappings for the marc_type of the instance? Including the normarc and unimarc for marc21 makes the file harder to read. Same the other way round to :-)
Comment 13 Fridolin SOMERS 2020-01-21 11:29:32 UTC
(In reply to Nick Clemens from comment #12)
> Question: Should we only load/export the mappings for the marc_type of the
> instance? Including the normarc and unimarc for marc21 makes the file harder
> to read. Same the other way round to :-)

There is the --type arg in the script to export only one marc flavour.
Comment 14 Bernardo Gonzalez Kriegel 2020-03-18 14:32:18 UTC
Created attachment 100960 [details] [review]
Bug 23204: script for exporting Elasticsearch mappings

Test plan:

  - launch perl misc/search_tools/export_elasticsearch_mappings.pl >
    /path/to/my_mappings.yaml
  - set koha-conf.elasticsearch_index_mappings to
    /path/to/my_mappings.yaml,
  - go to admin -> Search engine configuration,
  - click on "Reset mappins",
  - check that your search fields and mappings are as expected

Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Fixed commit message
Comment 15 Bernardo Gonzalez Kriegel 2020-03-18 14:32:22 UTC
Created attachment 100961 [details] [review]
Bug 23204: move code in a unit tested sub

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Test pass
Comment 16 Bernardo Gonzalez Kriegel 2020-03-18 14:32:25 UTC
Created attachment 100962 [details] [review]
Bug 23204: export field weights

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Comment 17 Bernardo Gonzalez Kriegel 2020-03-18 14:32:29 UTC
Created attachment 100963 [details] [review]
Bug 23204: add exec permission on the script

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Comment 18 Bernardo Gonzalez Kriegel 2020-03-18 14:32:32 UTC
Created attachment 100964 [details] [review]
Bug 23204: sort search fields by name

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Comment 19 Bernardo Gonzalez Kriegel 2020-03-18 14:32:35 UTC
Created attachment 100965 [details] [review]
Bug 23204: (follow-up) also sort by marc field

Test plan :
Launching script several times and see that marc field is alway ordered increasing

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Same order between runs
Comment 20 Jonathan Druart 2020-03-25 14:51:01 UTC
I think the exported file must export exactly the same file as the one we embed with Koha.

% perl misc/search_tools/export_elasticsearch_mappings.pl > /tmp/mappings.yaml
% diff admin/searchengine/elasticsearch/mappings.yaml /tmp/mappings.yaml
Should return no change.

Don't you think?
Comment 21 Nick Clemens 2020-03-25 17:11:06 UTC
Created attachment 101742 [details] [review]
Bug 23204: Fix ordering of fields in exported file
Comment 22 Nick Clemens 2020-03-25 17:17:07 UTC
(In reply to Jonathan Druart from comment #20)
> I think the exported file must export exactly the same file as the one we
> embed with Koha.
> 
> % perl misc/search_tools/export_elasticsearch_mappings.pl >
> /tmp/mappings.yaml
> % diff admin/searchengine/elasticsearch/mappings.yaml /tmp/mappings.yaml
> Should return no change.
> 
> Don't you think?

Yes and no - ordering fixed most, and not adding undefined weights or facet_order, but there are some remaining differences:
1 - Facet order not set in default file
2 - LC-card-number - comes out in lower case because we have it twice, once for authorities and once for biblios - these get combined when adding to DB because they all go in to 'search_field' which doesn't separate bibs/auths - I don't think the matters. There does seem to be convention of Auth fields being capitalized and bibs fields not, but I don't believe we rely on this or need to
3 - Default mapping file is not strictly alphabetized - date-of-publication moves on export

Probably a good compromise is to generate the mapping file from this bug once it passes QA and use the exported file as the default from now on. Consider it a cleanup of the file
Comment 23 Fridolin SOMERS 2020-03-26 13:03:35 UTC
(In reply to Nick Clemens from comment #22)
> 
> Probably a good compromise is to generate the mapping file from this bug
> once it passes QA and use the exported file as the default from now on.
> Consider it a cleanup of the file

+1
Comment 24 Jonathan Druart 2020-03-26 14:16:18 UTC
Created attachment 101850 [details] [review]
Bug 23204: script for exporting Elasticsearch mappings

Test plan:

  - launch perl misc/search_tools/export_elasticsearch_mappings.pl >
    /path/to/my_mappings.yaml
  - set koha-conf.elasticsearch_index_mappings to
    /path/to/my_mappings.yaml,
  - go to admin -> Search engine configuration,
  - click on "Reset mappins",
  - check that your search fields and mappings are as expected

Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Fixed commit message

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 25 Jonathan Druart 2020-03-26 14:16:22 UTC
Created attachment 101851 [details] [review]
Bug 23204: move code in a unit tested sub

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Test pass

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 26 Jonathan Druart 2020-03-26 14:16:25 UTC
Created attachment 101852 [details] [review]
Bug 23204: export field weights

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 27 Jonathan Druart 2020-03-26 14:16:29 UTC
Created attachment 101853 [details] [review]
Bug 23204: add exec permission on the script

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 28 Jonathan Druart 2020-03-26 14:16:33 UTC
Created attachment 101854 [details] [review]
Bug 23204: sort search fields by name

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 29 Jonathan Druart 2020-03-26 14:16:37 UTC
Created attachment 101855 [details] [review]
Bug 23204: (follow-up) also sort by marc field

Test plan :
Launching script several times and see that marc field is alway ordered increasing

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Same order between runs

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 30 Jonathan Druart 2020-03-26 14:16:40 UTC
Created attachment 101856 [details] [review]
Bug 23204: Fix ordering of fields in exported file

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 31 Jonathan Druart 2020-03-26 14:16:45 UTC
Created attachment 101857 [details] [review]
Bug 23204: Update the mappings.yaml file with the output generated

We need to be consistent and provide the mappings.yaml file generated by
the script.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 32 Jonathan Druart 2020-03-26 14:16:49 UTC
Created attachment 101858 [details] [review]
Bug 23204: Adjust copyright and license

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 33 Jonathan Druart 2020-03-26 14:18:50 UTC
(In reply to Jonathan Druart from comment #31)
> Created attachment 101857 [details] [review] [review]
> Bug 23204: Update the mappings.yaml file with the output generated
> 
> We need to be consistent and provide the mappings.yaml file generated by
> the script.
> 
> Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Alex, Nick, can you confirm you agree with that?

If so I think we need a test to make sure further changes will be put into the correct place in the file.
Comment 34 Alex Arnaud 2020-03-26 16:27:17 UTC
> 
> Alex, Nick, can you confirm you agree with that?
> 
> If so I think we need a test to make sure further changes will be put into
> the correct place in the file.

I agree !
Comment 35 Jonathan Druart 2020-03-27 07:50:06 UTC
(In reply to Jonathan Druart from comment #33)
> (In reply to Jonathan Druart from comment #31)
> > Created attachment 101857 [details] [review] [review] [review]
> > Bug 23204: Update the mappings.yaml file with the output generated
> > 
> > We need to be consistent and provide the mappings.yaml file generated by
> > the script.
> > 
> > Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
> 
> Alex, Nick, can you confirm you agree with that?
> 
> If so I think we need a test to make sure further changes will be put into
> the correct place in the file.

See bug 24997.
Comment 36 Martin Renvoize 2020-03-27 12:23:50 UTC
Nice work everyone!

Pushed to master for 20.05
Comment 37 Fridolin SOMERS 2020-03-27 12:59:26 UTC
We would really like this one in 19.11.x
Comment 38 Jonathan Druart 2020-03-27 15:37:28 UTC
Created attachment 102010 [details] [review]
Bug 23204: Fix tests swapping fields (they are not ordered)
Comment 39 Joy Nelson 2020-04-03 22:50:10 UTC
enhancement not backported to 19.11.x
Comment 40 Frédéric Demians 2020-04-30 05:09:16 UTC
Note : Since bug 20589, two fields have been added to 'search_field' table: staff_client and opac. Those fields are lacking in mappings.yaml. Its means that if this script is used to export and import (Reset mapping in Search engine configuration) indexing rules, the searchability (staff/opac) of fields will be lost.
Comment 41 Katrin Fischer 2020-04-30 21:21:11 UTC
(In reply to Frédéric Demians from comment #40)
> Note : Since bug 20589, two fields have been added to 'search_field' table:
> staff_client and opac. Those fields are lacking in mappings.yaml. Its means
> that if this script is used to export and import (Reset mapping in Search
> engine configuration) indexing rules, the searchability (staff/opac) of
> fields will be lost.

I've opened a new bug report: 

Bug 25337 - Export/import of Elasticsearch configuration is lacking information
Comment 42 Alex Arnaud 2020-05-29 10:18:41 UTC
(In reply to Jonathan Druart from comment #31)
> Created attachment 101857 [details] [review] [review]
> Bug 23204: Update the mappings.yaml file with the output generated
> 
> We need to be consistent and provide the mappings.yaml file generated by
> the script.
> 
> Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

I think i agreed a little to fast this commit. Removing quote on field number starting with 0 (030) make that number is badly interpreted (as octal).

So this mapping:

   coden:
    label: coden
    mappings:
      - facet: ''
        marc_field: 030
        marc_type: marc21
        sort: ~
        suggestible: ''

makes coden mapped with field 24

Indexing fail:

root@2dc1f5ace3f5:koha(master)$ perl misc/search_tools/rebuild_elasticsearch.pl -b -a -d -v
[1324] Checking state of biblios index
[1324] Dropping and recreating biblios index
[1324] Checking state of authorities index
[1324] Dropping and recreating authorities index
[1324] Indexing biblios
[1324] Committing final records...
Invalid MARC field expression: 24
Comment 43 Nick Clemens 2020-05-29 11:20:24 UTC
(In reply to Alex Arnaud from comment #42)
> (In reply to Jonathan Druart from comment #31)

> I think i agreed a little to fast this commit. Removing quote on field
> number starting with 0 (030) make that number is badly interpreted (as
> octal).

I experienced this once, however, I cannot recreate it now, what is your setup when this happens?
Comment 44 Frédéric Demians 2020-05-29 14:22:37 UTC
This issue may depend on the YAML parser used by default. I can't neither reproduce it on my setup. Replacing "use YAML" with "use YAML::Syck" could be a solution.
Comment 45 Alex Arnaud 2020-05-29 15:03:41 UTC
(In reply to Nick Clemens from comment #43)
> (In reply to Alex Arnaud from comment #42)
> 
> I experienced this once, however, I cannot recreate it now, what is your
> setup when this happens?

Now i can reproduce it almost each time:

 - start a fresh koha container (koha-testing-docker): ku-es5 or reset_all in an existing one,
 - Change searchengine syspref to "Elasticsearch",
 - Go to Admin -> Search engine configuration (Elasticsearch),
 - mapping ok (coden => 030, code-institution => 040 etc...)
 - reset mapping
 - mapping Ko (coden => 24, code-institution => 32 etc...)
Comment 46 Jonathan Druart 2020-06-05 10:34:39 UTC
Could someone deal with that? At least to open a new bug report.
Comment 47 Jonathan Druart 2020-06-17 09:38:46 UTC
I don't recreate with KOHA_IMAGE=master-bionic and docker-compose.es6.yml
Comment 48 Frédéric Demians 2020-06-17 10:49:05 UTC
If there is a new bug, assign it to me. I will do my best to reproduce it and fix it.