Bug 14899 - Mapping configuration page for Elastic search
Summary: Mapping configuration page for Elastic search
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Jonathan Druart
QA Contact: Testopia
URL:
Keywords:
Depends on: 12478 17091 17226 17372
Blocks: 17616 18235 18827
  Show dependency treegraph
 
Reported: 2015-09-25 14:38 UTC by Jonathan Druart
Modified: 2018-06-04 20:10 UTC (History)
10 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
This new page (admin/searchengine/elasticsearch/mappings.pl) will permit to manage the ES mappings. For the biblios and authorities indexes, the different mappings can be managed from this single page. The interface let you add, remove and update mappings and search fields. It's also possible to reorder the mappings, as the order can be important in the indexation process. Note that the table can be displayed in a different order that the one it was before saving, but the mappings are grouped by search field and the order inside the search field is preserved. Limitations: If something went wrong during the insertion/deletion/modification, the users will loose all these changes.
Version(s) released in:


Attachments
Bug 14899: Add Koha::SearchField[s] and Koha::SearchMarcMap[s] classes (5.73 KB, patch)
2015-09-25 15:34 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 14899: Add tableDND JS lib (29.84 KB, patch)
2015-09-25 15:34 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 14899: DB changes (5.88 KB, patch)
2015-09-25 15:34 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 14899: Add the mapping configuration page in the admin module (19.47 KB, patch)
2015-09-25 15:34 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 14899: Add a link to the new page in the admin (1.30 KB, patch)
2015-09-25 15:34 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 14899: Add Koha::SearchField[s] and Koha::SearchMarcMap[s] classes (5.73 KB, patch)
2015-10-02 09:49 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 14899: Add tableDND JS lib (29.82 KB, patch)
2015-10-02 09:50 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 14899: DB changes (6.01 KB, patch)
2015-10-02 09:50 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 14899: Add the mapping configuration page in the admin module (19.47 KB, patch)
2015-10-02 09:50 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 14899: Add a link to the new page in the admin (1.30 KB, patch)
2015-10-02 09:50 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 14899: schema changes to make the database better (3.73 KB, patch)
2015-10-05 03:22 UTC, Robin Sheat
Details | Diff | Splinter Review
Bug 14899: Add tableDND JS lib (29.82 KB, patch)
2015-10-12 16:25 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 14899: Add the mapping configuration page in the admin module (21.39 KB, patch)
2015-10-12 16:25 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 14899: Add a link to the new page in the admin (1.30 KB, patch)
2015-10-12 16:25 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 14899: Add tableDND JS lib (29.78 KB, patch)
2016-04-29 14:15 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 14899: Add the mapping configuration page in the admin module (21.39 KB, patch)
2016-04-29 14:15 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 14899: Add a link to the new page in the admin (1.29 KB, patch)
2016-04-29 14:15 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 14899: Add tableDND JS lib (29.88 KB, patch)
2016-09-29 11:08 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 14899: Add the mapping configuration page in the admin module (20.81 KB, patch)
2016-09-29 11:08 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 14899: Add a link to the new page in the admin (1.29 KB, patch)
2016-09-29 11:08 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 14899: Add a reset option (1.25 KB, patch)
2016-09-29 11:08 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 14899: Add tableDND JS lib (29.93 KB, patch)
2016-09-30 13:27 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 14899: Add the mapping configuration page in the admin module (20.87 KB, patch)
2016-09-30 13:27 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 14899: Add a link to the new page in the admin (1.35 KB, patch)
2016-09-30 13:27 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 14899: Add a reset option (1.31 KB, patch)
2016-09-30 13:27 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 14899: Add tableDND JS lib (29.99 KB, patch)
2016-10-11 13:12 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 14899: Add tableDND JS lib (29.99 KB, patch)
2016-10-11 13:25 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 14899: Add tableDND JS lib (29.99 KB, patch)
2016-10-11 13:48 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 14899: Add tableDND JS lib (29.99 KB, patch)
2016-10-11 13:50 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 14899: Add the mapping configuration page in the admin module (20.92 KB, patch)
2016-10-11 13:51 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 14899: Add a link to the new page in the admin (1.40 KB, patch)
2016-10-11 13:51 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 14899: Add a reset option (1.36 KB, patch)
2016-10-11 13:51 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 14899 - Convert links to buttons, add icons (2.02 KB, patch)
2016-10-11 13:51 UTC, Kyle M Hall
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Jonathan Druart 2015-09-25 14:38:52 UTC
To help the users and testers to use and understand how Elastic search works, Koha needs an interface to configure the mappings between the Elastic search indexes and the marc fields.
Comment 1 Jonathan Druart 2015-09-25 15:34:05 UTC Comment hidden (obsolete)
Comment 2 Jonathan Druart 2015-09-25 15:34:09 UTC Comment hidden (obsolete)
Comment 3 Jonathan Druart 2015-09-25 15:34:12 UTC Comment hidden (obsolete)
Comment 4 Jonathan Druart 2015-09-25 15:34:19 UTC Comment hidden (obsolete)
Comment 5 Jonathan Druart 2015-09-25 15:34:22 UTC Comment hidden (obsolete)
Comment 6 Jonathan Druart 2015-09-25 15:35:00 UTC
To apply on top of the ES work, see the catalyst/elastic_search branch.
Comment 7 Peter Zhao 2015-09-27 13:15:17 UTC
http://127.0.0.1:8080/cgi-bin/koha/admin/searchengine/elasticsearch/mappings.pl


An error has occurred!
Error 500

    In Koha this typically means that the Koha team is working on new features
    Wait while system maintenance is being done or email the Koha administrator.
    Use top menu bar to navigate to another part of Koha.
Comment 8 Jonathan Druart 2015-09-28 07:59:38 UTC
(In reply to Peter Zhao from comment #7)
> http://127.0.0.1:8080/cgi-bin/koha/admin/searchengine/elasticsearch/mappings.
> pl
> 
> 
> An error has occurred!
> Error 500
> 
>     In Koha this typically means that the Koha team is working on new
> features
>     Wait while system maintenance is being done or email the Koha
> administrator.
>     Use top menu bar to navigate to another part of Koha.

What is the error in the log file?
How did you test?

To test:
$ git remote -v
# should at least return
catalyst        git://git.catalyst.net.nz/koha.git (fetch)
catalyst        git://git.catalyst.net.nz/koha.git (push)
$ git fetch catalyst
$ git checkout -b bug_14899 catalyst/elastic_search
$ git bz apply 14899
$ perl installer/data/mysql/updatedatabase.pl
Comment 9 Jonathan Druart 2015-10-02 09:49:58 UTC Comment hidden (obsolete)
Comment 10 Jonathan Druart 2015-10-02 09:50:01 UTC Comment hidden (obsolete)
Comment 11 Jonathan Druart 2015-10-02 09:50:04 UTC Comment hidden (obsolete)
Comment 12 Jonathan Druart 2015-10-02 09:50:07 UTC Comment hidden (obsolete)
Comment 13 Jonathan Druart 2015-10-02 09:50:10 UTC Comment hidden (obsolete)
Comment 14 Jonathan Druart 2015-10-02 09:50:58 UTC
Patches updated, they now apply against the up-to-date catalyst/elastic_search branch.
Comment 15 Robin Sheat 2015-10-05 02:05:40 UTC
(In reply to Jonathan Druart from comment #11)
> 3/ Remove the index_name_2 unique constraint on the search_marc_map table.
> I don't understand how this could be useful, it was defined like (index_name,
> marc_type, marc_field), so it means a mapping cannot be defined twice with
> different values for facet, suggestible and sort.
> This limitation does not seem to make sense.
> Robin, please correct me if I am wrong :)

You are not wrong. I just wrote a big thing about what I was intending, realised I was wrong, so deleted it :)

I think I was confusing index_name and field name, or something. I'll merge this in as part of the ES branch like you suggest, as it'll solve some issues.
Comment 16 Robin Sheat 2015-10-05 02:15:26 UTC
Though, it'd be best to just put DB changes in the .sql file for now and let that load them in from there, rather than creating a whole set of transactions to manage.
Comment 17 Robin Sheat 2015-10-05 03:22:32 UTC Comment hidden (obsolete)
Comment 18 Jonathan Druart 2015-10-12 16:25:43 UTC Comment hidden (obsolete)
Comment 19 Jonathan Druart 2015-10-12 16:25:51 UTC Comment hidden (obsolete)
Comment 20 Jonathan Druart 2015-10-12 16:25:58 UTC Comment hidden (obsolete)
Comment 21 I'm just a bot 2016-03-06 21:50:45 UTC
.git/rebase-apply/patch:305: trailing whitespace.
        
warning: 1 line adds whitespace errors.
.git/rebase-apply/patch:305: trailing whitespace.
        
warning: 1 line applied after fixing whitespace errors.
error: Failed to merge in the changes. at ./getter.pl line 196.
Comment 22 Jonathan Druart 2016-04-29 14:15:48 UTC
Created attachment 50995 [details] [review]
Bug 14899: Add tableDND JS lib
Comment 23 Jonathan Druart 2016-04-29 14:15:53 UTC
Created attachment 50996 [details] [review]
Bug 14899: Add the mapping configuration page in the admin module

This new page (admin/searchengine/elasticsearch/mappings.pl) will permit
to manage the ES mappings.
For the biblios and authorities indexes, the different mappings can be
managed from this single page.
The interface let you add, remove and update mappings and search fields.
It's also possible to reorder the mappings, as the order can be important
in the indexation process. Note that the table can be displayed in a
different order that the one it was before saving, but the mappings are grouped
by search field and the order inside the search field is preserved.

Limitations:
- If something went wrong during the insertion/deletion/modification,
  the users will loose all these changes.

TODO:
- Add a specific permission (?)
- Add some data checks client side (JS)
- Use checkboxes for facet and suggestible (lazy today...)
- Understand the difference between the 3 values that sortable can have
  and improve the value for the options in the select box.
Comment 24 Jonathan Druart 2016-04-29 14:15:57 UTC
Created attachment 50997 [details] [review]
Bug 14899: Add a link to the new page in the admin
Comment 25 Jonathan Druart 2016-09-29 11:08:25 UTC
Created attachment 55915 [details] [review]
Bug 14899: Add tableDND JS lib
Comment 26 Jonathan Druart 2016-09-29 11:08:29 UTC
Created attachment 55916 [details] [review]
Bug 14899: Add the mapping configuration page in the admin module

This new page (admin/searchengine/elasticsearch/mappings.pl) will permit
to manage the ES mappings.
For the biblios and authorities indexes, the different mappings can be
managed from this single page.
The interface let you add, remove and update mappings and search fields.
It's also possible to reorder the mappings, as the order can be important
in the indexation process. Note that the table can be displayed in a
different order that the one it was before saving, but the mappings are grouped
by search field and the order inside the search field is preserved.

Limitations:
- If something went wrong during the insertion/deletion/modification,
  the users will loose all these changes.

TODO:
- Add a specific permission (?)
- Add some data checks client side (JS)
- Use checkboxes for facet and suggestible (lazy today...)
- Understand the difference between the 3 values that sortable can have
  and improve the value for the options in the select box.
Comment 27 Jonathan Druart 2016-09-29 11:08:32 UTC
Created attachment 55917 [details] [review]
Bug 14899: Add a link to the new page in the admin
Comment 28 Jonathan Druart 2016-09-29 11:08:37 UTC
Created attachment 55918 [details] [review]
Bug 14899: Add a reset option

For developpers only at the moment, if you want to reset your mappings,
you can hit:
  admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
Comment 29 Nick Clemens 2016-09-30 13:25:09 UTC
It seems to work, adding/deleting/updating values affects only the current marcflavour, adding a field to an index and performing a reindex results in good search results.

It would be nice to remain on the current tab when saving

Bad data here can break indexing ( save date-entered-on-file with '650a/4-8' ) so permissions are definitely needed
Comment 30 Nick Clemens 2016-09-30 13:27:38 UTC
Created attachment 55957 [details] [review]
Bug 14899: Add tableDND JS lib

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 31 Nick Clemens 2016-09-30 13:27:44 UTC
Created attachment 55958 [details] [review]
Bug 14899: Add the mapping configuration page in the admin module

This new page (admin/searchengine/elasticsearch/mappings.pl) will permit
to manage the ES mappings.
For the biblios and authorities indexes, the different mappings can be
managed from this single page.
The interface let you add, remove and update mappings and search fields.
It's also possible to reorder the mappings, as the order can be important
in the indexation process. Note that the table can be displayed in a
different order that the one it was before saving, but the mappings are grouped
by search field and the order inside the search field is preserved.

Limitations:
- If something went wrong during the insertion/deletion/modification,
  the users will loose all these changes.

TODO:
- Add a specific permission (?)
- Add some data checks client side (JS)
- Use checkboxes for facet and suggestible (lazy today...)
- Understand the difference between the 3 values that sortable can have
  and improve the value for the options in the select box.

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 32 Nick Clemens 2016-09-30 13:27:49 UTC
Created attachment 55959 [details] [review]
Bug 14899: Add a link to the new page in the admin

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 33 Nick Clemens 2016-09-30 13:27:55 UTC
Created attachment 55960 [details] [review]
Bug 14899: Add a reset option

For developpers only at the moment, if you want to reset your mappings,
you can hit:
  admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 34 Kyle M Hall 2016-10-11 13:12:51 UTC
Created attachment 56182 [details] [review]
Bug 14899: Add tableDND JS lib

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 35 Kyle M Hall 2016-10-11 13:25:25 UTC
Created attachment 56183 [details] [review]
Bug 14899: Add tableDND JS lib

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 36 Kyle M Hall 2016-10-11 13:48:43 UTC
Created attachment 56194 [details] [review]
Bug 14899: Add tableDND JS lib

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 37 Kyle M Hall 2016-10-11 13:50:43 UTC
Created attachment 56195 [details] [review]
Bug 14899: Add tableDND JS lib

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 38 Kyle M Hall 2016-10-11 13:51:00 UTC
Created attachment 56196 [details] [review]
Bug 14899: Add the mapping configuration page in the admin module

This new page (admin/searchengine/elasticsearch/mappings.pl) will permit
to manage the ES mappings.
For the biblios and authorities indexes, the different mappings can be
managed from this single page.
The interface let you add, remove and update mappings and search fields.
It's also possible to reorder the mappings, as the order can be important
in the indexation process. Note that the table can be displayed in a
different order that the one it was before saving, but the mappings are grouped
by search field and the order inside the search field is preserved.

Limitations:
- If something went wrong during the insertion/deletion/modification,
  the users will loose all these changes.

TODO:
- Add a specific permission (?)
- Add some data checks client side (JS)
- Use checkboxes for facet and suggestible (lazy today...)
- Understand the difference between the 3 values that sortable can have
  and improve the value for the options in the select box.

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 39 Kyle M Hall 2016-10-11 13:51:07 UTC
Created attachment 56197 [details] [review]
Bug 14899: Add a link to the new page in the admin

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 40 Kyle M Hall 2016-10-11 13:51:16 UTC
Created attachment 56198 [details] [review]
Bug 14899: Add a reset option

For developpers only at the moment, if you want to reset your mappings,
you can hit:
  admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 41 Kyle M Hall 2016-10-11 13:51:25 UTC
Created attachment 56199 [details] [review]
Bug 14899 - Convert links to buttons, add icons

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 42 Brendan Gallagher 2016-10-12 12:10:39 UTC
Pushed to Master - Should be in the November 2016 release.  Thanks!
Comment 43 Mason James 2016-12-14 11:37:54 UTC
Pushed to 16.05.x, for 16.05.06 release