Bug 19707 - Add new operations for synchronizing Elasticsearch mappings
Summary: Add new operations for synchronizing Elasticsearch mappings
Status: Failed QA
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: David Gustafsson
QA Contact:
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-11-28 10:40 UTC by David Gustafsson
Modified: 2022-08-04 08:26 UTC (History)
4 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 19707 - Add new operations for synchronizing Elasticsearch mappings (5.45 KB, patch)
2017-11-28 10:48 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 19707 - Add new operations for synchronizing Elasticsearch mappings (6.63 KB, patch)
2018-04-24 17:43 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 19707 - Add new operations for synchronizing Elasticsearch mappings (6.64 KB, patch)
2018-04-24 17:46 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 19707 - Add new operations for synchronizing Elasticsearch mappings (5.54 KB, patch)
2018-05-07 11:51 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 19707 - Add new operations for synchronizing Elasticsearch mappings (6.74 KB, patch)
2018-05-07 11:56 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 19707 - Add new operations for synchronizing Elasticsearch mappings (11.06 KB, patch)
2018-09-19 13:13 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 19707 - Add new operations for synchronizing Elasticsearch mappings (10.92 KB, patch)
2018-09-19 13:42 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 19707 - Add new operations for synchronizing Elasticsearch mappings (10.94 KB, patch)
2018-11-30 14:33 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 19707 - Add new operations for synchronizing Elasticsearch mappings (10.10 KB, patch)
2020-09-14 12:37 UTC, David Gustafsson
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description David Gustafsson 2017-11-28 10:40:47 UTC
It would be nice to have the option to synchronize possible upstream mappings with local onces, without overwritning local changes, so wrote this patch to "merge", "revert" and "add" operations for synchronizing Elasticsearch mappings stored in database with mappings.yaml
Comment 1 David Gustafsson 2017-11-28 10:48:30 UTC
Created attachment 69414 [details] [review]
Bug 19707 - Add new operations for synchronizing Elasticsearch mappings

Add new "merge", "revert" and "add" operations for synchronizing
Elasticsearch mappings stored in database with mappings.yaml
Comment 2 David Gustafsson 2017-11-28 11:02:54 UTC
"add": Only new fields will be added (present in mappings.yaml but not in database), mappings for fields stored in database will be left untouched.

"revert": Mappings for all fields in mappings.yaml will be reverted, mappings restored to default onces, and local modifications of those mappings removed.

"merge": New mappings for fields present in mappings.yaml will be merged in, keeping any local modifications of marc targets (this could also include old problematic mappings later fixed in mappings.yaml, so should be used with some caution).
Comment 3 David Gustafsson 2018-04-12 13:16:09 UTC
Just saw that commit message contained no How to test, I should probably fix this and will try to do so in the near future.
Comment 4 Séverine Queune 2018-04-16 21:40:31 UTC
Hi David,
I take a look to your patch to see if I can manage without a test plan but I failed :)
I can test it in the next few days if you can provide a test plan !
Comment 5 David Gustafsson 2018-04-24 17:43:36 UTC
Created attachment 74819 [details] [review]
Bug 19707 - Add new operations for synchronizing Elasticsearch mappings

Add new "merge", "revert" and "add" operations for synchronizing
Elasticsearch mappings stored in database with mappings.yaml

How to test:

1)  Make sure mappings.yaml contains mappings not present in database.
    (Also be aware that steps below will result in changed search engine
    configuration, so use a test environment and/or make sure to
    backup database before starting)
2)  Go to "Administration" and "Search Engine Configuration"
3)  Append ?i_know_what_i_am_doing=1&op=add to url and perform request
    in browser
4)  Vefify that any new mappings in mappings.yaml has been added, but
    no current mappings has been modified
5)  Edit mappings.yaml and add a new mapping for some field (new item
    below "mappings:" key)
6)  Add a new mapping for the same field in the database (through admin ui)
7)  Append ?i_know_what_i_am_doing=1&op=merge to url and perform request
    in browser
8)  Verify that the new mapping from mappings.yaml has been added for this field,
    while the field added in database was preserved
10) Append ?i_know_what_i_am_doing=1&op=revert to url and perform request
    in browser
11) Vefify that the mapping previously added the field in database has
    been deleted, and the mappings for this field are identical to those
    in mappings.yaml

Sponsored-by: Gothenburg University Library
Comment 6 David Gustafsson 2018-04-24 17:46:17 UTC
Created attachment 74820 [details] [review]
Bug 19707 - Add new operations for synchronizing Elasticsearch mappings

Add new "merge", "revert" and "add" operations for synchronizing
Elasticsearch mappings stored in database with mappings.yaml

How to test:

1)  Make sure mappings.yaml contains mappings not present in database.
    (Also be aware that steps below will result in changed search engine
    configuration, so use a test environment and/or make sure to
    backup database before starting)
2)  Go to "Administration" and "Search Engine Configuration"
3)  Append ?i_know_what_i_am_doing=1&op=add to url and perform request
    in browser
4)  Verify that any new mappings in mappings.yaml has been added, but
    no current mappings has been modified or deleted
5)  Edit mappings.yaml and add a new mapping for some field (new item
    below "mappings:" key)
6)  Add a new mapping for the same field in the database (through admin ui)
7)  Append ?i_know_what_i_am_doing=1&op=merge to url and perform request
    in browser
8)  Verify that the new mapping from mappings.yaml has been added for this field,
    while the field added in database was preserved
10) Append ?i_know_what_i_am_doing=1&op=revert to url and perform request
    in browser
11) Verify that the mapping previously added the field in database has
    been deleted, and the mappings for this field are identical to those
    in mappings.yaml

Sponsored-by: Gothenburg University Library
Comment 7 David Gustafsson 2018-04-24 17:48:33 UTC
Corrected some spelling mistakes in commit message.

@Séverine Queune: thankgs for taking an interest, there now is a test-plan.
Comment 8 Séverine Queune 2018-04-27 13:30:41 UTC
Hi David,
Thank you for your dev, it's a useful one so it's natural to be interested by it :)
I can't apply it on master branch (17.12.00.040) :

74820 - Bug 19707 - Add new operations for synchronizing Elasticsearch mappings

Apply? [(y)es, (n)o, (i)nteractive] y
Applying: Bug 19707 - Add new operations for synchronizing Elasticsearch mappings
Using index info to reconstruct a base tree...
M	Koha/SearchEngine/Elasticsearch.pm
Falling back to patching base and 3-way merge...
Auto-merging Koha/SearchEngine/Elasticsearch.pm
CONFLICT (content): Merge conflict in Koha/SearchEngine/Elasticsearch.pm
Failed to merge in the changes.
Patch failed at 0001 Bug 19707 - Add new operations for synchronizing Elasticsearch mappings
The copy of the patch that failed is found in:
   /var/repositories/koha/.git/rebase-apply/patch
Comment 9 David Gustafsson 2018-05-07 11:51:50 UTC
Created attachment 75107 [details] [review]
Bug 19707 - Add new operations for synchronizing Elasticsearch mappings

Add new "merge", "revert" and "add" operations for synchronizing
Elasticsearch mappings stored in database with mappings.yaml
Comment 10 David Gustafsson 2018-05-07 11:56:05 UTC
Created attachment 75108 [details] [review]
Bug 19707 - Add new operations for synchronizing Elasticsearch mappings

Add new "merge", "revert" and "add" operations for synchronizing
Elasticsearch mappings stored in database with mappings.yaml

How to test:

1)  Make sure mappings.yaml contains mappings not present in database.
    (Also be aware that steps below will result in changed search engine
    configuration, so use a test environment and/or make sure to
    backup database before starting)
2)  Go to "Administration" and "Search Engine Configuration"
3)  Append ?i_know_what_i_am_doing=1&op=add to url and perform request
    in browser
4)  Verify that any new mappings in mappings.yaml has been added, but
    no current mappings has been modified or deleted
5)  Edit mappings.yaml and add a new mapping for some field (new item
    below "mappings:" key)
6)  Add a new mapping for the same field in the database (through admin ui)
7)  Append ?i_know_what_i_am_doing=1&op=merge to url and perform request
    in browser
8)  Verify that the new mapping from mappings.yaml has been added for this field,
    while the field added in database was preserved
10) Append ?i_know_what_i_am_doing=1&op=revert to url and perform request
    in browser
11) Verify that the mapping previously added the field in database has
    been deleted, and the mappings for this field are identical to those
    in mappings.yaml

Sponsored-by: Gothenburg University Library
Comment 11 David Gustafsson 2018-05-07 11:56:48 UTC
Rebased against Koha master.
Comment 12 David Gustafsson 2018-09-19 13:13:46 UTC
Created attachment 79109 [details] [review]
Bug 19707 - Add new operations for synchronizing Elasticsearch mappings

Add new "merge", "revert" and "add" operations for synchronizing
Elasticsearch mappings stored in database with mappings.yaml

How to test:

1)  Make sure mappings.yaml contains mappings not present in database.
    (Also be aware that steps below will result in changed search engine
    configuration, so use a test environment and/or make sure to
    backup database before starting)
2)  Go to "Administration" and "Search Engine Configuration"
3)  Append ?i_know_what_i_am_doing=1&op=add to url and perform request
    in browser
4)  Verify that any new mappings in mappings.yaml has been added, but
    no current mappings has been modified or deleted
5)  Edit mappings.yaml and add a new mapping for some field (new item
    below "mappings:" key)
6)  Add a new mapping for the same field in the database (through admin ui)
7)  Append ?i_know_what_i_am_doing=1&op=merge to url and perform request
    in browser
8)  Verify that the new mapping from mappings.yaml has been added for this field,
    while the field added in database was preserved
10) Append ?i_know_what_i_am_doing=1&op=revert to url and perform request
    in browser
11) Verify that the mapping previously added the field in database has
    been deleted, and the mappings for this field are identical to those
    in mappings.yaml

Sponsored-by: Gothenburg University Library
Comment 13 David Gustafsson 2018-09-19 13:14:27 UTC
Something when wrong with the previous rebase, so rebased again and added buttons for all the new operations.
Comment 14 David Gustafsson 2018-09-19 13:42:35 UTC
Created attachment 79112 [details] [review]
Bug 19707 - Add new operations for synchronizing Elasticsearch mappings

Add new "merge", "revert" and "add" operations for synchronizing
Elasticsearch mappings stored in database with mappings.yaml

How to test:

1)  Make sure mappings.yaml contains mappings not present in database.
    (Also be aware that steps below will result in changed search engine
    configuration, so use a test environment and/or make sure to
    backup database before starting)
2)  Go to "Administration" and "Search Engine Configuration"
3)  Click "Add mappings" and confirm
4)  Verify that any new mappings in mappings.yaml has been added, but
    no current mappings has been modified or deleted
5)  Edit mappings.yaml and add a new mapping for some field (new item
    below "mappings:" key)
6)  Add a new mapping for the same field in the database (through admin ui)
7)  Click "Merge mappings" and confirm
8)  Verify that the new mapping from mappings.yaml has been added for this field,
    while the field added in database was preserved
10)  Click "Revert mappings" and confirm
11) Verify that the mapping previously added the field in database has
    been deleted, and the mappings for this field are identical to those
    in mappings.yaml

Sponsored-by: Gothenburg University Library
Comment 15 David Gustafsson 2018-09-19 13:42:56 UTC
Fixed commit-message to reflect latest changes.
Comment 16 David Gustafsson 2018-11-30 14:33:24 UTC
Created attachment 82795 [details] [review]
Bug 19707 - Add new operations for synchronizing Elasticsearch mappings

Add new "merge", "revert" and "add" operations for synchronizing
Elasticsearch mappings stored in database with mappings.yaml

How to test:

1)  Make sure mappings.yaml contains mappings not present in database.
    (Also be aware that steps below will result in changed search engine
    configuration, so use a test environment and/or make sure to
    backup database before starting)
2)  Go to "Administration" and "Search Engine Configuration"
3)  Click "Add mappings" and confirm
4)  Verify that any new mappings in mappings.yaml has been added, but
    no current mappings has been modified or deleted
5)  Edit mappings.yaml and add a new mapping for some field (new item
    below "mappings:" key)
6)  Add a new mapping for the same field in the database (through admin ui)
7)  Click "Merge mappings" and confirm
8)  Verify that the new mapping from mappings.yaml has been added for this field,
    while the field added in database was preserved
10)  Click "Revert mappings" and confirm
11) Verify that the mapping previously added the field in database has
    been deleted, and the mappings for this field are identical to those
    in mappings.yaml

Sponsored-by: Gothenburg University Library
Comment 17 David Gustafsson 2018-11-30 14:33:51 UTC
Rebase against master.
Comment 18 David Gustafsson 2020-09-14 12:37:03 UTC
Created attachment 110054 [details] [review]
Bug 19707 - Add new operations for synchronizing Elasticsearch mappings

Add new "merge", "revert" and "add" operations for synchronizing
Elasticsearch mappings stored in database with mappings.yaml

How to test:

1)  Make sure mappings.yaml contains mappings not present in database.
    (Also be aware that steps below will result in changed search engine
    configuration, so use a test environment and/or make sure to
    backup database before starting)
2)  Go to "Administration" and "Search Engine Configuration"
3)  Click "Add mappings" and confirm
4)  Verify that any new mappings in mappings.yaml has been added, but
    no current mappings has been modified or deleted
5)  Edit mappings.yaml and add a new mapping for some field (new item
    below "mappings:" key)
6)  Add a new mapping for the same field in the database (through admin ui)
7)  Click "Merge mappings" and confirm
8)  Verify that the new mapping from mappings.yaml has been added for this field,
    while the field added in database was preserved
10)  Click "Revert mappings" and confirm
11) Verify that the mapping previously added the field in database has
    been deleted, and the mappings for this field are identical to those
    in mappings.yaml

Sponsored-by: Gothenburg University Library
Comment 19 David Gustafsson 2020-09-14 12:37:27 UTC
Rebased against master.
Comment 20 Séverine Queune 2020-10-27 10:24:20 UTC
I've got an issue with merging option (steps 6-8) : I can add a new mapping in UI, no problem for saving it but after merging, the field disappear form UI and (obviously) it's not added to yaml file.
I didn't succeed to revert the mapping too, but maybe this option is only available combined with the merge option ?
Just for my curiosity, how the modifications are stored to enable a revert ?

This is such a great feature !!! Can't wait to see it in master !