Created attachment 71954 [details] [review] Improve Elasticsearch mapppings UI and rebuild_elastic_search.pl.
The patch provides the following improvements: 1.) Mappings UI now has button that allows one to reset the mappings. 2.) Mappings UI displays the items in alphabetical order. 3.) Indexing script drops and recreates the index right away, which helps prevent ES from autocreating a bad index if someone does something while the first batch of records is being processed. 4.) Indexing script has nicer output.
I think this should be splited into 2 parts. One for the indexing things and one for what is related to mapping UI. Are you ok?
I put all this in a single patch because the UI requires the indexer changes, but I'm fine with splitting it into two parts if that's required. It's not a large patch as it is, though.
1.) Mappings UI now has button that allows one to reset the mappings. It works for me. Maybe, we could be a little more specific in the confirm form. Something like "The current mappings you seen on the screen will be erased and relaced by the mappings in the mappings.yaml file" 2.) Mappings UI displays the items in alphabetical order. Ok it works for me. 3.) Indexing script drops and recreates the index right away, which helps prevent ES from autocreating a bad index if someone does something while the first batch of records is being processed. I don't understand this part. 4.) Indexing script has nicer output. Ok for me. I cannot SO with the doubt on 3. Thanks for your patch :)
3.) Currently the indexing script, when used with the -d parameter, does the following: 1. Drops the existing index 2. Start collecting records in a buffer 3. Commit the buffer when it's full The ES index will only be recreated at the first commit. This means that there's a time window between steps 1 and 3 during which it's possible that someone/something else saves a record, which causes the index to be automatically created by ES, but without all the settings it should have. That's why it's a good idea to recreate the index right away after dropping the old one. Another possibility is to instruct the ES instance to never autocreate indexes, but that's outside the scope of indexing here, and I believe it's in any case a good practice to at least have an empty but working index in place if the indexing script is interrupted before first commit.
Created attachment 72801 [details] [review] Bug 20248: Improve Elasticsearch mapppings UI and rebuild_elastic_search.pl. Improvements: 1) Mappings UI now has button that allows one to reset the mappings. 2) Mappings UI displays the items in alphabetical order. 3) Indexing script drops and recreates the index right away, which helps prevent ES from autocreating a bad index if someone does something while the first batch of records is being processed. 4) Indexing script has nicer output. To test: 1) Change mappings.yaml file 2) Reset mappings in UI in mappings.pl 3) Verify the mappings have been changed in UI 4) The field order is alphabetical 5) Rebuild script has an clean output Signed-off-by: Claire Hernandez <claire.hernandez@biblibre.com>
Here is my first QA review: 1) please check indenting, mainly in templates (i see some "2 spaces"), 2) you could provide unit tests for create_index function. Take a look at one wrote for update_index in t/db_dependent/Koha_Elasticsearch_Indexer.t. Don't forget to delete this "test" index after (which is not done for update_index).
Thanks for comments. 1) I tried to use the prevalent style in the file, but I'll change the indentation. 2) I'll add a test.
Created attachment 72861 [details] [review] Bug 20248 - Elasticsearch - Improvements to mappings UI and indexing script Improvements: 1) Mappings UI now has button that allows one to reset the mappings. 2) Mappings UI displays the items in alphabetical order. 3) Indexing script drops and recreates the index right away, which helps prevent ES from autocreating a bad index if someone does something while the first batch of records is being processed. 4) Indexing script has nicer output. To test: 1) Change mappings.yaml file 2) Reset mappings in UI in mappings.pl 3) Verify the mappings have been changed in UI 4) The field order is alphabetical 5) Rebuild script has clean output 6) Run test t/db_dependent/Koha_Elasticsearch_Indexer.t
I think I managed to address the issues. 1) I used correct indentation where it was possible without changing large blocks. 2) Added. Is this how it's supposed to be?
Created attachment 72907 [details] [review] Bug 20248 - Improve Elasticsearch mappings UI and rebuild_elastic_search.pl. Improvements: 1) Mappings UI now has button that allows one to reset the mappings. 2) Mappings UI displays the items in alphabetical order. 3) Indexing script drops and recreates the index right away, which helps prevent ES from autocreating a bad index if someone does something while the first batch of records is being processed. 4) Indexing script has nicer output. To test: 1) Change mappings.yaml file 2) Reset mappings in UI in mappings.pl 3) Verify the mappings have been changed in UI 4) The field order is alphabetical 5) Rebuild script has clean output 6) Run test t/db_dependent/Koha_Elasticsearch_Indexer.t Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
(In reply to Ere Maijala from comment #11) > I think I managed to address the issues. > > 1) I used correct indentation where it was possible without changing large > blocks. > 2) Added. Is this how it's supposed to be? Can you change the index name please? biblios => biblios_test for example. biblios is the one we use in real
Created attachment 72921 [details] [review] Bug 20248 - Improve Elasticsearch mappings UI and rebuild_elastic_search.pl. Improvements: 1) Mappings UI now has button that allows one to reset the mappings. 2) Mappings UI displays the items in alphabetical order. 3) Indexing script drops and recreates the index right away, which helps prevent ES from autocreating a bad index if someone does something while the first batch of records is being processed. 4) Indexing script has nicer output. To test: 1) Change mappings.yaml file 2) Reset mappings in UI in mappings.pl 3) Verify the mappings have been changed in UI 4) The field order is alphabetical 5) Rebuild script has clean output 6) Run test t/db_dependent/Koha_Elasticsearch_Indexer.t Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Created attachment 72922 [details] [review] Bug 20248: (follow-up) Improve Elasticsearch mappings UI and rebuild_elastic_search.pl - change index name
Created attachment 72932 [details] [review] Bug 20248 - Improve Elasticsearch mappings UI and rebuild_elastic_search.pl. Improvements: 1) Mappings UI now has button that allows one to reset the mappings. 2) Mappings UI displays the items in alphabetical order. 3) Indexing script drops and recreates the index right away, which helps prevent ES from autocreating a bad index if someone does something while the first batch of records is being processed. 4) Indexing script has nicer output. To test: 1) Change mappings.yaml file 2) Reset mappings in UI in mappings.pl 3) Verify the mappings have been changed in UI 4) The field order is alphabetical 5) Rebuild script has clean output 6) Run test t/db_dependent/Koha_Elasticsearch_Indexer.t Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Created attachment 72933 [details] [review] Bug 20248: (follow-up) Improve Elasticsearch mappings UI and rebuild_elastic_search.pl - change index name Signed-off-by: Koha User <support@biblibre.com> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Ok for me. So, passed QA
It does not look correct to me: https://screenshots.firefox.com/EeG5M4P78NVOdnr5/pro.kohadev.org - Nothing is displayed on the screen - The warning should not be displayed
Created attachment 73374 [details] [review] Bug 20248 - Improve Elasticsearch mappings UI and rebuild_elastic_search.pl. Improvements: 1) Mappings UI now has button that allows one to reset the mappings. 2) Mappings UI displays the items in alphabetical order. 3) Indexing script drops and recreates the index right away, which helps prevent ES from autocreating a bad index if someone does something while the first batch of records is being processed. 4) Indexing script has nicer output. To test: 1) Change mappings.yaml file 2) Reset mappings in UI in mappings.pl 3) Verify the mappings have been changed in UI 4) The field order is alphabetical 5) Rebuild script has clean output 6) Run test t/db_dependent/Koha_Elasticsearch_Indexer.t
Alex, please note that the test should already append __test to any index name (see line 71). I don't think 'biblios' in the new() call should be modified. It's always either biblios or authorities (might be better to use the constants, but the string was used in other places so I kept it).
Jonathan, thanks, that was indeed bad. Is it ok now? I wasn't quite sure what you meant with "The warning should not be displayed".
(In reply to Ere Maijala from comment #22) > Jonathan, thanks, that was indeed bad. Is it ok now? I wasn't quite sure > what you meant with "The warning should not be displayed". I was talking about this warning: "Warning: Any changes to the configuration [...]" I think it does not make sense to display it on the confirmation screen.
Created attachment 73796 [details] [review] Bug 20248 - Improve Elasticsearch mappings UI and rebuild_elastic_search.pl. Improvements: 1) Mappings UI now has button that allows one to reset the mappings. 2) Mappings UI displays the items in alphabetical order. 3) Indexing script drops and recreates the index right away, which helps prevent ES from autocreating a bad index if someone does something while the first batch of records is being processed. 4) Indexing script has nicer output. To test: 1) Change mappings.yaml file 2) Reset mappings in UI in mappings.pl 3) Verify the mappings have been changed in UI 4) The field order is alphabetical 5) Rebuild script has clean output 6) Run test t/db_dependent/Koha_Elasticsearch_Indexer.t Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Created attachment 73798 [details] [review] Bug 20248 - Improve Elasticsearch mappings UI and rebuild_elastic_search.pl. Improvements: 1) Mappings UI now has button that allows one to reset the mappings. 2) Mappings UI displays the items in alphabetical order. 3) Indexing script drops and recreates the index right away, which helps prevent ES from autocreating a bad index if someone does something while the first batch of records is being processed. 4) Indexing script has nicer output. To test: 1) Change mappings.yaml file 2) Reset mappings in UI in mappings.pl 3) Verify the mappings have been changed in UI 4) The field order is alphabetical 5) Rebuild script has clean output 6) Run test t/db_dependent/Koha_Elasticsearch_Indexer.t Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
(In reply to Jonathan Druart from comment #23) > (In reply to Ere Maijala from comment #22) > > Jonathan, thanks, that was indeed bad. Is it ok now? I wasn't quite sure > > what you meant with "The warning should not be displayed". > > I was talking about this warning: > "Warning: Any changes to the configuration [...]" > > I think it does not make sense to display it on the confirmation screen. Why not ? Resetting indexes is a change to the configuration, so the warning is relevant in my opinion.
Created attachment 77570 [details] [review] Bug 20248 - Improve Elasticsearch mappings UI and rebuild_elastic_search.pl. Improvements: 1) Mappings UI now has button that allows one to reset the mappings. 2) Mappings UI displays the items in alphabetical order. 3) Indexing script drops and recreates the index right away, which helps prevent ES from autocreating a bad index if someone does something while the first batch of records is being processed. 4) Indexing script has nicer output. To test: 1) Change mappings.yaml file 2) Reset mappings in UI in mappings.pl 3) Verify the mappings have been changed in UI 4) The field order is alphabetical 5) Rebuild script has clean output 6) Run test t/db_dependent/Koha_Elasticsearch_Indexer.t Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com> Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Created attachment 77571 [details] [review] Bug 20248: (QA follow-up) Remove unnecessary stuff - Remove useless 'action' attribute in forms - $op should never be equal to 'reset', so remove the test - Move 'op' parameter inside <button>s to avoid having different HTML tags for elements that have the same kind of behaviour Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Created attachment 77572 [details] [review] Bug 20248: Improve Elasticsearch mappings UI and rebuild_elastic_search.pl Improvements: 1) Mappings UI now has button that allows one to reset the mappings. 2) Mappings UI displays the items in alphabetical order. 3) Indexing script drops and recreates the index right away, which helps prevent ES from autocreating a bad index if someone does something while the first batch of records is being processed. 4) Indexing script has nicer output. To test: 1) Change mappings.yaml file 2) Reset mappings in UI in mappings.pl 3) Verify the mappings have been changed in UI 4) The field order is alphabetical 5) Rebuild script has clean output 6) Run test t/db_dependent/Koha_Elasticsearch_Indexer.t Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com> Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Created attachment 77573 [details] [review] Bug 20248: (QA follow-up) Remove unnecessary stuff - Remove useless 'action' attribute in forms - $op should never be equal to 'reset', so remove the test - Move 'op' parameter inside <button>s to avoid having different HTML tags for elements that have the same kind of behaviour Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
(In reply to Julian Maurice from comment #26) > (In reply to Jonathan Druart from comment #23) > > (In reply to Ere Maijala from comment #22) > > > Jonathan, thanks, that was indeed bad. Is it ok now? I wasn't quite sure > > > what you meant with "The warning should not be displayed". > > > > I was talking about this warning: > > "Warning: Any changes to the configuration [...]" > > > > I think it does not make sense to display it on the confirmation screen. > > Why not ? Resetting indexes is a change to the configuration, so the warning > is relevant in my opinion. I guess the important part was "on the confirmation screen".
> > > I was talking about this warning: > > > "Warning: Any changes to the configuration [...]" > > > > > > I think it does not make sense to display it on the confirmation screen. > > > > Why not ? Resetting indexes is a change to the configuration, so the warning > > is relevant in my opinion. > > I guess the important part was "on the confirmation screen". I've read this part too, and I don't understand why you would want to hide this warning on the confirmation screen. Can you explain why showing it does not make sense to you ?
(In reply to Julian Maurice from comment #32) > > > > I was talking about this warning: > > > > "Warning: Any changes to the configuration [...]" > > > > > > > > I think it does not make sense to display it on the confirmation screen. > > > > > > Why not ? Resetting indexes is a change to the configuration, so the warning > > > is relevant in my opinion. > > > > I guess the important part was "on the confirmation screen". > > I've read this part too, and I don't understand why you would want to hide > this warning on the confirmation screen. Can you explain why showing it does > not make sense to you ? It was minor, just saying that 2 warnings on the same page seems wrong. We could instead add a note about the full reindex to the confirmation box, like 'do not forget to reindex'. Let's push as it and improve later if needed.
Awesome work all! Pushed to master for 18.11
Enhancement, this will not be backported to 18.05.x series.