Bug 20248

Summary: Elasticsearch - Improvements to mappings UI and indexing script
Product: Koha Reporter: Ere Maijala <ere.maijala>
Component: Searching - ElasticsearchAssignee: Ere Maijala <ere.maijala>
Status: RESOLVED FIXED QA Contact:
Severity: enhancement    
Priority: P5 - low CC: alex.arnaud, bouzid.fergani, claire.hernandez, fridolin.somers, jonathan.druart, josef.moravec, julian.maurice, martin.renvoize, nick
Version: master   
Hardware: All   
OS: All   
Whiteboard:
Change sponsored?: --- Patch complexity: ---
Bot Control: --- When did the bot last check this:
Who signed the patch off: Text to go in the release notes:
Attachments: Improve Elasticsearch mapppings UI and rebuild_elastic_search.pl.
Bug 20248: Improve Elasticsearch mapppings UI and rebuild_elastic_search.pl.
Bug 20248 - Elasticsearch - Improvements to mappings UI and indexing script
Bug 20248 - Improve Elasticsearch mappings UI and rebuild_elastic_search.pl.
Bug 20248 - Improve Elasticsearch mappings UI and rebuild_elastic_search.pl.
Bug 20248: (follow-up) Improve Elasticsearch mappings UI and rebuild_elastic_search.pl
Bug 20248 - Improve Elasticsearch mappings UI and rebuild_elastic_search.pl.
Bug 20248: (follow-up) Improve Elasticsearch mappings UI and rebuild_elastic_search.pl
Bug 20248 - Improve Elasticsearch mappings UI and rebuild_elastic_search.pl.
Bug 20248 - Improve Elasticsearch mappings UI and rebuild_elastic_search.pl.
Bug 20248 - Improve Elasticsearch mappings UI and rebuild_elastic_search.pl.
Bug 20248 - Improve Elasticsearch mappings UI and rebuild_elastic_search.pl.
Bug 20248: (QA follow-up) Remove unnecessary stuff
Bug 20248: Improve Elasticsearch mappings UI and rebuild_elastic_search.pl
Bug 20248: (QA follow-up) Remove unnecessary stuff

Description Ere Maijala 2018-02-19 14:57:08 UTC

    
Comment 1 Ere Maijala 2018-02-19 14:58:36 UTC
Created attachment 71954 [details] [review]
Improve Elasticsearch mapppings UI and rebuild_elastic_search.pl.
Comment 2 Ere Maijala 2018-02-19 15:01:51 UTC
The patch provides the following improvements:

1.) Mappings UI now has button that allows one to reset the mappings.
2.) Mappings UI displays the items in alphabetical order.
3.) Indexing script drops and recreates the index right away, which helps prevent ES from autocreating a bad index if someone does something while the first batch of records is being processed.
4.) Indexing script has nicer output.
Comment 3 Alex Arnaud 2018-03-13 09:26:13 UTC
I think this should be splited into 2 parts. One for the indexing things and one for what is related to mapping UI. Are you ok?
Comment 4 Ere Maijala 2018-03-13 09:37:37 UTC
I put all this in a single patch because the UI requires the indexer changes, but I'm fine with splitting it into two parts if that's required. It's not a large patch as it is, though.
Comment 5 claire.hernandez@biblibre.com 2018-03-13 13:07:43 UTC
1.) Mappings UI now has button that allows one to reset the mappings.

It works for me. Maybe, we could be a little more specific in the confirm form. Something like "The current mappings you seen on the screen will be erased and relaced by the mappings in the mappings.yaml file"

2.) Mappings UI displays the items in alphabetical order.

Ok it works for me.

3.) Indexing script drops and recreates the index right away, which helps prevent ES from autocreating a bad index if someone does something while the first batch of records is being processed.

I don't understand this part.

4.) Indexing script has nicer output.

Ok for me.

I cannot SO with the doubt on 3. Thanks for your patch :)
Comment 6 Ere Maijala 2018-03-13 13:32:52 UTC
3.) Currently the indexing script, when used with the -d parameter, does the following:
  1. Drops the existing index
  2. Start collecting records in a buffer
  3. Commit the buffer when it's full

The ES index will only be recreated at the first commit. This means that there's a time window between steps 1 and 3 during which it's possible that someone/something else saves a record, which causes the index to be automatically created by ES, but without all the settings it should have. That's why it's a good idea to recreate the index right away after dropping the old one. Another possibility is to instruct the ES instance to never autocreate indexes, but that's outside the scope of indexing here, and I believe it's in any case a good practice to at least have an empty but working index in place if the indexing script is interrupted before first commit.
Comment 7 claire.hernandez@biblibre.com 2018-03-13 15:47:15 UTC
Created attachment 72801 [details] [review]
Bug 20248: Improve Elasticsearch mapppings UI and rebuild_elastic_search.pl.

Improvements:
1) Mappings UI now has button that allows one to reset the mappings.
2) Mappings UI displays the items in alphabetical order.
3) Indexing script drops and recreates the index right away, which
helps prevent ES from autocreating a bad index if someone does something
while the first batch of records is being processed.
4) Indexing script has nicer output.

To test:
1) Change mappings.yaml file
2) Reset mappings in UI in mappings.pl
3) Verify the mappings have been changed in UI
4) The field order is alphabetical
5) Rebuild script has an clean output

Signed-off-by: Claire Hernandez <claire.hernandez@biblibre.com>
Comment 8 Alex Arnaud 2018-03-13 16:15:30 UTC
Here is my first QA review:

1) please check indenting, mainly in templates (i see some "2 spaces"),
2) you could provide unit tests for create_index function. Take a look at one wrote for update_index in t/db_dependent/Koha_Elasticsearch_Indexer.t. Don't forget to delete this "test" index after (which is not done for update_index).
Comment 9 Ere Maijala 2018-03-14 10:58:33 UTC
Thanks for comments.

1) I tried to use the prevalent style in the file, but I'll change the indentation. 
2) I'll add a test.
Comment 10 Ere Maijala 2018-03-14 13:24:34 UTC
Created attachment 72861 [details] [review]
Bug 20248 - Elasticsearch - Improvements to mappings UI and indexing script

Improvements:
1) Mappings UI now has button that allows one to reset the mappings.
2) Mappings UI displays the items in alphabetical order.
3) Indexing script drops and recreates the index right away, which
helps prevent ES from autocreating a bad index if someone does something
while the first batch of records is being processed.
4) Indexing script has nicer output.

To test:
1) Change mappings.yaml file
2) Reset mappings in UI in mappings.pl
3) Verify the mappings have been changed in UI
4) The field order is alphabetical
5) Rebuild script has clean output
6) Run test t/db_dependent/Koha_Elasticsearch_Indexer.t
Comment 11 Ere Maijala 2018-03-14 13:26:15 UTC
I think I managed to address the issues.

1) I used correct indentation where it was possible without changing large blocks.
2) Added. Is this how it's supposed to be?
Comment 12 Nicolas Legrand 2018-03-15 08:43:42 UTC
Created attachment 72907 [details] [review]
Bug 20248 - Improve Elasticsearch mappings UI and rebuild_elastic_search.pl.

Improvements:
1) Mappings UI now has button that allows one to reset the mappings.
2) Mappings UI displays the items in alphabetical order.
3) Indexing script drops and recreates the index right away, which
helps prevent ES from autocreating a bad index if someone does something
while the first batch of records is being processed.
4) Indexing script has nicer output.

To test:
1) Change mappings.yaml file
2) Reset mappings in UI in mappings.pl
3) Verify the mappings have been changed in UI
4) The field order is alphabetical
5) Rebuild script has clean output
6) Run test t/db_dependent/Koha_Elasticsearch_Indexer.t

Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Comment 13 Alex Arnaud 2018-03-15 10:12:16 UTC
(In reply to Ere Maijala from comment #11)
> I think I managed to address the issues.
> 
> 1) I used correct indentation where it was possible without changing large
> blocks.
> 2) Added. Is this how it's supposed to be?
Can you change the index name please? biblios => biblios_test for example. biblios is the one we use in real
Comment 14 claire.hernandez@biblibre.com 2018-03-15 10:22:02 UTC
Created attachment 72921 [details] [review]
Bug 20248 - Improve Elasticsearch mappings UI and rebuild_elastic_search.pl.

Improvements:
1) Mappings UI now has button that allows one to reset the mappings.
2) Mappings UI displays the items in alphabetical order.
3) Indexing script drops and recreates the index right away, which
helps prevent ES from autocreating a bad index if someone does something
while the first batch of records is being processed.
4) Indexing script has nicer output.

To test:
1) Change mappings.yaml file
2) Reset mappings in UI in mappings.pl
3) Verify the mappings have been changed in UI
4) The field order is alphabetical
5) Rebuild script has clean output
6) Run test t/db_dependent/Koha_Elasticsearch_Indexer.t

Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Comment 15 claire.hernandez@biblibre.com 2018-03-15 10:22:11 UTC
Created attachment 72922 [details] [review]
Bug 20248: (follow-up) Improve Elasticsearch mappings UI and rebuild_elastic_search.pl

- change index name
Comment 16 Alex Arnaud 2018-03-15 11:17:12 UTC
Created attachment 72932 [details] [review]
Bug 20248 - Improve Elasticsearch mappings UI and rebuild_elastic_search.pl.

Improvements:
1) Mappings UI now has button that allows one to reset the mappings.
2) Mappings UI displays the items in alphabetical order.
3) Indexing script drops and recreates the index right away, which
helps prevent ES from autocreating a bad index if someone does something
while the first batch of records is being processed.
4) Indexing script has nicer output.

To test:
1) Change mappings.yaml file
2) Reset mappings in UI in mappings.pl
3) Verify the mappings have been changed in UI
4) The field order is alphabetical
5) Rebuild script has clean output
6) Run test t/db_dependent/Koha_Elasticsearch_Indexer.t

Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Comment 17 Alex Arnaud 2018-03-15 11:17:19 UTC
Created attachment 72933 [details] [review]
Bug 20248: (follow-up) Improve Elasticsearch mappings UI and rebuild_elastic_search.pl

- change index name

Signed-off-by: Koha User <support@biblibre.com>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Comment 18 Alex Arnaud 2018-03-15 11:19:04 UTC
Ok for me. So, passed QA
Comment 19 Jonathan Druart 2018-03-27 20:50:32 UTC
It does not look correct to me: https://screenshots.firefox.com/EeG5M4P78NVOdnr5/pro.kohadev.org

- Nothing is displayed on the screen
- The warning should not be displayed
Comment 20 Ere Maijala 2018-03-28 07:36:44 UTC
Created attachment 73374 [details] [review]
Bug 20248 - Improve Elasticsearch mappings UI and rebuild_elastic_search.pl.

Improvements:
1) Mappings UI now has button that allows one to reset the mappings.
2) Mappings UI displays the items in alphabetical order.
3) Indexing script drops and recreates the index right away, which
helps prevent ES from autocreating a bad index if someone does something
while the first batch of records is being processed.
4) Indexing script has nicer output.

To test:
1) Change mappings.yaml file
2) Reset mappings in UI in mappings.pl
3) Verify the mappings have been changed in UI
4) The field order is alphabetical
5) Rebuild script has clean output
6) Run test t/db_dependent/Koha_Elasticsearch_Indexer.t
Comment 21 Ere Maijala 2018-03-28 07:39:00 UTC
Alex, please note that the test should already append __test to any index name (see line 71). I don't think 'biblios' in the new() call should be modified. It's always either biblios or authorities (might be better to use the constants, but the string was used in other places so I kept it).
Comment 22 Ere Maijala 2018-03-28 07:40:33 UTC
Jonathan, thanks, that was indeed bad. Is it ok now? I wasn't quite sure what you meant with "The warning should not be displayed".
Comment 23 Jonathan Druart 2018-03-28 13:27:38 UTC
(In reply to Ere Maijala from comment #22)
> Jonathan, thanks, that was indeed bad. Is it ok now? I wasn't quite sure
> what you meant with "The warning should not be displayed".

I was talking about this warning:
"Warning: Any changes to the configuration [...]"

I think it does not make sense to display it on the confirmation screen.
Comment 24 Bouzid Fergani 2018-04-06 18:46:17 UTC Comment hidden (obsolete)
Comment 25 Bouzid Fergani 2018-04-06 18:55:18 UTC
Created attachment 73798 [details] [review]
Bug 20248 - Improve Elasticsearch mappings UI and rebuild_elastic_search.pl.

Improvements:
1) Mappings UI now has button that allows one to reset the mappings.
2) Mappings UI displays the items in alphabetical order.
3) Indexing script drops and recreates the index right away, which
helps prevent ES from autocreating a bad index if someone does something
while the first batch of records is being processed.
4) Indexing script has nicer output.

To test:
1) Change mappings.yaml file
2) Reset mappings in UI in mappings.pl
3) Verify the mappings have been changed in UI
4) The field order is alphabetical
5) Rebuild script has clean output
6) Run test t/db_dependent/Koha_Elasticsearch_Indexer.t

Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Comment 26 Julian Maurice 2018-08-08 12:16:58 UTC
(In reply to Jonathan Druart from comment #23)
> (In reply to Ere Maijala from comment #22)
> > Jonathan, thanks, that was indeed bad. Is it ok now? I wasn't quite sure
> > what you meant with "The warning should not be displayed".
> 
> I was talking about this warning:
> "Warning: Any changes to the configuration [...]"
> 
> I think it does not make sense to display it on the confirmation screen.

Why not ? Resetting indexes is a change to the configuration, so the warning is  relevant in my opinion.
Comment 27 Julian Maurice 2018-08-08 12:38:49 UTC
Created attachment 77570 [details] [review]
Bug 20248 - Improve Elasticsearch mappings UI and rebuild_elastic_search.pl.

Improvements:
1) Mappings UI now has button that allows one to reset the mappings.
2) Mappings UI displays the items in alphabetical order.
3) Indexing script drops and recreates the index right away, which
helps prevent ES from autocreating a bad index if someone does something
while the first batch of records is being processed.
4) Indexing script has nicer output.

To test:
1) Change mappings.yaml file
2) Reset mappings in UI in mappings.pl
3) Verify the mappings have been changed in UI
4) The field order is alphabetical
5) Rebuild script has clean output
6) Run test t/db_dependent/Koha_Elasticsearch_Indexer.t

Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Comment 28 Julian Maurice 2018-08-08 12:38:54 UTC
Created attachment 77571 [details] [review]
Bug 20248: (QA follow-up) Remove unnecessary stuff

- Remove useless 'action' attribute in forms
- $op should never be equal to 'reset', so remove the test
- Move 'op' parameter inside <button>s to avoid having different HTML
  tags for elements that have the same kind of behaviour

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Comment 29 Julian Maurice 2018-08-08 12:39:45 UTC
Created attachment 77572 [details] [review]
Bug 20248: Improve Elasticsearch mappings UI and rebuild_elastic_search.pl

Improvements:
1) Mappings UI now has button that allows one to reset the mappings.
2) Mappings UI displays the items in alphabetical order.
3) Indexing script drops and recreates the index right away, which
helps prevent ES from autocreating a bad index if someone does something
while the first batch of records is being processed.
4) Indexing script has nicer output.

To test:
1) Change mappings.yaml file
2) Reset mappings in UI in mappings.pl
3) Verify the mappings have been changed in UI
4) The field order is alphabetical
5) Rebuild script has clean output
6) Run test t/db_dependent/Koha_Elasticsearch_Indexer.t

Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Comment 30 Julian Maurice 2018-08-08 12:39:52 UTC
Created attachment 77573 [details] [review]
Bug 20248: (QA follow-up) Remove unnecessary stuff

- Remove useless 'action' attribute in forms
- $op should never be equal to 'reset', so remove the test
- Move 'op' parameter inside <button>s to avoid having different HTML
  tags for elements that have the same kind of behaviour

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Comment 31 Jonathan Druart 2018-08-08 15:51:27 UTC
(In reply to Julian Maurice from comment #26)
> (In reply to Jonathan Druart from comment #23)
> > (In reply to Ere Maijala from comment #22)
> > > Jonathan, thanks, that was indeed bad. Is it ok now? I wasn't quite sure
> > > what you meant with "The warning should not be displayed".
> > 
> > I was talking about this warning:
> > "Warning: Any changes to the configuration [...]"
> > 
> > I think it does not make sense to display it on the confirmation screen.
> 
> Why not ? Resetting indexes is a change to the configuration, so the warning
> is  relevant in my opinion.

I guess the important part was "on the confirmation screen".
Comment 32 Julian Maurice 2018-08-09 08:00:08 UTC
> > > I was talking about this warning:
> > > "Warning: Any changes to the configuration [...]"
> > > 
> > > I think it does not make sense to display it on the confirmation screen.
> > 
> > Why not ? Resetting indexes is a change to the configuration, so the warning
> > is  relevant in my opinion.
> 
> I guess the important part was "on the confirmation screen".

I've read this part too, and I don't understand why you would want to hide this warning on the confirmation screen. Can you explain why showing it does not make sense to you ?
Comment 33 Jonathan Druart 2018-08-09 14:33:55 UTC
(In reply to Julian Maurice from comment #32)
> > > > I was talking about this warning:
> > > > "Warning: Any changes to the configuration [...]"
> > > > 
> > > > I think it does not make sense to display it on the confirmation screen.
> > > 
> > > Why not ? Resetting indexes is a change to the configuration, so the warning
> > > is  relevant in my opinion.
> > 
> > I guess the important part was "on the confirmation screen".
> 
> I've read this part too, and I don't understand why you would want to hide
> this warning on the confirmation screen. Can you explain why showing it does
> not make sense to you ?

It was minor, just saying that 2 warnings on the same page seems wrong.
We could instead add a note about the full reindex to the confirmation box, like 'do not forget to reindex'.

Let's push as it and improve later if needed.
Comment 34 Nick Clemens 2018-08-22 15:00:06 UTC
Awesome work all!

Pushed to master for 18.11
Comment 35 Martin Renvoize 2018-08-24 12:36:11 UTC
Enhancement, this will not be backported to 18.05.x series.