This patchset aims to add the ability to leverage the ES alias feature during a reindex - when doing a full reindex or a deletion the indexer will look for an existing index under the alias - if not found it will create a new one, if found it will index into a new index and point the alias there when done and then delete the old index.
Created attachment 65086 [details] [review] Bug 18948 - Elasticsearch - Reindexes should use aliases to avoid down time while reindexing To test: 1 - Delete your current indices for ES (if any) 2 - Reindex your records 3 - curl -XGET 'localhost:9200/koha_kohadev?pretty' 4 - Note indices are now of form koha_kohadev_########## 5 - The numbers there are a result of appending 'time' command to name 6 - Ensure searching works as before the patches 7 - Reindex your records with a commit setting of 1 (to slow things down) you may need a large numebr fo records 8 - Ensure searching works during reindex 9 - Ensure reindexing completes successfully 10 - curl -XGET 'localhost:9200/koha_kohadev?pretty' 11 - There should only be one index each for biblios and authorities ** - Note number in above 12 - Perform a partial reindex by passing a biblionumber to the command 13 - Curl the indices again and ensure number/name has not changed
TODO: 1 - Ensure if records are indexed incrementally before first rebuild that an alias is used 2 - Unit tests Other comments welcome, this was first round proof of concept
Created attachment 65147 [details] [review] Bug 18948 - Elasticsearch - Reindexes should use aliases to avoid down time while reindexing Refactoring of rebuild_elastic_search.pl to move code into Koha::SearchEngine::Elasticsearch::Indexer
Created attachment 65159 [details] [review] Bug 18948 - Elasticsearch - Reindexes should use aliases to avoid down time while reindexing To test: 1 - Delete your current indices for ES (if any) 2 - Reindex your records 3 - curl 'localhost:9200/_cat/indices?v' 4 - Note indices are now of form koha_kohadev_########## 5 - The numbers there are a result of appending 'time' command to name 6 - Ensure searching works as before the patches 7 - Reindex your records with a commit setting of 1 (to slow things down) you may need a large numebr fo records 8 - Ensure searching works during reindex 9 - Ensure reindexing completes successfully 10 - curl 'localhost:9200/_cat/indices?v' 11 - There should only be one index each for biblios and authorities ** - Note number in above 12 - Perform a partial reindex by passing a biblionumber to the command 13 - Curl the indices again and ensure number/name has not changed 14 - Delete all indexes curl -XDELETE 'localhost:9200/koha_kohadev_*?pretty' 15 - Edit a known biblio: http://localhost:8081/cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber=1 16 - Add a word to the title "potato" 17 - Save the record 18 - Search for potato 19 - You should arrive at the detail page for that record 20 - Confirm your index is of form koha_kohadev_##########
Needs unit tests
Patch does not apply, SHA1 error for Indexer.pm
Created attachment 70815 [details] [review] Bug 18948 - Elasticsearch - Reindexes should use aliases to avoid down time while reindexing Refactoring of rebuild_elastic_search.pl to move code into Koha::SearchEngine::Elasticsearch::Indexer Current status: Patch doesn't apply
Created attachment 70816 [details] [review] Bug 18948 - Elasticsearch - Reindexes should use aliases to avoid down time while reindexing To test: 1 - Delete your current indices for ES (if any) 2 - Reindex your records 3 - curl 'localhost:9200/_cat/indices?v' 4 - Note indices are now of form koha_kohadev_########## 5 - The numbers there are a result of appending 'time' command to name 6 - Ensure searching works as before the patches 7 - Reindex your records with a commit setting of 1 (to slow things down) you may need a large numebr fo records 8 - Ensure searching works during reindex 9 - Ensure reindexing completes successfully 10 - curl 'localhost:9200/_cat/indices?v' 11 - There should only be one index each for biblios and authorities ** - Note number in above 12 - Perform a partial reindex by passing a biblionumber to the command 13 - Curl the indices again and ensure number/name has not changed 14 - Delete all indexes curl -XDELETE 'localhost:9200/koha_kohadev_*?pretty' 15 - Edit a known biblio: http://localhost:8081/cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber=1 16 - Add a word to the title "potato" 17 - Save the record 18 - Search for potato 19 - You should arrive at the detail page for that record 20 - Confirm your index is of form koha_kohadev_##########
Ugly rebase, still needs unit tests and testing to confirm it works
I'd like to see an option that uses the existing index also when reindexing all records. In my experience there are cases where you want to see the changes as soon as possible and not wait until the whole indexing process is complete. Testing indexing is just one of them but pretty important during development.
Additionally, if aliases are used, the there should be a mechanism to take care of indexing all the changes that were made in staff UI or such during the reindexing process. Otherwise they'd end up existing only in the old index that gets deleted, right?
Not working on this currently, leaving it for anyone who wants to look
+1 it would be great to use alias for not loosing search results during full reindexing \o/