Bug 18948 - Reindexing should use aliases to avoid down time
Summary: Reindexing should use aliases to avoid down time
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Bugs List
QA Contact:
URL:
Keywords:
Depends on:
Blocks: 18950
  Show dependency treegraph
 
Reported: 2017-07-17 20:20 UTC by Nick Clemens
Modified: 2019-04-24 09:38 UTC (History)
10 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 18948 - Elasticsearch - Reindexes should use aliases to avoid down time while reindexing (7.45 KB, patch)
2017-07-17 20:26 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 18948 - Elasticsearch - Reindexes should use aliases to avoid down time while reindexing (9.42 KB, patch)
2017-07-20 13:32 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 18948 - Elasticsearch - Reindexes should use aliases to avoid down time while reindexing (13.50 KB, patch)
2017-07-20 20:08 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 18948 - Elasticsearch - Reindexes should use aliases to avoid down time while reindexing (9.42 KB, patch)
2018-01-23 00:55 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 18948 - Elasticsearch - Reindexes should use aliases to avoid down time while reindexing (7.60 KB, patch)
2018-01-23 00:55 UTC, Nick Clemens
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Nick Clemens 2017-07-17 20:20:03 UTC
This patchset aims to add the ability to leverage the ES alias feature during a reindex - when doing a full reindex or a deletion the indexer will look for an existing index under the alias - if not found it will create a new one, if found it will index into a new index and point the alias there when done and then delete the old index.
Comment 1 Nick Clemens 2017-07-17 20:26:43 UTC
Created attachment 65086 [details] [review]
Bug 18948 - Elasticsearch - Reindexes should use aliases to avoid down time while reindexing

To test:
1  - Delete your current indices for ES (if any)
2  - Reindex your records
3  - curl -XGET 'localhost:9200/koha_kohadev?pretty'
4  - Note indices are now of form koha_kohadev_##########
5  - The numbers there are a result of appending 'time' command to name
6  - Ensure searching works as before the patches
7  - Reindex your records with a commit setting of 1 (to slow things
        down) you may need a large numebr fo records
8  - Ensure searching works during reindex
9  - Ensure reindexing completes successfully
10 - curl -XGET 'localhost:9200/koha_kohadev?pretty'
11 - There should only be one index each for biblios and authorities
** - Note number in above
12 - Perform a partial reindex by passing a biblionumber to the command
13 - Curl the indices again and ensure number/name has not changed
Comment 2 Nick Clemens 2017-07-17 20:28:39 UTC
TODO:
1 - Ensure if records are indexed incrementally before first rebuild that an alias is used
2 - Unit tests

Other comments welcome, this was first round proof of concept
Comment 3 Nick Clemens 2017-07-20 13:32:49 UTC
Created attachment 65147 [details] [review]
Bug 18948 - Elasticsearch - Reindexes should use aliases to avoid down time while reindexing

Refactoring of rebuild_elastic_search.pl to move code into
Koha::SearchEngine::Elasticsearch::Indexer
Comment 4 Nick Clemens 2017-07-20 20:08:45 UTC
Created attachment 65159 [details] [review]
Bug 18948 - Elasticsearch - Reindexes should use aliases to avoid down time while reindexing

To test:
1  - Delete your current indices for ES (if any)
2  - Reindex your records
3  - curl 'localhost:9200/_cat/indices?v'
4  - Note indices are now of form koha_kohadev_##########
5  - The numbers there are a result of appending 'time' command to name
6  - Ensure searching works as before the patches
7  - Reindex your records with a commit setting of 1 (to slow things
        down) you may need a large numebr fo records
8  - Ensure searching works during reindex
9  - Ensure reindexing completes successfully
10 - curl 'localhost:9200/_cat/indices?v'
11 - There should only be one index each for biblios and authorities
** - Note number in above
12 - Perform a partial reindex by passing a biblionumber to the command
13 - Curl the indices again and ensure number/name has not changed
14 - Delete all indexes
    curl -XDELETE 'localhost:9200/koha_kohadev_*?pretty'
15 - Edit a known biblio:
    http://localhost:8081/cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber=1
16 - Add a word to the title "potato"
17 - Save the record
18 - Search for potato
19 - You should arrive at the detail page for that record
20 - Confirm your index is of form koha_kohadev_##########
Comment 5 Nick Clemens 2017-07-20 20:09:32 UTC
Needs unit tests
Comment 6 David Bourgault 2017-11-03 20:24:08 UTC
Patch does not apply, SHA1 error for Indexer.pm
Comment 7 Nick Clemens 2018-01-23 00:55:13 UTC
Created attachment 70815 [details] [review]
Bug 18948 - Elasticsearch - Reindexes should use aliases to avoid down time while reindexing

Refactoring of rebuild_elastic_search.pl to move code into
Koha::SearchEngine::Elasticsearch::Indexer


Current status: Patch doesn't apply
Comment 8 Nick Clemens 2018-01-23 00:55:16 UTC
Created attachment 70816 [details] [review]
Bug 18948 - Elasticsearch - Reindexes should use aliases to avoid down time while reindexing

To test:
1  - Delete your current indices for ES (if any)
2  - Reindex your records
3  - curl 'localhost:9200/_cat/indices?v'
4  - Note indices are now of form koha_kohadev_##########
5  - The numbers there are a result of appending 'time' command to name
6  - Ensure searching works as before the patches
7  - Reindex your records with a commit setting of 1 (to slow things
        down) you may need a large numebr fo records
8  - Ensure searching works during reindex
9  - Ensure reindexing completes successfully
10 - curl 'localhost:9200/_cat/indices?v'
11 - There should only be one index each for biblios and authorities
** - Note number in above
12 - Perform a partial reindex by passing a biblionumber to the command
13 - Curl the indices again and ensure number/name has not changed
14 - Delete all indexes
    curl -XDELETE 'localhost:9200/koha_kohadev_*?pretty'
15 - Edit a known biblio:
    http://localhost:8081/cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber=1
16 - Add a word to the title "potato"
17 - Save the record
18 - Search for potato
19 - You should arrive at the detail page for that record
20 - Confirm your index is of form koha_kohadev_##########
Comment 9 Nick Clemens 2018-01-23 00:56:32 UTC
Ugly rebase, still needs unit tests and testing to confirm it works
Comment 10 Ere Maijala 2018-05-28 07:57:25 UTC
I'd like to see an option that uses the existing index also when reindexing all records. In my experience there are cases where you want to see the changes as soon as possible and not wait until the whole indexing process is complete. Testing indexing is just one of them but pretty important during development.
Comment 11 Ere Maijala 2018-12-05 11:51:02 UTC
Additionally, if aliases are used, the there should be a mechanism to take care of indexing all the changes that were made in staff UI or such during the reindexing process. Otherwise they'd end up existing only in the old index that gets deleted, right?
Comment 12 Nick Clemens 2019-02-21 13:44:40 UTC
Not working on this currently, leaving it for anyone who wants to look
Comment 13 claire.hernandez@biblibre.com 2019-03-15 10:48:57 UTC
+1 it would be great to use alias for not loosing search results during full reindexing \o/