Created attachment 129697 [details] [review] Bug 29921: Allow to index in ES all records modified since a given date This patch adds an option `--modified-since` to rebuild_elasticsearch.pl If used, all records modified since the given date will be indexed. For biblios it uses biblio_metadata.timestamp (so items modification time is not taken into account). For authorities it uses auth_header.modification_time. Test plan: 1. Create several biblio records at different times (or update manually the columns specified above) 2. Look at the records modification time (SELECT timestamp FROM biblio_metadata ORDER BY timestamp) 3. Run misc/search_tools/rebuild_elasticsearch.pl -b -v --modified-since "$DATE" $DATE can be a date (YYYY-MM-DD) or datetime (YYYY-MM-DD HH:MM[:SS]) 4. Verify that the correct number of records have been indexed (do the same for authorities:) 5. Create several authority records at different times (or update manually the columns specified above) 6. Look at the records modification time (SELECT modification_time FROM auth_header ORDER BY modification_time) 7. Run misc/search_tools/rebuild_elasticsearch.pl -a -v --modified-since "$DATE" $DATE can be a date (YYYY-MM-DD) or datetime (YYYY-MM-DD HH:MM[:SS]) 8. Verify that the correct number of records have been indexed
Created attachment 130594 [details] [review] Bug 29921: Allow to index in ES all records modified since a given date This patch adds an option `--modified-since` to rebuild_elasticsearch.pl If used, all records modified since the given date will be indexed. For biblios it uses biblio_metadata.timestamp and items.timestamp For authorities it uses auth_header.modification_time. Test plan: 1. Create several biblio records and items at different times (or update manually the columns specified above) 2. Look at the records modification time: SELECT MAX(GREATEST(bm.timestamp, COALESCE(i.timestamp, 0))) ts FROM biblio_metadata bm LEFT JOIN items i ON (bm.biblionumber = i.biblionumber) GROUP BY bm.biblionumber ORDER BY ts 3. Run misc/search_tools/rebuild_elasticsearch.pl -b -v --modified-since "$DATE" $DATE can be a date (YYYY-MM-DD) or a datetime (YYYY-MM-DD HH:MM[:SS]) 4. Verify that the correct number of records have been indexed (do the same for authorities:) 5. Create several authority records at different times (or update manually the columns specified above) 6. Look at the records modification time: SELECT modification_time FROM auth_header ORDER BY modification_time 7. Run misc/search_tools/rebuild_elasticsearch.pl -a -v --modified-since "$DATE" $DATE can be a date (YYYY-MM-DD) or a datetime (YYYY-MM-DD HH:MM[:SS]) 8. Verify that the correct number of records have been indexed
I just tested (see test plan below): it works, thank you test plan : koha@test-koha:~$ mysql MariaDB [koha]> select max(timestamp) from items; +---------------------+ | max(timestamp) | +---------------------+ | 2022-02-02 10:02:20 | +---------------------+ 1 row in set (0.001 sec) MariaDB [koha]> select max(timestamp) from biblio_metadata; +---------------------+ | max(timestamp) | +---------------------+ | 2022-02-02 10:40:16 | +---------------------+ 1 row in set (0.000 sec) MariaDB [koha]> select max(modification_time) from auth_header; +------------------------+ | max(modification_time) | +------------------------+ | 2022-01-31 16:22:58 | +------------------------+ MariaDB [koha]> select biblionumber from items where itemnumber=842565; +--------------+ | biblionumber | +--------------+ | 509975 | +--------------+ 1 row in set (0.001 sec) MariaDB [koha]> update items set timestamp='2022-02-14 10:02:20' where itemnumber=842565; Query OK, 0 rows affected (0.001 sec) Rows matched: 1 Changed: 0 Warnings: 0 MariaDB [koha]> update biblio_metadata set timestamp='2022-02-14 10:02:20' where biblionumber=509974; Query OK, 0 rows affected (0.001 sec) Rows matched: 1 Changed: 0 Warnings: 0 MariaDB [koha]> update auth_header set modification_time='2022-02-14 10:02:20' where authid=426547; Query OK, 0 rows affected (0.001 sec) Rows matched: 1 Changed: 0 Warnings: 0 MariaDB [koha]> exit; Bye koha@test-koha:~$ ~/src/misc/search_tools/rebuild_elasticsearch.pl -b -v --modified-since '2022-02-14 10:02:19' [16518] Checking state of biblios index [16518] Indexing biblios [16518] Committing final records... [16518] Total 2 records indexed koha@test-koha:~$ ~/src/misc/search_tools/rebuild_elasticsearch.pl -a -v --modified-since '2022-02-14 10:02:19' [16529] Checking state of authorities index [16529] Indexing authorities [16529] Committing final records... [16529] Total 1 records indexed
This surely needs a unit test for changes in get_all_biblios_iterator()
Bug 29921 - Allow to index in Elasticsearch all records modified since a given date 130594 - Bug 29921: Allow to index in ES all records modified since a given date Apply? [(y)es, (n)o, (i)nteractive] y Applying: Bug 29921: Allow to index in ES all records modified since a given date Using index info to reconstruct a base tree... M Koha/BiblioUtils.pm Falling back to patching base and 3-way merge... Auto-merging Koha/BiblioUtils.pm CONFLICT (content): Merge conflict in Koha/BiblioUtils.pm error: Failed to merge in the changes. Patch failed at 0001 Bug 29921: Allow to index in ES all records modified since a given date hint: Use 'git am --show-current-patch=diff' to see the failed patch When you have resolved this problem run "git bz apply --continue". If you would prefer to skip this patch, instead run "git bz apply --skip". To restore the original branch and stop patching run "git bz apply --abort". Patch left in /tmp/Bug-29921-Allow-to-index-in-ES-all-records-modifie-ZvCefZ.patch
The idea is a little similar to bug 33243
Note Bug 35345 has been pushed