Bug 29921

Created attachment 129697 [details] [review]
Bug 29921: Allow to index in ES all records modified since a given date

This patch adds an option `--modified-since` to rebuild_elasticsearch.pl
If used, all records modified since the given date will be indexed.

For biblios it uses biblio_metadata.timestamp (so items
modification time is not taken into account).
For authorities it uses auth_header.modification_time.

Test plan:
1. Create several biblio records at different times (or update manually
   the columns specified above)
2. Look at the records modification time (SELECT timestamp FROM
   biblio_metadata ORDER BY timestamp)
3. Run misc/search_tools/rebuild_elasticsearch.pl -b -v --modified-since "$DATE"
   $DATE can be a date (YYYY-MM-DD) or datetime (YYYY-MM-DD HH:MM[:SS])
4. Verify that the correct number of records have been indexed

(do the same for authorities:)
5. Create several authority records at different times (or update
   manually the columns specified above)
6. Look at the records modification time (SELECT modification_time FROM
   auth_header ORDER BY modification_time)
7. Run misc/search_tools/rebuild_elasticsearch.pl -a -v --modified-since "$DATE"
   $DATE can be a date (YYYY-MM-DD) or datetime (YYYY-MM-DD HH:MM[:SS])
8. Verify that the correct number of records have been indexed

Created attachment 130594 [details] [review]
Bug 29921: Allow to index in ES all records modified since a given date

This patch adds an option `--modified-since` to rebuild_elasticsearch.pl
If used, all records modified since the given date will be indexed.

For biblios it uses biblio_metadata.timestamp and items.timestamp
For authorities it uses auth_header.modification_time.

Test plan:
1. Create several biblio records and items at different times (or update
   manually the columns specified above)
2. Look at the records modification time:
    SELECT MAX(GREATEST(bm.timestamp, COALESCE(i.timestamp, 0))) ts
    FROM biblio_metadata bm
    LEFT JOIN items i ON (bm.biblionumber = i.biblionumber)
    GROUP BY bm.biblionumber
    ORDER BY ts
3. Run misc/search_tools/rebuild_elasticsearch.pl -b -v --modified-since "$DATE"
   $DATE can be a date (YYYY-MM-DD) or a datetime (YYYY-MM-DD HH:MM[:SS])
4. Verify that the correct number of records have been indexed

(do the same for authorities:)
5. Create several authority records at different times (or update
   manually the columns specified above)
6. Look at the records modification time:
    SELECT modification_time FROM auth_header ORDER BY modification_time
7. Run misc/search_tools/rebuild_elasticsearch.pl -a -v --modified-since "$DATE"
   $DATE can be a date (YYYY-MM-DD) or a datetime (YYYY-MM-DD HH:MM[:SS])
8. Verify that the correct number of records have been indexed

I just tested (see test plan below): it works, thank you

test plan :
koha@test-koha:~$ mysql
MariaDB [koha]> select max(timestamp) from items;
+---------------------+
| max(timestamp)      |
+---------------------+
| 2022-02-02 10:02:20 |
+---------------------+
1 row in set (0.001 sec)

MariaDB [koha]> select max(timestamp) from biblio_metadata;
+---------------------+
| max(timestamp)      |
+---------------------+
| 2022-02-02 10:40:16 |
+---------------------+
1 row in set (0.000 sec)

MariaDB [koha]> select max(modification_time) from auth_header;
+------------------------+
| max(modification_time) |
+------------------------+
| 2022-01-31 16:22:58    |
+------------------------+

MariaDB [koha]> select biblionumber from items where itemnumber=842565;
+--------------+
| biblionumber |
+--------------+
|       509975 |
+--------------+
1 row in set (0.001 sec)

MariaDB [koha]> update items set timestamp='2022-02-14 10:02:20' where itemnumber=842565;
Query OK, 0 rows affected (0.001 sec)
Rows matched: 1  Changed: 0  Warnings: 0

MariaDB [koha]> update biblio_metadata set timestamp='2022-02-14 10:02:20' where biblionumber=509974;
Query OK, 0 rows affected (0.001 sec)
Rows matched: 1  Changed: 0  Warnings: 0

MariaDB [koha]> update auth_header set modification_time='2022-02-14 10:02:20' where authid=426547;
Query OK, 0 rows affected (0.001 sec)
Rows matched: 1  Changed: 0  Warnings: 0
MariaDB [koha]> exit;
Bye
koha@test-koha:~$ ~/src/misc/search_tools/rebuild_elasticsearch.pl -b -v --modified-since '2022-02-14 10:02:19'
[16518] Checking state of biblios index
[16518] Indexing biblios
[16518] Committing final records...
[16518] Total 2 records indexed
koha@test-koha:~$ ~/src/misc/search_tools/rebuild_elasticsearch.pl -a -v --modified-since '2022-02-14 10:02:19'
[16529] Checking state of authorities index
[16529] Indexing authorities
[16529] Committing final records...
[16529] Total 1 records indexed

This surely needs a unit test for changes in get_all_biblios_iterator()

Bug 29921 - Allow to index in Elasticsearch all records modified since a given date

130594 - Bug 29921: Allow to index in ES all records modified since a given date

Apply? [(y)es, (n)o, (i)nteractive] y
Applying: Bug 29921: Allow to index in ES all records modified since a given date
Using index info to reconstruct a base tree...
M	Koha/BiblioUtils.pm
Falling back to patching base and 3-way merge...
Auto-merging Koha/BiblioUtils.pm
CONFLICT (content): Merge conflict in Koha/BiblioUtils.pm
error: Failed to merge in the changes.
Patch failed at 0001 Bug 29921: Allow to index in ES all records modified since a given date
hint: Use 'git am --show-current-patch=diff' to see the failed patch
When you have resolved this problem run "git bz apply --continue".
If you would prefer to skip this patch, instead run "git bz apply --skip".
To restore the original branch and stop patching run "git bz apply --abort".
Patch left in /tmp/Bug-29921-Allow-to-index-in-ES-all-records-modifie-ZvCefZ.patch

The idea is a little similar to bug 33243

Note Bug 35345 has been pushed