Bug 29921 - Allow to index in Elasticsearch all records modified since a given date
Summary: Allow to index in Elasticsearch all records modified since a given date
Status: Patch doesn't apply
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Julian Maurice
QA Contact:
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-01-21 13:45 UTC by Julian Maurice
Modified: 2024-04-22 10:44 UTC (History)
3 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 29921: Allow to index in ES all records modified since a given date (6.13 KB, patch)
2022-01-21 13:46 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 29921: Allow to index in ES all records modified since a given date (6.40 KB, patch)
2022-02-15 07:50 UTC, Julian Maurice
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Julian Maurice 2022-01-21 13:45:22 UTC

    
Comment 1 Julian Maurice 2022-01-21 13:46:07 UTC
Created attachment 129697 [details] [review]
Bug 29921: Allow to index in ES all records modified since a given date

This patch adds an option `--modified-since` to rebuild_elasticsearch.pl
If used, all records modified since the given date will be indexed.

For biblios it uses biblio_metadata.timestamp (so items
modification time is not taken into account).
For authorities it uses auth_header.modification_time.

Test plan:
1. Create several biblio records at different times (or update manually
   the columns specified above)
2. Look at the records modification time (SELECT timestamp FROM
   biblio_metadata ORDER BY timestamp)
3. Run misc/search_tools/rebuild_elasticsearch.pl -b -v --modified-since "$DATE"
   $DATE can be a date (YYYY-MM-DD) or datetime (YYYY-MM-DD HH:MM[:SS])
4. Verify that the correct number of records have been indexed

(do the same for authorities:)
5. Create several authority records at different times (or update
   manually the columns specified above)
6. Look at the records modification time (SELECT modification_time FROM
   auth_header ORDER BY modification_time)
7. Run misc/search_tools/rebuild_elasticsearch.pl -a -v --modified-since "$DATE"
   $DATE can be a date (YYYY-MM-DD) or datetime (YYYY-MM-DD HH:MM[:SS])
8. Verify that the correct number of records have been indexed
Comment 2 Julian Maurice 2022-02-15 07:50:36 UTC
Created attachment 130594 [details] [review]
Bug 29921: Allow to index in ES all records modified since a given date

This patch adds an option `--modified-since` to rebuild_elasticsearch.pl
If used, all records modified since the given date will be indexed.

For biblios it uses biblio_metadata.timestamp and items.timestamp
For authorities it uses auth_header.modification_time.

Test plan:
1. Create several biblio records and items at different times (or update
   manually the columns specified above)
2. Look at the records modification time:
    SELECT MAX(GREATEST(bm.timestamp, COALESCE(i.timestamp, 0))) ts
    FROM biblio_metadata bm
    LEFT JOIN items i ON (bm.biblionumber = i.biblionumber)
    GROUP BY bm.biblionumber
    ORDER BY ts
3. Run misc/search_tools/rebuild_elasticsearch.pl -b -v --modified-since "$DATE"
   $DATE can be a date (YYYY-MM-DD) or a datetime (YYYY-MM-DD HH:MM[:SS])
4. Verify that the correct number of records have been indexed

(do the same for authorities:)
5. Create several authority records at different times (or update
   manually the columns specified above)
6. Look at the records modification time:
    SELECT modification_time FROM auth_header ORDER BY modification_time
7. Run misc/search_tools/rebuild_elasticsearch.pl -a -v --modified-since "$DATE"
   $DATE can be a date (YYYY-MM-DD) or a datetime (YYYY-MM-DD HH:MM[:SS])
8. Verify that the correct number of records have been indexed
Comment 3 Stéphane Delaune 2022-02-15 08:51:31 UTC
I just tested (see test plan below): it works, thank you

test plan :
koha@test-koha:~$ mysql
MariaDB [koha]> select max(timestamp) from items;
+---------------------+
| max(timestamp)      |
+---------------------+
| 2022-02-02 10:02:20 |
+---------------------+
1 row in set (0.001 sec)

MariaDB [koha]> select max(timestamp) from biblio_metadata;
+---------------------+
| max(timestamp)      |
+---------------------+
| 2022-02-02 10:40:16 |
+---------------------+
1 row in set (0.000 sec)

MariaDB [koha]> select max(modification_time) from auth_header;
+------------------------+
| max(modification_time) |
+------------------------+
| 2022-01-31 16:22:58    |
+------------------------+

MariaDB [koha]> select biblionumber from items where itemnumber=842565;
+--------------+
| biblionumber |
+--------------+
|       509975 |
+--------------+
1 row in set (0.001 sec)

MariaDB [koha]> update items set timestamp='2022-02-14 10:02:20' where itemnumber=842565;
Query OK, 0 rows affected (0.001 sec)
Rows matched: 1  Changed: 0  Warnings: 0

MariaDB [koha]> update biblio_metadata set timestamp='2022-02-14 10:02:20' where biblionumber=509974;
Query OK, 0 rows affected (0.001 sec)
Rows matched: 1  Changed: 0  Warnings: 0

MariaDB [koha]> update auth_header set modification_time='2022-02-14 10:02:20' where authid=426547;
Query OK, 0 rows affected (0.001 sec)
Rows matched: 1  Changed: 0  Warnings: 0
MariaDB [koha]> exit;
Bye
koha@test-koha:~$ ~/src/misc/search_tools/rebuild_elasticsearch.pl -b -v --modified-since '2022-02-14 10:02:19'
[16518] Checking state of biblios index
[16518] Indexing biblios
[16518] Committing final records...
[16518] Total 2 records indexed
koha@test-koha:~$ ~/src/misc/search_tools/rebuild_elasticsearch.pl -a -v --modified-since '2022-02-14 10:02:19'
[16529] Checking state of authorities index
[16529] Indexing authorities
[16529] Committing final records...
[16529] Total 1 records indexed
Comment 4 Fridolin Somers 2022-11-30 06:26:28 UTC
This surely needs a unit test for changes in get_all_biblios_iterator()
Comment 5 solene.ngamga 2023-01-11 16:05:59 UTC
Bug 29921 - Allow to index in Elasticsearch all records modified since a given date

130594 - Bug 29921: Allow to index in ES all records modified since a given date

Apply? [(y)es, (n)o, (i)nteractive] y
Applying: Bug 29921: Allow to index in ES all records modified since a given date
Using index info to reconstruct a base tree...
M	Koha/BiblioUtils.pm
Falling back to patching base and 3-way merge...
Auto-merging Koha/BiblioUtils.pm
CONFLICT (content): Merge conflict in Koha/BiblioUtils.pm
error: Failed to merge in the changes.
Patch failed at 0001 Bug 29921: Allow to index in ES all records modified since a given date
hint: Use 'git am --show-current-patch=diff' to see the failed patch
When you have resolved this problem run "git bz apply --continue".
If you would prefer to skip this patch, instead run "git bz apply --skip".
To restore the original branch and stop patching run "git bz apply --abort".
Patch left in /tmp/Bug-29921-Allow-to-index-in-ES-all-records-modifie-ZvCefZ.patch
Comment 6 Katrin Fischer 2023-03-20 21:06:36 UTC
The idea is a little similar to bug 33243
Comment 7 Fridolin Somers 2024-04-22 10:44:11 UTC
Note Bug 35345 has been pushed