Bug 33243 - Reindex only Biblios changed in the last x minutes in Elasticsearch (ES)
Summary: Reindex only Biblios changed in the last x minutes in Elasticsearch (ES)
Status: Needs Signoff
Alias: None
Product: Koha
Classification: Unclassified
Component: Command-line Utilities (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement
Assignee: Mark Hofstetter
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-03-16 12:13 UTC by Mark Hofstetter
Modified: 2024-04-24 10:48 UTC (History)
5 users (show)

See Also:
Change sponsored?: Sponsored
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments
Bug 33243 - Reindex only Biblios changed in the last x minutes in Elasticsearch (ES) (2.62 KB, patch)
2023-03-16 12:30 UTC, Mark Hofstetter
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Mark Hofstetter 2023-03-16 12:13:59 UTC
add a command line option to 

rebuild_elasticsearch.pl

to only reindex Biblios changed in the last x minutes
rebuild_elasticsearch.pl -v -b --minutes 100
Comment 1 Mark Hofstetter 2023-03-16 12:30:32 UTC
Created attachment 148259 [details] [review]
Bug 33243 - Reindex only Biblios changed in the last x minutes in Elasticsearch (ES)

add a command line option to

rebuild_elasticsearch.pl

to only reindex Biblios changed in the last x minutes
rebuild_elasticsearch.pl -v -b --minutes 100

Reason: we (and others) sometime encounter problem when records are created (especially without items) AND are not automatically indexed in ES

To test:

instance has to use elasticsearch

0. apply patch
1. create or change new record (biblio)
2. run "rebuild_elasticsearch.pl -v -b --minutes 10"
3. if nothing else "happened" on this intance in the last 10 minutes, this command
should yield "1 record indexed"
4. record should be found in Web Search

Sponsored-by: Wien Museum
Comment 2 Katrin Fischer 2023-03-17 20:09:12 UTC
I know this is a quick fix and workaround, but it feels a bit like a step back. We had this behavior for Zebra, tean me moved on to the Koha indexer that gave us almost immediate indexing. And with Elasticsearch we have the promise of immediate indexing. It feels like we should explore first on how to fix the issues you and others experience.
Comment 3 Mark Hofstetter 2023-03-18 08:32:24 UTC
Yes, at the moment it is also used as a crutch. But it's not just useful for this purpose: quite often we change records just in the database and afterwards just this records need reindexing. 

I hope I'll find the time soon to fix the underlying issue
Comment 4 Nick Clemens (kidclamp) 2023-03-20 18:14:05 UTC
In other scripts we use a more generic 'where' option, this allows us to use any SQL to limit. I think the same would be better here
Comment 5 Katrin Fischer 2023-03-20 20:14:41 UTC
A where statement might be nice, then you could use the timestamp, but also use it after SQL updates/bulk changes.
Comment 6 Fridolin Somers 2024-04-15 14:30:39 UTC
A where would be great.

With also tables :
https://git.koha-community.org/Koha-community/Koha/src/branch/master/misc/migration_tools/rebuild_zebra.pl#L94
Comment 7 Fridolin Somers 2024-04-22 10:42:19 UTC
Duplicate of Bug 35345 ?