Bug 20384 - Elasticsearch rebuild script improvements - options for indexing
Summary: Elasticsearch rebuild script improvements - options for indexing
Status: Patch doesn't apply
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement (vote)
Assignee: axel Amghar
QA Contact:
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-03-13 10:11 UTC by claire.hernandez@biblibre.com
Modified: 2022-04-22 21:57 UTC (History)
6 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 20384 - Elastic rebuild script improvements - options --offset & --length (4.84 KB, patch)
2019-05-07 09:44 UTC, axel Amghar
Details | Diff | Splinter Review
Bug 20384 - Elastic rebuild script improvements - options --offset & --length (5.70 KB, patch)
2019-05-10 15:15 UTC, axel Amghar
Details | Diff | Splinter Review
Bug 20384: unit test for get_all_biblios_iterator & get_all_authorities_iterator (6.54 KB, patch)
2019-05-10 15:15 UTC, axel Amghar
Details | Diff | Splinter Review
Bug 20384 - Elastic rebuild script improvements : option --file (8.30 KB, patch)
2019-05-14 08:34 UTC, axel Amghar
Details | Diff | Splinter Review
Bug 20384 - Elastic rebuild script improvements - options --offset, --length & --file (13.62 KB, patch)
2019-05-17 11:54 UTC, axel Amghar
Details | Diff | Splinter Review
Bug 20384: unit test for get_all_biblios_iterator & get_all_authorities_iterator (6.67 KB, patch)
2019-05-17 11:54 UTC, axel Amghar
Details | Diff | Splinter Review
Bug 20384 - Elastic rebuild script improvements - options --offset, --length & --file (13.62 KB, patch)
2019-05-17 11:58 UTC, axel Amghar
Details | Diff | Splinter Review
Bug 20384: unit test for get_all_biblios_iterator & get_all_authorities_iterator (6.68 KB, patch)
2020-07-06 15:27 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 20384 - Elastic rebuild script improvements - options --offset, --length & --file (12.90 KB, patch)
2020-07-06 15:27 UTC, Nick Clemens
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description claire.hernandez@biblibre.com 2018-03-13 10:11:00 UTC
I would be glad to use the rebuild script with some more options like:

    -n 100             index 100 first records
    -n "100,2"       index 2 records after 100th (101 and 102)
    --from XXX     index biblio with biblionumber >= XXX
    --to YYY         index biblio with biblionumber <= YYY

bonus:
    -i                      gives elastic install information: syspref, numbers of biblios/auth indexed, name of indice used etc.

There is maybe an other way to name the options. Advice or opinion or objection on how to do this ?
Comment 1 dominique.rouger 2019-03-11 15:08:57 UTC Comment hidden (obsolete)
Comment 2 dominique.rouger 2019-03-11 15:13:21 UTC
It could also be useful to have an option to launch an indexing by decrementing the record number 

    -w "100,2" index 2 biblio with biblionumber after 100th (101 and 102)
and

    -w "100,-2" index 2 biblio with biblionumber before 100th (99 and 98)

This allows the highest biblionumbers, i.e. the most recent records, to be indexed first.
Comment 3 Fridolin Somers 2019-05-03 09:08:11 UTC
I'd say we should use same args as misc/migration_tools/rebuild_zebra.pl :

range of records :

    --length   1234         how many biblio you want to export
    --offset 1243           offset you want to start to

a where limit :

    --where                 let you specify a WHERE query, like itemtype='BOOK'
                            or something like that
    --table                 specify a table (can be items, biblioitems, biblio, biblio_metadata) to retrieve biblionumber to index.
Comment 4 Fridolin Somers 2019-05-03 09:13:19 UTC
something whould be awesome : 
limit on biblionumbers from a file.

allow to rebuild from a report results.
Comment 5 axel Amghar 2019-05-07 09:44:49 UTC Comment hidden (obsolete)
Comment 6 axel Amghar 2019-05-10 15:15:47 UTC Comment hidden (obsolete)
Comment 7 axel Amghar 2019-05-10 15:15:50 UTC Comment hidden (obsolete)
Comment 8 axel Amghar 2019-05-14 08:34:37 UTC Comment hidden (obsolete)
Comment 9 axel Amghar 2019-05-17 11:54:48 UTC Comment hidden (obsolete)
Comment 10 axel Amghar 2019-05-17 11:54:51 UTC
Created attachment 89887 [details] [review]
Bug 20384: unit test for get_all_biblios_iterator & get_all_authorities_iterator
Comment 11 axel Amghar 2019-05-17 11:58:38 UTC
Created attachment 89888 [details] [review]
Bug 20384 - Elastic rebuild script improvements - options --offset, --length & --file

This patch add the options --offset, --length & --file to the script:
misc/search_tools/rebuild_elastic_search.pl

To test :
- apply the patch
- verify that the script by default is working, launch :
	perl misc/search_tools/rebuild_elastic_search.pl -v -d -c 100

- launch with the option --offset
- then with the option --length
- launch with both options :
	perl misc/search_tools/rebuild_elastic_search.pl -v -d -c 100 --offset 1000 --length 200
- verify that the total records indexed is 200

launch perl misc/search_tools/rebuild_elastic_search.pl --man
for help

Option --file :

To test:
- after testing offset and length, apply this patch
- launch perl misc/search_tools/rebuild_elastic_search.pl -v -d -c 100 --file your_file.csv
your file need to contain a column of biblionumber from biblio or authid from auth_header (only one column)
You can go in Koha -> reports -> Create from sql
SELECT biblionumber FROM biblio ;
and then download in csv to have your csv file.
- You can test with others query like :
select biblionumber from biblio WHERE author LIKE 'a%';
Total number of results (411);
- verify that you have the rigth number of records indexed
Comment 12 Bernardo Gonzalez Kriegel 2020-03-18 15:06:23 UTC
Please rebase, does not apply.
Comment 13 Nick Clemens 2020-07-06 15:27:48 UTC
Created attachment 106587 [details] [review]
Bug 20384: unit test for get_all_biblios_iterator & get_all_authorities_iterator
Comment 14 Nick Clemens 2020-07-06 15:27:51 UTC
Created attachment 106588 [details] [review]
Bug 20384 - Elastic rebuild script improvements - options --offset, --length & --file

This patch add the options --offset, --length & --file to the script:
misc/search_tools/rebuild_elastic_search.pl

To test :
- apply the patch
- verify that the script by default is working, launch :
	perl misc/search_tools/rebuild_elastic_search.pl -v -d -c 100

- launch with the option --offset
- then with the option --length
- launch with both options :
	perl misc/search_tools/rebuild_elastic_search.pl -v -d -c 100 --offset 1000 --length 200
- verify that the total records indexed is 200

launch perl misc/search_tools/rebuild_elastic_search.pl --man
for help

Option --file :

To test:
- after testing offset and length, apply this patch
- launch perl misc/search_tools/rebuild_elastic_search.pl -v -d -c 100 --file your_file.csv
your file need to contain a column of biblionumber from biblio or authid from auth_header (only one column)
You can go in Koha -> reports -> Create from sql
SELECT biblionumber FROM biblio ;
and then download in csv to have your csv file.
- You can test with others query like :
select biblionumber from biblio WHERE author LIKE 'a%';
Total number of results (411);
- verify that you have the rigth number of records indexed