Bug 20384

Summary: Elasticsearch rebuild script improvements - options for indexing
Product: Koha Reporter: claire.hernandez <claire.hernandez>
Component: Searching - ElasticsearchAssignee: axel Amghar <axel.amghar>
Status: Patch doesn't apply --- QA Contact:
Severity: enhancement    
Priority: P5 - low CC: axel.amghar, bgkriegel, black23, dominique.rouger, fridolin.somers, lucas
Version: Main   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22826
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Attachments: Bug 20384 - Elastic rebuild script improvements - options --offset & --length
Bug 20384 - Elastic rebuild script improvements - options --offset & --length
Bug 20384: unit test for get_all_biblios_iterator & get_all_authorities_iterator
Bug 20384 - Elastic rebuild script improvements : option --file
Bug 20384 - Elastic rebuild script improvements - options --offset, --length & --file
Bug 20384: unit test for get_all_biblios_iterator & get_all_authorities_iterator
Bug 20384 - Elastic rebuild script improvements - options --offset, --length & --file
Bug 20384: unit test for get_all_biblios_iterator & get_all_authorities_iterator
Bug 20384 - Elastic rebuild script improvements - options --offset, --length & --file

Description claire.hernandez@biblibre.com 2018-03-13 10:11:00 UTC
I would be glad to use the rebuild script with some more options like:

    -n 100             index 100 first records
    -n "100,2"       index 2 records after 100th (101 and 102)
    --from XXX     index biblio with biblionumber >= XXX
    --to YYY         index biblio with biblionumber <= YYY

bonus:
    -i                      gives elastic install information: syspref, numbers of biblios/auth indexed, name of indice used etc.

There is maybe an other way to name the options. Advice or opinion or objection on how to do this ?
Comment 1 dominique.rouger 2019-03-11 15:08:57 UTC Comment hidden (obsolete)
Comment 2 dominique.rouger 2019-03-11 15:13:21 UTC
It could also be useful to have an option to launch an indexing by decrementing the record number 

    -w "100,2" index 2 biblio with biblionumber after 100th (101 and 102)
and

    -w "100,-2" index 2 biblio with biblionumber before 100th (99 and 98)

This allows the highest biblionumbers, i.e. the most recent records, to be indexed first.
Comment 3 Fridolin Somers 2019-05-03 09:08:11 UTC
I'd say we should use same args as misc/migration_tools/rebuild_zebra.pl :

range of records :

    --length   1234         how many biblio you want to export
    --offset 1243           offset you want to start to

a where limit :

    --where                 let you specify a WHERE query, like itemtype='BOOK'
                            or something like that
    --table                 specify a table (can be items, biblioitems, biblio, biblio_metadata) to retrieve biblionumber to index.
Comment 4 Fridolin Somers 2019-05-03 09:13:19 UTC
something whould be awesome : 
limit on biblionumbers from a file.

allow to rebuild from a report results.
Comment 5 axel Amghar 2019-05-07 09:44:49 UTC Comment hidden (obsolete)
Comment 6 axel Amghar 2019-05-10 15:15:47 UTC Comment hidden (obsolete)
Comment 7 axel Amghar 2019-05-10 15:15:50 UTC Comment hidden (obsolete)
Comment 8 axel Amghar 2019-05-14 08:34:37 UTC Comment hidden (obsolete)
Comment 9 axel Amghar 2019-05-17 11:54:48 UTC Comment hidden (obsolete)
Comment 10 axel Amghar 2019-05-17 11:54:51 UTC
Created attachment 89887 [details] [review]
Bug 20384: unit test for get_all_biblios_iterator & get_all_authorities_iterator
Comment 11 axel Amghar 2019-05-17 11:58:38 UTC
Created attachment 89888 [details] [review]
Bug 20384 - Elastic rebuild script improvements - options --offset, --length & --file

This patch add the options --offset, --length & --file to the script:
misc/search_tools/rebuild_elastic_search.pl

To test :
- apply the patch
- verify that the script by default is working, launch :
	perl misc/search_tools/rebuild_elastic_search.pl -v -d -c 100

- launch with the option --offset
- then with the option --length
- launch with both options :
	perl misc/search_tools/rebuild_elastic_search.pl -v -d -c 100 --offset 1000 --length 200
- verify that the total records indexed is 200

launch perl misc/search_tools/rebuild_elastic_search.pl --man
for help

Option --file :

To test:
- after testing offset and length, apply this patch
- launch perl misc/search_tools/rebuild_elastic_search.pl -v -d -c 100 --file your_file.csv
your file need to contain a column of biblionumber from biblio or authid from auth_header (only one column)
You can go in Koha -> reports -> Create from sql
SELECT biblionumber FROM biblio ;
and then download in csv to have your csv file.
- You can test with others query like :
select biblionumber from biblio WHERE author LIKE 'a%';
Total number of results (411);
- verify that you have the rigth number of records indexed
Comment 12 Bernardo Gonzalez Kriegel 2020-03-18 15:06:23 UTC
Please rebase, does not apply.
Comment 13 Nick Clemens (kidclamp) 2020-07-06 15:27:48 UTC
Created attachment 106587 [details] [review]
Bug 20384: unit test for get_all_biblios_iterator & get_all_authorities_iterator
Comment 14 Nick Clemens (kidclamp) 2020-07-06 15:27:51 UTC
Created attachment 106588 [details] [review]
Bug 20384 - Elastic rebuild script improvements - options --offset, --length & --file

This patch add the options --offset, --length & --file to the script:
misc/search_tools/rebuild_elastic_search.pl

To test :
- apply the patch
- verify that the script by default is working, launch :
	perl misc/search_tools/rebuild_elastic_search.pl -v -d -c 100

- launch with the option --offset
- then with the option --length
- launch with both options :
	perl misc/search_tools/rebuild_elastic_search.pl -v -d -c 100 --offset 1000 --length 200
- verify that the total records indexed is 200

launch perl misc/search_tools/rebuild_elastic_search.pl --man
for help

Option --file :

To test:
- after testing offset and length, apply this patch
- launch perl misc/search_tools/rebuild_elastic_search.pl -v -d -c 100 --file your_file.csv
your file need to contain a column of biblionumber from biblio or authid from auth_header (only one column)
You can go in Koha -> reports -> Create from sql
SELECT biblionumber FROM biblio ;
and then download in csv to have your csv file.
- You can test with others query like :
select biblionumber from biblio WHERE author LIKE 'a%';
Total number of results (411);
- verify that you have the rigth number of records indexed