Bug 22828

Summary: Add display of errors encountered during indexing on the command line
Product: Koha Reporter: Nick Clemens <nick>
Component: Searching - ElasticsearchAssignee: Nick Clemens <nick>
Status: CLOSED FIXED QA Contact:
Severity: enhancement    
Priority: P5 - low CC: 1joynelson, black23, bouzid.fergani, ere.maijala, glasklas, jonathan.druart, lucas, marjorie.barry-vila, martin.renvoize
Version: master   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22829
Change sponsored?: --- Patch complexity: Trivial patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
20.05.00, 19.11.06, 19.05.11
Attachments: Bug 22828: Elasticsearch - display errors encountered during indexing on the command line
Bug 22828: Elasticsearch - display errors encountered during indexing on the command line
Bug 22828: Elasticsearch - display errors encountered during indexing on the command line
Bug 22828: Elasticsearch - display errors encountered during indexing on the command line
Bug 22828: Elasticsearch - display errors encountered during indexing on the command line
Bug 22828: Elasticsearch - display errors encountered during indexing on the command line
Bug 22828: Add tests
Bug 22828: Elasticsearch - display errors encountered during indexing on the command line
Bug 22828: Add tests

Description Nick Clemens 2019-05-02 11:31:05 UTC
Currently any records that do not index simply fail silently

To recreate:
1 - Use the Koha sample data, or insert a blank 245$b into a record (easiest way is using advanced cataloging editor
2 - Reindex elasticsearch
3 - Check the ES count on the about page
4 - Check the count in the DB (SELECT count(*) FROM biblio)
5 - They don't match!
6 - perl misc/search_tools/rebuild_elastic_search.pl -v -v
7 - No errors indicated
Comment 1 Nick Clemens 2019-05-02 11:34:16 UTC Comment hidden (obsolete)
Comment 2 Nick Clemens 2019-05-02 11:36:52 UTC
Created attachment 89228 [details] [review]
Bug 22828: Elasticsearch - display errors encountered during indexing on the command line

To test:
 1 - Use the Koha sample data, or insert a blank 245$b into a record (easiest way is using advanced cataloging editor
 2 - Reindex elasticsearch
 3 - Check the ES count on the about page
 4 - Check the count in the DB (SELECT count(*) FROM biblio)
 5 - They don't match!
 6 - perl misc/search_tools/rebuild_elastic_search.pl -v -v
 7 - No errors indicated
 8 - Apply patch
 9 - perl misc/search_tools/rebuild_elastic_search.pl -v
10 - You should be notified of an error
11 - perl misc/search_tools/rebuild_elastic_search.pl -v -v
12 - You should be notified of the specific biblio with an error and a (somewhat) readable reason
13 - perl misc/search_tools/rebuild_elastic_search.pl
14 - No output
Comment 3 Ere Maijala 2019-05-02 11:47:16 UTC
I agree errors should be displayed, but I don't think we should fail indexing a record with an empty field.
Comment 4 Nick Clemens 2019-11-06 03:21:46 UTC
(In reply to Ere Maijala from comment #3)
> I agree errors should be displayed, but I don't think we should fail
> indexing a record with an empty field.

I don't know what I meant there, but I cannot recreate a record that will not index, at least not by simply adding an empty field. All sample records index successfully now. 

I don't think this patch is a bad idea still in the case that we do encounter errors
Comment 5 Ere Maijala 2019-11-06 07:39:00 UTC
Agreed. Could you rebase, please?
Comment 6 Nick Clemens 2019-11-06 10:51:44 UTC
Created attachment 95096 [details] [review]
Bug 22828: Elasticsearch - display errors encountered during indexing on the command line

To test:
 1 - Use the Koha sample data, or insert a blank 245$b into a record (easiest way is using advanced cataloging editor
 2 - Reindex elasticsearch
 3 - Check the ES count on the about page
 4 - Check the count in the DB (SELECT count(*) FROM biblio)
 5 - They don't match!
 6 - perl misc/search_tools/rebuild_elasticsearch.pl -v -v
 7 - No errors indicated
 8 - Apply patch
 9 - perl misc/search_tools/rebuild_elasticsearch.pl -v
10 - You should be notified of an error
11 - perl misc/search_tools/rebuild_elasticsearch.pl -v -v
12 - You should be notified of the specific biblio with an error and a (somewhat) readable reason
13 - perl misc/search_tools/rebuild_elasticsearch.pl
14 - No output
Comment 7 Ere Maijala 2019-11-06 11:04:42 UTC
Created attachment 95098 [details] [review]
Bug 22828: Elasticsearch - display errors encountered during indexing on the command line

To test:
 1 - Use the Koha sample data, or insert a blank 245$b into a record (easiest way is using advanced cataloging editor
 2 - Reindex elasticsearch
 3 - Check the ES count on the about page
 4 - Check the count in the DB (SELECT count(*) FROM biblio)
 5 - They don't match!
 6 - perl misc/search_tools/rebuild_elastic_search.pl -v -v
 7 - No errors indicated
 8 - Apply patch
 9 - perl misc/search_tools/rebuild_elastic_search.pl -v
10 - You should be notified of an error
11 - perl misc/search_tools/rebuild_elastic_search.pl -v -v
12 - You should be notified of the specific biblio with an error and a (somewhat) readable reason
13 - perl misc/search_tools/rebuild_elastic_search.pl
14 - No output

Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Comment 8 Séverine Queune 2019-11-19 09:57:30 UTC
Created attachment 95530 [details] [review]
Bug 22828: Elasticsearch - display errors encountered during indexing on the command line

To test:
 1 - Use the Koha sample data, or insert a blank 245$b into a record (easiest way is using advanced cataloging editor
 2 - Reindex elasticsearch
 3 - Check the ES count on the about page
 4 - Check the count in the DB (SELECT count(*) FROM biblio)
 5 - They don't match!
 6 - perl misc/search_tools/rebuild_elastic_search.pl -v -v
 7 - No errors indicated
 8 - Apply patch
 9 - perl misc/search_tools/rebuild_elastic_search.pl -v
10 - You should be notified of an error
11 - perl misc/search_tools/rebuild_elastic_search.pl -v -v
12 - You should be notified of the specific biblio with an error and a (somewhat) readable reason
13 - perl misc/search_tools/rebuild_elastic_search.pl
14 - No output

Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Comment 9 Jonathan Druart 2020-01-14 10:28:52 UTC
Do we really want to continue without adding tests?
Comment 10 Jonathan Druart 2020-01-14 10:37:01 UTC
(In reply to Nick Clemens from comment #0)
> Currently any records that do not index simply fail silently
> 
> To recreate:
> 1 - Use the Koha sample data, or insert a blank 245$b into a record (easiest
> way is using advanced cataloging editor
> 2 - Reindex elasticsearch
> 3 - Check the ES count on the about page
> 4 - Check the count in the DB (SELECT count(*) FROM biblio)
> 5 - They don't match!

They match for me. Which record is supposed to be wrong in the sample data?
I added a blank 245$b but the reindex is ok.
Comment 11 Bouzid Fergani 2020-02-14 21:02:11 UTC
Created attachment 99040 [details] [review]
Bug 22828: Elasticsearch - display errors encountered during indexing on the command line

To test:
 1 - Use the Koha sample data, or insert a blank 245$b into a record (easiest way is using advanced cataloging editor
 2 - Reindex elasticsearch
 3 - Check the ES count on the about page
 4 - Check the count in the DB (SELECT count(*) FROM biblio)
 5 - They don't match!
 6 - perl misc/search_tools/rebuild_elastic_search.pl -v -v
 7 - No errors indicated
 8 - Apply patch
 9 - perl misc/search_tools/rebuild_elastic_search.pl -v
10 - You should be notified of an error
11 - perl misc/search_tools/rebuild_elastic_search.pl -v -v
12 - You should be notified of the specific biblio with an error and a (somewhat) readable reason
13 - perl misc/search_tools/rebuild_elastic_search.pl
14 - No output

Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Comment 12 Jonathan Druart 2020-02-21 15:10:29 UTC
Created attachment 99388 [details] [review]
Bug 22828: Add tests
Comment 13 Jonathan Druart 2020-02-21 15:13:01 UTC
kohadev-koha@df2c0b4bf86c:/kohadevbox/koha$ perl misc/search_tools/rebuild_elasticsearch.pl -v
[9614] Checking state of biblios index
[9614] Checking state of authorities index
[9614] Indexing biblios
[9614] Committing final records...
[9614] There were errors during indexing
[9614] Total 436 records indexed
[9614] Indexing authorities
[9614] 1000 records processed
[9614] Committing final records...
[9614] Total 1706 records indexed


with -v -v the output is not readable and the error is lost in the middle of the id. Should not we have it at the end?
Comment 14 Jonathan Druart 2020-03-26 14:58:52 UTC
Waiting for an answer.
Comment 15 Nick Clemens 2020-04-24 10:23:44 UTC
(In reply to Jonathan Druart from comment #14)
> Waiting for an answer.

I cannot recreate an error at this time, so I am not sure what you mean.

(In reply to Jonathan Druart from comment #13)
> with -v -v the output is not readable and the error is lost in the middle of
> the id. Should not we have it at the end?

Can you show a sample and or what you propose?

I don't think it actually matters where it falls. I expect in practice the output will be sent to a file. I am not saying there isn't room for improvement, but ensuring they are returned in any form is better than ignoring them as we do now
Comment 16 Jonathan Druart 2020-04-24 11:02:39 UTC
Let improve later then!
Comment 17 Jonathan Druart 2020-04-24 11:03:51 UTC
Created attachment 103645 [details] [review]
Bug 22828: Elasticsearch - display errors encountered during indexing on the command line

To test:
 1 - Use the Koha sample data, or insert a blank 245$b into a record (easiest way is using advanced cataloging editor
 2 - Reindex elasticsearch
 3 - Check the ES count on the about page
 4 - Check the count in the DB (SELECT count(*) FROM biblio)
 5 - They don't match!
 6 - perl misc/search_tools/rebuild_elastic_search.pl -v -v
 7 - No errors indicated
 8 - Apply patch
 9 - perl misc/search_tools/rebuild_elastic_search.pl -v
10 - You should be notified of an error
11 - perl misc/search_tools/rebuild_elastic_search.pl -v -v
12 - You should be notified of the specific biblio with an error and a (somewhat) readable reason
13 - perl misc/search_tools/rebuild_elastic_search.pl
14 - No output

Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 18 Jonathan Druart 2020-04-24 11:03:56 UTC
Created attachment 103646 [details] [review]
Bug 22828: Add tests

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 19 Martin Renvoize 2020-04-29 16:35:50 UTC
Nice work everyone!

Pushed to master for 20.05
Comment 20 Joy Nelson 2020-05-08 22:34:14 UTC
Backported to 19.11.x for 19.11.06
Comment 21 Lucas Gass 2020-05-15 15:37:39 UTC
backported to 19.05.x for 19.05.11