Bug 22828 - Elasticsearch - display errors encountered during indexing on the command line
Summary: Elasticsearch - display errors encountered during indexing on the command line
Status: Failed QA
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Nick Clemens
QA Contact:
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-05-02 11:31 UTC by Nick Clemens
Modified: 2020-03-26 14:58 UTC (History)
6 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 22828: Elasticsearch - display errors encountered during indexing on the command line (4.74 KB, patch)
2019-05-02 11:34 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 22828: Elasticsearch - display errors encountered during indexing on the command line (4.73 KB, patch)
2019-05-02 11:36 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 22828: Elasticsearch - display errors encountered during indexing on the command line (4.30 KB, patch)
2019-11-06 10:51 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 22828: Elasticsearch - display errors encountered during indexing on the command line (4.34 KB, patch)
2019-11-06 11:04 UTC, Ere Maijala (away 16Mar-10May)
Details | Diff | Splinter Review
Bug 22828: Elasticsearch - display errors encountered during indexing on the command line (4.48 KB, patch)
2019-11-19 09:57 UTC, Séverine Queune
Details | Diff | Splinter Review
Bug 22828: Elasticsearch - display errors encountered during indexing on the command line (4.54 KB, patch)
2020-02-14 21:02 UTC, Bouzid Fergani
Details | Diff | Splinter Review
Bug 22828: Add tests (1.42 KB, patch)
2020-02-21 15:10 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Nick Clemens 2019-05-02 11:31:05 UTC
Currently any records that do not index simply fail silently

To recreate:
1 - Use the Koha sample data, or insert a blank 245$b into a record (easiest way is using advanced cataloging editor
2 - Reindex elasticsearch
3 - Check the ES count on the about page
4 - Check the count in the DB (SELECT count(*) FROM biblio)
5 - They don't match!
6 - perl misc/search_tools/rebuild_elastic_search.pl -v -v
7 - No errors indicated
Comment 1 Nick Clemens 2019-05-02 11:34:16 UTC Comment hidden (obsolete)
Comment 2 Nick Clemens 2019-05-02 11:36:52 UTC
Created attachment 89228 [details] [review]
Bug 22828: Elasticsearch - display errors encountered during indexing on the command line

To test:
 1 - Use the Koha sample data, or insert a blank 245$b into a record (easiest way is using advanced cataloging editor
 2 - Reindex elasticsearch
 3 - Check the ES count on the about page
 4 - Check the count in the DB (SELECT count(*) FROM biblio)
 5 - They don't match!
 6 - perl misc/search_tools/rebuild_elastic_search.pl -v -v
 7 - No errors indicated
 8 - Apply patch
 9 - perl misc/search_tools/rebuild_elastic_search.pl -v
10 - You should be notified of an error
11 - perl misc/search_tools/rebuild_elastic_search.pl -v -v
12 - You should be notified of the specific biblio with an error and a (somewhat) readable reason
13 - perl misc/search_tools/rebuild_elastic_search.pl
14 - No output
Comment 3 Ere Maijala (away 16Mar-10May) 2019-05-02 11:47:16 UTC
I agree errors should be displayed, but I don't think we should fail indexing a record with an empty field.
Comment 4 Nick Clemens 2019-11-06 03:21:46 UTC
(In reply to Ere Maijala from comment #3)
> I agree errors should be displayed, but I don't think we should fail
> indexing a record with an empty field.

I don't know what I meant there, but I cannot recreate a record that will not index, at least not by simply adding an empty field. All sample records index successfully now. 

I don't think this patch is a bad idea still in the case that we do encounter errors
Comment 5 Ere Maijala (away 16Mar-10May) 2019-11-06 07:39:00 UTC
Agreed. Could you rebase, please?
Comment 6 Nick Clemens 2019-11-06 10:51:44 UTC
Created attachment 95096 [details] [review]
Bug 22828: Elasticsearch - display errors encountered during indexing on the command line

To test:
 1 - Use the Koha sample data, or insert a blank 245$b into a record (easiest way is using advanced cataloging editor
 2 - Reindex elasticsearch
 3 - Check the ES count on the about page
 4 - Check the count in the DB (SELECT count(*) FROM biblio)
 5 - They don't match!
 6 - perl misc/search_tools/rebuild_elasticsearch.pl -v -v
 7 - No errors indicated
 8 - Apply patch
 9 - perl misc/search_tools/rebuild_elasticsearch.pl -v
10 - You should be notified of an error
11 - perl misc/search_tools/rebuild_elasticsearch.pl -v -v
12 - You should be notified of the specific biblio with an error and a (somewhat) readable reason
13 - perl misc/search_tools/rebuild_elasticsearch.pl
14 - No output
Comment 7 Ere Maijala (away 16Mar-10May) 2019-11-06 11:04:42 UTC
Created attachment 95098 [details] [review]
Bug 22828: Elasticsearch - display errors encountered during indexing on the command line

To test:
 1 - Use the Koha sample data, or insert a blank 245$b into a record (easiest way is using advanced cataloging editor
 2 - Reindex elasticsearch
 3 - Check the ES count on the about page
 4 - Check the count in the DB (SELECT count(*) FROM biblio)
 5 - They don't match!
 6 - perl misc/search_tools/rebuild_elastic_search.pl -v -v
 7 - No errors indicated
 8 - Apply patch
 9 - perl misc/search_tools/rebuild_elastic_search.pl -v
10 - You should be notified of an error
11 - perl misc/search_tools/rebuild_elastic_search.pl -v -v
12 - You should be notified of the specific biblio with an error and a (somewhat) readable reason
13 - perl misc/search_tools/rebuild_elastic_search.pl
14 - No output

Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Comment 8 Séverine Queune 2019-11-19 09:57:30 UTC
Created attachment 95530 [details] [review]
Bug 22828: Elasticsearch - display errors encountered during indexing on the command line

To test:
 1 - Use the Koha sample data, or insert a blank 245$b into a record (easiest way is using advanced cataloging editor
 2 - Reindex elasticsearch
 3 - Check the ES count on the about page
 4 - Check the count in the DB (SELECT count(*) FROM biblio)
 5 - They don't match!
 6 - perl misc/search_tools/rebuild_elastic_search.pl -v -v
 7 - No errors indicated
 8 - Apply patch
 9 - perl misc/search_tools/rebuild_elastic_search.pl -v
10 - You should be notified of an error
11 - perl misc/search_tools/rebuild_elastic_search.pl -v -v
12 - You should be notified of the specific biblio with an error and a (somewhat) readable reason
13 - perl misc/search_tools/rebuild_elastic_search.pl
14 - No output

Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Comment 9 Jonathan Druart 2020-01-14 10:28:52 UTC
Do we really want to continue without adding tests?
Comment 10 Jonathan Druart 2020-01-14 10:37:01 UTC
(In reply to Nick Clemens from comment #0)
> Currently any records that do not index simply fail silently
> 
> To recreate:
> 1 - Use the Koha sample data, or insert a blank 245$b into a record (easiest
> way is using advanced cataloging editor
> 2 - Reindex elasticsearch
> 3 - Check the ES count on the about page
> 4 - Check the count in the DB (SELECT count(*) FROM biblio)
> 5 - They don't match!

They match for me. Which record is supposed to be wrong in the sample data?
I added a blank 245$b but the reindex is ok.
Comment 11 Bouzid Fergani 2020-02-14 21:02:11 UTC
Created attachment 99040 [details] [review]
Bug 22828: Elasticsearch - display errors encountered during indexing on the command line

To test:
 1 - Use the Koha sample data, or insert a blank 245$b into a record (easiest way is using advanced cataloging editor
 2 - Reindex elasticsearch
 3 - Check the ES count on the about page
 4 - Check the count in the DB (SELECT count(*) FROM biblio)
 5 - They don't match!
 6 - perl misc/search_tools/rebuild_elastic_search.pl -v -v
 7 - No errors indicated
 8 - Apply patch
 9 - perl misc/search_tools/rebuild_elastic_search.pl -v
10 - You should be notified of an error
11 - perl misc/search_tools/rebuild_elastic_search.pl -v -v
12 - You should be notified of the specific biblio with an error and a (somewhat) readable reason
13 - perl misc/search_tools/rebuild_elastic_search.pl
14 - No output

Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Comment 12 Jonathan Druart 2020-02-21 15:10:29 UTC
Created attachment 99388 [details] [review]
Bug 22828: Add tests
Comment 13 Jonathan Druart 2020-02-21 15:13:01 UTC
kohadev-koha@df2c0b4bf86c:/kohadevbox/koha$ perl misc/search_tools/rebuild_elasticsearch.pl -v
[9614] Checking state of biblios index
[9614] Checking state of authorities index
[9614] Indexing biblios
[9614] Committing final records...
[9614] There were errors during indexing
[9614] Total 436 records indexed
[9614] Indexing authorities
[9614] 1000 records processed
[9614] Committing final records...
[9614] Total 1706 records indexed


with -v -v the output is not readable and the error is lost in the middle of the id. Should not we have it at the end?
Comment 14 Jonathan Druart 2020-03-26 14:58:52 UTC
Waiting for an answer.