Bug 27667 - Display the number of non-indexed records
Summary: Display the number of non-indexed records
Status: Needs documenting
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Fridolin Somers
QA Contact: Victor Grousset/tuxayo
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-02-10 08:50 UTC by Séverine Queune
Modified: 2022-09-14 19:13 UTC (History)
7 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
This enhancement adds information about non-indexed records when using the Elasticsearch search engine on the About Koha > System information page. For example: Records are not indexed in Elasticsearch - Warning 1 record(s) missing on a total of 435 in index koha_kohadev_biblios.record(s). - Warning 1 record(s) missing on a total of 1705 in index koha_kohadev_authorities.
Version(s) released in:
22.11.00, 22.05.05, 21.11.12


Attachments
Bug 27667: Display a warning for records missing in Elasticsearch (7.78 KB, patch)
2022-05-11 19:49 UTC, Fridolin Somers
Details | Diff | Splinter Review
Bug 27667: Display a warning for records missing in Elasticsearch (7.83 KB, patch)
2022-06-27 06:18 UTC, David Nind
Details | Diff | Splinter Review
Bug 27667: Display a warning for records missing in Elasticsearch (7.89 KB, patch)
2022-07-09 23:19 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review
Bug 27667: Display a warning for records missing in Elasticsearch (7.95 KB, patch)
2022-07-15 13:25 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 27667: (QA follow-up) Fix singular index and rephrase heading (1.51 KB, patch)
2022-07-15 13:25 UTC, Katrin Fischer
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Séverine Queune 2021-02-10 08:50:10 UTC
On "About Koha" page are displayed the numbers of indexed biblios and authorities with ES.
It could be great if we could have comparison with the number of records in the database to display the number of non-indexed datas.
We could have on the same page something like :
    Indices: koha_robin_biblios (count: 834999 on 1056480 = 221481 missing)
Or on the "System information" tab with the other "Data problems".
Comment 1 Fridolin Somers 2022-05-11 19:49:12 UTC
Created attachment 134904 [details] [review]
Bug 27667: Display a warning for records missing in Elasticsearch

On "About Koha" page are displayed the numbers of indexed biblios and authorities with ES.
It could be great if we could have comparison with the number of records in the database to display the number of missing records.
In the "System information" tab with the other "Data problems".

If Searchengine has any issue (index count will be -1), we do not
display missing records.

Test plan :
1) Use Elasticsearch searchengine
2) Rebuild all records : koha-elasticsearch --rebuild -d kohadev
3) Go to "About Koha" page and look at System information tab
=> you see no warning
4) On a biblio record delete the MARCXML in biblio_metadata.metadata
5) Rebuild all records : koha-elasticsearch --rebuild -d kohadev
6) Go to "About Koha" page and look at System information tab
=> you see warning : 1 record(s) missing on a total of 435 in indice koha_kohadev_biblios.
7) On a authority record delete the MARCXML in auth_header.marcxml
8) Rebuild all records : koha-elasticsearch --rebuild -d kohadev
9) Go to "About Koha" page and look at System information tab
=> you see warning : 1 record(s) missing on a total of 1705 in indice koha_kohadev_authorities.
Comment 2 Fridolin Somers 2022-05-11 19:49:57 UTC
I was digging into ES bugs/enh and found this easy one.
Comment 3 David Nind 2022-06-27 06:18:19 UTC
Created attachment 136565 [details] [review]
Bug 27667: Display a warning for records missing in Elasticsearch

On "About Koha" page are displayed the numbers of indexed biblios and authorities with ES.
It could be great if we could have comparison with the number of records in the database to display the number of missing records.
In the "System information" tab with the other "Data problems".

If Searchengine has any issue (index count will be -1), we do not
display missing records.

Test plan :
1) Use Elasticsearch searchengine
2) Rebuild all records : koha-elasticsearch --rebuild -d kohadev
3) Go to "About Koha" page and look at System information tab
=> you see no warning
4) On a biblio record delete the MARCXML in biblio_metadata.metadata
5) Rebuild all records : koha-elasticsearch --rebuild -d kohadev
6) Go to "About Koha" page and look at System information tab
=> you see warning : 1 record(s) missing on a total of 435 in indice koha_kohadev_biblios.
7) On a authority record delete the MARCXML in auth_header.marcxml
8) Rebuild all records : koha-elasticsearch --rebuild -d kohadev
9) Go to "About Koha" page and look at System information tab
=> you see warning : 1 record(s) missing on a total of 1705 in indice koha_kohadev_authorities.

Signed-off-by: David Nind <david@davidnind.com>
Comment 4 David Nind 2022-06-27 06:26:39 UTC
Testing notes (using koha-testing-docker with sample data):

- For step 4, SQL query is: update biblio_metadata set metadata="" where id=436;

- For step 7, SQL query is: update auth_header set marcxml="" where authid=1707;
Comment 5 Victor Grousset/tuxayo 2022-07-09 23:19:30 UTC
> [% index.missing_count | html %] record(s) missing on a total of [% index.index_count | html %] in indice [% index.index_name | html %].

Any reason to use indice instead of index? IIUC, only the plural form is used, for the singular, it's index.

Otherwise, all looks good.
Comment 6 Victor Grousset/tuxayo 2022-07-09 23:19:55 UTC
Created attachment 137458 [details] [review]
Bug 27667: Display a warning for records missing in Elasticsearch

On "About Koha" page are displayed the numbers of indexed biblios and authorities with ES.
It could be great if we could have comparison with the number of records in the database to display the number of missing records.
In the "System information" tab with the other "Data problems".

If Searchengine has any issue (index count will be -1), we do not
display missing records.

Test plan :
1) Use Elasticsearch searchengine
2) Rebuild all records : koha-elasticsearch --rebuild -d kohadev
3) Go to "About Koha" page and look at System information tab
=> you see no warning
4) On a biblio record delete the MARCXML in biblio_metadata.metadata
5) Rebuild all records : koha-elasticsearch --rebuild -d kohadev
6) Go to "About Koha" page and look at System information tab
=> you see warning : 1 record(s) missing on a total of 435 in indice koha_kohadev_biblios.
7) On a authority record delete the MARCXML in auth_header.marcxml
8) Rebuild all records : koha-elasticsearch --rebuild -d kohadev
9) Go to "About Koha" page and look at System information tab
=> you see warning : 1 record(s) missing on a total of 1705 in indice koha_kohadev_authorities.

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 7 Katrin Fischer 2022-07-15 13:18:00 UTC
I followed the test plan, but the system information note doesn't show up for me. I deleted the MARCXML as follows:

update biblio_metadata set metadata = "" where biblionumber = 2;

Reindexing then gives an error:

No record passed at /kohadevbox/koha/Koha/MetadataIterator.pm line 71.
Something went wrong reading record for biblio Koha::Schema::Result::Biblio=HASH(0x557fea5e8d78)->biblionumber: Can't bless non-reference value at /kohadevbox/koha/Koha/BiblioUtils.pm line 69.

Cannot determine authority type for record: 1 at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch.pm line 564.


BUT: The system information tab remains empty.
Comment 8 Katrin Fischer 2022-07-15 13:19:24 UTC
Ok, good test: you also need to set the SearchEngine to Elasticsearch :)
Comment 9 Katrin Fischer 2022-07-15 13:25:06 UTC
Created attachment 137755 [details] [review]
Bug 27667: Display a warning for records missing in Elasticsearch

On "About Koha" page are displayed the numbers of indexed biblios and authorities with ES.
It could be great if we could have comparison with the number of records in the database to display the number of missing records.
In the "System information" tab with the other "Data problems".

If Searchengine has any issue (index count will be -1), we do not
display missing records.

Test plan :
1) Use Elasticsearch searchengine
2) Rebuild all records : koha-elasticsearch --rebuild -d kohadev
3) Go to "About Koha" page and look at System information tab
=> you see no warning
4) On a biblio record delete the MARCXML in biblio_metadata.metadata
5) Rebuild all records : koha-elasticsearch --rebuild -d kohadev
6) Go to "About Koha" page and look at System information tab
=> you see warning : 1 record(s) missing on a total of 435 in indice koha_kohadev_biblios.
7) On a authority record delete the MARCXML in auth_header.marcxml
8) Rebuild all records : koha-elasticsearch --rebuild -d kohadev
9) Go to "About Koha" page and look at System information tab
=> you see warning : 1 record(s) missing on a total of 1705 in indice koha_kohadev_authorities.

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Comment 10 Katrin Fischer 2022-07-15 13:25:11 UTC
Created attachment 137756 [details] [review]
Bug 27667: (QA follow-up) Fix singular index and rephrase heading
Comment 11 Tomás Cohen Arazi 2022-07-18 17:55:44 UTC
Pushed to master for 22.11.

Nice work everyone, thanks!
Comment 12 David Cook 2022-07-19 23:53:24 UTC
Some day, it would be nice to make about.pl use more asynchronous calls. 

I often look at about.pl when I want to double-check a Koha instance's version and it takes so long to load these days...
Comment 13 Fridolin Somers 2022-07-27 18:45:52 UTC
That is why I proposed Bug 27497 :)

We at Biblibre have patch to add a toolip on Koha logo showing version, marcflavour and searchengine.
Those info are super useful when managing hundreds of sites.
One can have them even without login into staff interface, via logo in auth page.

There is also this plugin, we use it for demo and dev envs :
https://github.com/biblibre/koha-plugin-intranet-version-in-header
Comment 14 Lucas Gass 2022-08-23 19:28:20 UTC
Backported to 22.05.x for 22.05.05
Comment 15 Arthur Suzuki 2022-08-30 09:24:06 UTC
thx pushed to 21.11.x for 21.11.12
Comment 16 Victor Grousset/tuxayo 2022-09-14 19:13:07 UTC
Not backported to oldoldstable (21.05.x). Feel free to ask if it's needed.

Maybe worth documenting?