Summary: | Display the number of non-indexed records | ||
---|---|---|---|
Product: | Koha | Reporter: | Séverine Queune <severine.queune> |
Component: | Searching - Elasticsearch | Assignee: | Fridolin Somers <fridolin.somers> |
Status: | RESOLVED FIXED | QA Contact: | Victor Grousset/tuxayo <victor> |
Severity: | enhancement | ||
Priority: | P5 - low | CC: | arthur.suzuki, david, dcook, fridolin.somers, jonathan.druart, lucas, philip.orr, victor |
Version: | Main | ||
Hardware: | All | ||
OS: | All | ||
Change sponsored?: | --- | Patch complexity: | Trivial patch |
Documentation contact: | Philip Orr | Documentation submission: | https://gitlab.com/koha-community/koha-manual/-/merge_requests/760 |
Text to go in the release notes: |
This enhancement adds information about non-indexed records when using the Elasticsearch search engine on the About Koha > System information page.
For example:
Records are not indexed in Elasticsearch
- Warning 1 record(s) missing on a total of 435 in index koha_kohadev_biblios.record(s).
- Warning 1 record(s) missing on a total of 1705 in index koha_kohadev_authorities.
|
Version(s) released in: |
22.11.00, 22.05.05, 21.11.12
|
Attachments: |
Bug 27667: Display a warning for records missing in Elasticsearch
Bug 27667: Display a warning for records missing in Elasticsearch Bug 27667: Display a warning for records missing in Elasticsearch Bug 27667: Display a warning for records missing in Elasticsearch Bug 27667: (QA follow-up) Fix singular index and rephrase heading |
Description
Séverine Queune
2021-02-10 08:50:10 UTC
Created attachment 134904 [details] [review] Bug 27667: Display a warning for records missing in Elasticsearch On "About Koha" page are displayed the numbers of indexed biblios and authorities with ES. It could be great if we could have comparison with the number of records in the database to display the number of missing records. In the "System information" tab with the other "Data problems". If Searchengine has any issue (index count will be -1), we do not display missing records. Test plan : 1) Use Elasticsearch searchengine 2) Rebuild all records : koha-elasticsearch --rebuild -d kohadev 3) Go to "About Koha" page and look at System information tab => you see no warning 4) On a biblio record delete the MARCXML in biblio_metadata.metadata 5) Rebuild all records : koha-elasticsearch --rebuild -d kohadev 6) Go to "About Koha" page and look at System information tab => you see warning : 1 record(s) missing on a total of 435 in indice koha_kohadev_biblios. 7) On a authority record delete the MARCXML in auth_header.marcxml 8) Rebuild all records : koha-elasticsearch --rebuild -d kohadev 9) Go to "About Koha" page and look at System information tab => you see warning : 1 record(s) missing on a total of 1705 in indice koha_kohadev_authorities. I was digging into ES bugs/enh and found this easy one. Created attachment 136565 [details] [review] Bug 27667: Display a warning for records missing in Elasticsearch On "About Koha" page are displayed the numbers of indexed biblios and authorities with ES. It could be great if we could have comparison with the number of records in the database to display the number of missing records. In the "System information" tab with the other "Data problems". If Searchengine has any issue (index count will be -1), we do not display missing records. Test plan : 1) Use Elasticsearch searchengine 2) Rebuild all records : koha-elasticsearch --rebuild -d kohadev 3) Go to "About Koha" page and look at System information tab => you see no warning 4) On a biblio record delete the MARCXML in biblio_metadata.metadata 5) Rebuild all records : koha-elasticsearch --rebuild -d kohadev 6) Go to "About Koha" page and look at System information tab => you see warning : 1 record(s) missing on a total of 435 in indice koha_kohadev_biblios. 7) On a authority record delete the MARCXML in auth_header.marcxml 8) Rebuild all records : koha-elasticsearch --rebuild -d kohadev 9) Go to "About Koha" page and look at System information tab => you see warning : 1 record(s) missing on a total of 1705 in indice koha_kohadev_authorities. Signed-off-by: David Nind <david@davidnind.com> Testing notes (using koha-testing-docker with sample data): - For step 4, SQL query is: update biblio_metadata set metadata="" where id=436; - For step 7, SQL query is: update auth_header set marcxml="" where authid=1707; > [% index.missing_count | html %] record(s) missing on a total of [% index.index_count | html %] in indice [% index.index_name | html %].
Any reason to use indice instead of index? IIUC, only the plural form is used, for the singular, it's index.
Otherwise, all looks good.
Created attachment 137458 [details] [review] Bug 27667: Display a warning for records missing in Elasticsearch On "About Koha" page are displayed the numbers of indexed biblios and authorities with ES. It could be great if we could have comparison with the number of records in the database to display the number of missing records. In the "System information" tab with the other "Data problems". If Searchengine has any issue (index count will be -1), we do not display missing records. Test plan : 1) Use Elasticsearch searchengine 2) Rebuild all records : koha-elasticsearch --rebuild -d kohadev 3) Go to "About Koha" page and look at System information tab => you see no warning 4) On a biblio record delete the MARCXML in biblio_metadata.metadata 5) Rebuild all records : koha-elasticsearch --rebuild -d kohadev 6) Go to "About Koha" page and look at System information tab => you see warning : 1 record(s) missing on a total of 435 in indice koha_kohadev_biblios. 7) On a authority record delete the MARCXML in auth_header.marcxml 8) Rebuild all records : koha-elasticsearch --rebuild -d kohadev 9) Go to "About Koha" page and look at System information tab => you see warning : 1 record(s) missing on a total of 1705 in indice koha_kohadev_authorities. Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> I followed the test plan, but the system information note doesn't show up for me. I deleted the MARCXML as follows: update biblio_metadata set metadata = "" where biblionumber = 2; Reindexing then gives an error: No record passed at /kohadevbox/koha/Koha/MetadataIterator.pm line 71. Something went wrong reading record for biblio Koha::Schema::Result::Biblio=HASH(0x557fea5e8d78)->biblionumber: Can't bless non-reference value at /kohadevbox/koha/Koha/BiblioUtils.pm line 69. Cannot determine authority type for record: 1 at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch.pm line 564. BUT: The system information tab remains empty. Ok, good test: you also need to set the SearchEngine to Elasticsearch :) Created attachment 137755 [details] [review] Bug 27667: Display a warning for records missing in Elasticsearch On "About Koha" page are displayed the numbers of indexed biblios and authorities with ES. It could be great if we could have comparison with the number of records in the database to display the number of missing records. In the "System information" tab with the other "Data problems". If Searchengine has any issue (index count will be -1), we do not display missing records. Test plan : 1) Use Elasticsearch searchengine 2) Rebuild all records : koha-elasticsearch --rebuild -d kohadev 3) Go to "About Koha" page and look at System information tab => you see no warning 4) On a biblio record delete the MARCXML in biblio_metadata.metadata 5) Rebuild all records : koha-elasticsearch --rebuild -d kohadev 6) Go to "About Koha" page and look at System information tab => you see warning : 1 record(s) missing on a total of 435 in indice koha_kohadev_biblios. 7) On a authority record delete the MARCXML in auth_header.marcxml 8) Rebuild all records : koha-elasticsearch --rebuild -d kohadev 9) Go to "About Koha" page and look at System information tab => you see warning : 1 record(s) missing on a total of 1705 in indice koha_kohadev_authorities. Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de> Created attachment 137756 [details] [review] Bug 27667: (QA follow-up) Fix singular index and rephrase heading Pushed to master for 22.11. Nice work everyone, thanks! Some day, it would be nice to make about.pl use more asynchronous calls. I often look at about.pl when I want to double-check a Koha instance's version and it takes so long to load these days... That is why I proposed Bug 27497 :) We at Biblibre have patch to add a toolip on Koha logo showing version, marcflavour and searchengine. Those info are super useful when managing hundreds of sites. One can have them even without login into staff interface, via logo in auth page. There is also this plugin, we use it for demo and dev envs : https://github.com/biblibre/koha-plugin-intranet-version-in-header Backported to 22.05.x for 22.05.05 thx pushed to 21.11.x for 21.11.12 Not backported to oldoldstable (21.05.x). Feel free to ask if it's needed. Maybe worth documenting? Added to the manual. |