| Summary: | Zebra phrase register is incorrectly tokenized when using ICU | ||
|---|---|---|---|
| Product: | Koha | Reporter: | David Cook <dcook> |
| Component: | Searching - Zebra | Assignee: | David Cook <dcook> |
| Status: | CLOSED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | P5 - low | CC: | andrew, caroline.cyr-la-rose, fridolin.somers, janet.mcgowan, jonathan.druart, marjorie.barry-vila, martin.renvoize, victor |
| Version: | Main | ||
| Hardware: | All | ||
| OS: | All | ||
| See Also: |
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18017 https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13751 https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11158 https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24562 |
||
| Change sponsored?: | --- | Patch complexity: | --- |
| Documentation contact: | Documentation submission: | ||
| Text to go in the release notes: |
Previously, Zebra indexing in ICU mode was incorrectly tokenizing text for the "p" register. This meant that particular phrase searches were not working as expected. With this change, phrase searching works the same in ICU and CHR modes.
|
Version(s) released in: |
21.05.00,20.11.02
|
| Circulation function: | |||
| Bug Depends on: | 10729 | ||
| Bug Blocks: | 11158, 18017 | ||
| Attachments: |
Bug 27299: Remove tokenization for phrase register/index with ICU
Bib record used for testing auth linking Bug 27299: Remove tokenization for phrase register/index with ICU Bug 27299: Remove tokenization for phrase register/index with ICU |
||
|
Description
David Cook
2020-12-22 22:56:39 UTC
Created attachment 114646 [details] [review] Bug 27299: Remove tokenization for phrase register/index with ICU This patch removes tokenization for phrase register/index when using ICU. This should cause "exact" searches to work the same way with ICU as they do with CHR. Here's a line for testing phrases-icu.xml: echo "Gontarski, S. E." | yaz-icu -c ./etc/zebradb/etc/phrases-icu.xml With the patch, I get the following: 1 1 'gontarski s e ' 'Gontarski S E ' Without the patch, I get the following: 1 1 'gontarski ' 'Gontarski ' 2 1 's ' 'S ' 3 1 'e ' 'E ' Actually, I have tested my change using the test_files.zip that I provided at https://github.com/indexdata/idzebra/issues/24, and this patch works well. I'm still going to wait to hear from Indexdata, but I think this is probably the right approach. Once I've confirmed with them, then I could look at adding a more Koha-centric test plan. Based on https://github.com/indexdata/idzebra/issues/24#issuecomment-751260285, I think that Adam is confirming that we should not be using tokenization in phrases-icu.xml, but that they're not going to change the default phrases-icu.xml file in Zebra, because they're worried that it would negatively impact users upgrading. So I think at this point we can move forward with my patch. I think you need to restart zebra server and indexer after applying patch (In reply to Fridolin Somers from comment #5) > I think you need to restart zebra server and indexer after applying patch Oops. I didn't include a test plan for Koha! I'll remedy that. Test plan: 0. Do not apply the patch yet 1. vi /etc/koha/zebradb/etc/default.idx 2. Replace the first charmap line with "icuchain words-icu.xml" 3. Replace the second charmap line with "icuchain phrases-icu.xml" 4. koha-zebra --stop kohadev 5. pkill zebrasrv 6. koha-zebra --start kohadev 7. koha-indexer --restart kohadev 8. koha-rebuild-zebra -f kohadev 9. Go to http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=au%2Cphr%2Cext%3DE 10. Note that 6 results are returned 11. Apply the patch 12. cp /etc/koha/zebradb/etc/phrases-icu.xml /etc/koha/zebradb/etc/phrases-icu.bak 13. cp etc/zebradb/etc/phrases-icu.xml /etc/koha/zebradb/etc/phrases-icu.xml 14. koha-zebra --stop kohadev 15. pkill zebrasrv 16. koha-zebra --start kohadev 17. koha-indexer --restart kohadev 18. koha-rebuild-zebra -f kohadev 19. Go to http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=au%2Cphr%2Cext%3DE 20. Note that 0 results are returned 21. Go to http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=au%2Cphr%2Cext%3DGontarski%2C+S+E 22. Note that you're redirected to http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=25&found1=1 Admittedly this test plan targets biblio searching rather than authority searching, but that's because the user has some control over biblio searching and the authority searching is terrible (and misleading)... But I should do something to prove the authority linking... Alternate test plan: 0a. Do not apply the patch yet 0b. Change "BiblioAddsAuthorities" system preference to "allow" 1. vi /etc/koha/zebradb/etc/default.idx 2. Replace the first charmap line with "icuchain words-icu.xml" 3. Replace the second charmap line with "icuchain phrases-icu.xml" 4. koha-zebra --stop kohadev 5. pkill zebrasrv 6. koha-zebra --start kohadev 7. koha-indexer --restart kohadev 8. koha-rebuild-zebra -f kohadev 9. Go to http://localhost:8081/cgi-bin/koha/tools/stage-marc-import.pl 10. Choose bib-24.marcxml 11. Click "Upload file" 12. Choose "KohaBiblio (999$c)" for "Record matching rule" 13. Choose "No" for "Check for embedded item record data" 14. Click "Stage for import" 15. Click "Manage staged records" 16. Click "Import this batch into the catalog" 17. In a new tab, go to http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=25 18. Hover over "Gontarski, Bad" and note that it shows the following URL: http://localhost:8081/cgi-bin/koha/authorities/detail.pl?authid=236 19. Go to http://localhost:8081/cgi-bin/koha/authorities/detail.pl?authid=236 20. Note this authority is for "Gontarski, S. E." 21. Click on "Used in 1 record(s)" 22. Note that you're taken to the biblio record with a Contributor of "Gontarski, Bad" 23. Click "Undo import into catalog" on the import page 24. Apply the patch 25. cp /etc/koha/zebradb/etc/phrases-icu.xml /etc/koha/zebradb/etc/phrases-icu.bak 26. cp etc/zebradb/etc/phrases-icu.xml /etc/koha/zebradb/etc/phrases-icu.xml 27. koha-zebra --stop kohadev 28. pkill zebrasrv 29. koha-zebra --start kohadev 30. koha-indexer --restart kohadev 31. koha-rebuild-zebra -f kohadev 32. On the import page, click "Import this batch into the catalog" 33. Go to http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=25 34. Hover over "Gontarski, Bad" and note it shows the following URL: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=au:%22Gontarski%2C%20Bad.%22 35. Go to http://localhost:8081/cgi-bin/koha/authorities/detail.pl?authid=236 36. Note that it says "This authority is not used in any records" Created attachment 114827 [details]
Bib record used for testing auth linking
Bib record used for testing auth linking
Note that the "p" register very rarely gets used in Koha. It's only when a "complete field" search is done, which basically is only an "exact" search. "Phrase" searching in Koha actually uses the "w" register. For more info, see https://software.indexdata.com/zebra/doc/querymodel-zebra.html#querymodel-pqf-apt-mapping-structuretype *** Bug 13751 has been marked as a duplicate of this bug. *** Created attachment 115061 [details] [review] Bug 27299: Remove tokenization for phrase register/index with ICU This patch removes tokenization for phrase register/index when using ICU. This should cause "exact" searches to work the same way with ICU as they do with CHR. Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> This makes allot of sense and having now tested, I agree the tokenization was the issue. Signing off.. I believe Frido is working through testing too so hopefully he can provide a quick QA :) (In reply to Martin Renvoize from comment #14) > This makes allot of sense and having now tested, I agree the tokenization > was the issue. > > Signing off.. I believe Frido is working through testing too so hopefully he > can provide a quick QA :) Excellent! Created attachment 115185 [details] [review] Bug 27299: Remove tokenization for phrase register/index with ICU This patch removes tokenization for phrase register/index when using ICU. This should cause "exact" searches to work the same way with ICU as they do with CHR. Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com> Should this be Passed QA now or still Signed Off? I think with Fridolin and I haveing both tested we can safely PQA this. Pushed to master for 21.05, thanks to everybody involved! Pushed to 20.11.x for 20.11.02 We need a release note to explain this (In reply to Fridolin Somers from comment #21) > We need a release note to explain this Done :D Pushed to 20.05.x for 20.05.09 Backported: Pushed to 19.11.x branch for 19.11.15 You know... in hindsight... this might not be a good one to backport. If we push out this change and people install it via a package upgrade without paying attention, their Zebra search is going to break until they do a full Zebra re-index. That could make for a lot of very unhappy confused people. The Zebrasrv will also need a restart. I'm not sure if we actually do that when upgrading Koha... 20.11.x is still young so I think this fix can be into it. But maybe not in older branches. (In reply to Fridolin Somers from comment #27) > 20.11.x is still young so I think this fix can be into it. > But maybe not in older branches. How would they know a reindex is needed? You should at least add a big warning during the update DB process. > 20.11.x is still young
Hea lists 750 Kohas 20.11.x
Reverted from 19.11.x Reverted from 20.05 |