Elasticsearch is currently splitting all fields and doing other non-optimal stuff. Will provide a patch to improve the indexing.
Created attachment 71919 [details] [review] Improve ES indexing: don't split everything, handle linked fields and index ISBN versions.
Could you please provide a test plan?
Ere, see bug 19581, can you rebase using that and provide a test plan/tests?
Oh, cool. I'll rebase as soon as I can.
Created attachment 73372 [details] Greek Sample MARC21
Created attachment 73373 [details] [review] Bug 20244: Improve ES indexing: handle linked fields and index ISBN versions. https://bugs.koha-community.org/show_bug.cgi?id=20244 Test plan: 1. Make sure you have recent Catmandu::Identifiers module installed. 2. Add records with alternate scripts (e.g. the attached sample record). 3. Make sure the records can be found also with the alternate script. 4. Add a record with an ISBN-10 or ISBN-13 that can be converted to ISBN-10 (e.g. 1-56619-909-3). 5. Verify that the record can be found by searching for "1-56619-909-3", "1566199093", "978-1-56619-909-4" and "9781566199094".
Comment on attachment 73373 [details] [review] Bug 20244: Improve ES indexing: handle linked fields and index ISBN versions. I added a new patch with a test plan. This version does not contain any changes to the splitting. That can be handled in bug 19581. Nick, does this work for you?
Oops, edited wrong bug.
Created attachment 74321 [details] [review] Bug 20244: Improve ES indexing: join subfields with at least a space, handle linked fields, index ISBN versions and create single-valued sort fields. https://bugs.koha-community.org/show_bug.cgi?id=20244 Test plan (ISBN): 1. Make sure you have recent Catmandu::Identifiers module installed. 2. Add records with alternate scripts (e.g. the attached sample record). 3. Make sure the records can be found also with the alternate script. 4. Add a record with an ISBN-10 or ISBN-13 that can be converted to ISBN-10 (e.g. 1-56619-909-3). 5. Verify that the record can be found by searching for "1-56619-909-3", "1566199093", "978-1-56619-909-4" and "9781566199094". Test plan (joining fields): 1. Add a record that has a 650 field with multiple subfields and no punctuation between fields (e.g. record 19 from https://github.com/joubu/koha-misc4dev/tree/master/data/sql/marc21/1611/after_17196). 2. Verify that you can find the record by entering all terms from the 650 field. Test plan (sorting): 1. Add three records with "Novels" as 240a field and "A", "B" and "C" as 245a field. 2. Search for "Novels", sort by title and make sure the results are in correct order.
Okay, here goes. The join option is needed in case there are fields without punctuation. Otherwise Catmandu will join the fields together without any space. Sorting has been fixed so that for each field only a single-value is created by joining the values (there's no meaningful way to sort by multivalued string fields in ES).
Comment on attachment 74321 [details] [review] Bug 20244: Improve ES indexing: join subfields with at least a space, handle linked fields, index ISBN versions and create single-valued sort fields. Marking obsolete since it seems to make sense to rather wait for bug 19893 and work based on that.
Created attachment 76646 [details] [review] Bug 20244: Improve Elasticsearch ISBN indexing https://bugs.koha-community.org/show_bug.cgi?id=20244 Test plan: 1. Add a record with an ISBN-10 or ISBN-13 that can be converted to ISBN-10 (e.g. 1-56619-909-3). 2. Verify that the record can be found by searching for "1-56619-909-3", "1566199093", "978-1-56619-909-4" and "9781566199094".
Created attachment 76647 [details] [review] Bug 20244: Add alt script indexing and fix sort field indexing https://bugs.koha-community.org/show_bug.cgi?id=20244 Test plan: 1. Add a record with alternate script fields in 880 (sample attached in the bug). 2. Make sure the the record can be found e.g. with the alternate script title. 3. Add a record with multiple authors. 4. Check that in the index there is only a single author__sort field.
Created attachment 76648 [details] [review] Bug 20244: Do not add records to zebraqueue when Elasticsearch is active https://bugs.koha-community.org/show_bug.cgi?id=20244 Otherwise the table will just keep growing. Test plan: 1. Add or edit some biblio records. 2. Verify in the database that they were not added to the zebraqueue table. 3. Verify that the changes are still reflected in the Elasticsearch index.
Created attachment 76649 [details] [review] Bug 20244: Improve indexing so that multiple subfields are joined together https://bugs.koha-community.org/show_bug.cgi?id=20244 Also adds 245b to the title field. Joining the subfields in a single field affects relevancy, exact phrase search etc. It is especially important for authorities where working phrase search is required for finding matches. Test plan: 1. Add a personal name authority record with 100 $aTester $b1999 $eaut 2. Add a personal name authority record with 100 $aTester $baut $c1999 2. Make sure you can find the first and only the first record by doing a "Search main heading" search for Personal name with "starts with" option (N.B. the exact match option may still have issues).
Note that bug 19365 provides further fixes to authority searches.
Hi Ere, Did you considered this way of configuring indexing for this purpose ? https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20389 I think we should let the librarian or the person who configure Elasticsearch mapping, choosing his way of concatenation for fields. Regards.
That'd be a nice extension to the current functionality, but I didn't consider it yet since there are already so many moving parts here. The problem with adding punctuation is that the rules can get complicated pretty quickly. It's easy to add punctuation between two subfields, but when there are more than two subfields with some of them optional and punctuation depends on which subfields are concatenated, the rule language required would be complicated. A middle ground might be to allow simple rules and then use Perl-based plugins for the more complicated ones, but that's definitely out of scope here.
Created attachment 78727 [details] [review] Bug 20244: Improve Elasticsearch ISBN indexing https://bugs.koha-community.org/show_bug.cgi?id=20244 Test plan: 1. Add a record with an ISBN-10 or ISBN-13 that can be converted to ISBN-10 (e.g. 1-56619-909-3). 2. Verify that the record can be found by searching for "1-56619-909-3", "1566199093", "978-1-56619-909-4" and "9781566199094".
Created attachment 78728 [details] [review] Bug 20244: Add alt script indexing and fix sort field indexing https://bugs.koha-community.org/show_bug.cgi?id=20244 Test plan: 1. Add a record with alternate script fields in 880 (sample attached in the bug). 2. Make sure the the record can be found e.g. with the alternate script title. 3. Add a record with multiple authors. 4. Check that in the index there is only a single author__sort field.
Created attachment 78729 [details] [review] Bug 20244: Do not add records to zebraqueue when Elasticsearch is active https://bugs.koha-community.org/show_bug.cgi?id=20244 Otherwise the table will just keep growing. Test plan: 1. Add or edit some biblio records. 2. Verify in the database that they were not added to the zebraqueue table. 3. Verify that the changes are still reflected in the Elasticsearch index.
Created attachment 79012 [details] [review] Bug 20244: Improve Elasticsearch ISBN indexing https://bugs.koha-community.org/show_bug.cgi?id=20244 Test plan: 1. Add a record with an ISBN-10 or ISBN-13 that can be converted to ISBN-10 (e.g. 1-56619-909-3). 2. Verify that the record can be found by searching for "1-56619-909-3", "1566199093", "978-1-56619-909-4" and "9781566199094". Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Created attachment 79013 [details] [review] Bug 20244: Add alt script indexing and fix sort field indexing https://bugs.koha-community.org/show_bug.cgi?id=20244 Test plan: 1. Add a record with alternate script fields in 880 (sample attached in the bug). 2. Make sure the the record can be found e.g. with the alternate script title. 3. Add a record with multiple authors. 4. Check that in the index there is only a single author__sort field. Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Created attachment 79014 [details] [review] Bug 20244: Do not add records to zebraqueue when Elasticsearch is active https://bugs.koha-community.org/show_bug.cgi?id=20244 Otherwise the table will just keep growing. Test plan: 1. Add or edit some biblio records. 2. Verify in the database that they were not added to the zebraqueue table. 3. Verify that the changes are still reflected in the Elasticsearch index. Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Marking as blocked pending bug 19893
Created attachment 82160 [details] [review] Bug 20244: Improve Elasticsearch ISBN indexing https://bugs.koha-community.org/show_bug.cgi?id=20244 Test plan: 1. Add a record with an ISBN-10 or ISBN-13 that can be converted to ISBN-10 (e.g. 1-56619-909-3). 2. Verify that the record can be found by searching for "1-56619-909-3", "1566199093", "978-1-56619-909-4" and "9781566199094". Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Created attachment 82161 [details] [review] Bug 20244: Add alt script indexing and fix sort field indexing https://bugs.koha-community.org/show_bug.cgi?id=20244 Test plan: 1. Add a record with alternate script fields in 880 (sample attached in the bug). 2. Make sure the the record can be found e.g. with the alternate script title. 3. Add a record with multiple authors. 4. Check that in the index there is only a single author__sort field. Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Created attachment 82162 [details] [review] Bug 20244: Do not add records to zebraqueue when Elasticsearch is active https://bugs.koha-community.org/show_bug.cgi?id=20244 Otherwise the table will just keep growing. Test plan: 1. Add or edit some biblio records. 2. Verify in the database that they were not added to the zebraqueue table. 3. Verify that the changes are still reflected in the Elasticsearch index. Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Rebased on top of latest version of bug 19893.
Comment on attachment 82162 [details] [review] Bug 20244: Do not add records to zebraqueue when Elasticsearch is active Obsoleted the last patch since Zebra might still be used for Z39.50/SRU.
Created attachment 82272 [details] [review] Bug 20244: Improve Elasticsearch ISBN indexing https://bugs.koha-community.org/show_bug.cgi?id=20244 Test plan: 1. Add a record with an ISBN-10 or ISBN-13 that can be converted to ISBN-10 (e.g. 1-56619-909-3). 2. Verify that the record can be found by searching for "1-56619-909-3", "1566199093", "978-1-56619-909-4" and "9781566199094". Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Created attachment 82273 [details] [review] Bug 20244: Add alt script indexing and fix sort field indexing https://bugs.koha-community.org/show_bug.cgi?id=20244 Test plan: 1. Add a record with alternate script fields in 880 (sample attached in the bug). 2. Make sure the the record can be found e.g. with the alternate script title. 3. Add a record with multiple authors. 4. Check that in the index there is only a single author__sort field. Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
No regressions found.. a sensible enhancement and tests added. Passing QA
Awesome work all! Pushed to master for 18.11
Enhancement, will not be backported to 18.05.x series.
'Indexing improvement' is a bit vague.. any chance you could either clarify the bug title a bit or add a tiny summary of what this does for the release notes?
I've added a summary myself :)