Bug 20244 - Elasticsearch - Indexing improvements
Summary: Elasticsearch - Indexing improvements
Status: RESOLVED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Target Milestone: ---
Assignee: Ere Maijala
QA Contact:
URL:
Whiteboard:
Keywords: release-notes-needed, rel_18_11_candidate
Depends on: 19893 20073
Blocks:
  Show dependency treegraph
 
Reported: 2018-02-19 08:29 UTC by Ere Maijala
Modified: 2018-11-23 16:17 UTC (History)
7 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Bot Control: ---
When did the bot last check this:
Who signed the patch off:
Text to go in the release notes:
* Index both ISBN10 and ISBN13 where possible. * Add handling for alternative scripts. * Improve sort field handling


Attachments
Improve ES indexing: don't split everything, handle linked fields and index ISBN versions. (4.74 KB, patch)
2018-02-19 08:32 UTC, Ere Maijala
Details | Diff | Splinter Review
Greek Sample MARC21 (4.10 KB, application/xml)
2018-03-28 07:03 UTC, Ere Maijala
Details
Bug 20244: Improve ES indexing: handle linked fields and index ISBN versions. (4.85 KB, patch)
2018-03-28 07:09 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 20244: Improve ES indexing: join subfields with at least a space, handle linked fields, index ISBN versions and create single-valued sort fields. (4.94 KB, patch)
2018-04-17 13:24 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 20244: Improve Elasticsearch ISBN indexing (5.86 KB, patch)
2018-07-03 11:16 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 20244: Add alt script indexing and fix sort field indexing (4.88 KB, patch)
2018-07-03 11:16 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 20244: Do not add records to zebraqueue when Elasticsearch is active (907 bytes, patch)
2018-07-03 11:16 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 20244: Improve indexing so that multiple subfields are joined together (6.94 KB, patch)
2018-07-03 11:16 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 20244: Improve Elasticsearch ISBN indexing (5.47 KB, patch)
2018-09-14 21:11 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 20244: Add alt script indexing and fix sort field indexing (5.08 KB, patch)
2018-09-14 21:11 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 20244: Do not add records to zebraqueue when Elasticsearch is active (907 bytes, patch)
2018-09-14 21:11 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 20244: Improve Elasticsearch ISBN indexing (5.54 KB, patch)
2018-09-16 21:44 UTC, Brendan Gallagher
Details | Diff | Splinter Review
Bug 20244: Add alt script indexing and fix sort field indexing (5.14 KB, patch)
2018-09-16 21:44 UTC, Brendan Gallagher
Details | Diff | Splinter Review
Bug 20244: Do not add records to zebraqueue when Elasticsearch is active (972 bytes, patch)
2018-09-16 21:44 UTC, Brendan Gallagher
Details | Diff | Splinter Review
Bug 20244: Improve Elasticsearch ISBN indexing (5.57 KB, patch)
2018-11-09 12:16 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 20244: Add alt script indexing and fix sort field indexing (7.33 KB, patch)
2018-11-09 12:16 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 20244: Do not add records to zebraqueue when Elasticsearch is active (972 bytes, patch)
2018-11-09 12:16 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 20244: Improve Elasticsearch ISBN indexing (5.65 KB, patch)
2018-11-13 13:24 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 20244: Add alt script indexing and fix sort field indexing (7.39 KB, patch)
2018-11-13 13:24 UTC, Martin Renvoize
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Ere Maijala 2018-02-19 08:29:18 UTC
Elasticsearch is currently splitting all fields and doing other non-optimal stuff. Will provide a patch to improve the indexing.
Comment 1 Ere Maijala 2018-02-19 08:32:41 UTC
Created attachment 71919 [details] [review]
Improve ES indexing: don't split everything, handle linked fields and index ISBN versions.
Comment 2 Nicolas Legrand 2018-03-14 09:07:29 UTC
Could you please provide a test plan?
Comment 3 Nick Clemens 2018-03-16 09:07:06 UTC
Ere, see bug 19581, can you rebase using that and provide a test plan/tests?
Comment 4 Ere Maijala 2018-03-16 15:56:42 UTC
Oh, cool. I'll rebase as soon as I can.
Comment 5 Ere Maijala 2018-03-28 07:03:14 UTC
Created attachment 73372 [details]
Greek Sample MARC21
Comment 6 Ere Maijala 2018-03-28 07:09:40 UTC Comment hidden (obsolete)
Comment 7 Ere Maijala 2018-03-28 07:11:50 UTC Comment hidden (obsolete)
Comment 8 Ere Maijala 2018-04-05 15:15:21 UTC
Oops, edited wrong bug.
Comment 9 Ere Maijala 2018-04-17 13:24:36 UTC Comment hidden (obsolete)
Comment 10 Ere Maijala 2018-04-17 13:28:45 UTC
Okay, here goes. The join option is needed in case there are fields without punctuation. Otherwise Catmandu will join the fields together without any space. Sorting has been fixed so that for each field only a single-value is created by joining the values (there's no meaningful way to sort by multivalued string fields in ES).
Comment 11 Ere Maijala 2018-04-19 10:37:35 UTC Comment hidden (obsolete)
Comment 12 Ere Maijala 2018-07-03 11:16:31 UTC
Created attachment 76646 [details] [review]
Bug 20244: Improve Elasticsearch ISBN indexing

https://bugs.koha-community.org/show_bug.cgi?id=20244

Test plan:

1. Add a record with an ISBN-10 or ISBN-13 that can be converted to ISBN-10 (e.g. 1-56619-909-3).
2. Verify that the record can be found by searching for "1-56619-909-3", "1566199093", "978-1-56619-909-4" and "9781566199094".
Comment 13 Ere Maijala 2018-07-03 11:16:35 UTC
Created attachment 76647 [details] [review]
Bug 20244: Add alt script indexing and fix sort field indexing

https://bugs.koha-community.org/show_bug.cgi?id=20244

Test plan:

1. Add a record with alternate script fields in 880 (sample attached in the bug).
2. Make sure the the record can be found e.g. with the alternate script title.
3. Add a record with multiple authors.
4. Check that in the index there is only a single author__sort field.
Comment 14 Ere Maijala 2018-07-03 11:16:38 UTC
Created attachment 76648 [details] [review]
Bug 20244: Do not add records to zebraqueue when Elasticsearch is active

https://bugs.koha-community.org/show_bug.cgi?id=20244

Otherwise the table will just keep growing.

Test plan:

1. Add or edit some biblio records.
2. Verify in the database that they were not added to the zebraqueue table.
3. Verify that the changes are still reflected in the Elasticsearch index.
Comment 15 Ere Maijala 2018-07-03 11:16:42 UTC
Created attachment 76649 [details] [review]
Bug 20244: Improve indexing so that multiple subfields are joined together

https://bugs.koha-community.org/show_bug.cgi?id=20244

Also adds 245b to the title field. Joining the subfields in a single field affects
relevancy, exact phrase search etc. It is especially important for authorities where
working phrase search is required for finding matches.

Test plan:

1. Add a personal name authority record with 100 $aTester $b1999 $eaut
2. Add a personal name authority record with 100 $aTester $baut $c1999
2. Make sure you can find the first and only the first record by doing a "Search main heading" search for Personal name with "starts with" option (N.B. the exact match option may still have issues).
Comment 16 Ere Maijala 2018-07-03 11:21:48 UTC
Note that bug 19365 provides further fixes to authority searches.
Comment 17 claire.hernandez@biblibre.com 2018-09-06 08:02:26 UTC
Hi Ere,

Did you considered this way of configuring indexing for this purpose ? https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20389

I think we should let the librarian or the person who configure Elasticsearch mapping, choosing his way of concatenation for fields.

Regards.
Comment 18 Ere Maijala 2018-09-07 10:10:23 UTC
That'd be a nice extension to the current functionality, but I didn't consider it yet since there are already so many moving parts here. The problem with adding punctuation is that the rules can get complicated pretty quickly. It's easy to add punctuation between two subfields, but when there are more than two subfields with some of them optional and punctuation depends on which subfields are concatenated, the rule language required would be complicated. A middle ground might be to allow simple rules and then use Perl-based plugins for the more complicated ones, but that's definitely out of scope here.
Comment 19 Ere Maijala 2018-09-14 21:11:22 UTC
Created attachment 78727 [details] [review]
Bug 20244: Improve Elasticsearch ISBN indexing

https://bugs.koha-community.org/show_bug.cgi?id=20244

Test plan:

1. Add a record with an ISBN-10 or ISBN-13 that can be converted to ISBN-10 (e.g. 1-56619-909-3).
2. Verify that the record can be found by searching for "1-56619-909-3", "1566199093", "978-1-56619-909-4" and "9781566199094".
Comment 20 Ere Maijala 2018-09-14 21:11:28 UTC
Created attachment 78728 [details] [review]
Bug 20244: Add alt script indexing and fix sort field indexing

https://bugs.koha-community.org/show_bug.cgi?id=20244

Test plan:

1. Add a record with alternate script fields in 880 (sample attached in the bug).
2. Make sure the the record can be found e.g. with the alternate script title.
3. Add a record with multiple authors.
4. Check that in the index there is only a single author__sort field.
Comment 21 Ere Maijala 2018-09-14 21:11:34 UTC
Created attachment 78729 [details] [review]
Bug 20244: Do not add records to zebraqueue when Elasticsearch is active

https://bugs.koha-community.org/show_bug.cgi?id=20244

Otherwise the table will just keep growing.

Test plan:

1. Add or edit some biblio records.
2. Verify in the database that they were not added to the zebraqueue table.
3. Verify that the changes are still reflected in the Elasticsearch index.
Comment 22 Brendan Gallagher 2018-09-16 21:44:33 UTC
Created attachment 79012 [details] [review]
Bug 20244: Improve Elasticsearch ISBN indexing

https://bugs.koha-community.org/show_bug.cgi?id=20244

Test plan:

1. Add a record with an ISBN-10 or ISBN-13 that can be converted to ISBN-10 (e.g. 1-56619-909-3).
2. Verify that the record can be found by searching for "1-56619-909-3", "1566199093", "978-1-56619-909-4" and "9781566199094".

Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Comment 23 Brendan Gallagher 2018-09-16 21:44:46 UTC
Created attachment 79013 [details] [review]
Bug 20244: Add alt script indexing and fix sort field indexing

https://bugs.koha-community.org/show_bug.cgi?id=20244

Test plan:

1. Add a record with alternate script fields in 880 (sample attached in the bug).
2. Make sure the the record can be found e.g. with the alternate script title.
3. Add a record with multiple authors.
4. Check that in the index there is only a single author__sort field.

Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Comment 24 Brendan Gallagher 2018-09-16 21:44:53 UTC
Created attachment 79014 [details] [review]
Bug 20244: Do not add records to zebraqueue when Elasticsearch is active

https://bugs.koha-community.org/show_bug.cgi?id=20244

Otherwise the table will just keep growing.

Test plan:

1. Add or edit some biblio records.
2. Verify in the database that they were not added to the zebraqueue table.
3. Verify that the changes are still reflected in the Elasticsearch index.

Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Comment 25 Martin Renvoize 2018-10-08 14:37:50 UTC
Marking as blocked pending bug 19893
Comment 26 Ere Maijala 2018-11-09 12:16:15 UTC
Created attachment 82160 [details] [review]
Bug 20244: Improve Elasticsearch ISBN indexing

https://bugs.koha-community.org/show_bug.cgi?id=20244

Test plan:

1. Add a record with an ISBN-10 or ISBN-13 that can be converted to ISBN-10 (e.g. 1-56619-909-3).
2. Verify that the record can be found by searching for "1-56619-909-3", "1566199093", "978-1-56619-909-4" and "9781566199094".

Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Comment 27 Ere Maijala 2018-11-09 12:16:19 UTC
Created attachment 82161 [details] [review]
Bug 20244: Add alt script indexing and fix sort field indexing

https://bugs.koha-community.org/show_bug.cgi?id=20244

Test plan:

1. Add a record with alternate script fields in 880 (sample attached in the bug).
2. Make sure the the record can be found e.g. with the alternate script title.
3. Add a record with multiple authors.
4. Check that in the index there is only a single author__sort field.

Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Comment 28 Ere Maijala 2018-11-09 12:16:23 UTC
Created attachment 82162 [details] [review]
Bug 20244: Do not add records to zebraqueue when Elasticsearch is active

https://bugs.koha-community.org/show_bug.cgi?id=20244

Otherwise the table will just keep growing.

Test plan:

1. Add or edit some biblio records.
2. Verify in the database that they were not added to the zebraqueue table.
3. Verify that the changes are still reflected in the Elasticsearch index.

Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Comment 29 Ere Maijala 2018-11-09 12:17:21 UTC
Rebased on top of latest version of bug 19893.
Comment 30 Ere Maijala 2018-11-13 13:05:15 UTC
Comment on attachment 82162 [details] [review]
Bug 20244: Do not add records to zebraqueue when Elasticsearch is active

Obsoleted the last patch since Zebra might still be used for Z39.50/SRU.
Comment 31 Martin Renvoize 2018-11-13 13:24:02 UTC
Created attachment 82272 [details] [review]
Bug 20244: Improve Elasticsearch ISBN indexing

https://bugs.koha-community.org/show_bug.cgi?id=20244

Test plan:

1. Add a record with an ISBN-10 or ISBN-13 that can be converted to ISBN-10 (e.g. 1-56619-909-3).
2. Verify that the record can be found by searching for "1-56619-909-3", "1566199093", "978-1-56619-909-4" and "9781566199094".

Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 32 Martin Renvoize 2018-11-13 13:24:07 UTC
Created attachment 82273 [details] [review]
Bug 20244: Add alt script indexing and fix sort field indexing

https://bugs.koha-community.org/show_bug.cgi?id=20244

Test plan:

1. Add a record with alternate script fields in 880 (sample attached in the bug).
2. Make sure the the record can be found e.g. with the alternate script title.
3. Add a record with multiple authors.
4. Check that in the index there is only a single author__sort field.

Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 33 Martin Renvoize 2018-11-13 13:24:47 UTC
No regressions found.. a sensible enhancement and tests added.

Passing QA
Comment 34 Nick Clemens 2018-11-16 13:09:02 UTC
Awesome work all!

Pushed to master for 18.11
Comment 35 Martin Renvoize 2018-11-23 16:13:56 UTC
Enhancement, will not be backported to 18.05.x series.
Comment 36 Martin Renvoize 2018-11-23 16:14:59 UTC
'Indexing improvement' is a bit vague.. any chance you could either clarify the bug title a bit or add a tiny summary of what this does for the release notes?
Comment 37 Martin Renvoize 2018-11-23 16:17:09 UTC
I've added a summary myself :)