Bug 22258 - Elasticsearch - full record is not indexed in plain text
Summary: Elasticsearch - full record is not indexed in plain text
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal (vote)
Assignee: Ere Maijala
QA Contact: Alex Arnaud
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-02-01 13:57 UTC by Nick Clemens
Modified: 2020-11-30 21:44 UTC (History)
13 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
19.11.00


Attachments
Bug 22258: Elasticsearch: Add array as an alternative MARC format (14.83 KB, patch)
2019-02-15 11:34 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 22258: Elasticsearch: Add array as an alternative MARC format (14.83 KB, patch)
2019-04-05 07:31 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 22258: Increase Elasticsearch maximum field count to 10000 (1004 bytes, patch)
2019-04-05 07:31 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 22258: Elasticsearch: Add array as an alternative MARC format (14.97 KB, patch)
2019-04-05 08:57 UTC, Michal Denar
Details | Diff | Splinter Review
Bug 22258: Increase Elasticsearch maximum field count to 10000 (1.08 KB, patch)
2019-04-05 08:57 UTC, Michal Denar
Details | Diff | Splinter Review
Bug 22258: Elasticsearch: Add array as an alternative MARC format (15.11 KB, patch)
2019-08-23 14:47 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 22258: Increase Elasticsearch maximum field count to 10000 (1.13 KB, patch)
2019-08-23 14:47 UTC, Nick Clemens
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Nick Clemens 2019-02-01 13:57:27 UTC
This is to start a discussion - at some point we dropped the inclusion of the full record in plain text in the ES index 

A consequence is that data in a field that is not specifically indexed is no longer searchable.

This is a change from zebra I believe where the entire record can be searched.

I am not sure of the affect on relevancy, but I think we need t o find a way to support full record searching.
Comment 1 Ere Maijala 2019-02-01 15:03:25 UTC
I'd suggest something like allowing wildcards in the index mappings so that you could index all the needed fields in the _all field using the index mappings. This would make it possible to e.g. concatenate the subfields for proper phrase searching. Additionally, there could be an option to index the MARC record so that it's easily searchable by field. I wouldn't make it the default since it can be sort of a really big hammer used on a really small nail, but having it as an option would give nice flexibility.
Comment 2 Nicolas Legrand 2019-02-01 15:06:16 UTC
Hey Nick,

thank you for working on ES!

I think we have a better relevancy without having the whole notice indexed. I guess it produces also a smaller index volume too and maybe a faster indexation.

It looks redundant too me to do otherwise. Like doing something super precisely very very neatly and in the same move throwing whatever you have in a big bucket. Some people wants it because they fear to lose something in the way. But, this may make noise like a cluster of thousands fuzz pedals with all knobs all the way up (OK maybe I'm a bit dramatic here :)).

In the UNIMARC fields 4XX for example we have a lot of references. If the notice is the notice of an article, you'll get the name of the serial in some 4XX fields. Yup, typing the name of a serial may bring up all his minions and they will hide it in the tenth result page or something like that. So we have to be sure how to sort it. Ok, with weighting on the title index, the serial ends up on the top of the list, so maybe it is not that bad.

I'm not eager for this one, if it has to be, I'll put a super low weight on the index entry to test it first. I'll also check the speed and the weight.

Cheers!
Comment 3 Ere Maijala 2019-02-04 09:13:45 UTC
Nicolas, don't worry, it will be completely optional. :)
Comment 4 Nicolas Legrand 2019-02-04 09:16:55 UTC
He he, I just fear the moment when my librarian colleagues will know about this option :)
Comment 5 Ere Maijala 2019-02-04 09:20:44 UTC
Ah, yes :) Then again you could already add rules for all MARC fields, it's just more cumbersome.
Comment 6 Ere Maijala 2019-02-15 11:34:42 UTC
Created attachment 85158 [details] [review]
Bug 22258: Elasticsearch: Add array as an alternative MARC format

Adds preference ElasticsearchMARCFormat that controls whether MARC records are stored as ISO2709/MARCXML or array. Array is searchable by field and also indexes all subfields in the _all field for searching.

Test plan:
1. Test that searching and indexing works with the patch without any changes.
2. Switch to array format and index some records.
3. Check e.g. the 008 field of a record and verify that the record can be found with the contents enclosed in quotes.
4. Check that it's possible to search for a specific field/subfield. Search query: marc_data_array.fields.655.subfields.a:Diaries
5. Check that tests still pass, especially t/Koha/SearchEngine/Elasticsearch.t
Comment 7 Ere Maijala 2019-02-15 11:38:38 UTC
So, the patch allows one to change the format MARC is stored in Elasticsearch. Additionally the new field type uses default search field settings that cause the content to be indexed in _all for keyword searching.

I just realized the test plan doesn't quite work. Step 1 needs to be changed:

1. Test that searching and indexing works with the patch after recreating the index.

Nick, do you think this makes sense and supports the needed use cases? I've yet to implement the wildcard handling for mappings, and that can be left for another issue.
Comment 8 Nick Clemens 2019-02-15 12:38:16 UTC
(In reply to Ere Maijala from comment #7)
Yes, this seems an easy way to ensure the whole record can be searchable, I like it as an optional format. Will test when I can
Comment 9 Michal Denar 2019-04-04 10:06:47 UTC
After aplly of patch are marc searchable at index, reindex works byt Koha serach is broken.
Comment 10 Katrin Fischer 2019-04-04 17:20:03 UTC
see comment#9
Comment 11 Ere Maijala 2019-04-05 07:31:48 UTC
Created attachment 87436 [details] [review]
Bug 22258: Elasticsearch: Add array as an alternative MARC format

Adds preference ElasticsearchMARCFormat that controls whether MARC records are stored as ISO2709/MARCXML or array. Array is searchable by field and also indexes all subfields in the _all field for searching.

Test plan:
1. Test that searching and indexing works with the patch without any changes.
2. Switch to array format and index some records.
3. Check e.g. the 008 field of a record and verify that the record can be found with the contents enclosed in quotes.
4. Check that it's possible to search for a specific field/subfield. Search query: marc_data_array.fields.655.subfields.a:Diaries
5. Check that tests still pass, especially t/Koha/SearchEngine/Elasticsearch.t
Comment 12 Ere Maijala 2019-04-05 07:31:53 UTC
Created attachment 87437 [details] [review]
Bug 22258: Increase Elasticsearch maximum field count to 10000

Increases maximum field count from the default 1000 to 10000 to accommodate large records and MARC as an array.
Comment 13 Ere Maijala 2019-04-05 07:34:53 UTC
Right. We have some new index fields and together with the MARC record as array they blew the Elasticsearch default limit of 1000 fields per record. I added configuration to raise the limit to 10000. The limit is just a safeguard in Elasticsearch to guard against broken indexer blowing up the index.
Comment 14 Michal Denar 2019-04-05 08:57:13 UTC
Created attachment 87440 [details] [review]
Bug 22258: Elasticsearch: Add array as an alternative MARC format

Adds preference ElasticsearchMARCFormat that controls whether MARC records are stored as ISO2709/MARCXML or array. Array is searchable by field and also indexes all subfields in the _all field for searching.

Test plan:
1. Test that searching and indexing works with the patch without any changes.
2. Switch to array format and index some records.
3. Check e.g. the 008 field of a record and verify that the record can be found with the contents enclosed in quotes.
4. Check that it's possible to search for a specific field/subfield. Search query: marc_data_array.fields.655.subfields.a:Diaries
5. Check that tests still pass, especially t/Koha/SearchEngine/Elasticsearch.t

Signed-off-by: Michal Denar <black23@gmail.com>
Comment 15 Michal Denar 2019-04-05 08:57:17 UTC
Created attachment 87441 [details] [review]
Bug 22258: Increase Elasticsearch maximum field count to 10000

Increases maximum field count from the default 1000 to 10000 to accommodate large records and MARC as an array.

Signed-off-by: Michal Denar <black23@gmail.com>

Signed-off-by: Michal Denar <black23@gmail.com>
Comment 16 Alex Arnaud 2019-04-25 16:24:39 UTC
All is working here and i don't have any QA warning.

But i have concerns about Bug 20589. This last disable searching on _all and could make this one useless. Am i right ?

if (yes) {
    Ere, David: Is there something planned to build these 2 bugs together ?
}
Comment 17 Nick Clemens 2019-08-09 11:08:57 UTC
(In reply to Alex Arnaud from comment #16)
> All is working here and i don't have any QA warning.
> 
> But i have concerns about Bug 20589. This last disable searching on _all and
> could make this one useless. Am i right ?
> 
> if (yes) {
>     Ere, David: Is there something planned to build these 2 bugs together ?
> }

Bug 20589 allows for specifying whether a specific field will be included in a keyword search, so it removes _all but preserve functionality

This patch just extends the formats allowed for storing marc in the 'marc_record' field of the ES document so will work well with 20589. This would actually allow for querying the full record if desired, and then 20589 would allow excluding it from general searches to not affect relevancy
Comment 18 Nick Clemens 2019-08-23 14:47:06 UTC
Created attachment 92451 [details] [review]
Bug 22258: Elasticsearch: Add array as an alternative MARC format

Adds preference ElasticsearchMARCFormat that controls whether MARC records are stored as ISO2709/MARCXML or array. Array is searchable by field and also indexes all subfields in the _all field for searching.

Test plan:
1. Test that searching and indexing works with the patch without any changes.
2. Switch to array format and index some records.
3. Check e.g. the 008 field of a record and verify that the record can be found with the contents enclosed in quotes.
4. Check that it's possible to search for a specific field/subfield. Search query: marc_data_array.fields.655.subfields.a:Diaries
5. Check that tests still pass, especially t/Koha/SearchEngine/Elasticsearch.t

Signed-off-by: Michal Denar <black23@gmail.com>

Signed-off-by: Michal Denar <black23@gmail.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 19 Nick Clemens 2019-08-23 14:47:09 UTC
Created attachment 92452 [details] [review]
Bug 22258: Increase Elasticsearch maximum field count to 10000

Increases maximum field count from the default 1000 to 10000 to accommodate large records and MARC as an array.

Signed-off-by: Michal Denar <black23@gmail.com>

Signed-off-by: Michal Denar <black23@gmail.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 20 Nick Clemens 2019-08-23 14:49:18 UTC
I think this one can move ahead, ES is still in flux and we have decisions to make

Even if 20589 removes the ability to search this when it goes in we can easily restore it

I would love to see the ability to have a checkbox 'search entire record' so that we could isolate this functionality if the record is stored this way.

Either way, it works as expected, PQA
Comment 21 Martin Renvoize 2019-09-06 12:12:07 UTC
Nice work!

Pushed to master for 19.11.00