This is to start a discussion - at some point we dropped the inclusion of the full record in plain text in the ES index A consequence is that data in a field that is not specifically indexed is no longer searchable. This is a change from zebra I believe where the entire record can be searched. I am not sure of the affect on relevancy, but I think we need t o find a way to support full record searching.
I'd suggest something like allowing wildcards in the index mappings so that you could index all the needed fields in the _all field using the index mappings. This would make it possible to e.g. concatenate the subfields for proper phrase searching. Additionally, there could be an option to index the MARC record so that it's easily searchable by field. I wouldn't make it the default since it can be sort of a really big hammer used on a really small nail, but having it as an option would give nice flexibility.
Hey Nick, thank you for working on ES! I think we have a better relevancy without having the whole notice indexed. I guess it produces also a smaller index volume too and maybe a faster indexation. It looks redundant too me to do otherwise. Like doing something super precisely very very neatly and in the same move throwing whatever you have in a big bucket. Some people wants it because they fear to lose something in the way. But, this may make noise like a cluster of thousands fuzz pedals with all knobs all the way up (OK maybe I'm a bit dramatic here :)). In the UNIMARC fields 4XX for example we have a lot of references. If the notice is the notice of an article, you'll get the name of the serial in some 4XX fields. Yup, typing the name of a serial may bring up all his minions and they will hide it in the tenth result page or something like that. So we have to be sure how to sort it. Ok, with weighting on the title index, the serial ends up on the top of the list, so maybe it is not that bad. I'm not eager for this one, if it has to be, I'll put a super low weight on the index entry to test it first. I'll also check the speed and the weight. Cheers!
Nicolas, don't worry, it will be completely optional. :)
He he, I just fear the moment when my librarian colleagues will know about this option :)
Ah, yes :) Then again you could already add rules for all MARC fields, it's just more cumbersome.
Created attachment 85158 [details] [review] Bug 22258: Elasticsearch: Add array as an alternative MARC format Adds preference ElasticsearchMARCFormat that controls whether MARC records are stored as ISO2709/MARCXML or array. Array is searchable by field and also indexes all subfields in the _all field for searching. Test plan: 1. Test that searching and indexing works with the patch without any changes. 2. Switch to array format and index some records. 3. Check e.g. the 008 field of a record and verify that the record can be found with the contents enclosed in quotes. 4. Check that it's possible to search for a specific field/subfield. Search query: marc_data_array.fields.655.subfields.a:Diaries 5. Check that tests still pass, especially t/Koha/SearchEngine/Elasticsearch.t
So, the patch allows one to change the format MARC is stored in Elasticsearch. Additionally the new field type uses default search field settings that cause the content to be indexed in _all for keyword searching. I just realized the test plan doesn't quite work. Step 1 needs to be changed: 1. Test that searching and indexing works with the patch after recreating the index. Nick, do you think this makes sense and supports the needed use cases? I've yet to implement the wildcard handling for mappings, and that can be left for another issue.
(In reply to Ere Maijala from comment #7) Yes, this seems an easy way to ensure the whole record can be searchable, I like it as an optional format. Will test when I can
After aplly of patch are marc searchable at index, reindex works byt Koha serach is broken.
see comment#9
Created attachment 87436 [details] [review] Bug 22258: Elasticsearch: Add array as an alternative MARC format Adds preference ElasticsearchMARCFormat that controls whether MARC records are stored as ISO2709/MARCXML or array. Array is searchable by field and also indexes all subfields in the _all field for searching. Test plan: 1. Test that searching and indexing works with the patch without any changes. 2. Switch to array format and index some records. 3. Check e.g. the 008 field of a record and verify that the record can be found with the contents enclosed in quotes. 4. Check that it's possible to search for a specific field/subfield. Search query: marc_data_array.fields.655.subfields.a:Diaries 5. Check that tests still pass, especially t/Koha/SearchEngine/Elasticsearch.t
Created attachment 87437 [details] [review] Bug 22258: Increase Elasticsearch maximum field count to 10000 Increases maximum field count from the default 1000 to 10000 to accommodate large records and MARC as an array.
Right. We have some new index fields and together with the MARC record as array they blew the Elasticsearch default limit of 1000 fields per record. I added configuration to raise the limit to 10000. The limit is just a safeguard in Elasticsearch to guard against broken indexer blowing up the index.
Created attachment 87440 [details] [review] Bug 22258: Elasticsearch: Add array as an alternative MARC format Adds preference ElasticsearchMARCFormat that controls whether MARC records are stored as ISO2709/MARCXML or array. Array is searchable by field and also indexes all subfields in the _all field for searching. Test plan: 1. Test that searching and indexing works with the patch without any changes. 2. Switch to array format and index some records. 3. Check e.g. the 008 field of a record and verify that the record can be found with the contents enclosed in quotes. 4. Check that it's possible to search for a specific field/subfield. Search query: marc_data_array.fields.655.subfields.a:Diaries 5. Check that tests still pass, especially t/Koha/SearchEngine/Elasticsearch.t Signed-off-by: Michal Denar <black23@gmail.com>
Created attachment 87441 [details] [review] Bug 22258: Increase Elasticsearch maximum field count to 10000 Increases maximum field count from the default 1000 to 10000 to accommodate large records and MARC as an array. Signed-off-by: Michal Denar <black23@gmail.com> Signed-off-by: Michal Denar <black23@gmail.com>
All is working here and i don't have any QA warning. But i have concerns about Bug 20589. This last disable searching on _all and could make this one useless. Am i right ? if (yes) { Ere, David: Is there something planned to build these 2 bugs together ? }
(In reply to Alex Arnaud from comment #16) > All is working here and i don't have any QA warning. > > But i have concerns about Bug 20589. This last disable searching on _all and > could make this one useless. Am i right ? > > if (yes) { > Ere, David: Is there something planned to build these 2 bugs together ? > } Bug 20589 allows for specifying whether a specific field will be included in a keyword search, so it removes _all but preserve functionality This patch just extends the formats allowed for storing marc in the 'marc_record' field of the ES document so will work well with 20589. This would actually allow for querying the full record if desired, and then 20589 would allow excluding it from general searches to not affect relevancy
Created attachment 92451 [details] [review] Bug 22258: Elasticsearch: Add array as an alternative MARC format Adds preference ElasticsearchMARCFormat that controls whether MARC records are stored as ISO2709/MARCXML or array. Array is searchable by field and also indexes all subfields in the _all field for searching. Test plan: 1. Test that searching and indexing works with the patch without any changes. 2. Switch to array format and index some records. 3. Check e.g. the 008 field of a record and verify that the record can be found with the contents enclosed in quotes. 4. Check that it's possible to search for a specific field/subfield. Search query: marc_data_array.fields.655.subfields.a:Diaries 5. Check that tests still pass, especially t/Koha/SearchEngine/Elasticsearch.t Signed-off-by: Michal Denar <black23@gmail.com> Signed-off-by: Michal Denar <black23@gmail.com> Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Created attachment 92452 [details] [review] Bug 22258: Increase Elasticsearch maximum field count to 10000 Increases maximum field count from the default 1000 to 10000 to accommodate large records and MARC as an array. Signed-off-by: Michal Denar <black23@gmail.com> Signed-off-by: Michal Denar <black23@gmail.com> Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
I think this one can move ahead, ES is still in flux and we have decisions to make Even if 20589 removes the ability to search this when it goes in we can easily restore it I would love to see the ability to have a checkbox 'search entire record' so that we could isolate this functionality if the record is stored this way. Either way, it works as expected, PQA
Nice work! Pushed to master for 19.11.00