Bug 24428 - Elasticsearch index does not contain consistent item information
Summary: Elasticsearch index does not contain consistent item information
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: Main
Hardware: PC Linux
: P5 - low enhancement (vote)
Assignee: Bugs List
QA Contact:
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-01-15 14:57 UTC by david holoshka
Modified: 2020-01-28 21:28 UTC (History)
4 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description david holoshka 2020-01-15 14:57:37 UTC
Probably when catmandu was removed from the indexing process the item information organization was lost.  

For example whenever there is more than a one copy of an item any item's missing information is just omitted from the es array - no place holder is used.  Previously the 3rd item's information was always found in the 3 position in the items fields now this can no longer be assumed because if item 1 or 2 is missing data for this field then the 3rd item's information is placed in either position 1 or 2. 

This means that es can not be used to extract item information extra database look ups are required.
Comment 1 david holoshka 2020-01-15 15:05:37 UTC
also the local-classification for a single item is stored using multiple array positions in elastic search instead of a single array element.  This breaks with the previous structure of storage.
Comment 2 Ere Maijala 2020-01-17 08:02:47 UTC
I don't think you can ever rely on the field order being correct. In Elasticsearch the documents are JSON objects, and a JSON object is by definition "an unordered set of name/value pairs".

The way around this would be to either include item information in MARC or index each item as a subdocument. This should probably be optional.
Comment 3 Katrin Fischer 2020-01-17 09:44:19 UTC
This is a problem we currently have with Zebra, repeated 952 fields are indexed as without keeping the information separate per item. So you can't search for something like 'item of itemtype X in library B'. The search will also bring up all results where one item is in library B and antoher item has itemtype X.
Comment 4 Katrin Fischer 2020-01-17 09:44:43 UTC
Saying: it would be really great to fix this with Elastic.
Comment 5 Jonathan Druart 2020-01-27 11:25:40 UTC
I would consider this more like an enhancement request than a critical bug. We will need to create a new index for items, which implies a quite big work.
Comment 6 Ere Maijala 2020-01-27 11:43:44 UTC
Workaround, or a solution depending on how you view this, is to set the ElasticsearchMARCFormat pref to "Searchable array" and reindex everything. Then the item information would be available in the 952 field.
Comment 7 david holoshka 2020-01-28 08:04:55 UTC
Hi,

I don't really follow how breaking functionality then say that restoring the functionality is a enhancement.

We have several modifications to koha that depended on finding organized item information in the es index.  It wase previously structured so that index 0, for example, for all item info always belonged to the same item.  I thought the community was moving towards es I so I used this fast searching ability to build new functionality. 

It is a waste of time to lookup something in an index that contains all the info then look up the item again in a database  or in koha's case look up biblio info again in the biblio tables.  If the item information is organized in es we can make use of es fast search capabilities to improve koha's performance for a lot more scenarios eg subscriptions.

We are working to optimize the koha search here, but it is going to take awhile.

Best Regards,
David
Comment 8 Nick Clemens 2020-01-28 12:25:11 UTC
(In reply to david holoshka from comment #7) 
> We have several modifications to koha that depended on finding organized
> item information in the es index. 

Hi David,

Are these modifications available to the community in bugzilla or github so that we can see how they work?

We are striving to improve the Elasticsearch functionality and most of the work has been put into efforts to improve the basic search functionality. While we have talked about using ES as a data store, we have not focused on this area yet, which is why I think we would call this an enhancement.

Classifying it this way doesn't mean we don't think it is important or necessary, just that it doesn't break the features built in to Koha at this time.

I would try switching the system preference ElasticsearchMARCFormat as Ere mentions, as this indexes the full marc record as a searchable array and may help with your issue.

-Nick
Comment 9 david holoshka 2020-01-28 13:40:06 UTC
Hi Nick,

Does this mean you are not against making the es index backward compatible ? 

I also noticed that my code broke because the marc record in es is now in some encoded format (or in some cases marcxml)  instead of json.  

I will work on a patch for the indexer to make it backward compatible.  We will try to add es 6.5 support with aliasing to make index rebuilds possible during working hours.  We are setting the default for marc to marcxml as it is easier to debug.  I guess the base64 format uses a little less space.

Thanks for considering this problem.

Best Regards,
David
Comment 10 Ere Maijala 2020-01-28 13:43:33 UTC
David, please see comment #6. There's no need to to additional work.
Comment 11 Katrin Fischer 2020-01-28 21:28:24 UTC
I slightly disagree with Nick in that I think item searching is part of the basic search we are aiming at right now. The itemtype, location and collection searches and facetting on item values are prominent features in our OPAC and basically they never worked correctly with Zebra. You can argue that this is not a regression compared to how Zebra worked (it's just the same bug as before...), but you could also say we should work to get rid if our old problems with a new implementation :)