Summary: | Elasticsearch index does not contain consistent item information | ||
---|---|---|---|
Product: | Koha | Reporter: | david holoshka <david.holoshka> |
Component: | Searching - Elasticsearch | Assignee: | Bugs List <koha-bugs> |
Status: | NEW --- | QA Contact: | |
Severity: | enhancement | ||
Priority: | P5 - low | CC: | ere.maijala, jonathan.druart, nick |
Version: | Main | ||
Hardware: | PC | ||
OS: | Linux | ||
GIT URL: | Change sponsored?: | --- | |
Patch complexity: | --- | Documentation contact: | |
Documentation submission: | Text to go in the release notes: | ||
Version(s) released in: | Circulation function: |
Description
david holoshka
2020-01-15 14:57:37 UTC
also the local-classification for a single item is stored using multiple array positions in elastic search instead of a single array element. This breaks with the previous structure of storage. I don't think you can ever rely on the field order being correct. In Elasticsearch the documents are JSON objects, and a JSON object is by definition "an unordered set of name/value pairs". The way around this would be to either include item information in MARC or index each item as a subdocument. This should probably be optional. This is a problem we currently have with Zebra, repeated 952 fields are indexed as without keeping the information separate per item. So you can't search for something like 'item of itemtype X in library B'. The search will also bring up all results where one item is in library B and antoher item has itemtype X. Saying: it would be really great to fix this with Elastic. I would consider this more like an enhancement request than a critical bug. We will need to create a new index for items, which implies a quite big work. Workaround, or a solution depending on how you view this, is to set the ElasticsearchMARCFormat pref to "Searchable array" and reindex everything. Then the item information would be available in the 952 field. Hi, I don't really follow how breaking functionality then say that restoring the functionality is a enhancement. We have several modifications to koha that depended on finding organized item information in the es index. It wase previously structured so that index 0, for example, for all item info always belonged to the same item. I thought the community was moving towards es I so I used this fast searching ability to build new functionality. It is a waste of time to lookup something in an index that contains all the info then look up the item again in a database or in koha's case look up biblio info again in the biblio tables. If the item information is organized in es we can make use of es fast search capabilities to improve koha's performance for a lot more scenarios eg subscriptions. We are working to optimize the koha search here, but it is going to take awhile. Best Regards, David (In reply to david holoshka from comment #7) > We have several modifications to koha that depended on finding organized > item information in the es index. Hi David, Are these modifications available to the community in bugzilla or github so that we can see how they work? We are striving to improve the Elasticsearch functionality and most of the work has been put into efforts to improve the basic search functionality. While we have talked about using ES as a data store, we have not focused on this area yet, which is why I think we would call this an enhancement. Classifying it this way doesn't mean we don't think it is important or necessary, just that it doesn't break the features built in to Koha at this time. I would try switching the system preference ElasticsearchMARCFormat as Ere mentions, as this indexes the full marc record as a searchable array and may help with your issue. -Nick Hi Nick, Does this mean you are not against making the es index backward compatible ? I also noticed that my code broke because the marc record in es is now in some encoded format (or in some cases marcxml) instead of json. I will work on a patch for the indexer to make it backward compatible. We will try to add es 6.5 support with aliasing to make index rebuilds possible during working hours. We are setting the default for marc to marcxml as it is easier to debug. I guess the base64 format uses a little less space. Thanks for considering this problem. Best Regards, David David, please see comment #6. There's no need to to additional work. I slightly disagree with Nick in that I think item searching is part of the basic search we are aiming at right now. The itemtype, location and collection searches and facetting on item values are prominent features in our OPAC and basically they never worked correctly with Zebra. You can argue that this is not a regression compared to how Zebra worked (it's just the same bug as before...), but you could also say we should work to get rid if our old problems with a new implementation :) |