Bug 18085 - Elasticsearch ids are not being stored correctly
Summary: Elasticsearch ids are not being stored correctly
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: master
Hardware: All All
: P5 - low major (vote)
Assignee: Galen Charlton
QA Contact: Testopia
Depends on: 17255
  Show dependency treegraph
Reported: 2017-02-08 16:54 UTC by Nick Clemens
Modified: 2019-06-27 09:24 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:


Note You need to log in before you can comment on or make changes to this bug.
Description Nick Clemens 2017-02-08 16:54:35 UTC
The upgrade from 1.X to 5.X works very well for indexing existing records, however, it seems the '_id' field is not being given to elastic at save so it is generating a random id.

The problem with this is that each record update will save a new record as opposed to overwriting the existing record and you will get multiple items in your search.

To reproduce:
1 - Have ES5 running and have some records
2 - Choose a biblionumber and reindex just that record, deleting the index before:
perl /home/vagrant/kohaclone/misc/search_tools/rebuild_elastic_search.pl -d -b -bn 2365
3 - View that record on the command line:
curl -XGET 'localhost:9200/koha_kohadev_biblios/_search/?pretty=true&q=es_id:2365'
4 - Note the '_id' field is of the form: "_id" : "AVoemHc9LtgvIQ1yOOU6",
5 - Search for the record in koha - should be one
6 - Add or edit the items on that record
7 - Search again, the record returns multiple times

This should be a small fix, not sure where the disconnect is though
Comment 1 Jonathan Druart 2017-02-10 08:04:42 UTC
Olli, any chances you can try and fix this one?
Comment 2 Olli-Antti Kivilahti 2017-02-10 08:50:21 UTC
Sorry. Busy.
I (or somebody in Koha-Suomi) will solve the ES issues when we decide to go to ES.

Can't tell from the top of my hat, where the problem is. Sounds like something to do with interfacing via Catmandu. Catmandu needs some special care to handle _id properly. Maybe the issue is with one of the multiple Catmandu modules needed to get the MARC::Record to ES.

Do we really need Catmandu at this phase? We can directly index to ES, or any search engine because Koha already has the infrastucture to facilitate multiple search engines.
Now we need to wait for the Catmandu-guys to port Catmandu::Store::ElasticSearch to 5.2 or supply a patch (if the problem is in Catmandu)

Also We need to implement Catmandu in the JavaScript-side if we want to consume the results directly to the browser. We would have to implement any parser there though.
AJAX-searches would be nice!

I guess Catmandu defines some kind of a "standard" for working with library data, but so does MARCXML.
Turning MARCXML to JSON is really simple.
Or even better... JSON-LD!!

But this is not an answer to the bug at hand :)

My bet is Catmandu. Giving it the right parameters to handle _id.
Comment 3 Nick Clemens 2017-02-10 12:05:21 UTC
Nvm - this is a kohadevbox issue - version of Catmandu installed is not the .deb from bug 17255

was using version 3.03 - need .5 or greater
Comment 4 Jonathan Druart 2017-02-16 11:25:12 UTC
Ran into this too.

For the record:
ES log says "MapperParsingException[Field [_id] is a metadata field and cannot be added inside a document. Use the index API request parameters.]"

You need to cpan Catmandu::Store::ElasticSearch, not Catmandu::Store::Elasticsearch

S vs s!