Elasticsearch indexing uses 999$c to store record id by deleting the all field first ! So you can not store anything in field 999, even in UNIMARC and even in authorities records. Looks like it is quick fix code added to start Elasticsearch use.
Created attachment 92971 [details] [review] Bug 23630: Do not remove field 999 in Elasticsearch indexing Elasticsearch indexing uses 999$c to store record id by deleting the all field first ! So you can not store anything in field 999, even in UNIMARC and even in authorities records. Looks like it is quick fix code added to start Elasticsearch use. This behavior is disturbing and very strange for UNIMARC flavour. This patch corrects by defining record ids mandatory in Koha::SearchEngine::Elasticsearch::Indexer::update_index(). This ids array is actually always given (except in UT). I think it is useless to allow adding a record without its id. Test plan : 1) Use Elasticsearch as SearchEngine 2) Create a subfield 999$z in default framework 3) Create a record with default framework 4) Enter a random string (never used in catalog) like "tototata" in 999$z 5) In Search engine configuration, define search field "subject" for 999$z 6) Rebuild record : misc/search_tools/rebuild_elasticsearch.pl -b -bn <biblionumber> -v 7) Search for the random string => You get a result 8) Optionnaly look at records in ES : <es server>:9200/<es index name>/data/<biblionumber>
Created attachment 93242 [details] [review] Bug 23630: Do not remove field 999 in Elasticsearch indexing Elasticsearch indexing uses 999$c to store record id by deleting the all field first ! So you can not store anything in field 999, even in UNIMARC and even in authorities records. Looks like it is quick fix code added to start Elasticsearch use. This behavior is disturbing and very strange for UNIMARC flavour. This patch corrects by defining record ids mandatory in Koha::SearchEngine::Elasticsearch::Indexer::update_index(). This ids array is actually always given (except in UT). I think it is useless to allow adding a record without its id. Test plan : 1) Use Elasticsearch as SearchEngine 2) Create a subfield 999$z in default framework 3) Create a record with default framework 4) Enter a random string (never used in catalog) like "tototata" in 999$z 5) In Search engine configuration, define search field "subject" for 999$z 6) Rebuild record : misc/search_tools/rebuild_elasticsearch.pl -b -bn <biblionumber> -v 7) Search for the random string => You get a result 8) Optionnaly look at records in ES : <es server>:9200/<es index name>/data/<biblionumber> Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Following the test plan worked, additionally I forced the record to have a 999c and 999d that did not match the biblionumber - the only issue I found was that the links were odd in the opac, however, they still worked, If someone has this issue with their records, it is something they should fix, and we shouldn't clear the entire field for a rare occurrence. Editing the record in Koha basic editor strips the 999c and 999d as it is.
Created attachment 94823 [details] [review] Bug 23630: Do not remove field 999 in Elasticsearch indexing Elasticsearch indexing uses 999$c to store record id by deleting the all field first ! So you can not store anything in field 999, even in UNIMARC and even in authorities records. Looks like it is quick fix code added to start Elasticsearch use. This behavior is disturbing and very strange for UNIMARC flavour. This patch corrects by defining record ids mandatory in Koha::SearchEngine::Elasticsearch::Indexer::update_index(). This ids array is actually always given (except in UT). I think it is useless to allow adding a record without its id. Test plan : 1) Use Elasticsearch as SearchEngine 2) Create a subfield 999$z in default framework 3) Create a record with default framework 4) Enter a random string (never used in catalog) like "tototata" in 999$z 5) In Search engine configuration, define search field "subject" for 999$z 6) Rebuild record : misc/search_tools/rebuild_elasticsearch.pl -b -bn <biblionumber> -v 7) Search for the random string => You get a result 8) Optionnaly look at records in ES : <es server>:9200/<es index name>/data/<biblionumber> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Seems like a sane enough change, works as expected and passes qa scripts. Passing QA
Nice work! Pushed to master for 19.11.00
Created attachment 94912 [details] [review] Bug 23630: (QA follow-up) Fix tests marc_records_to_documents is now an arrayref of hashes, not an arrayref of arrays _sanitise_records has been removed, we don't need those tests anymore
Pushed to 19.05.x for 19.05.05