This affects sorting, when encountering a null field the title in question will sort out of place.
Created attachment 68947 [details] [review] Bug 19581: Unit tests
Created attachment 68948 [details] [review] Bug 19581: Remove 'split' option from Catmandu fixes for marc This option in combination with append seems to add null entries in the indexes for fields that don't exist. These fields then affect sorting. Removing it solves the issue To test: 1 - Have an ES setup 2 - Catalog and index some records 3 - View a record in ES: curl 'localhost:9200/koha_kohadev_biblios/data/4?pretty=yes' 4 - Note there are 'null' entries in the indexes, save this result for comparison 5 - Apply patch 6 - Reindex, amking sure to delete existing indexes perl misc/search_tools/rebuild_elastic_search.pl -d 7 - View the record again 8 - Note the null entries are removed 9 - Note indices are otherwise intact
Created attachment 68949 [details] [review] Bug 19581: Unit tests This patch makes a few changes to the tests to take into account the change to indexing. To test: prove t/db_dependent/Koha_Elasticsearch_Indexer.t Should return green
Created attachment 69092 [details] [review] Bug 19581: Remove 'split' option from Catmandu fixes for marc This option in combination with append seems to add null entries in the indexes for fields that don't exist. These fields then affect sorting. Removing it solves the issue To test: 1 - Have an ES setup 2 - Catalog and index some records 3 - View a record in ES: curl 'localhost:9200/koha_kohadev_biblios/data/4?pretty=yes' 4 - Note there are 'null' entries in the indexes, save this result for comparison 5 - Apply patch 6 - Reindex, amking sure to delete existing indexes perl misc/search_tools/rebuild_elastic_search.pl -d 7 - View the record again 8 - Note the null entries are removed 9 - Note indices are otherwise intact Signed-off-by: David Bourgault <david.bourgault@inlibro.com>
Created attachment 69093 [details] [review] Bug 19581: Unit tests This patch makes a few changes to the tests to take into account the change to indexing. To test: prove t/db_dependent/Koha_Elasticsearch_Indexer.t Should return green Signed-off-by: David Bourgault <david.bourgault@inlibro.com>
QA tools green Tests passed Work as described. I could still values being set to null, but only as indicators of 00X fields, which I assume is on purpose.
The code looks weird now, as we have my $options = ''; ... $options = '' unless ... So this line is completely useless. Also, the comment above is about the split option which is not used anymore, can you update it please ?
Created attachment 72745 [details] [review] Bug 19581: Remove unused comment/variable
Created attachment 72767 [details] [review] Bug 19581: Remove unused comment and line
Created attachment 72997 [details] [review] Bug 19581: (follow-up) Fix handling of new and old arrays for auth compat Stolen from bug 20244
I think the join option would be useful (also in bug 20244), like this: $options = "join:' '" unless $marc_field =~ m|_/| || $type eq 'sum'; Otherwise subfields of a single field get split to separate fields in ES.
(In reply to Ere Maijala from comment #11) > I think the join option would be useful (also in bug 20244), like this: > > $options = "join:' '" unless $marc_field =~ m|_/| || $type eq 'sum'; > > Otherwise subfields of a single field get split to separate fields in ES. I don't actually see a difference here - I added the line and reindexed and the record appeared the same, I tried changing title from 245a to 245 and 245abc - in all cases I got a single entry in the index - maybe I don't understand the issue?
Thanks for checking. I'll try again to verify. It might have been a configuration issue on my end too.
Created attachment 73864 [details] [review] Bug 19581: Remove 'split' option from Catmandu fixes for marc This option in combination with append seems to add null entries in the indexes for fields that don't exist. These fields then affect sorting. Removing it solves the issue To test: 1 - Have an ES setup 2 - Catalog and index some records 3 - View a record in ES: curl 'localhost:9200/koha_kohadev_biblios/data/4?pretty=yes' 4 - Note there are 'null' entries in the indexes, save this result for comparison 5 - Apply patch 6 - Reindex, amking sure to delete existing indexes perl misc/search_tools/rebuild_elastic_search.pl -d 7 - View the record again 8 - Note the null entries are removed 9 - Note indices are otherwise intact Signed-off-by: David Bourgault <david.bourgault@inlibro.com> Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Created attachment 73865 [details] [review] Bug 19581: Unit tests This patch makes a few changes to the tests to take into account the change to indexing. To test: prove t/db_dependent/Koha_Elasticsearch_Indexer.t Should return green Signed-off-by: David Bourgault <david.bourgault@inlibro.com> Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Created attachment 73866 [details] [review] Bug 19581: Remove unused comment and line Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Created attachment 73867 [details] [review] Bug 19581: (follow-up) Fix handling of new and old arrays for auth compat Stolen from bug 20244 Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Pushed to master for 18.05, thanks to everybody involved!
Lot of failures from t/db_dependent/Koha_Elasticsearch.t: # Failed test at t/db_dependent/Koha_Elasticsearch.t line 153. # got: 'marc_map('110a','author__sort.$append', )' # expected: 'marc_map('110a','author__sort.$append', -split => 1)' # Looks like you failed 23 tests of 49. Please fix ASAP https://jenkins.koha-community.org/job/Koha_Master_D8/425/consoleFull
Created attachment 73896 [details] [review] Bug 19581: Fix Koha_Elasticsearch.t
Ok done, pushed to master.
Pushed to stable for 17.11.06 Awesome work all!
Not pushed to 17.05.x