Bug 17255 - Upgrade Elastic Search code to work with version 5.1
Summary: Upgrade Elastic Search code to work with version 5.1
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Olli-Antti Kivilahti
QA Contact: Tomás Cohen Arazi
URL:
Keywords:
Depends on: 12478 17372
Blocks: 17885 18085 18189
  Show dependency treegraph
 
Reported: 2016-09-05 20:48 UTC by Chris Cormack
Modified: 2017-12-07 22:16 UTC (History)
10 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
libsearch-elasticsearch-perl_2.03-1_all.deb (229.07 KB, application/x-debian-package)
2016-09-05 22:09 UTC, Mirko Tietgen
Details
libcatmandu-store-elasticsearch-perl_0.0501-1_all.deb (18.91 KB, application/x-debian-package)
2016-09-05 22:09 UTC, Mirko Tietgen
Details
Bug 17255 - Upgrade Elastic Search code to work with version 2.4+ - rebased wip (10.99 KB, patch)
2017-01-09 16:24 UTC, Olli-Antti Kivilahti
Details | Diff | Splinter Review
Bug 17255 - Upgrade Elastic Search code to work with version 5.1 (6.98 KB, patch)
2017-01-10 14:45 UTC, Olli-Antti Kivilahti
Details | Diff | Splinter Review
Bug 17255 - Upgrade Elastic Search code to work with version 2.4+ - rebased wip (10.77 KB, patch)
2017-01-24 11:33 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 17255 - Upgrade Elastic Search code to work with version 5.1 (7.04 KB, patch)
2017-01-24 11:33 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 17255 - Upgrade Elastic Search code to work with version 2.4+ - rebased wip (10.85 KB, patch)
2017-02-06 16:51 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 17255 - Upgrade Elastic Search code to work with version 5.1 (7.12 KB, patch)
2017-02-06 16:51 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Chris Cormack 2016-09-05 20:48:10 UTC
Before building more things on Elastic (I have 3 things ready to go) We need to upgrade to work with the latest stable (2.4)

This will involve getting a 2.4 cluster running, pointing Koha at it, and fixing any errors that occur.
Comment 1 Mirko Tietgen 2016-09-05 21:45:01 UTC
It will also need newer Catmandu than 0.9x I guess?
Comment 2 Chris Cormack 2016-09-05 21:47:07 UTC
That's what we will find out. I suspect so. But apparently 2.4 is more backward compatible than earlier 2.x versions.
Comment 3 Mirko Tietgen 2016-09-05 22:09:12 UTC
Created attachment 55205 [details]
libsearch-elasticsearch-perl_2.03-1_all.deb
Comment 4 Mirko Tietgen 2016-09-05 22:09:56 UTC
Created attachment 55206 [details]
libcatmandu-store-elasticsearch-perl_0.0501-1_all.deb
Comment 5 Mirko Tietgen 2016-09-05 22:11:50 UTC
I have attached newer versions of the ES dependencies that I plan to submit to Debian.
Comment 6 Chris Cormack 2016-09-05 22:17:54 UTC
Awesome, thank you Mirko
Comment 7 Jonathan Druart 2016-09-06 07:11:06 UTC
(In reply to Chris Cormack from comment #0)
> Before building more things on Elastic (I have 3 things ready to go)

Where are they? I have not found anything on the catalyst git repo.
Comment 8 Chris Cormack 2016-09-06 07:12:19 UTC
Three things ready to build that is.
Comment 9 Jonathan Druart 2016-09-29 10:08:40 UTC
Please build it on top of bug 17372.
Comment 10 Jonathan Druart 2016-10-12 08:16:14 UTC
It seems that the code is available at http://git.catalyst.co.nz/gw?p=koha.git;a=shortlog;h=refs/heads/wr262182
Comment 11 Jonathan Druart 2016-10-18 15:50:32 UTC
(In reply to Jonathan Druart from comment #10)
> It seems that the code is available at
> http://git.catalyst.co.nz/gw?p=koha.git;a=shortlog;h=refs/heads/wr262182

What's the status of this branch? Is it considered as testable/wip/done?
Comment 12 Chris Cormack 2016-10-18 20:28:52 UTC
(In reply to Jonathan Druart from comment #11)
> (In reply to Jonathan Druart from comment #10)
> > It seems that the code is available at
> > http://git.catalyst.co.nz/gw?p=koha.git;a=shortlog;h=refs/heads/wr262182
> 
> What's the status of this branch? Is it considered as testable/wip/done?

It is testable, however


* Facets most probably do not work (to be done)

* We had to change Catmandu::Store::ElasticSearch::Bag
https://github.com/LibreCat/Catmandu-Store-Elasticsearch/issues/10

But we would love some testing. (It is built on top of your bug to move the elasticsearch code (that is already in that branch))
Comment 13 Chris Cormack 2016-10-18 20:29:29 UTC
Once the facets are working we will submit them as patches for sign off, but testing now would help catch any issues
Comment 14 Jonathan Druart 2016-10-19 11:39:51 UTC
Note: The errors don't go to the error file but to stderr
Comment 15 Jonathan Druart 2016-10-19 11:40:23 UTC
perl misc/search_tools/rebuild_elastic_search.pl -d -a -b -v
 problem with :16 : :1: parser error : Start tag expected, '<' not found
podigpodfigo
^
 
podigpodfigo at C4/Biblio.pm line 1353.
No record passed at Koha/MetadataIterator.pm line 71.
Something went wrong reading record for biblio Koha::Schema::Result::Biblio=HASH(0xd904770)->biblionumber: Can't bless non-reference value at Koha/BiblioUtils.pm line 70.
Comment 16 Jonathan Druart 2016-10-19 11:47:06 UTC Comment hidden (obsolete)
Comment 17 Jonathan Druart 2016-10-19 11:52:09 UTC
I did not install the second .deb, and it fails for me:

% sudo dpkg -i attachment.cgi?id=55206 #  libcatmandu-store-elasticsearch-perl_0.0501-1_all.deb 
(Reading database ... 83863 files and directories currently installed.)
Preparing to unpack attachment.cgi?id=55206 ...
Unpacking libcatmandu-store-elasticsearch-perl (0.0501-1) over (0.0304-1) ...
dpkg: dependency problems prevent configuration of libcatmandu-store-elasticsearch-perl:
 libcatmandu-store-elasticsearch-perl depends on libcatmandu-perl (>= 1.01); however:
  Version of libcatmandu-perl on system is 0.9206-1.

dpkg: error processing package libcatmandu-store-elasticsearch-perl (--install):
 dependency problems - leaving unconfigured
Processing triggers for man-db (2.7.5-1) ...
Errors were encountered while processing:
 libcatmandu-store-elasticsearch-perl


% apt policy libcatmandu-perl
libcatmandu-perl:
  Installed: 0.9206-1
  Candidate: 0.9505-1
  Version table:
     0.9505-1 500
        500 http://ftp.fr.debian.org/debian testing/main i386 Packages
 *** 0.9206-1 100
        100 /var/lib/dpkg/status


I am willing to test but I will need a test plan to upgrade my packages please.
Comment 19 Jonathan Druart 2016-10-20 08:38:11 UTC
% sudo dpkg -i libcatmandu-perl_1.0002-1_all.deb libsearch-elasticsearch-perl_2.03-1_all.deb libcatmandu-store-elasticsearch-perl_0.0501-1_all.deb
(Reading database ... 83849 files and directories currently installed.)
Preparing to unpack libcatmandu-perl_1.0002-1_all.deb ...
Unpacking libcatmandu-perl (1.0002-1) over (0.9505-1) ...
Preparing to unpack libsearch-elasticsearch-perl_2.03-1_all.deb ...
Unpacking libsearch-elasticsearch-perl (2.03-1) over (2.03-1) ...
Preparing to unpack libcatmandu-store-elasticsearch-perl_0.0501-1_all.deb ...
Unpacking libcatmandu-store-elasticsearch-perl (0.0501-1) over (0.0304-1) ...
dpkg: dependency problems prevent configuration of libcatmandu-perl:
 libcatmandu-perl depends on libmoox-role-logger-perl (>= 0.005); however:
  Package libmoox-role-logger-perl is not installed.
 libcatmandu-perl depends on libdata-uuid-perl (>= 1.217); however:
  Package libdata-uuid-perl is not installed.
  Version of libdata-uuid-perl on system, provided by libossp-uuid-perl:i386, is <none>.
 libcatmandu-perl depends on liburi-template-perl (>= 0.22); however:
  Version of liburi-template-perl on system is 0.18-1.

dpkg: error processing package libcatmandu-perl (--install):
 dependency problems - leaving unconfigured
Setting up libsearch-elasticsearch-perl (2.03-1) ...
dpkg: dependency problems prevent configuration of libcatmandu-store-elasticsearch-perl:
 libcatmandu-store-elasticsearch-perl depends on libcatmandu-perl (>= 1.01); however:
  Package libcatmandu-perl is not configured yet.

dpkg: error processing package libcatmandu-store-elasticsearch-perl (--install):
 dependency problems - leaving unconfigured
Processing triggers for man-db (2.7.5-1) ...
Errors were encountered while processing:
 libcatmandu-perl
 libcatmandu-store-elasticsearch-perl


% sudo apt-get install -f
The following additional packages will be installed:
  libalgorithm-combinatorics-perl libalgorithm-diff-xs-perl libanyevent-perl libapt-pkg-perl libasync-interrupt-perl libb-compiling-perl libb-hooks-op-check-entersubforcv-perl
  libb-hooks-op-check-perl libb-utils-perl libbareword-filehandles-perl libbit-vector-perl libcache-fastmmap-perl libcache-memcached-fast-perl libcairo-perl libclass-c3-xs-perl
  libclass-load-xs-perl libclass-xsaccessor-perl libclone-perl libcommon-sense-perl libcpanel-json-xs-perl libcrypt-eksblowfish-perl libcrypt-gcrypt-perl libcrypt-ssleay-perl
  libcss-minifier-xs-perl libdata-messagepack-perl libdata-util-perl libdate-calc-xs-perl libdatetime-format-mail-perl libdatetime-locale-perl libdatetime-perl libdatetime-timezone-perl
  libdbd-mysql-perl libdbd-sqlite2-perl libdbd-sqlite3-perl libdbi-perl libdevel-caller-perl libdevel-cover-perl libdevel-declare-perl libdevel-lexalias-perl libdevel-nytprof-perl
  libdevel-size-perl libdigest-sha-perl libev-perl libfcgi-perl libfile-fcntllock-perl libfilter-perl libfont-freetype-perl libgd-perl libglib-perl libgraphics-magick-perl libgssapi-perl
  libguard-perl libhtml-parser-perl libhtml-template-pro-perl libhtml-tidy-perl libhttp-tiny-perl libindirect-perl libio-pty-perl libjavascript-minifier-xs-perl libjson-xs-perl
  liblexical-sealrequirehints-perl liblingua-stem-snowball-perl liblinux-inotify2-perl liblist-moreutils-perl liblocale-gettext-perl libmarpa-r2-perl libmodule-metadata-perl libmoose-perl
  libmoox-role-logger-perl libmultidimensional-perl libnet-ssleay-perl libnet-z3950-zoom-perl libossp-uuid-perl libpackage-stash-xs-perl libpadwalker-perl libpango-perl
  libparams-classify-perl libparams-util-perl libparams-validate-perl libperl5.24 libperlio-gzip-perl libpng16-16 librdf-perl libreadonly-xs-perl libscalar-list-utils-perl
  libsereal-decoder-perl libsereal-encoder-perl libsocket6-perl libstring-crc32-perl libsub-identify-perl libsub-name-perl libtemplate-perl libterm-readkey-perl libtext-bidi-perl
  libtext-charwidth-perl libtext-csv-xs-perl libtext-iconv-perl libtext-soundex-perl libtext-unaccent-perl libtidy5 libtype-tiny-xs-perl libunicode-map-perl libunicode-utf8-perl
  libuuid-perl libvariable-magic-perl libversion-perl libwebp6 libxml-libxml-perl libxml-libxslt-perl libxml-parser-perl libyaml-libyaml-perl libyaml-syck-perl lintian perl perl-base
  perl-doc perl-modules-5.24
Suggested packages:
  libevent-perl libio-async-perl libmldbm-perl libbrowser-open-perl libtest-differences-perl graphicsmagick-dbg iwatch libscalar-number-perl libtemplate-perl-doc libtemplate-plugin-gd-perl
  libtemplate-plugin-xml-perl binutils-multiarch libterm-readline-gnu-perl | libterm-readline-perl-perl
The following packages will be REMOVED:
  libcatmandu-marc-perl libcatmandu-perl libcatmandu-store-elasticsearch-perl libdata-alias-perl libdevel-beginlift-perl libparse-cpan-meta-perl
The following NEW packages will be installed:
  libmoox-role-logger-perl libperl5.24 libpng16-16 libscalar-list-utils-perl libtidy5 libwebp6 perl-modules-5.24
The following packages will be upgraded:
  libalgorithm-combinatorics-perl libalgorithm-diff-xs-perl libanyevent-perl libapt-pkg-perl libasync-interrupt-perl libb-compiling-perl libb-hooks-op-check-entersubforcv-perl
  libb-hooks-op-check-perl libb-utils-perl libbareword-filehandles-perl libbit-vector-perl libcache-fastmmap-perl libcache-memcached-fast-perl libcairo-perl libclass-c3-xs-perl
  libclass-load-xs-perl libclass-xsaccessor-perl libclone-perl libcommon-sense-perl libcpanel-json-xs-perl libcrypt-eksblowfish-perl libcrypt-gcrypt-perl libcrypt-ssleay-perl
  libcss-minifier-xs-perl libdata-messagepack-perl libdata-util-perl libdate-calc-xs-perl libdatetime-format-mail-perl libdatetime-locale-perl libdatetime-perl libdatetime-timezone-perl
  libdbd-mysql-perl libdbd-sqlite2-perl libdbd-sqlite3-perl libdbi-perl libdevel-caller-perl libdevel-cover-perl libdevel-declare-perl libdevel-lexalias-perl libdevel-nytprof-perl
  libdevel-size-perl libdigest-sha-perl libev-perl libfcgi-perl libfile-fcntllock-perl libfilter-perl libfont-freetype-perl libgd-perl libglib-perl libgraphics-magick-perl libgssapi-perl
  libguard-perl libhtml-parser-perl libhtml-template-pro-perl libhtml-tidy-perl libhttp-tiny-perl libindirect-perl libio-pty-perl libjavascript-minifier-xs-perl libjson-xs-perl
  liblexical-sealrequirehints-perl liblingua-stem-snowball-perl liblinux-inotify2-perl liblist-moreutils-perl liblocale-gettext-perl libmarpa-r2-perl libmodule-metadata-perl libmoose-perl
  libmultidimensional-perl libnet-ssleay-perl libnet-z3950-zoom-perl libossp-uuid-perl libpackage-stash-xs-perl libpadwalker-perl libpango-perl libparams-classify-perl libparams-util-perl
  libparams-validate-perl libperlio-gzip-perl librdf-perl libreadonly-xs-perl libsereal-decoder-perl libsereal-encoder-perl libsocket6-perl libstring-crc32-perl libsub-identify-perl
  libsub-name-perl libtemplate-perl libterm-readkey-perl libtext-bidi-perl libtext-charwidth-perl libtext-csv-xs-perl libtext-iconv-perl libtext-soundex-perl libtext-unaccent-perl
  libtype-tiny-xs-perl libunicode-map-perl libunicode-utf8-perl libuuid-perl libvariable-magic-perl libversion-perl libxml-libxml-perl libxml-libxslt-perl libxml-parser-perl
  libyaml-libyaml-perl libyaml-syck-perl lintian perl perl-base perl-doc

% pmvers Catmandu
IO.c: loadable library and perl binaries are mismatched (got handshake key 0x8180080, needed 0x81c0080)

% sudo updatedb; locate Catmandu.pm
/usr/share/perl5/Catmandu.pm

% grep VERSION /usr/share/perl5/Catmandu.pm
our $VERSION = '1.0002';

Everything looks... more or less ok
Comment 20 Jonathan Druart 2016-10-20 08:40:44 UTC
(In reply to Jonathan Druart from comment #19)
> Everything looks... more or less ok

% perl misc/search_tools/rebuild_elastic_search.pl -d -a -b -v
misc/search_tools/rebuild_elastic_search.pl: Perl lib version (5.22.2) doesn't match executable '/usr/bin/perl' version (5.24.1) at /usr/lib/i386-linux-gnu/perl/5.22/Config.pm line 62.

more or less

I am using debian testing.
Comment 21 Jonathan Druart 2016-10-20 09:41:56 UTC
So, I have completely exploded my lxc under testing because of https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=836991 (that's the game)

Tried again with kohadevbox (so jessie) and I got the same errors http://pastebin.com/JYSVGfGM
Comment 22 Jonathan Druart 2016-10-20 10:35:25 UTC
I seems better with using the debs from
 deb https://apt.abunchofthings.net/debian jessie main

So:

% sudo apt install libcatmandu-perl
 libcatmandu-perl : Depends: libdata-uuid-perl (>= 1.217)

% sudo apt install libdata-uuid-perl
% sudo apt install libcatmandu-perl
% sudo apt install libcatmandu-store-elasticsearch-perl

rebuild_elastic_search will raise "Can't locate Catmandu/Importer/MARC.pm"

% sudo apt install libcatmandu-marc-perl
 libcatmandu-marc-perl : Depends: libmarc-xml-perl (>= 1.0.3) but 1.0.2-1kohadev1 is to be installed

Because it takes the one from the Koha repo, specify you want the one from Jessie:
% sudo apt install -t jessie libmarc-xml-perl
% sudo apt install libcatmandu-marc-perl

Here you go!
Comment 23 Jonathan Druart 2016-10-20 10:36:39 UTC
Catmandu is 1.0002
Search::Elasticsearch is 2.03
Catmandu::Store::ElasticSearch is 0.0501
Comment 24 Mirko Tietgen 2016-10-20 11:43:18 UTC
(In reply to Chris Cormack from comment #12)

> * We had to change Catmandu::Store::ElasticSearch::Bag
> https://github.com/LibreCat/Catmandu-Store-Elasticsearch/issues/10

Is that maybe related to ES claiming all underscore prefix names?
Comment 25 Nick Clemens 2016-12-30 15:53:14 UTC
I finally got ES2 to index and return results in kohadevbox, I found I had to do a few things:

I used the two .debs attached here and also installed Catmandu from CPAN

In Koha/SearchEngine/Elasticsearch.pm I added the line below just before the return:
+    push @rules, "move_field(_id,my_id)";

In koha-conf I added at the bottom of the elasticsearch stanza:
      <key_prefix>my_</key_prefix>

In koha/SearchEngine/Elasticsearch/Indexer.pm

-      Catmandu::Importer::MARC->new( records => $records, _id => '999c' );
+      Catmandu::Importer::MARC->new( records => $records, id => '999c' );
Comment 26 Olli-Antti Kivilahti 2017-01-09 16:24:46 UTC
Created attachment 58690 [details] [review]
Bug 17255 - Upgrade Elastic Search code to work with version 2.4+ - rebased wip

-Changed deprecated facets to aggregations
-Fixed boolean datatypes not allowing analyzers to be specified
-Fixed deprecated '_id' to 'es_id'. Now the ES-index has the correct id==biblionumber

ZE TEST PLAN

1. Reset Zebra index since facets are hard coded to dynamic search_marc_mappings.
2. perl misc/search_tools/rebuild_elastic_search.pl
3. Fetch all indexed records and the facet for subject__facet

curl -XGET localhost:9200/koha_biblios/data/_search?pretty -d '{
  "aggregations": {
    "my_agg": {
      "terms": {
        "field": "subject__facet"
      }
    }
  }
}'
Comment 27 Olli-Antti Kivilahti 2017-01-09 16:26:18 UTC
Has there been any discussion about migrating to ElasticSearch 5.1 instead of 2.4?

I ran a small test setup back home (using Ubuntu16.04) during the weekends and discovered that I had accidentally installed ElasticSearch 5.1 and Catmandu dependencies from cpan, and that indexing and searching and deleting worked with no probs using the Catmandu::Store::Elasticsearch perldoc examples with "the" compatibility patch in the perldoc.

Will look into that tomorrow and hopefully provide a patch for running with ES 5.1
Comment 28 Chris Cormack 2017-01-09 21:02:27 UTC
(In reply to Olli-Antti Kivilahti from comment #27)
> Has there been any discussion about migrating to ElasticSearch 5.1 instead
> of 2.4?
> 
> I ran a small test setup back home (using Ubuntu16.04) during the weekends
> and discovered that I had accidentally installed ElasticSearch 5.1 and
> Catmandu dependencies from cpan, and that indexing and searching and
> deleting worked with no probs using the Catmandu::Store::Elasticsearch
> perldoc examples with "the" compatibility patch in the perldoc.
> 
> Will look into that tomorrow and hopefully provide a patch for running with
> ES 5.1

Nope no discussion yet (5.1 only came out in December) But I think going to 5.1 seems sensible enough to me
Comment 29 Olli-Antti Kivilahti 2017-01-10 14:45:11 UTC
Created attachment 58757 [details] [review]
Bug 17255 - Upgrade Elastic Search code to work with version 5.1

Builds on top of commit:
    Bug 17255 - Upgrade Elastic Search code to work with version 2.4+ - rebased wip

-Fix data type 'string' to 'keyword' and 'text'
-index: not_analyzed deprecated, replaced with type: keyword which is equivalent
-store: yes was deprecated, use store: true

TODO: Installer bindings to both the debian package install and the raw developer install.
A taster in Buug 17851

ZE TEST PLAN

0. Remove existing ES and reinstall ES 5.1
   apt-get purge elasticsearch
   Follow instruction here:
   https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html

1. Reset Elasticsearch index since facets are hard coded to dynamic search_marc_mappings.
1a. perl -e 'use Koha::SearchEngine::Elasticsearch; Koha::SearchEngine::Elasticsearch->reset_elasticsearch_mappings();'
1b. If you get trouble, simply DELETE FROM [search_fields|search_marc_to_fields|search_marc_mapping];
    and retry 1a.
1c. Destroy elasticsearch index
    curl -XDELETE localhost:9200/koha_biblios
    so it can be recreated

2. Recreate the index:
   perl misc/search_tools/rebuild_elastic_search.pl
2a. Add something to index if your koha.biblio-table is empty

3. Fetch all indexed records and the facet for subject__facet

curl -XGET localhost:9200/koha_biblios/data/_search?pretty -d '{
  "aggregations": {
    "my_agg": {
      "terms": {
        "field": "subject__facet"
      }
    }
  }
}'

4. Run the included tests:
   perl t/db_dependent/Koha_Elasticsearch_Indexer.t
   perl t/db_dependent/Koha_Elasticsearch.t
   perl t/db_dependent/Koha_SearchEngine_Elasticsearch_Search.t

Have fun with your new ES 5.1 cluster!
Comment 30 Olli-Antti Kivilahti 2017-01-10 14:49:05 UTC
Tested on Ubuntu16.04 with:
-ES from their own apt-repo.
-cpanm Catmandu::Store::Elasticsearch

intra search works, facets work, adding records works, full reindex works
Comment 31 Nick Clemens 2017-01-24 11:33:48 UTC
Created attachment 59493 [details] [review]
Bug 17255 - Upgrade Elastic Search code to work with version 2.4+ - rebased wip

-Changed deprecated facets to aggregations
-Fixed boolean datatypes not allowing analyzers to be specified
-Fixed deprecated '_id' to 'es_id'. Now the ES-index has the correct id==biblionumber

ZE TEST PLAN

1. Reset Zebra index since facets are hard coded to dynamic search_marc_mappings.
2. perl misc/search_tools/rebuild_elastic_search.pl
3. Fetch all indexed records and the facet for subject__facet

curl -XGET localhost:9200/koha_biblios/data/_search?pretty -d '{
  "aggregations": {
    "my_agg": {
      "terms": {
        "field": "subject__facet"
      }
    }
  }
}'

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 32 Nick Clemens 2017-01-24 11:33:55 UTC
Created attachment 59494 [details] [review]
Bug 17255 - Upgrade Elastic Search code to work with version 5.1

Builds on top of commit:
    Bug 17255 - Upgrade Elastic Search code to work with version 2.4+ - rebased wip

-Fix data type 'string' to 'keyword' and 'text'
-index: not_analyzed deprecated, replaced with type: keyword which is equivalent
-store: yes was deprecated, use store: true

TODO: Installer bindings to both the debian package install and the raw developer install.
A taster in Buug 17851

ZE TEST PLAN

0. Remove existing ES and reinstall ES 5.1
   apt-get purge elasticsearch
   Follow instruction here:
   https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html

1. Reset Elasticsearch index since facets are hard coded to dynamic search_marc_mappings.
1a. perl -e 'use Koha::SearchEngine::Elasticsearch; Koha::SearchEngine::Elasticsearch->reset_elasticsearch_mappings();'
1b. If you get trouble, simply DELETE FROM [search_fields|search_marc_to_fields|search_marc_mapping];
    and retry 1a.
1c. Destroy elasticsearch index
    curl -XDELETE localhost:9200/koha_biblios
    so it can be recreated

2. Recreate the index:
   perl misc/search_tools/rebuild_elastic_search.pl
2a. Add something to index if your koha.biblio-table is empty

3. Fetch all indexed records and the facet for subject__facet

curl -XGET localhost:9200/koha_biblios/data/_search?pretty -d '{
  "aggregations": {
    "my_agg": {
      "terms": {
        "field": "subject__facet"
      }
    }
  }
}'

4. Run the included tests:
   perl t/db_dependent/Koha_Elasticsearch_Indexer.t
   perl t/db_dependent/Koha_Elasticsearch.t
   perl t/db_dependent/Koha_SearchEngine_Elasticsearch_Search.t

Have fun with your new ES 5.1 cluster!

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 33 Nick Clemens 2017-01-24 11:35:38 UTC
I had to apply the patches interactively, one at a time, but they worked then without issue.
Comment 34 Tomás Cohen Arazi 2017-02-06 16:51:19 UTC
Created attachment 59938 [details] [review]
Bug 17255 - Upgrade Elastic Search code to work with version 2.4+ - rebased wip

-Changed deprecated facets to aggregations
-Fixed boolean datatypes not allowing analyzers to be specified
-Fixed deprecated '_id' to 'es_id'. Now the ES-index has the correct id==biblionumber

ZE TEST PLAN

1. Reset Zebra index since facets are hard coded to dynamic search_marc_mappings.
2. perl misc/search_tools/rebuild_elastic_search.pl
3. Fetch all indexed records and the facet for subject__facet

curl -XGET localhost:9200/koha_biblios/data/_search?pretty -d '{
  "aggregations": {
    "my_agg": {
      "terms": {
        "field": "subject__facet"
      }
    }
  }
}'

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 35 Tomás Cohen Arazi 2017-02-06 16:51:28 UTC
Created attachment 59939 [details] [review]
Bug 17255 - Upgrade Elastic Search code to work with version 5.1

Builds on top of commit:
    Bug 17255 - Upgrade Elastic Search code to work with version 2.4+ - rebased wip

-Fix data type 'string' to 'keyword' and 'text'
-index: not_analyzed deprecated, replaced with type: keyword which is equivalent
-store: yes was deprecated, use store: true

TODO: Installer bindings to both the debian package install and the raw developer install.
A taster in Buug 17851

ZE TEST PLAN

0. Remove existing ES and reinstall ES 5.1
   apt-get purge elasticsearch
   Follow instruction here:
   https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html

1. Reset Elasticsearch index since facets are hard coded to dynamic search_marc_mappings.
1a. perl -e 'use Koha::SearchEngine::Elasticsearch; Koha::SearchEngine::Elasticsearch->reset_elasticsearch_mappings();'
1b. If you get trouble, simply DELETE FROM [search_fields|search_marc_to_fields|search_marc_mapping];
    and retry 1a.
1c. Destroy elasticsearch index
    curl -XDELETE localhost:9200/koha_biblios
    so it can be recreated

2. Recreate the index:
   perl misc/search_tools/rebuild_elastic_search.pl
2a. Add something to index if your koha.biblio-table is empty

3. Fetch all indexed records and the facet for subject__facet

curl -XGET localhost:9200/koha_biblios/data/_search?pretty -d '{
  "aggregations": {
    "my_agg": {
      "terms": {
        "field": "subject__facet"
      }
    }
  }
}'

4. Run the included tests:
   perl t/db_dependent/Koha_Elasticsearch_Indexer.t
   perl t/db_dependent/Koha_Elasticsearch.t
   perl t/db_dependent/Koha_SearchEngine_Elasticsearch_Search.t

Have fun with your new ES 5.1 cluster!

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 36 Kyle M Hall 2017-02-07 17:18:57 UTC
Pushed to master for 17.05, thanks Olli!
Comment 37 Katrin Fischer 2017-02-07 21:01:33 UTC
This won't get ported back to 16.11.x as it is an enhancement.

But: yay!
Comment 38 Jonathan Druart 2017-02-27 14:41:57 UTC
Authority search seems broken, someone faced it?

[Request] ** [http://localhost:9200]-[400] [illegal_argument_exception] Fielddata is disabled on text fields by default. Set fielddata=true on [Heading.phrase] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory., called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /usr/local/share/perl/5.20.2/Catmandu/Store/ElasticSearch/Bag.pm line 159. With vars: {'request' => {'serialize' => 'std','method' => 'GET','body' => {'sort' => [{'Heading.phrase' => 'asc'}],'from' => 0,'size' => 20,'query' => {'bool' => {'should' => [{'match' => {'Heading-Main' => ''}}]}}},'ignore' => [],'mime_type' => 'application/json','path' => '/koha_kohadev_authorities/data/_search','qs' => {}},'status_code' => 400,'body' => {'status' => 400,'error' => {'caused_by' => {'reason' => 'Fielddata is disabled on text fields by default. Set fielddata=true on [Heading.phrase] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory.','type' => 'illegal_argument_exception'},'phase' => 'query','grouped' => bless( do{\(my $o = 1)}, 'JSON::PP::Boolean' ),'failed_shards' => [{'shard' => 0,'node' => 'or12QGCqT2mEEFuFMr4wFA','index' => 'koha_kohadev_authorities','reason' => {'reason' => 'Fielddata is disabled on text fields by default. Set fielddata=true on [Heading.phrase] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory.','type' => 'illegal_argument_exception'}}],'type' => 'search_phase_execution_exception','root_cause' => [{'type' => 'illegal_argument_exception','reason' => 'Fielddata is disabled on text fields by default. Set fielddata=true on [Heading.phrase] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory.'}],'reason' => 'all shards failed'}}}
Unable to perform your search. Please try again.