Information and large patch dumps will go here periodically, so people can see what's happening.
Active development is happening here: http://git.catalyst.net.nz/gw?p=koha.git;a=shortlog;h=refs/heads/elastic_search you can add this to your own repo if you like (be aware it'll be periodically rebased to keep up with master.)
Created attachment 29204 [details] [review] Bug 12478 - an almagamation of all the Elasticsearch code so far
This won't work straight off, to make it work you need to: * set the system preference 'SearchEngine' to 'Elasticsearch' * load installer/data/mysql/elasticsearch_mapping.sql into your database * add something like: <elasticsearch> <server>es-server:9200</server> <index_name>koha_instance</index_name> </elasticsearch> to the config section of koha-conf.xml Perhaps some other things I forgot.
(In reply to Robin Sheat from comment #4) > <elasticsearch> > <server>es-server:9200</server> <server> can be repeated to point to each server in your cluster, too.
Perl dependencies are needed to run Catmandu. Packages aimed at Debian Testing are here: http://www.kallisti.net.nz/~robin/catmandu-deps.tar.gz they might work on Wheezy if you're lucky, but they haven't been tested there.
Created attachment 31128 [details] [review] Bug 12478 - an almagamation of all the Elasticsearch code so far
Created attachment 31129 [details] [review] Bug 12478 - set up database tables for elasticsearch This sets up the database changes needed to run ES. This isn't intended to be ready for upstreaming, just to make testing easier.
Hi, is there somewhere a target described with features that could be provided ? like a rfc or summary or something like that ?
(In reply to claire.hernandez@biblibre.com from comment #9) > Hi, is there somewhere a target described with features that could be > provided ? like a rfc or summary or something like that ? There isn't really, at this stage. For now it's still a case of replicating the functionality of zebra. Once that is done, then new features can be added.
http://git.catalyst.net.nz/gw?p=koha.git;a=shortlog;h=refs/heads/elasticsearch_browser This has the browser (subject browse etc) code on it.
Created attachment 33136 [details] [review] Bug 12478 - add some base objects that the ES code will depend on
Created attachment 33137 [details] [review] Bug 12478 - add some base objects that the ES code will depend on
Created attachment 33138 [details] [review] Bug 12478 - pile of elasticsearch code
Created attachment 33139 [details] [review] Bug 12478 - set up database tables for elasticsearch This sets up the database changes needed to run ES. This isn't intended to be ready for upstreaming, just to make testing easier.
Created attachment 33140 [details] [review] Bug 12478 - authorities can now be stored in ES (Not fetched yet though.)
Created attachment 33141 [details] [review] Bug 12478 - add test cases
I've added a dump of the current state of patches. It's in the process of being split up and having unit tests written, in particular of the underlying modules, i.e. the things that will be needed but aren't central to ES itself. The last functional change was the ability for authorities to be indexed too. Currently, they can't be searched, but they are there. The next functional change (clearly) is to make them be searchable. I'm expecting that 90% of this will leverage the existing query builder type stuff. I'm hoping that someone can have a go setting this up on their own installation and trying it out, to see if there are any particular points that need explained. dcook, I'll be seeing you at the conference tomorrow, I suggest you have a laptop with a VM ready to go on it ;)
(In reply to Robin Sheat from comment #18) > I've added a dump of the current state of patches. It's in the process of > being split up and having unit tests written, in particular of the > underlying modules, i.e. the things that will be needed but aren't central > to ES itself. > > The last functional change was the ability for authorities to be indexed > too. Currently, they can't be searched, but they are there. The next > functional change (clearly) is to make them be searchable. I'm expecting > that 90% of this will leverage the existing query builder type stuff. > > I'm hoping that someone can have a go setting this up on their own > installation and trying it out, to see if there are any particular points > that need explained. > > dcook, I'll be seeing you at the conference tomorrow, I suggest you have a > laptop with a VM ready to go on it ;) Alas, I'll have to try it out another time. Working on some OAI stuff at the moment, but will have to look at this sometime soon!
http://elasticsearch.koha.catalystdemo.net.nz/cgi-bin/koha/opac-main.pl This is running Ubuntu 14.10 (I couldn't convince things to work on 14.04, I'd expect Debian Jessie to work also), it has zebra turned off, and the OPAC search interface is running through ES. The code it's running is taken directly from the catalyst repo, though I probably won't update it when I know things are broken. Otherwise I'll try to keep it current. If you notice any weirdnesses, feel free to let me know.
Is there a doc somewhere? I did not find it... I tried but failed: $ git remote add catalyst git://git.catalyst.net.nz/koha.git $ git checkout -b elastic_search catalyst/elastic_search $ perl installer/data/mysql/updatedatabase.pl $ sudo apt-get install elasticsearch $ perl misc/search_tools/rebuild_elastic_search.pl -h Can't locate Elasticsearch.pm in @INC (you may need to install the Elasticsearch module) (@INC contains: blablabla) $ apt-cache search elasticsearch | grep perl libcatmandu-perl - metadata toolkit $ sudo apt-get install libcatmandu-perl But got the same error. The ElasticSearch module on the cpan is marked as deprecated: http://search.cpan.org/~drtech/ElasticSearch-0.68/lib/ElasticSearch.pm
I also added the lines in the $KOHA_CONF file.
(In reply to Jonathan Druart from comment #21) > $ perl misc/search_tools/rebuild_elastic_search.pl -h > Can't locate Elasticsearch.pm in @INC (you may need to install the > Elasticsearch module) (@INC contains: blablabla) > > $ apt-cache search elasticsearch | grep perl > libcatmandu-perl - metadata toolkit In comment #6 I added a link to the dependencies needed. It's a bit old now, I've been working on making some new ones, but that archive may still work.
Back here, $ mkdir catmandu-deps $ cd catmandu-deps $ wget www.kallisti.net.nz/~robin/catmandu-deps.tar.gz $ tar zxvf catmandu-deps.tar.gz $ sudo dpkg -i *.deb I got some dpkg: warning: downgrading $package from $version_a+ to $version_a And it finished with: dpkg: dependency problems prevent configuration of libdata-messagepack-perl: libdata-messagepack-perl depends on perlapi-5.18.2; however: Package perlapi-5.18.2 is not installed. dpkg: error processing package libdata-messagepack-perl (--install): dependency problems - leaving unconfigured Setting up libdata-spreadpagination-perl (0.1.2-1) ... Setting up libdispatch-class-perl (0.01-1) ... Setting up libelasticsearch-perl (1.04-1) ... Setting up libhijk-perl (0.13-1) ... Setting up libhttp-tiny-perl (0.043-1) ... Setting up libjson-maybexs-perl (1.002002-1) ... Setting up liblog-any-adapter-callback-perl (0.07-1) ... dpkg: dependency problems prevent configuration of libmarpa-r2-perl: libmarpa-r2-perl depends on perlapi-5.18.2; however: Package perlapi-5.18.2 is not installed. dpkg: error processing package libmarpa-r2-perl (--install): dependency problems - leaving unconfigured Setting up libmoox-log-any-perl (0.001-1) ... Setting up libsearch-elasticsearch-perl (1.11-1) ... Setting up libtry-tiny-byclass-perl (0.01-1) ... Setting up libyaml-perl (0.90-1) ... dpkg: dependency problems prevent configuration of libcatmandu-perl: libcatmandu-perl depends on libmarpa-r2-perl (>= 2.084000); however: Package libmarpa-r2-perl is not configured yet. dpkg: error processing package libcatmandu-perl (--install): dependency problems - leaving unconfigured dpkg: dependency problems prevent configuration of libcatmandu-store-elasticsearch-perl: libcatmandu-store-elasticsearch-perl depends on libcatmandu-perl (>= 0.8); however: Package libcatmandu-perl is not configured yet. dpkg: error processing package libcatmandu-store-elasticsearch-perl (--install): dependency problems - leaving unconfigured Setting up libelasticsearch-compat-perl (0.03-1) ... Setting up libsearch-elasticsearch-compat-perl (0.10-1) ... dpkg: dependency problems prevent configuration of libcatmandu-marc-perl: libcatmandu-marc-perl depends on libcatmandu-perl (>= 0.08); however: Package libcatmandu-perl is not configured yet. dpkg: error processing package libcatmandu-marc-perl (--install): dependency problems - leaving unconfigured Processing triggers for man-db (2.6.7.1-1) ... Errors were encountered while processing: libdata-messagepack-perl libmarpa-r2-perl libcatmandu-perl libcatmandu-store-elasticsearch-perl libcatmandu-marc-perl Note: libossp-uuid-perl was missing, I installed it. I have perl v.5.20 installed.
Yeah, they worked on Ubuntu 14.10 last I looked. I'm in the process of making a new set for Debian Jessie. As for the deprecated bits, there are some issues there that make it not quite worth messing with the new versions just yet. Check out the discussion thread here: http://mail.librecat.org/pipermail/librecat-dev/2015-January/000322.html There should be very few, if any, API changes as the result of switching to the new version when it's suitable, too.
You can grab the new dependencies from here: http://debian.koha-community.org/koha/otherthings/ make sure you uninstall liblog-any-adapter-perl before doing anything, it conflicts with the new liblog-any-perl, and is no longer needed. I've added a dummy version of it to keep those packages that think they need it happy. It should just be a matter of doing: sudo dpkg -i *.deb sudo apt-get -f install Oh, and this requires Debian Jessie, as most of the old dependencies made it into there.
Looks better with these new packages. Continuing debugging... Catmandu::Importer::MARC was missing, I installed it using cpanm: $ sudo cpanm Catmandu::Importer::MARC Now I got: $ perl rebuild_elastic_search.pl 1 to 115 is displayed Committing... And nothing else... $ ps aux | grep elasticsearch returns nothing $ sudo service elasticsearch start $ ps aux | grep elasticsearch returns nothing $ curl -X GET http://localhost:9200/ curl: (7) Failed to connect to localhost port 9200: Connection refused Editing /etc/init.d/elasticsearch, I found the binary: /usr/share/elasticsearch/bin/elasticsearch $ cd /usr/share/elasticsearch $ slog4j:WARN No appenders could be found for logger (node). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.udo bin/elasticsearch $ apt-cache show elasticsearch Package: elasticsearch Version: 1.0.3+dfsg-5 Should I installed another version? Something is missing in my jvm configuration? Something to do in the elasticsearch config?
(In reply to Jonathan Druart from comment #27) > Looks better with these new packages. > Continuing debugging... > > Catmandu::Importer::MARC was missing, I installed it using cpanm: > $ sudo cpanm Catmandu::Importer::MARC Catmandu::Importer::MARC is provided by libcatmandu-marc-perl which is in Jessie already. > Now I got: > $ perl rebuild_elastic_search.pl > 1 to 115 is displayed > Committing... > > And nothing else... > > $ ps aux | grep elasticsearch > returns nothing > $ sudo service elasticsearch start > $ ps aux | grep elasticsearch > returns nothing That's not really my code's fault :) you may need to configure elasticsearch before it starts. Don't forget to change the clustername in /etc/elasticsearch/elasticsearch.yml, otherwise you may have it clustering with other ES instances on the same network. This said, it shouldn't just sit there if the daemon isn't available, I've added a note to look into that. > $ curl -X GET http://localhost:9200/ > curl: (7) Failed to connect to localhost port 9200: Connection refused Yep, that there is your issue. $ curl -X GET http://koha-es:9200/ { "status" : 200, "name" : "koha-es", "version" : { "number" : "1.3.7", "build_hash" : "3042293e4b219dfb855a4e6c64241c530d1abeb0", "build_timestamp" : "2014-12-16T13:59:32Z", "build_snapshot" : false, "lucene_version" : "4.9" }, "tagline" : "You Know, for Search" } > Editing /etc/init.d/elasticsearch, I found the binary: > /usr/share/elasticsearch/bin/elasticsearch > $ cd /usr/share/elasticsearch > $ slog4j:WARN No appenders could be found for logger (node). > log4j:WARN Please initialize the log4j system properly. > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for > more info.udo bin/elasticsearch > > $ apt-cache show elasticsearch > Package: elasticsearch > Version: 1.0.3+dfsg-5 $ apt-cache policy elasticsearch elasticsearch: Installed: 1.3.7 Candidate: 1.3.7 Version table: *** 1.3.7 0 500 http://packages.elasticsearch.org/elasticsearch/1.3/debian/ stable/main amd64 Packages 100 /var/lib/dpkg/status > Should I installed another version? Something is missing in my jvm > configuration? Something to do in the elasticsearch config? Try the one from the official repo, I'm not sure what differences are in the debian provided version. It might just be you need some log4j stuff, but I can only guess. http://www.elasticsearch.org/blog/apt-and-yum-repositories/
$ sudo apt-get install libcatmandu-marc-perl Try with the lastest (1.4.2): $ wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.2.deb $ sudo dpkg -i elasticsearch-1.4.2.deb Selecting previously unselected package elasticsearch. (Reading database ... 59178 files and directories currently installed.) Preparing to unpack .../koha/elasticsearch-1.4.2.deb ... Unpacking elasticsearch (1.4.2) ... Setting up elasticsearch (1.4.2) ... Adding system user `elasticsearch' (UID 106) ... Adding new user `elasticsearch' (UID 106) with group `elasticsearch' ... Not creating home directory `/usr/share/elasticsearch'. ### NOT starting elasticsearch by default on bootup, please execute sudo update-rc.d elasticsearch defaults 95 10 ### In order to start elasticsearch, execute sudo /etc/init.d/elasticsearch start $ sudo update-rc.d elasticsearch defaults 95 10 $ sudo /etc/init.d/elasticsearch start $ ps aux | grep elasticsearch elastic+ 5715 75.4 1.6 1326700 131172 ? Sl 09:57 0:03 /usr/lib/jvm/java-7-openjdk-i386//bin/java -Xms256m -Xmx1g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.4.2.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch better :) But the indexing is still stuck on "Committing..." I am using an UNIMARC DB, does the problem could be related to a bad mapping?
(In reply to Jonathan Druart from comment #29) > Try with the lastest (1.4.2): > > $ wget > https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch- > 1.4.2.deb For what it's worth, I'm using 1.3.7 which is the latest stable. I wouldn't think that would make a difference really though. > But the indexing is still stuck on "Committing..." > > I am using an UNIMARC DB, does the problem could be related to a bad mapping? It shouldn't be, the mapping process is pretty simple and I'd expect it to just not succeed rather than cause it to lock up. Sitting on "committing" is consistent with not being able to talk to the ES server, and there's currently no error handling for that (I'd expect it to time out eventually, but who knows how long that'll take.) If I were you, that's where I'd start looking. Make sure you can hit it with curl, and that the config in koha-conf.xml is correct. fwiw, mine is: <elasticsearch> <server>koha-es:9200</server> <index_name>koha_robin</index_name> </elasticsearch> If you're keen, fire it up in the debugger (perl -d) and do: c Catmandu::Store::ElasticSearch::Bag::commit and trace through from there, but after that point it gets pretty hard to drill down further I found. In theory, adding a <timeout> value to the elasticsearch block in the config should cause that to be passed along to the other code, but that didn't seem to happen with a quick test, so I might be wrong there.
Hi Robin, having a discussion about his on IRC right now - we are all quite curious about your work - could you write up some installation docs and/or a summary what the ES patches will include/not include?
(In reply to Robin Sheat from comment #30) > (In reply to Jonathan Druart from comment #29) > > Try with the lastest (1.4.2): > > > > $ wget > > https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch- > > 1.4.2.deb > > For what it's worth, I'm using 1.3.7 which is the latest stable. I wouldn't > think that would make a difference really though. Will try with 1.3.7, but I believe I got the same error as before ( with this version. > Sitting on "committing" is consistent with not being able to talk to the ES > server, and there's currently no error handling for that (I'd expect it to > time out eventually, but who knows how long that'll take.) If I were you, > that's where I'd start looking. Make sure you can hit it with curl, and that > the config in koha-conf.xml is correct. fwiw, mine is: > > <elasticsearch> > <server>koha-es:9200</server> > <index_name>koha_robin</index_name> > </elasticsearch> Yes, I have something similar: 303 <elasticsearch> 304 <server>http://localhost:9200/</server> 305 <index_name>koha_instance</index_name> 306 </elasticsearch> > If you're keen, fire it up in the debugger (perl -d) and do: > > c Catmandu::Store::ElasticSearch::Bag::commit Actually it never reach commit: % perl -d misc/search_tools/rebuild_elastic_search.pl Loading DB routines from perl5db.pl version 1.44 Editor support available. Enter h or 'h h' for help, or 'man perldebug' for more help. main::(misc/search_tools/rebuild_elastic_search.pl:92): 92: my $verbose = 0; DB<1> c Catmandu::Addable::add_many Indexing biblios 3 Catmandu::Addable::add_many(/usr/share/perl5/Catmandu/Addable.pm:84): 84: my ($self, $many) = @_; DB<2> n Catmandu::Addable::add_many(/usr/share/perl5/Catmandu/Addable.pm:86): 86: if (is_hash_ref($many)) { DB<2> Catmandu::Addable::add_many(/usr/share/perl5/Catmandu/Addable.pm:91): 91: if (is_array_ref($many)) { DB<2> Catmandu::Addable::add_many(/usr/share/perl5/Catmandu/Addable.pm:96): 96: if (is_invocant($many)) { DB<2> Catmandu::Addable::add_many(/usr/share/perl5/Catmandu/Addable.pm:97): 97: $many = check_able($many, 'generator')->generator; DB<2> Catmandu::Addable::add_many(/usr/share/perl5/Catmandu/Addable.pm:100): 100: check_code_ref($many); DB<2> Catmandu::Addable::add_many(/usr/share/perl5/Catmandu/Addable.pm:102): 102: my $data; DB<2> Catmandu::Addable::add_many(/usr/share/perl5/Catmandu/Addable.pm:103): 103: my $n = 0; DB<2> Catmandu::Addable::add_many(/usr/share/perl5/Catmandu/Addable.pm:104): 104: while (defined($data = $many->())) { DB<2> use Data::Dumper; warn Dumper $many; $VAR1 = sub { "DUMMY" }; at (eval 1256)[/usr/share/perl/5.20/perl5db.pl:732] line 2. eval 'no strict; ($@, $!, $^E, $,, $/, $\\, $^W) = @DB::saved;package Catmandu::Addable; $^D = $^D | $DB::db_stop; use Data::Dumper; warn Dumper $many;; ' called at /usr/share/perl/5.20/perl5db.pl line 732 DB::eval called at /usr/share/perl/5.20/perl5db.pl line 3094 DB::DB called at /usr/share/perl5/Catmandu/Addable.pm line 104 Catmandu::Addable::add_many(Catmandu::Store::ElasticSearch::Bag=HASH(0xd5c503c), Catmandu::Iterator=CODE(0xc565080)) called at Koha/ElasticSearch/Indexer.pm line 80 Koha::ElasticSearch::Indexer::update_index(Koha::ElasticSearch::Indexer=HASH(0xcd60c00), ARRAY(0xca8ca80), ARRAY(0xca8cae4)) called at misc/search_tools/rebuild_elastic_search.pl line 184 main::do_reindex(CODE(0xcae4dc0), "biblios") called at misc/search_tools/rebuild_elastic_search.pl line 132 DB<3> warn ref($many); CODE at (eval 1257)[/usr/share/perl/5.20/perl5db.pl:732] line 2. at (eval 1257)[/usr/share/perl/5.20/perl5db.pl:732] line 2. eval 'no strict; ($@, $!, $^E, $,, $/, $\\, $^W) = @DB::saved;package Catmandu::Addable; $^D = $^D | $DB::db_stop; warn ref($many);; ' called at /usr/share/perl/5.20/perl5db.pl line 732 DB::eval called at /usr/share/perl/5.20/perl5db.pl line 3094 DB::DB called at /usr/share/perl5/Catmandu/Addable.pm line 104 Catmandu::Addable::add_many(Catmandu::Store::ElasticSearch::Bag=HASH(0xd5c503c), Catmandu::Iterator=CODE(0xc565080)) called at Koha/ElasticSearch/Indexer.pm line 80 Koha::ElasticSearch::Indexer::update_index(Koha::ElasticSearch::Indexer=HASH(0xcd60c00), ARRAY(0xca8ca80), ARRAY(0xca8cae4)) called at misc/search_tools/rebuild_elastic_search.pl line 184 main::do_reindex(CODE(0xcae4dc0), "biblios") called at misc/search_tools/rebuild_elastic_search.pl line 132 DB<4> n And nothing else (no CPU activity neither). the ref($many) returning "CODE" does not smell good :) continuing DB<4> n ^CqMARC::File::USMARC::_next(/usr/share/perl5/MARC/File/USMARC.pm:53): 53: local $/ = END_OF_RECORD; DB<4> q % pmvers MARC::File::USMARC /usr/bin/pmvers: unknown version for module `MARC::File::USMARC' hum... Since I installed it using the CPAN first, I removed it: % sudo cpanm -U Catmandu::Importer::MARC [...] Successfully uninstalled Catmandu::Importer::MARC % dpkg -l libcatmandu-marc-perl ii libcatmandu-marc-perl | 0.206-2 | all | modules for working with MARC data within the Catmandu framework % perl misc/search_tools/rebuild_elastic_search.pl [lot of logs] 25888 records indexed. \o/ Sorry about that! I should have tried pmpath MARC::File::USMARC to catch the problem quickly. Now, search records! OPAC: cgi-bin/koha/opac-search.pl returns Can't locate Koha/ElasticSearch/Search.pm in @INC It's seem to be caused by the last commit Bug 12478 - authority work in progress diff --git a/Koha/ElasticSearch/Search.pm b/Koha/ElasticSearch/Search.pm deleted file mode 100644 I tried to fix some compilation errors (patch coming), but the branch looks to be let in an unusable state. The next error is: Can't locate object method "mk_accessors" via package "Koha::SearchEngine::ElasticSearch::Search" I don't want to continue and add conflicts with something you have already fixed.
previous comment tldr: indexing: OK searching: KO, branch in an unusable state.
Created attachment 35654 [details] [review] Bug 12478: fix some compilation errors Feel free to obsolete this patch.
(In reply to Jonathan Druart from comment #33) > previous comment tldr: > indexing: OK Yay! > searching: KO, branch in an unusable state. Yeah, I forgot to mention that sorry. Roll it back a bit, I'm in the middle of a bit of refactoring (just moving a module to a more consistent place, but I haven't updated the references to it yet), and also developing the authorities searching (though only some of that has been pushed so far.) I'll try to get it to a properly working state again today.
(In reply to Robin Sheat from comment #35) > I'll try to get it to a properly working state again today. And done, it should work again. I'll try to put together a wiki page with some info.
(In reply to Robin Sheat from comment #36) > I'll try to put together a wiki page with some info. http://wiki.koha-community.org/wiki/Elasticsearch It's a quick brain dump, so feel free to update and polish if I've missed things.
Thanks Robin for the wiki page. I have quickly tested this morning, to confirm the search works. Just some quick notes: - The indexation should commit every 1k biblio (minimum), the commit is a slow op indeed. - You have removed the use of search/results.tt in your last commit, is it intended? I am pretty sure it's not a good idea to use the same template for ES and Zebra, but maybe it's temporary. Note that there are encoding errors in facets (and in the table results also), the "Show more" link appears even if only 1 entry is displayed. I only get 2 facets: authors and itemtype. Do you plan to rebase your work against master? It would be great to see these patches on top of bug 11944.
(In reply to Jonathan Druart from comment #38) > Thanks Robin for the wiki page. > I have quickly tested this morning, to confirm the search works. > Just some quick notes: > - The indexation should commit every 1k biblio (minimum), the commit is a > slow op indeed. Yeah. The commit rate I'm defining is currently actually not useful as Catmandu has its own buffering and committing system. I think that's changeable though. I also think the whole indexing process can be optimised a fair bit. I just haven't looked into it yet. > - You have removed the use of search/results.tt in your last commit, is it > intended? I am pretty sure it's not a good idea to use the same template for > ES and Zebra, but maybe it's temporary. Hmm? I'm not totally sure what you mean here. I am totally deliberately using the same template to show ES results as also shows zebra results, that's by design. Any other way would require copy-pasting 99% of the template, when there's already a perfectly good one that shows search results. I'm also trying to make a bit of an abstract search layer thing, not totally perfectly, to make it easier to work with this sort of thing in the future (very much based off your solr stuff in that respect.) > Note that there are encoding errors in facets (and in the table results > also), the "Show more" link appears even if only 1 entry is displayed. Yep, definitely. There'll be a lot of things like that that just need to be polished. > I only get 2 facets: authors and itemtype. MARC21 or UNIMARC? You should get more, though I haven't looked at that for a while. For example, http://elasticsearch.koha.catalystdemo.net.nz/cgi-bin/koha/opac-search.pl?q=chicken give availability, item types, authors and topics. But the facet stuff as a whole will need more work, I mostly got it to the point where it works at all and then moved on. > Do you plan to rebase your work against master? It would be great to see > these patches on top of bug 11944. Yes, I do. I'm just a little afraid to as I know it'll conflict with many, many things. One day soon I'll suck it up and do it though.
(In reply to Robin Sheat from comment #39) > MARC21 or UNIMARC? You should get more, though I haven't looked at that for > a while. > > For example, > http://elasticsearch.koha.catalystdemo.net.nz/cgi-bin/koha/opac-search. > pl?q=chicken > > give availability, item types, authors and topics. But the facet stuff as a > whole will need more work, I mostly got it to the point where it works at > all and then moved on. Oh, whether a field is suitable for faceting is now defined in the mapping table in the database. This might be why it's not working if you're in unimarc, the field definitions may not be correct for it. I want (eventually) to extend this to make what is facetable configurable, at the moment it's hardcoded into the template.
FYI, the catalyst repo branch now has basic authority search working. There's still a good bit to do (paging, biblio counts, many more things aren't there yet), but results are coming out so I'm counting that as a win :) You can see it in action here: http://elasticsearch.koha.catalystdemo.net.nz/cgi-bin/koha/opac-authorities-home.pl?op=do_search&type=opac&operator=contains&value=robert&marclist=any&and_or=and&orderby=HeadingAsc I think the fact that there are so many Jordan, Roberts is due to the data, but I haven't actually checked yet. At the moment it just replicates how the zebra version works, but I do want to push some of the things into the indexing side so that there's less computation needed to display results and things can perhaps be made a bit simpler. We can do this because we can store arbitrary fields in elasticsearch alongside the actual records.
(In reply to Robin Sheat from comment #41) > FYI, the catalyst repo branch now has basic authority search working. > There's still a good bit to do (paging, biblio counts, many more things > aren't there yet), but results are coming out so I'm counting that as a win > :) > > You can see it in action here: > > http://elasticsearch.koha.catalystdemo.net.nz/cgi-bin/koha/opac-authorities- > home. > pl?op=do_search&type=opac&operator=contains&value=robert&marclist=any&and_or= > and&orderby=HeadingAsc > > I think the fact that there are so many Jordan, Roberts is due to the data, > but I haven't actually checked yet. At the moment it just replicates how the > zebra version works, but I do want to push some of the things into the > indexing side so that there's less computation needed to display results and > things can perhaps be made a bit simpler. We can do this because we can > store arbitrary fields in elasticsearch alongside the actual records. Dear Robin, It is a great job! Thanks a lot. I tried to install a ES. It seems to index the biblio. But I can't search the record."No results found!" I can find the record by Zebra. koha@koha:~$ /home/koha/kohaclone/misc/search_tools/rebuild_elastic_search.pl -v -d Indexing biblios 1 Could you give some advice?
(In reply to Peter Zhao from comment #42) > It is a great job! Thanks a lot. I tried to install a ES. > It seems to index the biblio. But I can't search the record."No > results found!" I can find the record by Zebra. > koha@koha:~$ > /home/koha/kohaclone/misc/search_tools/rebuild_elastic_search.pl -v -d > Indexing biblios > 1 > > Could you give some advice? There's not really enough info to go on there. I'd start by looking in the logs. Many operations should be filling them full of search traces and so on, so that might have useful things in it. Have you followed the steps in here: http://wiki.koha-community.org/wiki/Elasticsearch in particular, adding the mapping SQL file into the database.
(In reply to Robin Sheat from comment #43) > (In reply to Peter Zhao from comment #42) > > It is a great job! Thanks a lot. I tried to install a ES. > > It seems to index the biblio. But I can't search the record."No > > results found!" I can find the record by Zebra. > > koha@koha:~$ > > /home/koha/kohaclone/misc/search_tools/rebuild_elastic_search.pl -v -d > > Indexing biblios > > 1 > > > > Could you give some advice? > > There's not really enough info to go on there. I'd start by looking in the > logs. Many operations should be filling them full of search traces and so > on, so that might have useful things in it. > > Have you followed the steps in here: > http://wiki.koha-community.org/wiki/Elasticsearch > in particular, adding the mapping SQL file into the database. I followed the steps in "http://wiki.koha-community.org/wiki/Elasticsearch",and also added the mapping SQL file into the database. I useed Ubuntu 14.10 to install the ES Koha. The following is the information of elasticsearch log file. [2015-03-06 12:27:18,359][INFO ][node ] [koha] version[1.3.7], pid[3448], build[3042293/2014-12-16T13:59:32Z] [2015-03-06 12:27:18,360][INFO ][node ] [koha] initializing ... [2015-03-06 12:27:18,367][INFO ][plugins ] [koha] loaded [], sites [] [2015-03-06 12:27:23,291][INFO ][node ] [koha] initialized [2015-03-06 12:27:23,292][INFO ][node ] [koha] starting ... [2015-03-06 12:27:23,476][INFO ][transport ] [koha] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.1.3:9300]} [2015-03-06 12:27:23,532][INFO ][discovery ] [koha] koha/yPYdtMzrTtucBGGFgGYAnw [2015-03-06 12:27:26,588][INFO ][cluster.service ] [koha] new_master [koha][yPYdtMzrTtucBGGFgGYAnw][koha][inet[/192.168.1.3:9300]], reason: zen-disco-join (elected_as_master) [2015-03-06 12:27:26,644][INFO ][http ] [koha] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.1.3:9200]} [2015-03-06 12:27:26,649][INFO ][node ] [koha] started [2015-03-06 12:27:26,728][INFO ][gateway ] [koha] recovered [0] indices into cluster_state [2015-03-06 12:28:45,881][INFO ][cluster.metadata ] [koha] [koha_biblios] creating index, cause [api], shards [5]/[1], mappings [data] [2015-03-06 12:28:46,740][INFO ][cluster.metadata ] [koha] [koha_biblios] deleting index [2015-03-06 12:29:35,050][INFO ][cluster.metadata ] [koha] [koha_biblios] creating index, cause [api], shards [5]/[1], mappings [] [2015-03-06 12:31:41,222][INFO ][node ] [koha] stopping ... [2015-03-06 12:31:41,338][INFO ][node ] [koha] stopped [2015-03-06 12:31:41,339][INFO ][node ] [koha] closing ... [2015-03-06 12:31:41,352][INFO ][node ] [koha] closed
(In reply to Robin Sheat from comment #43) > (In reply to Peter Zhao from comment #42) > > It is a great job! Thanks a lot. I tried to install a ES. > > It seems to index the biblio. But I can't search the record."No > > results found!" I can find the record by Zebra. > > koha@koha:~$ > > /home/koha/kohaclone/misc/search_tools/rebuild_elastic_search.pl -v -d > > Indexing biblios > > 1 > > > > Could you give some advice? > > There's not really enough info to go on there. I'd start by looking in the > logs. Many operations should be filling them full of search traces and so > on, so that might have useful things in it. > > Have you followed the steps in here: > http://wiki.koha-community.org/wiki/Elasticsearch > in particular, adding the mapping SQL file into the database. The following is the "koha-opac-error log " [Fri Mar 06 12:30:19.551872 2015] [cgi:error] [pid 3674] [client 127.0.0.1:45890] AH01215: [Fri Mar 6 12:30:19 2015] opac-search.pl: Use of uninitialized value $f in hash element at /usr/share/koha/lib/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm line 479., referer: http://127.0.1.1/cgi-bin/koha/opac-search.pl?idx=ti&q=theology [Fri Mar 06 12:33:43.232092 2015] [cgi:error] [pid 3759] [client 127.0.0.1:45904] AH01215: [Fri Mar 6 12:33:43 2015] opac-search.pl: Use of uninitialized value $f in hash element at /usr/share/koha/lib/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm line 479., referer: http://127.0.1.1/cgi-bin/koha/opac-search.pl?idx=&q=theology [Fri Mar 06 12:33:43.398916 2015] [cgi:error] [pid 3759] [client 127.0.0.1:45904] AH01215: [Fri Mar 6 12:33:43 2015] opac-search.pl: Use of uninitialized value $error in concatenation (.) or string at /usr/share/koha/opac/cgi-bin/opac/opac-search.pl line 568., referer: http://127.0.1.1/cgi-bin/koha/opac-search.pl?idx=&q=theology
It might be useful to ensure that they indexed right, you can do this: curl -XGET 'http://localhost:9200/koha_biblios/_search?pretty=1' where 'koha_biblios' is whatever your index is called. This'll give you a dump of everything in the index.
(In reply to Robin Sheat from comment #46) > It might be useful to ensure that they indexed right, you can do this: > > curl -XGET 'http://localhost:9200/koha_biblios/_search?pretty=1' > > where 'koha_biblios' is whatever your index is called. This'll give you a > dump of everything in the index. ----------- $ curl -XGET 'http://localhost:9200/koha_biblios/_search?pretty=1' { "error" : "IndexMissingException[[koha_biblios] missing]", "status" : 404 } ---------- $ /home/koha/kohaclone/misc/search_tools/rebuild_elastic_search.pl -v -d Indexing biblios 1 ----------------- I just catalogued one record to try. But it always stay there. I think it cannot finish to build the index. How to handle this problem?
404 means there is no index with that name, you need to verify that the index name that you're providing is correct. You can do that with: curl 'koha-es:9200/_cat/indices?v' It is possible that there's a bug that if there aren't enough records to commit, nothing happens. I haven't tested that case yet. Try setting the commit option to the rebuild to '1'
Yay, the fundamentals of authority searching are now working (biblio count and paging.) Next up is making sure the various types of searches result in the right results.
(In reply to Robin Sheat from comment #48) > 404 means there is no index with that name, you need to verify that the > index name that you're providing is correct. You can do that with: > > curl 'koha-es:9200/_cat/indices?v' > > It is possible that there's a bug that if there aren't enough records to > commit, nothing happens. I haven't tested that case yet. Try setting the > commit option to the rebuild to '1' I try to set the commit option to the rebuild to '1'. And then when rebuild the record, it stays on “ is commiting”. ~$ /home/koha/kohaclone/misc/search_tools/rebuild_elastic_search.pl -v -d Indexing biblios 1 Committing... ----------- ~$ curl 'localhost:9200/_cat/indices?v' health index pri rep docs.count docs.deleted store.size pri.store.size yellow koha_biblios 5 1 0 0 575b 575b ------------- ~$ curl -XGET 'http://localhost:9200/koha_biblios/_search?pretty=1' { "took" : 1, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 0, "max_score" : null, "hits" : [ ] } } ----------- koha-conf.xml <elasticsearch> <server>localhost:9200</server> <index_name>koha</index_name> </elasticsearch> ---------- elasticsearch.yml cluster.name: koha node.name: "koha" -------------- Should I change more setting information?
How long did you let it sit on 'Committing', the first one can take some time (in the order of 10 seconds or so perhaps.) Otherwise, it's hard to see what's going on without real information but it does look like things aren't indexing well. In Koha/ElasticSearch/Indexer.pm there are a couple of commented out 'trace_calls' entries. If you make that active, you may get more information about what's going on. Otherwise perhaps trace through it using the Perl debugger and see where it stops.
(In reply to Robin Sheat from comment #51) > How long did you let it sit on 'Committing', the first one can take some > time (in the order of 10 seconds or so perhaps.) > > Otherwise, it's hard to see what's going on without real information but it > does look like things aren't indexing well. > > In Koha/ElasticSearch/Indexer.pm there are a couple of commented out > 'trace_calls' entries. If you make that active, you may get more information > about what's going on. > > Otherwise perhaps trace through it using the Perl debugger and see where it > stops. ~$ perl -d /home/koha/kohaclone/misc/search_tools/rebuild_elastic_search.pl Loading DB routines from perl5db.pl version 1.33 Editor support available. Enter h or `h h' for help, or `man perldebug' for more help. main::(/home/koha/kohaclone/misc/search_tools/rebuild_elastic_search.pl:92): 92: my $verbose = 0; DB<1>
(In reply to Peter Zhao from comment #52) > (In reply to Robin Sheat from comment #51) > > How long did you let it sit on 'Committing', the first one can take some > > time (in the order of 10 seconds or so perhaps.) > > > > Otherwise, it's hard to see what's going on without real information but it > > does look like things aren't indexing well. > > > > In Koha/ElasticSearch/Indexer.pm there are a couple of commented out > > 'trace_calls' entries. If you make that active, you may get more information > > about what's going on. > > > > Otherwise perhaps trace through it using the Perl debugger and see where it > > stops. > > ~$ perl -d /home/koha/kohaclone/misc/search_tools/rebuild_elastic_search.pl > > Loading DB routines from perl5db.pl version 1.33 > Editor support available. > > Enter h or `h h' for help, or `man perldebug' for more help. > > main::(/home/koha/kohaclone/misc/search_tools/rebuild_elastic_search.pl:92): > 92: my $verbose = 0; > DB<1> The problem is fixed by "cpan Task::Catmandu". Index works well. ~$ /home/koha/kohaclone/misc/search_tools/rebuild_elastic_search.pl -v -d Indexing biblios 1 Committing... 1 records indexed. Indexing authorities 0 records indexed.
(In reply to Peter Zhao from comment #52) > Enter h or `h h' for help, or `man perldebug' for more help. > > main::(/home/koha/kohaclone/misc/search_tools/rebuild_elastic_search.pl:92): > 92: my $verbose = 0; > DB<1> You would actually need to trace it through after this point :) (In reply to Peter Zhao from comment #53) > The problem is fixed by "cpan Task::Catmandu". > Index works well. Unfortunately, that doesn't solve the cause of the problem. It shouldn't be necessary to cpan anything. It would be better to know what the missing module is so that I can make it fail properly if it's missing.
Authority searching is now "complete", by which I mean it seems to work but there are bound to be issues in it.
I've rebased the branch on top of current master* and pushed it into the catalyst repo. This may cause the SearchEngine preference to vanish, if so, just go and re-set it from the system preferences. * git can get upset if a file is in the thing you're rebasing onto, but there are still patches that have to go against it. It ended up grabbing a totally unrelated file that presumably was the closest match and shoving the patches in there. That's why there's the occasionally weird touching of Koha::Template::Plugin::Price.
Just a note that I've got basic staff client working on my demo server: http://elasticsearch.koha.catalystdemo.net.nz/
(In reply to Robin Sheat from comment #57) > Just a note that I've got basic staff client working on my demo server: > http://elasticsearch.koha.catalystdemo.net.nz/ Staff client /searching/
(In reply to Robin Sheat from comment #57) > Just a note that I've got basic staff client working on my demo server: > http://elasticsearch.koha.catalystdemo.net.nz/ "Basic searching there should be working, anything else may explode or otherwise fail." Does it mean we can make you some feedbacks or you know where bugs are? :) (for instance a search result for "harry" sorted by author az is not the reverse of author za. The "more" link does not do anything.)
(In reply to Jonathan Druart from comment #59) > "Basic searching there should be working, anything else may explode or > otherwise fail." > > Does it mean we can make you some feedbacks or you know where bugs are? :) Feedback is welcome :) I'll add things to my todo list. > (for instance a search result for "harry" sorted by author az is not the > reverse of author za. Strictly speaking, the way it does it now is the most correct. But what I'm probably going to do based on this thread: https://lists.katipo.co.nz/pipermail/koha/2015-May/042746.html is have it sort only on the 1x0$a (primary author) field. However I need to figure out a way to try to be as consistent as possible with that, as it causes a problem in that now author search will use a different field than author sort. Hmm, I wonder if I can add named sort fields to my database mapping sorta like I do with facets... > The "more" link does not do anything.) I assume you mean the "more" link on facets? Yeah, that's known. It's because it's not currently truncating the list to 5 or whatever the default is. I don't think that's on my todo list though, I should add it...
Just a heads up that I've refactored how the mappings get stored in the database. If you have a local setup, you'll want to re-import the elasticsearch_mapping.sql (which currently is a weird hybrid, it loads things in the old way, uses SQL to generate new tables from them, and drops the old one. This is temporary until I get around to verifying that the new version is near enough to correct.
I'm testing ES. My problem is that Koha is always trying to connect to localhost:9200, but my koha-conf.xml has: <elasticsearch> <server>192.168.0.213:9200</server> <index_name>core</index_name> </elasticsearch> my instance of ES it not in localhost. Do I have to do something to force to connect to 192.168.0.213 and not to localhost? This is the error: [NoNodes] ** No nodes are available: [http://localhost:9200], called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /usr/local/share/perl/5.10.1/Catmandu/Store/ElasticSearch.pm line 61
(In reply to Juan Romay Sieira from comment #62) > I'm testing ES. My problem is that Koha is always trying to connect to > localhost:9200, but my koha-conf.xml has: > > <elasticsearch> > <server>192.168.0.213:9200</server> > <index_name>core</index_name> > </elasticsearch> > > my instance of ES it not in localhost. Do I have to do something to force to > connect to 192.168.0.213 and not to localhost? > > This is the error: [NoNodes] ** No nodes are available: > [http://localhost:9200], called from sub > Search::Elasticsearch::Role::Client::Direct::__ANON__ at > /usr/local/share/perl/5.10.1/Catmandu/Store/ElasticSearch.pm line 61 It definitely doesn't require localhost, my own test system runs against a remote ES server. I'm not sure what could cause that, are you sure you've got the right koha-conf? I'll attach the whole of mine to make sure the context is right.
Created attachment 40903 [details] My koha-conf.xml as an example
Yes, its the right koha-conf, and Koha gets the right configuration. Now I change it to put the timeout tag and . I put a warn in Koha/SearchEngine/Elasticsearch/Search.pm warn Data::Dumper::Dumper(\%$params); $self->store( Catmandu::Store::ElasticSearch->new( %$params, trace_calls => 1, ) ) unless $self->store; this is what it show in koha opac log $VAR1 = { 'index_name' => 'core_biblios', 'timeout' => '10', 'servers' => [ '192.168.0.213:9200' ] }; The ES instance is running too, if I visit the URL of ES in a browser it returns: { "status" : 200, "name" : "koha-es", "cluster_name" : "koha-cluster", "version" : { "number" : "1.6.0", "build_hash" : "cdd3ac4dde4f69524ec0a14de3828cb95bbb86d0", "build_timestamp" : "2015-06-09T13:36:34Z", "build_snapshot" : false, "lucene_version" : "4.10.4" }, "tagline" : "You Know, for Search" } To temporaly fix it I wrote a iptables rule to forward my 9200 port to the other machine...
Finally I can do index, search, ... and with the same configuration. My problem is that I was using Squeeze instead Jessie. Thank you!
(In reply to Juan Romay Sieira from comment #66) > Finally I can do index, search, ... and with the same configuration. My > problem is that I was using Squeeze instead Jessie. Thank you! Oh, excellent! I'm surprised it even let you install the modules on squeeze, to be honest.
I've just pushed up some updates to the Catalyst git repo, the main thing being that updates now work. That is, you can update or delete records in Koha and they're (immediately) updated in Elasticsearch. There is a provision for background updating (e.g. if the ES server isn't there), but I haven't actually done it yet. I'm updating the ES demo server (http://elasticsearch.koha.catalystdemo.net.nz/) at the moment. Have a play and let me know if^Wwhen you spot problems. I'm keeping a rough to-do list here: https://tree.taiga.io/project/robins-koha-elasticsearch/kanban (which wasn't publicly visible, now is as taiga lets that happen.) If you spot something that's not on that list, let me know and I'll add it to the list. I'm taking a small break from ES specifically because I need to do stuff on a related thing. But I'll review anything posted here :)
Hi Robin, thx for updating! Some things I tried/found/wondered about, mostly about facetting as I am getting a lot of feedback on our current implementation: 1) Can you show the number of results behind the facets? There is currently a preference for this, but I couldn't turn it on/chefk if it turned on. 2) Can facets be sorted by most used to less used in a result list? 3) Can you show all entries for a big result list? How does it limit which facets to show? 4) If you search for () it explodes rather spectacularly.
(In reply to Katrin Fischer from comment #69) > 1) Can you show the number of results behind the facets? There is currently > a preference for this, but I couldn't turn it on/chefk if it turned on. I've turned it on now. > 2) Can facets be sorted by most used to less used in a result list? This is a Koha question, not a search question. We can re-order them however we like (though, I don't know how meaningful that'd be tbh.) I'm trying to not change the behaviour significantly beyond what zebra can do. > 3) Can you show all entries for a big result list? How does it limit which > facets to show? That's not how facets work in this case. It calculates facets across all the results (roughly, but I think mostly accurate for the scale of results in a Koha system.) It then returns the most frequent 10. At the moment, that's what's displaying. When I get to that point in my todo list, it'll show 5 until you click on the "show more" thing. Which forces a full page reload for hysterical raisins. > 4) If you search for () it explodes rather spectacularly. It sure does. There's an option to make the string parsing more lenient I need to enable. I also need to catch that error better. Basically, if you give it anything that can't be parsed as a properly lucene query, you'll see this. The real solution is to parse the query in Koha into a tree, then turn that tree into a proper Elasticsearch query. For obvious reasons (it's hard!), I'm not doing that yet. The other part of the real solution is to catch and parse that ES response and say something sensible, rather than just output it verbatim.
(In reply to Robin Sheat from comment #70) > (In reply to Katrin Fischer from comment #69) > > 1) Can you show the number of results behind the facets? There is currently > > a preference for this, but I couldn't turn it on/chefk if it turned on. > > I've turned it on now. Thx! > > > 2) Can facets be sorted by most used to less used in a result list? > > This is a Koha question, not a search question. We can re-order them however > we like (though, I don't know how meaningful that'd be tbh.) I'm trying to > not change the behaviour significantly beyond what zebra can do. We currently do it alphabetically - which is kind of ok, but as we pick the facets quite randomly out of 20 records... I understand about not changing behaviour significantly for now, but I think there are a lot of weirdnesses that we shouldn't translate to Elastic Search if we can help it. And our facets are quite weird right now. > > 3) Can you show all entries for a big result list? How does it limit which > > facets to show? > > That's not how facets work in this case. It calculates facets across all the > results (roughly, but I think mostly accurate for the scale of results in a > Koha system.) It then returns the most frequent 10. At the moment, that's > what's displaying. When I get to that point in my todo list, it'll show 5 > until you click on the "show more" thing. Which forces a full page reload > for hysterical raisins. Sorry, I don't understand, what do you mean by "(roughly, but I think mostly accurate for the scale of results in a Koha system.)"? The demo database seems to be quite small and from here it's not very fast, so wondering about how performance looks for you. Have you done testing with a bigger database by chance? (100.000+?). We are running into a lot of never-noticed problems right now with the bigger databases and searching. About the number - I think limiting to 10 could be problematic for things like itemtypes, locations and branches. It could be good to have that configurable at some point or an option to just show 'all' for some of the facets.
(In reply to Katrin Fischer from comment #71) > We currently do it alphabetically - which is kind of ok, but as we pick the > facets quite randomly out of 20 records... Oh, this should be presenting the facets with the most results across the full set of search results. > I understand about not changing behaviour significantly for now, but I think > there are a lot of weirdnesses that we shouldn't translate to Elastic Search > if we can help it. And our facets are quite weird right now. Well, they're probably a bit more sane in ES anyway. > Sorry, I don't understand, what do you mean by "(roughly, but I think mostly > accurate for the scale of results in a Koha system.)"? Oh, just that when you get to many results over a clustered system, after a point it starts estimating the counts. Like google's "Results 1-10 of about 1,200,000", it's an estimate, not an exact count. But for smaller result counts it's going to be pretty exact. > The demo database seems to be quite small and from here it's not very fast, > so wondering about how performance looks for you. Have you done testing with > a bigger database by chance? (100.000+?). We are running into a lot of > never-noticed problems right now with the bigger databases and searching. http://elasticsearch.koha.catalystdemo.net.nz/files/es-search/ Most of the time is spent in Koha::Database. This is something that has to be fixed in general for Koha to use it (alternately, using a persistent environment like Plack where the init only has to happen once.) There is a bit of extra time spent in Catmandu that could be reduced if necessary, we're using it as a fairly thin wrapper over search (as opposed to indexing where it's critical) and it'd be OK to use the elasticsearch libraries directly. However that's a bridge to burn when we get to it. The database I'm using has 8,679 biblios and 14,841 items. So not very large at all. At some stage I might have a look at putting a bigger one in, but that does impact my reindexing time :) > About the number - I think limiting to 10 could be problematic for things > like itemtypes, locations and branches. It could be good to have that > configurable at some point or an option to just show 'all' for some of the > facets. Sure, but is that a search thing or a Koha thing? I mean, putting hooks in to make this possible is certainly a search thing (they don't exist yet), but the decision on how it should be done is out of scope. I have enough problems to worry about as it is :)
Sorry, no intention to scope creep :) So far it sounds like facets will be nicer - and we can worry about other things later.
I've just pushed a new commit that makes the code work with the newer Catmandu::Store::Elasticsearch version. Until I re-roll the dependencies tarball, you'll want to grab the .deb file here: http://debian.koha-community.org/koha/otherthings/
Created attachment 42030 [details] opac_search_for_d_sort_by_relevance
Created attachment 42031 [details] opac_search_for_d_sort_by_title
Created attachment 42032 [details] opac_search_for_harry_sort_by_title
Created attachment 42033 [details] limit_by_book_sort_by_pubdate
Me again :) So, I have tried to do some tests locally using your branch (OPAC biblio search only). The first problem I got was to find a MARC21 DB (since the UNIMARC mappings are not defined, I cannot test with an UNIMARC DB). I have used the one created for the sandboxes (http://git.koha-community.org/gitweb/?p=contrib/global.git;a=blob;f=sandbox/sql/sandbox1.sql.gz;h=19268bccb43b2a33d5644b7d86cbb1abb323016b;hb=HEAD). But there are only 436 biblios, it's not enough to test some stuffs (facets for instance). Or maybe you can share your DB? Here some notes: 1/ Add deps to C4/Installer/PerlDependencies.pm 2/ The number of tests provided is very low. 3/ catalyst/elastic_search is 1004 commits behind origin/master, please rebase 4/ The message "No 'elasticsearch' block is defined in koha-conf.xml" should be raised before starting the indexation process, and not on commiting the first batch. 5/ You really need to tune the default value for the commit :) commit 100: perl misc/search_tools/rebuild_elastic_search.pl -b 77.57s user 0.86s system 91% cpu 1:25.62 total commit 1000: perl misc/search_tools/rebuild_elastic_search.pl -b 24.68s user 0.52s system 79% cpu 31.595 total For Solr, we used 5000. Yes I know, it's configurable. 6/ Verbose does not work as expected, it could be fixed with - print $msg if ($verbose <= $level); + print $msg if ($verbose >= $level); 7/ perl -e "use Pod::Checker;podchecker('misc/search_tools/rebuild_elastic_search.pl')"; *** WARNING: empty section in previous paragraph at line 36 in file misc/search_tools/rebuild_elastic_search.pl *** ERROR: =over on line 38 without closing =back at line EOF in file misc/search_tools/rebuild_elastic_search.pl 8/ 2 occurrences of "Solr" reintroduced in installer/data/mysql/sysprefs.sql and koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/admin.pref 9/ Test! I have launched some searches, with the same DB (the one from the sandbox). On a local using your remote branch and another one using master (sandbox7 provided by BibLibre). a. Search for 'd' (screentshot opac_search_for_d_sort_by_relevance.png ES on the left, Zebra on the right). Main differences: - 183 vs 182 results (?) - the order is not the same (make sense) - Locations and Places facets are missing - 6 entries are displayed in the facets for ES (current behavior is 5). b. Search for 'd', sort by title AZ (screenshot opac_search_for_d_sort_by_title.png) - Zebra displayes only 1 facet - The order is still completely different c. Search for 'harry', sort by title AZ (screenshot opac_search_for_harry_sort_by_title.png) - 'Show more' links is displayed even if only 2 entries for a facet are available - The order is still different ("The discovery of heaven" should be sorted either before Dollhouse (if the is a stopword) either after "Hareios*" - The availability is wrong for ES (The item for Dollhouse is not for loan) d. Search for Books (limit by item type in the adv search), sort by pubdate (screenshot limit_by_book_sort_by_pubdate.png) - "Return to the last advanced search" link is not displayed - The item types facet contains several entries, which does not make sense - The number of results highly differ (395 vs 364) - The order is still completely different. I had a look in the index and found: "Pictura murală*" has "pubdate":"||||" (/_search?q=_id:39&pretty) The Korean Go Association's learn to play go "pubdate":"uuuu" (/_search?q=_id:155&pretty) Where do come from these values? Shouldn't be a date, or at least an integer? It's not easy to know what is indexed where. Did you have a look at the indexes configuration page the Solr stuff had? It provided an interface to configure the different mappings, it was very useful.
(In reply to Jonathan Druart from comment #79) > The first problem I got was to find a MARC21 DB (since the UNIMARC mappings > are not defined, I cannot test with an UNIMARC DB). The UNIMARC mappings should be defined, though not tested. > I have used the one created for the sandboxes > (http://git.koha-community.org/gitweb/?p=contrib/global.git;a=blob;f=sandbox/ > sql/sandbox1.sql.gz;h=19268bccb43b2a33d5644b7d86cbb1abb323016b;hb=HEAD). But > there are only 436 biblios, it's not enough to test some stuffs (facets for > instance). > Or maybe you can share your DB? I could, but I think we'll get more useful results from different databases. > Here some notes: > > 1/ Add deps to C4/Installer/PerlDependencies.pm Yeah, I'm mostly waiting for things to settle (which they have now.) > 2/ The number of tests provided is very low. Yes, I've been meaning to go back and add a pile more. > 3/ catalyst/elastic_search is 1004 commits behind origin/master, please > rebase It's just a tedious process, so I keep putting it off :) should do that soon though. > 4/ The message "No 'elasticsearch' block is defined in koha-conf.xml" should > be raised before starting the indexation process, and not on commiting the > first batch. Added to my TODO. > 5/ You really need to tune the default value for the commit :) > commit 100: perl misc/search_tools/rebuild_elastic_search.pl -b 77.57s > user 0.86s system 91% cpu 1:25.62 total > commit 1000: perl misc/search_tools/rebuild_elastic_search.pl -b 24.68s > user 0.52s system 79% cpu 31.595 total > For Solr, we used 5000. > Yes I know, it's configurable. I just picked a number and haven't gone back to it. I'm also thinking that maybe dropping the committing entirely and just feeding straight into Catmandu and letting it do its own batching, rather than doubling up on it. More experimentation needed really, but definitely increasing the default is a sensible thing to do. FWIW, committing at 5,000: real 2m14.627s user 1m13.272s sys 0m2.228s 100: real 6m6.280s user 4m45.268s sys 0m2.828s That's a fair difference :) > 6/ Verbose does not work as expected, it could be fixed with Oops. TODOed. > > 7/ perl -e "use > Pod::Checker;podchecker('misc/search_tools/rebuild_elastic_search.pl')"; > *** WARNING: empty section in previous paragraph at line 36 in file > misc/search_tools/rebuild_elastic_search.pl > *** ERROR: =over on line 38 without closing =back at line EOF in file > misc/search_tools/rebuild_elastic_search.pl TODOed. > 8/ 2 occurrences of "Solr" reintroduced in installer/data/mysql/sysprefs.sql > and koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/admin.pref Must have come about when merging. TODOed. > 9/ Test! > I have launched some searches, with the same DB (the one from the sandbox). > On a local using your remote branch and another one using master (sandbox7 > provided by BibLibre). > > a. Search for 'd' (screentshot opac_search_for_d_sort_by_relevance.png ES on > the left, Zebra on the right). > Main differences: > - 183 vs 182 results (?) I wouldn't necessarily expect them to be the same, especially for a fairly meaningless search. > - the order is not the same (make sense) > - Locations and Places facets are missing Yeah, they're not faceted yet. Added that to my TODO list before I forget again. > - 6 entries are displayed in the facets for ES (current behavior is 5). > > b. Search for 'd', sort by title AZ (screenshot > opac_search_for_d_sort_by_title.png) > - Zebra displayes only 1 facet That's probably zebra being wrong then :) > - The order is still completely different I'm not sure which is right in this case, though I'm doing some work on the sorting at the moment that would allow you to pick which of the fields that end up in title you want to sort by. For example, it might be that ES is putting the ones with a lower series title near the start, even though it displays a different title. That'll be tuneable when I'm done with the current stuff. > c. Search for 'harry', sort by title AZ (screenshot > opac_search_for_harry_sort_by_title.png) > - 'Show more' links is displayed even if only 2 entries for a facet are > available Thought I'd fixed that, I'll have to have a look again. > - The order is still different ("The discovery of heaven" should be sorted > either before Dollhouse (if the is a stopword) either after "Hareios*" Dollhouse probably has another title field that's actually being used, as noted above. > - The availability is wrong for ES (The item for Dollhouse is not for loan) Why is it not for loan? Is it by policy, because there are no items, or because all items are issued? > d. Search for Books (limit by item type in the adv search), sort by pubdate > (screenshot limit_by_book_sort_by_pubdate.png) > - "Return to the last advanced search" link is not displayed I wonder how it knows to show that... I can't actually find that string in my checkout at all. > - The item types facet contains several entries, which does not make sense Curious. Are there situations where you have a biblio-level itemtype that differs from the item-level item type, or where one biblio might have multiple items with different item types? At the moment, I think they're all being thrown into one facet pot. > - The number of results highly differ (395 vs 364) Probably due to biblio-vs-item itemtype selection not being supported yet. If you can find it giving you a record that plain shouldn't match though, that'd be interesting. > - The order is still completely different. I had a look in the index and > found: > "Pictura murală*" has "pubdate":"||||" (/_search?q=_id:39&pretty) > The Korean Go Association's learn to play go "pubdate":"uuuu" > (/_search?q=_id:155&pretty) > Where do come from these values? Shouldn't be a date, or at least an integer? Could be the mapping is funny/broken for that. My test system has things like: "pubdate":"1998" though, which implies that it's correct. The actual mapping comes from: INSERT INTO `elasticsearch_mapping` (`indexname`, `mapping`, `facet`, `suggestible`, `type`, `marc21`, `unimarc`, `normarc`) VALUES ('biblios','pubdate',FALSE,FALSE,'','008_/7-10','100a_/9-12','008_/7-10'); On the other hand, it does have: "date-entered-on-file":"61006" which doesn't look right no matter how you carve it. > It's not easy to know what is indexed where. Did you have a look at the > indexes configuration page the Solr stuff had? > It provided an interface to configure the different mappings, it was very > useful. I haven't yet got to the point where I have the time to make an interface. At the moment it's all configured in elasticsearch_mapping.sql, which is somewhat human readable/editable. After loading the data into a table, it rewrites all those tables into a form that'll be more conducive for having a GUI on top of, but is less human readable. BTW, if you add <trace_to>Stderr</trace_to> to the <elasticsearch> block, it'll dump all the chatter with ES out to stderr, which is useful for seeing what exactly is going on. I warn you, there is a lot there though. Thanks for testing, even if I have a pile more things to fix now :)
(In reply to Robin Sheat from comment #80) > (In reply to Jonathan Druart from comment #79) > > The first problem I got was to find a MARC21 DB (since the UNIMARC mappings > > are not defined, I cannot test with an UNIMARC DB). > > The UNIMARC mappings should be defined, though not tested. Well, it's defined yes, but does not work at all (the marc21 mappings are used) :) It is caused by some errors in the sql file. Patch's coming. Note the following: MariaDB [koha_es_unimarc]> insert into search_field (name, type) select distinct mapping, type from elasticsearch_mapping; Query OK, 73 rows affected, 57 warnings (0.05 sec) Records: 73 Duplicates: 0 Warnings: 57 MariaDB [koha_es_unimarc]> show warnings; +---------+------+--------------------------------------------+ | Level | Code | Message | +---------+------+--------------------------------------------+ | Warning | 1265 | Data truncated for column 'type' at row 1 | and 72 others. > > I have used the one created for the sandboxes > > (http://git.koha-community.org/gitweb/?p=contrib/global.git;a=blob;f=sandbox/ > > sql/sandbox1.sql.gz;h=19268bccb43b2a33d5644b7d86cbb1abb323016b;hb=HEAD). But > > there are only 436 biblios, it's not enough to test some stuffs (facets for > > instance). > > Or maybe you can share your DB? > > I could, but I think we'll get more useful results from different databases. Yes of course, but I am not a real tester, I am a developer, and it would be useful to share info on specific data. I am fine to use the sandbox DB, if it's ok for you. > > Here some notes: > > > > 1/ Add deps to C4/Installer/PerlDependencies.pm > > Yeah, I'm mostly waiting for things to settle (which they have now.) > > > 2/ The number of tests provided is very low. > > Yes, I've been meaning to go back and add a pile more. Ok, I let it that for you :) > > 6/ Verbose does not work as expected, it could be fixed with > > Oops. TODOed. Patch is coming. > > 7/ perl -e "use > > Pod::Checker;podchecker('misc/search_tools/rebuild_elastic_search.pl')"; > > *** WARNING: empty section in previous paragraph at line 36 in file > > misc/search_tools/rebuild_elastic_search.pl > > *** ERROR: =over on line 38 without closing =back at line EOF in file > > misc/search_tools/rebuild_elastic_search.pl > > TODOed. Patch is coming. > > 8/ 2 occurrences of "Solr" reintroduced in installer/data/mysql/sysprefs.sql > > and koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/admin.pref > > Must have come about when merging. TODOed. Patch is coming. > > 9/ Test! > > c. Search for 'harry', sort by title AZ (screenshot > > opac_search_for_harry_sort_by_title.png) > > - 'Show more' links is displayed even if only 2 entries for a facet are > > available > > Thought I'd fixed that, I'll have to have a look again. Patch is coming. > > - The order is still different ("The discovery of heaven" should be sorted > > either before Dollhouse (if the is a stopword) either after "Hareios*" > > Dollhouse probably has another title field that's actually being used, as > noted above. Yes it has: title":["Dollhouse"],["Seasons one & two."]] 245$a Dollhouse 490$a Seasons one & two. But 245$a should be used for sorting :) > > - The availability is wrong for ES (The item for Dollhouse is not for loan) > > Why is it not for loan? Is it by policy, because there are no items, or > because all items are issued? The item is a "Visual Materials" which has a itemtype.notforloan flag set. > > d. Search for Books (limit by item type in the adv search), sort by pubdate > > (screenshot limit_by_book_sort_by_pubdate.png) > > - "Return to the last advanced search" link is not displayed > > I wonder how it knows to show that... > > I can't actually find that string in my checkout at all. Yes sorry, introduced by Bug 13307: Create a link to the last advanced search in search result page (OPAC). Which is not in your branch yet. > > - The item types facet contains several entries, which does not make sense > > Curious. Are there situations where you have a biblio-level itemtype that > differs from the item-level item type, or where one biblio might have > multiple items with different item types? At the moment, I think they're all > being thrown into one facet pot. It comes from biblioitems.itemtype=2WEEK Not sure if the data I used are correct... > > - The number of results highly differ (395 vs 364) > > Probably due to biblio-vs-item itemtype selection not being supported yet. > If you can find it giving you a record that plain shouldn't match though, > that'd be interesting. Outch, not sure how I could find that easily. > > - The order is still completely different. I had a look in the index and > > found: > > "Pictura murală*" has "pubdate":"||||" (/_search?q=_id:39&pretty) > > The Korean Go Association's learn to play go "pubdate":"uuuu" > > (/_search?q=_id:155&pretty) > > Where do come from these values? Shouldn't be a date, or at least an integer? > > Could be the mapping is funny/broken for that. My test system has things > like: > > "pubdate":"1998" > > though, which implies that it's correct. The actual mapping comes from: > > INSERT INTO `elasticsearch_mapping` (`indexname`, `mapping`, `facet`, > `suggestible`, `type`, `marc21`, `unimarc`, `normarc`) VALUES > ('biblios','pubdate',FALSE,FALSE,'','008_/7-10','100a_/9-12','008_/7-10'); It comes from the 008 > "Pictura murală*" has "pubdate":"||||" (/_search?q=_id:39&pretty) 008 090409|||||||||xx |||||||||||||| ||und|| > The Korean Go Association's learn to play go "pubdate":"uuuu" 008 971030muuuu9999nyua 000 0 eng But the index should not contain an invalid date. For Solr (you can find the code on the BibLibre repo at https://git.biblibre.com/biblibre/koha_biblibre/commits/dev/solr Browse C4/Search/), we used a system of plugins. And there is a Date plugin (https://git.biblibre.com/biblibre/koha_biblibre/blob/bd38ce1811289fcfbd75a37ec99fc4cd3c5d37f4/C4/Search/Plugins/Date.pm) which does this job. A plugin can be linked to a mapping. Just a note: I know nobody has ever had a look at the Solr code, but it is used in production by several (4 or 5) customers for more than 4 years now. And I have already had all the issues and problems you will encounter. > > It's not easy to know what is indexed where. Did you have a look at the > > indexes configuration page the Solr stuff had? > > It provided an interface to configure the different mappings, it was very > > useful. > > I haven't yet got to the point where I have the time to make an interface. > At the moment it's all configured in elasticsearch_mapping.sql, which is > somewhat human readable/editable. After loading the data into a table, it > rewrites all those tables into a form that'll be more conducive for having a > GUI on top of, but is less human readable. > > BTW, if you add > > <trace_to>Stderr</trace_to> > > to the <elasticsearch> block, it'll dump all the chatter with ES out to > stderr, which is useful for seeing what exactly is going on. I warn you, > there is a lot there though. I will try and see if I can find some time and propose something here, I you want some help.
Created attachment 42062 [details] [review] Bug 12478: Fix the UNIMARC and NORMARC indexing 1/ This patch fixes a major issue on UNIMARC and NORMARC installations (Compulsive c/p on insert into insert into search_marc_to_field for unimarc and normarc) 2/ After fixing that, I got invalid marc path at /usr/share/perl5/Catmandu/Fix/marc_map.pm line 35. on reindexing. It was caused by some "" values instead of NULL. 3/ Then, a "duplicate entry" error was caused by 3 lines for Local-number. It seems to be an error, 1 is enough (I suppose 0909 was an error too). Note that the following is not fixed by this patch: MariaDB [koha_es_unimarc]> insert into search_field (name, type) select distinct mapping, type from elasticsearch_mapping; Query OK, 73 rows affected, 57 warnings (0.05 sec) Records: 73 Duplicates: 0 Warnings: 57 MariaDB [koha_es_unimarc]> show warnings; +---------+------+--------------------------------------------+ | Level | Code | Message | +---------+------+--------------------------------------------+ | Warning | 1265 | Data truncated for column 'type' at row 1 |
Created attachment 42063 [details] [review] Bug 12478: Change the commit count to 5k It will improve the indexing time.
Created attachment 42064 [details] [review] Bug 12478: Fix the verbose flag on reindexing
Created attachment 42065 [details] [review] Bug 12478: Fix pod in the rebuild_ES.pl script
Created attachment 42066 [details] [review] Bug 12478: Remove Solr occurrences reintroduced by a previous patch
Created attachment 42067 [details] [review] Bug 12478: Display the correct number of facets (5 instead of 6) The loop on facet entries was wrong, it should stop after 5 iterations.
Created attachment 42068 [details] [review] Bug 12478: Do not display the 'Show more' links if no more facet available
Created attachment 42069 [details] [review] Bug 12478: Fix encoding issue on facets Filer on "Ślez, Ts." => Can't escape \x{015A}, try uri_escape_utf8() instead at /home/koha/src/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm line 221.
Something else, there is a sort issue in the facets: [Some entries] Zeitoun, Ariel, Ó Cadhain, Máirtín. Ślez, Ts.. Ó should be after O, not after Z.
(In reply to Jonathan Druart from comment #81) > Well, it's defined yes, but does not work at all (the marc21 mappings are > used) :) > It is caused by some errors in the sql file. Patch's coming. Ah, ta. > > Note the following: > MariaDB [koha_es_unimarc]> insert into search_field (name, type) select > distinct mapping, type from elasticsearch_mapping; > Query OK, 73 rows affected, 57 warnings (0.05 sec) > Records: 73 Duplicates: 0 Warnings: 57 > > MariaDB [koha_es_unimarc]> show warnings; > +---------+------+--------------------------------------------+ > | Level | Code | Message | > +---------+------+--------------------------------------------+ > | Warning | 1265 | Data truncated for column 'type' at row 1 | Hmm, I remember that, but I'm not 100% sure it mattered. Could be wrong though. > Yes of course, but I am not a real tester, I am a developer, and it would be > useful to share info on specific data. > I am fine to use the sandbox DB, if it's ok for you. Fair point. Let me see if I can tidy the database some for uploading somewhere. Here it is: http://elasticsearch.koha.catalystdemo.net.nz/files/koha_es_marc21.sql.bz2 it's not the best data, but it's good enough for messing about with. > > > 2/ The number of tests provided is very low. > > Yes, I've been meaning to go back and add a pile more. > Ok, I let it that for you :) Oh, you don't have to. I don't mind if you go and write them all for me :) > Patch is coming. > Patch is coming. > Patch is coming. > Patch is coming. Thanks! > > Yes it has: > title":["Dollhouse"],["Seasons one & two."]] > > 245$a Dollhouse > 490$a Seasons one & two. > > But 245$a should be used for sorting :) Yes, that's something I'm trying to fix at the moment :) > The item is a "Visual Materials" which has a itemtype.notforloan flag set. Good to know, I've not tested that case yet. > Outch, not sure how I could find that easily. Probably easiest to construct a case manually. > It comes from the 008 > > "Pictura murală*" has "pubdate":"||||" (/_search?q=_id:39&pretty) > 008 090409|||||||||xx |||||||||||||| ||und|| > > The Korean Go Association's learn to play go "pubdate":"uuuu" > 008 971030muuuu9999nyua 000 0 eng > > But the index should not contain an invalid date. Hmm. I don't know if we can put validation into the fixer rules. I'll have to explore that some further. Possibly also telling ES that this must be a number could cause bad data to get rejected, but it may reject the whole record, not sure. Do you happen to know how zebra handles that? > For Solr (you can find the code on the BibLibre repo at > https://git.biblibre.com/biblibre/koha_biblibre/commits/dev/solr Browse > C4/Search/), we used a system of plugins. And there is a Date plugin > (https://git.biblibre.com/biblibre/koha_biblibre/blob/ > bd38ce1811289fcfbd75a37ec99fc4cd3c5d37f4/C4/Search/Plugins/Date.pm) which > does this job. > A plugin can be linked to a mapping. We probably can't directly reuse that, at present we're using Catmandu do do the data conversion and interfacing with ES for the most part. But it's possible I can hook something in somewhere. > Just a note: I know nobody has ever had a look at the Solr code, but it is > used in production by several (4 or 5) customers for more than 4 years now. > And I have already had all the issues and problems you will encounter. I'm sure I'll encounter some exciting new ones :) > I will try and see if I can find some time and propose something here, I you > want some help. Sure, anything is welcome. (In reply to Jonathan Druart from comment #90) > Something else, there is a sort issue in the facets: > > [Some entries] > Zeitoun, Ariel, > Ó Cadhain, Máirtín. > Ślez, Ts.. > > Ó should be after O, not after Z. Line 573 of opac/opac-search.pl does a sort with cmp, which isn't very unicode aware. I'm putting that in the not-my-problem bin as it's in upstream :)
I'v install a ES Koha. Index works well and ES server can search records, but the Opac cannot search. "No results found! You did not specify any search criteria. Error: Unable to perform your search. Please try again. "
(In reply to Peter Zhao from comment #92) > I'v install a ES Koha. > Index works well and ES server can search records, but the Opac cannot > search. Can you provide more detail? Comment #80, at the bottom, shows how to debug the traces between ES and Koha. This will tell you what search request is actually being made.
(In reply to Robin Sheat from comment #93) > (In reply to Peter Zhao from comment #92) > > I'v install a ES Koha. > > Index works well and ES server can search records, but the Opac cannot > > search. > > Can you provide more detail? Comment #80, at the bottom, shows how to debug > the traces between ES and Koha. This will tell you what search request is > actually being made. koha-opac-error_log shows: [Tue Sep 01 10:03:30 2015] [error] [client 127.0.0.1] # Request to: http://localhost:9200, referer: http://127.0.1.1/ [Tue Sep 01 10:03:30 2015] [error] [client 127.0.0.1] curl -XHEAD 'http://localhost:9200/koha_biblios?pretty=1', referer: http://127.0.1.1/ [Tue Sep 01 10:03:30 2015] [error] [client 127.0.0.1] , referer: http://127.0.1.1/ [Tue Sep 01 10:03:30 2015] [error] [client 127.0.0.1] # Response: 200, Took: 14 ms, referer: http://127.0.1.1/ [Tue Sep 01 10:03:30 2015] [error] [client 127.0.0.1] # 1, referer: http://127.0.1.1/ [Tue Sep 01 10:03:30 2015] [error] [client 127.0.0.1] [Tue Sep 1 10:03:30 2015] opac-search.pl: [Serializer] ** encountered object '1', but neither allow_blessed nor convert_blessed settings are enabled at /usr/local/share/perl/5.14.2/Search/Elasticsearch/Role/Serializer/JSON.pm line 24., referer: http://127.0.1.1/ [Tue Sep 01 10:03:30 2015] [error] [client 127.0.0.1] [Tue Sep 1 10:03:30 2015] opac-search.pl: , called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /usr/local/share/perl/5.14.2/Catmandu/Store/ElasticSearch/Bag.pm line 127. With vars: {'var' => {'from' => 0,'query' => {'query_string' => {'fuzziness' => 'auto','default_field' => '_all','query' => '(title:best)','default_operator' => 'AND','lenient' => bless( do{\\(my $o = 1)}, 'JSON::PP::Boolean' )}},'size' => 20,'facets' => {'subject' => {'terms' => {'field' => 'subject__facet'}},'author' => {'terms' => {'field' => 'author__facet'}},'itype' => {'terms' => {'field' => 'itype__facet'}}}}}, referer: http://127.0.1.1/ [Tue Sep 01 10:03:30 2015] [error] [client 127.0.0.1] [Tue Sep 1 10:03:30 2015] opac-search.pl: Use of uninitialized value $error in concatenation (.) or string at /home/koha/kohaes/opac/opac-search.pl line 578., referer: http://127.0.1.1/
(In reply to Peter Zhao from comment #94) > [Tue Sep 01 10:03:30 2015] [error] [client 127.0.0.1] [Tue Sep 1 10:03:30 > 2015] opac-search.pl: [Serializer] ** encountered object '1', but neither > allow_blessed nor convert_blessed settings are enabled at > /usr/local/share/perl/5.14.2/Search/Elasticsearch/Role/Serializer/JSON.pm > line 24., referer: http://127.0.1.1/ > [Tue Sep 01 10:03:30 2015] [error] [client 127.0.0.1] [Tue Sep 1 10:03:30 > 2015] opac-search.pl: , called from sub > Search::Elasticsearch::Role::Client::Direct::__ANON__ at > /usr/local/share/perl/5.14.2/Catmandu/Store/ElasticSearch/Bag.pm line 127. > With vars: {'var' => {'from' => 0,'query' => {'query_string' => {'fuzziness' > => 'auto','default_field' => '_all','query' => > '(title:best)','default_operator' => 'AND','lenient' => bless( do{\\(my $o = > 1)}, 'JSON::PP::Boolean' )}},'size' => 20,'facets' => {'subject' => {'terms' > => {'field' => 'subject__facet'}},'author' => {'terms' => {'field' => > 'author__facet'}},'itype' => {'terms' => {'field' => 'itype__facet'}}}}}, > referer: http://127.0.1.1/ Interesting. What do you get as the output of: $ perl -MData::Dumper -MJSON -e 'print Dumper JSON::true;' Also, what about: perl -MJSON::XS::Boolean -e '' Thirdly, does installing libjson-xs-perl make things work?
(In reply to Robin Sheat from comment #95) > (In reply to Peter Zhao from comment #94) > > [Tue Sep 01 10:03:30 2015] [error] [client 127.0.0.1] [Tue Sep 1 10:03:30 > > 2015] opac-search.pl: [Serializer] ** encountered object '1', but neither > > allow_blessed nor convert_blessed settings are enabled at > > /usr/local/share/perl/5.14.2/Search/Elasticsearch/Role/Serializer/JSON.pm > > line 24., referer: http://127.0.1.1/ > > [Tue Sep 01 10:03:30 2015] [error] [client 127.0.0.1] [Tue Sep 1 10:03:30 > > 2015] opac-search.pl: , called from sub > > Search::Elasticsearch::Role::Client::Direct::__ANON__ at > > /usr/local/share/perl/5.14.2/Catmandu/Store/ElasticSearch/Bag.pm line 127. > > With vars: {'var' => {'from' => 0,'query' => {'query_string' => {'fuzziness' > > => 'auto','default_field' => '_all','query' => > > '(title:best)','default_operator' => 'AND','lenient' => bless( do{\\(my $o = > > 1)}, 'JSON::PP::Boolean' )}},'size' => 20,'facets' => {'subject' => {'terms' > > => {'field' => 'subject__facet'}},'author' => {'terms' => {'field' => > > 'author__facet'}},'itype' => {'terms' => {'field' => 'itype__facet'}}}}}, > > referer: http://127.0.1.1/ > > Interesting. What do you get as the output of: > > $ perl -MData::Dumper -MJSON -e 'print Dumper JSON::true;' > > Also, what about: > > perl -MJSON::XS::Boolean -e '' > > Thirdly, does installing libjson-xs-perl make things work? koha@koha:~$ perl -MData::Dumper -MJSON -e 'print Dumper JSON::true;' $VAR1 = bless( do{\(my $o = 1)}, 'JSON::PP::Boolean' ); koha@koha:~$ perl -MJSON::XS::Boolean -e '' koha@koha:~$ koha@koha:~$ sudo apt-get install libjson-xs-perl [sudo] password for koha: 正在读取软件包列表... 完成 正在分析软件包的依赖关系树 正在读取状态信息... 完成 libjson-xs-perl 已经是最新的版本了。 libjson-xs-perl 被设置为手动安装。 升级了 0 个软件包，新安装了 0 个软件包，要卸载 0 个软件包，有 0 个软件包未被升级。 (In reply to Robin Sheat from comment #95) > (In reply to Peter Zhao from comment #94) > > [Tue Sep 01 10:03:30 2015] [error] [client 127.0.0.1] [Tue Sep 1 10:03:30 > > 2015] opac-search.pl: [Serializer] ** encountered object '1', but neither > > allow_blessed nor convert_blessed settings are enabled at > > /usr/local/share/perl/5.14.2/Search/Elasticsearch/Role/Serializer/JSON.pm > > line 24., referer: http://127.0.1.1/ > > [Tue Sep 01 10:03:30 2015] [error] [client 127.0.0.1] [Tue Sep 1 10:03:30 > > 2015] opac-search.pl: , called from sub > > Search::Elasticsearch::Role::Client::Direct::__ANON__ at > > /usr/local/share/perl/5.14.2/Catmandu/Store/ElasticSearch/Bag.pm line 127. > > With vars: {'var' => {'from' => 0,'query' => {'query_string' => {'fuzziness' > > => 'auto','default_field' => '_all','query' => > > '(title:best)','default_operator' => 'AND','lenient' => bless( do{\\(my $o = > > 1)}, 'JSON::PP::Boolean' )}},'size' => 20,'facets' => {'subject' => {'terms' > > => {'field' => 'subject__facet'}},'author' => {'terms' => {'field' => > > 'author__facet'}},'itype' => {'terms' => {'field' => 'itype__facet'}}}}}, > > referer: http://127.0.1.1/ > > Interesting. What do you get as the output of: > > $ perl -MData::Dumper -MJSON -e 'print Dumper JSON::true;' > > Also, what about: > > perl -MJSON::XS::Boolean -e '' > > Thirdly, does installing libjson-xs-perl make things work? koha@koha:~$ perl -MData::Dumper -MJSON -e 'print Dumper JSON::true;' $VAR1 = bless( do{\(my $o = 1)}, 'JSON::PP::Boolean' ); koha@koha:~$ perl -MJSON::XS::Boolean -e '' koha@koha:~$ koha@koha:~$ sudo cpanm JSON::XS After "JSON::XS" was installed, Opac works well. Thanks a lot!
(In reply to Peter Zhao from comment #96) > koha@koha:~$ perl -MData::Dumper -MJSON -e 'print Dumper JSON::true;' > $VAR1 = bless( do{\(my $o = 1)}, 'JSON::PP::Boolean' ); > koha@koha:~$ perl -MJSON::XS::Boolean -e '' > koha@koha:~$ > koha@koha:~$ sudo cpanm JSON::XS > > After "JSON::XS" was installed, Opac works well. Thanks a lot! Cool. I suspect that your setup is particularly non-standard. I tried to uninstall JSON::XS to test, but I couldn't as it would have cause other critical things to uninstall also.
ES Opac cannot support Chinese language? I try to search Chinese word in http://elasticsearch.koha.catalystdemo.net.nz/ and my ES, they also show "Software error". Software error: Can't escape \x{57FA}, try uri_escape_utf8() instead at /opt/kohaclones/elasticsearch/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm line 217. For help, please send mail to the webmaster ([no address given]), giving this error message and the time and date of the error. Software error: Can't escape \x{57FA}, try uri_escape_utf8() instead at /home/koha/kohaes/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm line 217. For help, please send mail to the webmaster (webmaster@koha), giving this error message and the time and date of the error.
(In reply to Peter Zhao from comment #98) > ES Opac cannot support Chinese language? I try to search Chinese word in > http://elasticsearch.koha.catalystdemo.net.nz/ and my ES, they also show > "Software error". > > Software error: > > Can't escape \x{57FA}, try uri_escape_utf8() instead at > /opt/kohaclones/elasticsearch/Koha/SearchEngine/Elasticsearch/QueryBuilder. > pm line 217. > > For help, please send mail to the webmaster ([no address given]), giving > this error message and the time and date of the error. > > Software error: > > Can't escape \x{57FA}, try uri_escape_utf8() instead at > /home/koha/kohaes/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm line 217. > > For help, please send mail to the webmaster (webmaster@koha), giving this > error message and the time and date of the error. This is fixed by Bug 12478: Fix encoding issue on facets Try to add the last 8 patches from this bug report. The Robin's branch does not contain these fixes.
(In reply to Robin Sheat from comment #91) > (In reply to Jonathan Druart from comment #81) > > Note the following: > > MariaDB [koha_es_unimarc]> insert into search_field (name, type) select > > distinct mapping, type from elasticsearch_mapping; > > Query OK, 73 rows affected, 57 warnings (0.05 sec) > > Records: 73 Duplicates: 0 Warnings: 57 > > > > MariaDB [koha_es_unimarc]> show warnings; > > +---------+------+--------------------------------------------+ > > | Level | Code | Message | > > +---------+------+--------------------------------------------+ > > | Warning | 1265 | Data truncated for column 'type' at row 1 | > > Hmm, I remember that, but I'm not 100% sure it mattered. Could be wrong > though. It's caused by the fact that you insert a an empty string into an enum field. I am not sure about the consequences. > Here it is: > > http://elasticsearch.koha.catalystdemo.net.nz/files/koha_es_marc21.sql.bz2 > > it's not the best data, but it's good enough for messing about with. Great, thanks. Another set of data :) > > It comes from the 008 > > > "Pictura murală*" has "pubdate":"||||" (/_search?q=_id:39&pretty) > > 008 090409|||||||||xx |||||||||||||| ||und|| > > > The Korean Go Association's learn to play go "pubdate":"uuuu" > > 008 971030muuuu9999nyua 000 0 eng > > > > But the index should not contain an invalid date. > > Hmm. I don't know if we can put validation into the fixer rules. I'll have > to explore that some further. Possibly also telling ES that this must be a > number could cause bad data to get rejected, but it may reject the whole > record, not sure. > > Do you happen to know how zebra handles that? Absolutely no idea. > > For Solr (you can find the code on the BibLibre repo at > > https://git.biblibre.com/biblibre/koha_biblibre/commits/dev/solr Browse > > C4/Search/), we used a system of plugins. And there is a Date plugin > > (https://git.biblibre.com/biblibre/koha_biblibre/blob/ > > bd38ce1811289fcfbd75a37ec99fc4cd3c5d37f4/C4/Search/Plugins/Date.pm) which > > does this job. > > A plugin can be linked to a mapping. > > We probably can't directly reuse that, at present we're using Catmandu do do > the data conversion and interfacing with ES for the most part. But it's > possible I can hook something in somewhere. We will have to do some data pre-processing before indexing the records. I need to learn more about ES, but with Solr we had to process the date values for date type mappings. Otherwise it is not possible to correctly request on this index (for instance dates range, or sort by, etc.). By the way, the date type is only used on acqdate and copydate, why not on other dates (at least pubdate)? > (In reply to Jonathan Druart from comment #90) > > Something else, there is a sort issue in the facets: > > > > [Some entries] > > Zeitoun, Ariel, > > Ó Cadhain, Máirtín. > > Ślez, Ts.. > > > > Ó should be after O, not after Z. > > Line 573 of opac/opac-search.pl does a sort with cmp, which isn't very > unicode aware. I'm putting that in the not-my-problem bin as it's in > upstream :) Yes, and, IMO, there is a design issue here. We should not reuse the pl and tt files. How do you plan to add features that Zebra cannot provide? :) Not sure it will be maintainable to add conditions (if SE == 'ES') in the TT. For instance, for the facets, we would like to display them as ES retrieve them (order by most used), and add the number of occurrences.
95 # TODO implement in the future - I don't know the best way of doing this yet. 96 # If fork: make sure process group is changed so apache doesn't wait for us. 97 98 =cut 99 100 sub update_index_background { FWIW: I have developed an "index queue" for Solr. It's inconceivable to index on the fly one by one in a real world. Koha just added a recordid in a file and the index queue watches the file and indexes a batch when needed (x minutes passed or y records in the file). I can provide you more information if needed.
Ok, just saw the comment about the date format: Koha/ElasticSearch.pm 185 # TODO be aware of date formats, but this requires pre-parsing 186 # as ES will simply reject anything with an invalid date. We would pre-process the data at this point, if a "plugin is linked to" (to define) this field.
(In reply to Peter Zhao from comment #98) > ES Opac cannot support Chinese language? I try to search Chinese word in > http://elasticsearch.koha.catalystdemo.net.nz/ and my ES, they also show > "Software error". > > Software error: > > Can't escape \x{57FA}, try uri_escape_utf8() instead at > /opt/kohaclones/elasticsearch/Koha/SearchEngine/Elasticsearch/QueryBuilder. > pm line 217. > > For help, please send mail to the webmaster ([no address given]), giving > this error message and the time and date of the error. > > Software error: > > Can't escape \x{57FA}, try uri_escape_utf8() instead at > /home/koha/kohaes/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm line 217. > > For help, please send mail to the webmaster (webmaster@koha), giving this > error message and the time and date of the error. After I changed "uri_escape" to "uri_escape_utf8", ES can index and Opac can search Chinese in MARC21 structure. But it does not work in UNIMARC structure. It shows messy code. koha@koha:~$ curl -XGET 'http://localhost:9200/koha_biblios/_search?pretty=1' { "took" : 132, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 2, "max_score" : 1.0, "hits" : [ { "_index" : "koha_biblios", "_type" : "data", "_id" : "1", "_score" : 1.0, "_source":{"pubdate":" ","subject":[["åºç£æ"],["åºç£æ"],["ç¥å¦"]],"author__facet":[["赵红å"]],"_id":"1","author":[["赵红å"]],"subject__facet":[["åºç£æ"],["åºç£æ"],["ç¥å¦"]],"record":[["LDR",null,null,"_","00261nam 22001213 4500"],["001",null,null,"_","1"],["005",null,null,"_","20150903102757.0"],["090"," "," ","a","1"],["100"," "," ","a","20150903d u||y0chiy50 ea"],["200"," "," ","a","åºç£æææ³å²","c","Peter","f","赵红å"],["600"," "," ","a","åºç£æ"],["601"," "," ","a","ç¥å¦"],["942"," "," ","c","BK"],["999"," "," ","c","1","d","1"]],"ta":"u","title":[["åºç£æææ³å²"],["Peter"]],"onloan":"0","Local-number":[["1"]]} }, { "_index" : "koha_biblios", "_type" : "data", "_id" : "2", "_score" : 1.0, "_source":{"pubdate":" ","subject":[["åºç£æ"],["åºç£æ"],["ç¥å¦"]],"author__facet":[["peter"]],"_id":"2","author":[["peter"]],"subject__facet":[["åºç£æ"],["åºç£æ"],["ç¥å¦"]],"record":[["LDR",null,null,"_","00257nam 22001213 4500"],["001",null,null,"_","2"],["005",null,null,"_","20150903105155.0"],["090"," "," ","a","2"],["100"," "," ","a","20150903d u||y0chiy50 ea"],["200"," "," ","a","åºç£æææ³å²","c","Peter","f","peter"],["600"," "," ","a","åºç£æ"],["601"," "," ","a","ç¥å¦"],["942"," "," ","c","BK"],["999"," "," ","c","2","d","2"]],"ta":"u","title":[["åºç£æææ³å²"],["Peter"]],"onloan":"0","Local-number":[["2"]]} } ] } } koha@koha:~$
(In reply to Peter Zhao from comment #103) > After I changed "uri_escape" to "uri_escape_utf8", ES can index and Opac can > search Chinese in MARC21 structure. > > But it does not work in UNIMARC structure. It shows messy code. The code paths between MARC21 and UNIMARC shouldn't differ in ways that would make a difference here. The only place where it comes up is determining that XXXy goes into "author" or whatever.
(In reply to Robin Sheat from comment #104) > (In reply to Peter Zhao from comment #103) > > After I changed "uri_escape" to "uri_escape_utf8", ES can index and Opac can > > search Chinese in MARC21 structure. > > > > But it does not work in UNIMARC structure. It shows messy code. > > The code paths between MARC21 and UNIMARC shouldn't differ in ways that > would make a difference here. The only place where it comes up is > determining that XXXy goes into "author" or whatever. The following are 3 records, Chinese words show messy code, English words show correct, but they cannot search through Opac. koha@koha:~$ curl -XGET 'http://localhost:9200/koha_biblios/_search?pretty=1' { "took" : 3, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 3, "max_score" : 1.0, "hits" : [ { "_index" : "koha_biblios", "_type" : "data", "_id" : "1", "_score" : 1.0, "_source":{"pubdate":" ","subject":[["åºç£æ"],["åºç£æ"],["ç¥å¦"]],"author__facet":[["赵红å"]],"_id":"1","author":[["赵红å"]],"subject__facet":[["åºç£æ"],["åºç£æ"],["ç¥å¦"]],"record":[["LDR",null,null,"_","00261nam 22001213 4500"],["001",null,null,"_","1"],["005",null,null,"_","20150903102757.0"],["090"," "," ","a","1"],["100"," "," ","a","20150903d u||y0chiy50 ea"],["200"," "," ","a","åºç£æææ³å²","c","Peter","f","赵红å"],["600"," "," ","a","åºç£æ"],["601"," "," ","a","ç¥å¦"],["942"," "," ","c","BK"],["999"," "," ","c","1","d","1"]],"ta":"u","title":[["åºç£æææ³å²"],["Peter"]],"onloan":"0","Local-number":[["1"]]} }, { "_index" : "koha_biblios", "_type" : "data", "_id" : "2", "_score" : 1.0, "_source":{"pubdate":" ","subject":[["åºç£æ"],["åºç£æ"],["ç¥å¦"]],"author__facet":[["peter"]],"_id":"2","author":[["peter"]],"subject__facet":[["åºç£æ"],["åºç£æ"],["ç¥å¦"]],"record":[["LDR",null,null,"_","00257nam 22001213 4500"],["001",null,null,"_","2"],["005",null,null,"_","20150903105155.0"],["090"," "," ","a","2"],["100"," "," ","a","20150903d u||y0chiy50 ea"],["200"," "," ","a","åºç£æææ³å²","c","Peter","f","peter"],["600"," "," ","a","åºç£æ"],["601"," "," ","a","ç¥å¦"],["942"," "," ","c","BK"],["999"," "," ","c","2","d","2"]],"ta":"u","title":[["åºç£æææ³å²"],["Peter"]],"onloan":"0","Local-number":[["2"]]} }, { "_index" : "koha_biblios", "_type" : "data", "_id" : "3", "_score" : 1.0, "_source":{"pubdate":" ","subject":[["History"],["History"],["Christianity"]],"author__facet":[["Peter"]],"_id":"3","author":[["Peter"]],"subject__facet":[["History"],["History"],["Christianity"]],"record":[["LDR",null,null,"_","00259nam 22001213 4500"],["001",null,null,"_","3"],["005",null,null,"_","20150903125354.0"],["090"," "," ","a","3"],["100"," "," ","a","20150903d u||y0frey50 ba"],["200"," "," ","a","History of Christianity","f","Peter"],["600"," "," ","a","History"],["601"," "," ","a","Christianity"],["942"," "," ","c","BK"],["999"," "," ","c","3","d","3"]],"ta":"u","title":[["History of Christianity"]],"onloan":"0","Local-number":[["3"]]} } ] } } koha@koha:~$
Just a headsup that I'll be out of touch until about the 17th of September, so that's why there'll be radio silence for a while :) Jonathan, I was hoping to pull your patches in on top of my sorting fixes, unfortunately I didn't quite get them finished in time.
Created attachment 42392 [details] [review] Bug 12478: Fix error on indexing a specific record % perl misc/search_tools/rebuild_elastic_search.pl -bn 42 Can't locate object method "idnumber" via package "MARC::Record" at misc/search_tools/rebuild_elastic_search.pl line 171.
Created attachment 42393 [details] [review] Bug 12478: Fix encoding issues on indexing There were encoding issues on indexing, the way to build a marc record was the old way (retrieved from the noxml style from rebuild_zebra). Here we should either c/p the code from C4::Biblio::GetMarcBiblio or call it directly (what this patch does).
Peter, the last patch should fix your issue.
(In reply to Jonathan Druart from comment #109) > Peter, the last patch should fix your issue. Jonathan, it works well. Thanks a lot. ES indexing and Opac works wonderful. But in staff interface when I input non-latin words (e.g. Chinese) to search (search the catalog), it shows "Software error:Cannot decode string with wide characters at /usr/lib/i386-linux-gnu/perl/5.20/Encode.pm line 215. For help, please send mail to the webmaster (webmaster@koha), giving this error message and the time and date of the error. " If I input English words, it can show results, but when I click the record, it shows "The record you requested does not exist ()."
I haven't focused on the intranet side yet :) I have found an error this morning: you cannot access the detail page from the result page.
I've pushed up my latest changes with the sorting updates. Here's the commit message for a description of how it works: Bug 12478: allow more granular sorting configuration This allows sorting to be configured within a field. For example, while many values are included for search on author, sorting should only be done on the main entry values. This permits that by having a sort value, which can be true, false, or null. true and null are pretty much the same, but false means that a field isn't available for sorting on. By default (null), fields can be sorted on.
Jonathan, I've added all your patches to the branch. Thanks!
(In reply to Robin Sheat from comment #80) > > 3/ catalyst/elastic_search is 1004 commits behind origin/master, please > > rebase > > It's just a tedious process, so I keep putting it off :) should do that soon > though. Now? Please. :)
(In reply to Mirko Tietgen from comment #114) > Now? Please. :) I have a rebase done, though it's untested. I'll force push it up, which will require resetting your local copy. Will get some testing in tomorrow to make sure not too much has blown up. Apologies Jonathan, due to Koha/Biblio.pm being introduced into master, I renamed the ES version to BiblioUtils.pm, but did it by ignoring all the rebase conflicts and just cleaning it up at the end (otherwise it was going to be a nightmare.) This means some of your commits will have vanished and been rolled into my top commit. Note to self in case things go horribly wrong: the previous branch head is at commit 9669bbc.
(In reply to Robin Sheat from comment #115) > I have a rebase done, though it's untested. Now it's had a quick test and some things have been fixed. It appears on the surface to be working again.
I am getting error on the DB update process: C4::Installer::load_sql returned the following errors while attempting to load /home/koha/src/installer/data/mysql/elasticsearch_mapping.sql: ERROR 1062 (23000) at line 294: Duplicate entry 'biblios-marc21-490a' for key 'index_name_2' ERROR 1062 (23000) at line 294: Duplicate entry 'biblios-marc21-490a' for key 'index_name_2'
(In reply to Jonathan Druart from comment #117) > I am getting error on the DB update process: > C4::Installer::load_sql returned the following errors while attempting to > load /home/koha/src/installer/data/mysql/elasticsearch_mapping.sql: > ERROR 1062 (23000) at line 294: Duplicate entry 'biblios-marc21-490a' for > key 'index_name_2' > ERROR 1062 (23000) at line 294: Duplicate entry 'biblios-marc21-490a' for > key 'index_name_2' Robin, have a look at the patch "DB changes" I have attached on bug 14899, maybe it should be integrated here.
Does the authorities index is supposed to work? I get Use of uninitialized value $id in concatenation (.) or string at misc/search_tools/rebuild_elastic_search.pl line 173. after a perl misc/search_tools/rebuild_elastic_search.pl -a -v and on /koha-es_authorities/_search?q=*&pretty "_index" : "koha-es_authorities", "_type" : "data", "_id" : "7c8462c8-c813-4013-9ab7-1ca77069bbc4",
Robin, It seems that something was wrong during the rebase process. I have tried the updated branch without any success: $ git reset --hard catalyst/elastic_search $ installer/data/mysql/updatedatabase.pl # see error in previous comment, but I should not that the following is still true if I remove the unique key $ perl misc/search_tools/rebuild_elastic_search.pl -b -v # no error I can now only search for "*", other patterns return nothing, and /koha-es_biblios/_search?q=*&pretty shows entries like: { "_index" : "koha-es_biblios", "_type" : "data", "_id" : "11", "_score" : 1.0, "_source":{"record":[["LDR",null,null,"_","01199cam a22003134a 4500"],["001",null,null,"_","12011929"],["005",null,null,"_","20140507153623.0"],["008",null,null,"_","000518s2000 ch a b 001 0 eng "],["010"," "," ","a"," 00041664 "],["020"," "," ","a","1565924193"],["040"," "," ","a","DLC","c","DLC","d","DLC"],["042"," "," ","a","pcc"],["050","0","0","a","QA76.73.P22","b","G84 2000"],["082","0","0","a","005.2/762","2","21"],["100","1"," ","a","Guelich, Scott.","9","8"],["245","1","0","a","CGI programming with Perl /","c","Scott Guelich, Shishir Gundavaram and Gunther Birznieks."],["250"," "," ","a","2nd ed."],["260"," "," ","a","Beijing ;","a","Cambridge, Mass. :","b","O'Reilly,","c","2000."],["300"," "," ","a","xv, 451 p.","b","ill.","c","24 cm."],["504"," "," ","a","Includes bibliographical references (p. 403-406) and index."],["650"," ","0","a","Perl (Computer program language)","9","35"],["650"," ","0","a","CGI (Computer network protocol)","9","37"],["650"," ","0","a","Internet programming.","9","7"],["700","1"," ","a","Birznieks, Gunther.","9","38"],["856","4","2","3","Publisher description","u","http://www.loc.gov/catdir/enhancements/fy0715/00041664-d.html"],["906"," "," ","a","7","b","cbc","c","orignew","d","1","e","ocip","f","20","g","y-gencatlg"],["942"," "," ","2","ddc","c","BK"],["955"," "," ","a","to ASCD pc16 05-18-00; jf05 (desc.) 05/18/00 ; jf11 to sl 5-19-00; jf12 to Dewey 05-23-00;aa03 5-24-00;CIP ver jf0504/10/01; jf12 to BCCD 04-11-01"],["952"," "," ","0","0","1","0","4","0","6","_","7","0","8","GEN","9","32","a","ALPHA","b","ALPHA","c","GEN","d","2014-09-04","p","39999000000498","r","2014-09-04","w","2014-09-04","y","BK"],["999"," "," ","c","11","d","11"]],"_id":"11"} } There is something wrong when the tables are populated.
(In reply to Jonathan Druart from comment #119) > Does the authorities index is supposed to work? > > I get > Use of uninitialized value $id in concatenation (.) or string at > misc/search_tools/rebuild_elastic_search.pl line 173. That shouldn't happen, I think there was something underlying that changed in the rebase. Lemme sort that out...
Fixed the authorities thing, there was an API change where what was 'idnumber' had to become 'id', which is now done. (In reply to Jonathan Druart from comment #120) > I can now only search for "*", other patterns return nothing, and > /koha-es_biblios/_search?q=*&pretty shows entries like: That's weird. Are you sure you reloaded the tables? I just have, and end up with: "_source":{"title__suggestion":{"input":[[["Harmonization of international accounting standards"]]]},"record":[["LDR",null,null,"_","01476pam a2200205 a 4500"],["110"," "," ","a","International Capital Markets Group"],["245"," "," ","a","Harmonization of international accounting standards","c","prepared by the International Federation of Accountants with the assistance of Federation Internationale des Bourses de Valeurs and the International Bar Association Section on Business Law"],["260"," "," ","a","London, United Kingdom","b","International Capital Markets Group","c","?"],["300"," "," ","a","67 p.","c","23 cm."],["650"," "," ","a","INTERNATIONAL ACCOUNTING STANDARDS"],["650"," "," ","a","HARMONI100 33019 100 33019 0 0 388k 0 --:--:-- --:--:-- --:--:-- 447k TION"],["650"," "," ","a","ACCOUNTING STANDARDS"],["650"," "," ","a","ACCOUNTING POLICIES"],["653"," "," ","a","INTERNATIONAL ACCOUNTING STANDARDS"],["505"," "," ","a","Introduction ; Accounting standards around the world - why are they different? ; Accounting standard setting in five countries ; What are the major differences? ; The costs of disharmony ; The barriers to harmonisation ; Toward global harmonisation ; International accounting standards ; Appendices"],["520"," "," ","a","The report examines the reasons for differences in accounting standards around the world, and, in the context of the pressures for global harmonisation of standards, examines the factors that have to be overcome if global harmonisation is to be achieved. It also examines what form global harmonisation might take, and considers the structure and organisation of accounting standard setting in this context."],["942"," "," ","c","Book"],["024"," "," ","a","16"],["900"," "," ","a","AJ9304"],["952"," "," ","0","0","1","0","2","ddc","4","0","6","100_600000000000000_HAR","7","0","9","16","a","WLGTN","b","WLGTN","d","2011-12-19","o","100.6 HAR","p","CA002438","r","2011-12-19","w","2011-12-19","y","BOOK"],["999"," "," ","c","16","d","16"]],"_id":"16","author__facet":[["International Capital Markets Group"]],"title":[["Harmonization of international accounting standards"]],"acqdate":[["2011-12-19"]],"Local-number":[["16"]],"subject":[["INTERNATIONAL ACCOUNTING STANDARDS"],["HARMONISATION"],["ACCOUNTING STANDARDS"],["ACCOUNTING POLICIES"],["INTERNATIONAL ACCOUNTING STANDARDS"]],"publisher":[["International Capital Markets Group"]],"place":[["London, United Kingdom"]],"subject__suggestion":{"input":[[["INTERNATIONAL ACCOUNTING STANDARDS"]],[["HARMONISATION"]],[["ACCOUNTING STANDARDS"]],[["ACCOUNTING POLICIES"]],[["INTERNATIONAL ACCOUNTING STANDARDS"]]]},"homebranch":[["WLGTN"]],"itype__facet":[["Book"],["BOOK"]],"local-classification__suggestion":{"input":[[["100.6 HAR"]]]},"author__suggestion":{"input":[[["International Capital Markets Group"]]]},"itemnumber":[["16"]],"copydate":[["?"]],"copydate__facet":[["?"]],"place__facet":[["London, United Kingdom"]],"author":[["International Capital Markets Group"],["prepared by the International Federation of Accountants with the assistance of Federation Internationale des Bourses de Valeurs and the International Bar Association Section on Business Law"]],"onloan":"0","subject__facet":[["INTERNATIONAL ACCOUNTING STANDARDS"],["HARMONISATION"],["ACCOUNTING STANDARDS"],["ACCOUNTING POLICIES"],["INTERNATIONAL ACCOUNTING STANDARDS"]],"publisher__facet":[["International Capital Markets Group"]],"local-classification":[["100.6 HAR"]],"author__sort":[["International Capital Markets Group"]],"holdingbranch":[["WLGTN"]],"itype":[["Book"],["BOOK"]],"homebranch__facet":[["WLGTN"]]} Which is what I would have expected. (sorry about the wall of text there)
Robin, just retried now with and everything works fine. Not sure where did it come from, but it looks to be fixed! Just got: Upgrade to XXX done (Bug 12478 - set up elasticsearch tables) DBD::mysql::db do failed: Can't DROP 'index_name_2'; check that column/key exists at installer/data/mysql/updatedatabase.pl line 10967.DBD::mysql::db do failed: Duplicate column name 'label' at installer/data/mysql/updatedatabase.pl line 10970. Upgrade to XXX done (Bug 12478/14899 - DB Changes to the elasticsearch tables) when updating the DB. You have updated the structure in the elasticsearch_mapping.sql file, so no need to remove the unique key and add the label field in the last updatedatabase entry. Not a big deal anyway :)
I have found an encoding bug on the "Show more" link, it will be fixed by bug 14955. Already exists on master.
Robin, Another set of patches is coming. 1/ Reintroduce the SearchEngine pref 2/ Fix to correct the number of facets displayed (again) 3/ Remove an unnecessary "&limit=" (minor) 4/ Replace the Koha::ItemType[s] classes you have introduced with a Koha::Object[s] way 5/ Display description instead of code for locations in facets. About 4: I know you are now agreeing completely with Koha::Object[s] and how it has been implemented. But I think we need to have an homogeneous code, and all new classes in the Koha namespace should follow the same directions. I would please you to have a look at bug 14828 (and the see also ones), which introduces the Koha::ItemType[s] classes as well and rewrites the admin/itemtypes.pl script, to see how easy it is to use it. To avoid future conflicts and don't play to a race against each other we should agree on this kind of decisions now.
Created attachment 43127 [details] [review] Bug 12478: Reintroduce the SearchEngine system preference It is in the sysprefs.sql file, but could have been removed by the previous entry (3.17.00.027).
Created attachment 43128 [details] [review] Bug 12478: (follow-up) Display the correct number of facets This patch fixes a side-effect of the previous "Display the correct number of facets (5 instead of 6)" patch. If you ask for more facets ('show more' link) and that the number of facets is > 5 but < 10, 1 facet will be missing.
Created attachment 43129 [details] [review] Bug 12478: Remove empty limit parameter If no limit are passed, the url will contain '&limit=' anyway. It is not necessary and can be avoided easily Test plan: 1/ Search for a term in your catalogue 2/ Hover over a link in the facet area 3/ The link is cgi-bin/koha/opac-search.pl?idx=kw&q=your_term&limit=&limit=[...] With this patch, the empty limit parameter does not appear.
Created attachment 43130 [details] [review] Bug 12478: Remove Koha::ItemType[s] Class::Accessor classes
Created attachment 43131 [details] [review] Bug 12478: Add Koha::ItemType[s] classes These 2 classes come from bug 14828. Tests in t/db_dependent/Koha/ItemTypes.t should be updated.
Created attachment 43132 [details] [review] Bug 12478: Use the new Koha::ItemTypes to retrieve itypes descriptions
Created attachment 43133 [details] [review] Bug 12478: Add Koha::AuthorisedValue[s] class These 2 classes come from bug 10363.
Created attachment 43134 [details] [review] Bug 12478: facets - Display description instead of code for locations
Created attachment 43140 [details] [review] Bug 12478: Display facet terms ordered by number of occurrences By default ES returns the facet terms ordered by most used, which makes sense. This patch removes resort done in the scripts (catalogue/search.pl and opac/opac-search.pl) and moves it to the module. For Zebra it's now done in C4::Search::getRecords, and there is no change to expect (still alphabetically). On the Elastic search side, we could imagine to let the library define the order of the facets. The facet terms are now sorted by most used. To test easily this change, turn on the displayFacetCount pref.
> (In reply to Jonathan Druart from comment #90) > > > Something else, there is a sort issue in the facets: > > > > > > [Some entries] > > > Zeitoun, Ariel, > > > Ó Cadhain, Máirtín. > > > Ślez, Ts.. > > > > > > Ó should be after O, not after Z. > > > > Line 573 of opac/opac-search.pl does a sort with cmp, which isn't very > > unicode aware. I'm putting that in the not-my-problem bin as it's in > > upstream :) > > Yes, and, IMO, there is a design issue here. > We should not reuse the pl and tt files. > How do you plan to add features that Zebra cannot provide? :) > Not sure it will be maintainable to add conditions (if SE == 'ES') in the TT. > For instance, for the facets, we would like to display them as ES retrieve > them (order by most used), and add the number of occurrences. In the last patch I suggest to display the facet terms in the same order as ES build them (i.e. most used first).
Created attachment 43141 [details] [review] Bug 12478: Remove empty limit parameter If no limit are passed, the url will contain '&limit=' anyway. It is not necessary and can be avoided easily Test plan: 1/ Search for a term in your catalogue 2/ Hover over a link in the facet area 3/ The link is cgi-bin/koha/opac-search.pl?idx=kw&q=your_term&limit=&limit=[...] With this patch, the empty limit parameter does not appear.
Created attachment 43142 [details] [review] Bug 12478: Remove empty limit parameter If no limit are passed, the url will contain '&limit=' anyway. It is not necessary and can be avoided easily Test plan: 1/ Search for a term in your catalogue 2/ Hover over a link in the facet area 3/ The link is cgi-bin/koha/opac-search.pl?idx=kw&q=your_term&limit=&limit=[...] With this patch, the empty limit parameter does not appear.
Created attachment 43148 [details] [review] Bug 12478: Take the FacetMaxCount pref into account The system preference FacetMaxCount should work as expected with ES.
(In reply to Jonathan Druart from comment #123) > You have updated the structure in the elasticsearch_mapping.sql file, so no > need to remove the unique key and add the label field in the last > updatedatabase entry. > Not a big deal anyway :) Oh, yes. That should have been removed. (In reply to Jonathan Druart from comment #125) > I know you are now agreeing completely with Koha::Object[s] and how it has > been implemented. > But I think we need to have an homogeneous code, and all new classes in the > Koha namespace should follow the same directions. > I would please you to have a look at bug 14828 (and the see also ones), > which introduces the Koha::ItemType[s] classes as well and rewrites the > admin/itemtypes.pl script, to see how easy it is to use it. > To avoid future conflicts and don't play to a race against each other we > should agree on this kind of decisions now. Sure, that's fine. Consistency is generally better than reinventing everything. (In reply to Jonathan Druart from comment #135) > In the last patch I suggest to display the facet terms in the same order as > ES build them (i.e. most used first). While I've added this, in general I'm avoiding changing of behaviour too much. We can always remove it later if it's decided we don't want it (and personally, I think it'll be better.) All these patches are added. Thanks Jonathan! FYI, I'm looking into the notforloan stuff at the moment.
sorting facets by number of use sounds better to me than alphabetic - maybe a good change :)
Robin, Thanks for your responsiveness! Yesterday I have read a bit of doc about facets and on the Terms Facet page of the ES doc ( https://www.elastic.co/guide/en/elasticsearch/reference/current/search-facets-terms-facet.html), there is a big warning on top " Warning Facets are deprecated and will be removed in a future release. You are encouraged to migrate to aggregations instead.". Did you have time to look into the terms aggregations? Or maybe is it what you are already using?
On a unimarc installation, there are no mapping at all for authorities. If an unimarc user follows this bug report...
Robin, Maybe I have not drunk enough tea this morning but... I am trying to improve the mappings area to have a consistent interface to manage them. The idea is to 1) move the elastic_mapping.sql to a elastic_mapping.json file (easier to modify and read), 2) provide methods to serialize/unserialize mappings and then 3) introduce a backup/import/reset mappings feature and finally 4) make easier the mapping progression to get a good basis to use ES. I have managed to create a json file from the sql file, the structure is something like: { biblio => { title => { label => 'Title', type => 'string', mappings => [ { suggestible => 1, facet => 1, marc21 => '245a', unimarc => '200a', normarc => '245a', }, ] }, } And I have some questions :) - Do you agree with the idea? - Don't you think the index_name should be a column of the search_fields table? - Some of the fields don't have a type, should we assign "string" as the default value? - wordings: 'sortable' and 'facetable' sounds more appropriate than 'sort' and 'facet' - (/me is clearing his throat) I think that all the mappings of a field should be removed if the field is removed. In other word, there is a 1-n relationship beetwen search_field and search_marc_map, which means that the join table (search_marc_to_field) is not needed and we could simplify the structure removing it. I am going to wait for an answer before starting anything :)
Ha, something else: the biblionumber should be a field of the biblios index. It will fix a couple of issues (click on a result at the intranet and non-xslt views). And certainly some others, for instance I get a lot of warnings in the logs: opac-search.pl: GetCOinSBiblio called with undefined record at /home/koha/src/opac/opac-search.pl line 636. opac-search.pl: GetMarcBiblio called with undefined biblionumber at /home/koha/src/opac/opac-search.pl line 635.
(In reply to Jonathan Druart from comment #141) > Did you have time to look into the terms aggregations? > Or maybe is it what you are already using? Yeah, I'm aware of that. Unfortunately, it only became deprecated after I'd implemented it, and I haven't got around to going back and reworking it. I don't think it'll be a big change, it's possible it's sufficient to just switch the ES type we're using. Anyway, I've put it on the "deal with later" pile. (In reply to Jonathan Druart from comment #142) > On a unimarc installation, there are no mapping at all for authorities. > If an unimarc user follows this bug report... Working out the MARC21 mappings was tedious enough :) (In reply to Jonathan Druart from comment #143) > Maybe I have not drunk enough tea this morning but... > I am trying to improve the mappings area to have a consistent interface to > manage them. > The idea is to 1) move the elastic_mapping.sql to a elastic_mapping.json > file (easier to modify and read), 2) provide methods to > serialize/unserialize mappings and then 3) introduce a backup/import/reset > mappings feature and finally 4) make easier the mapping progression to get a > good basis to use ES. > > I have managed to create a json file from the sql file, the structure is > something like: > > { > biblio => { > title => { > label => 'Title', > type => 'string', > mappings => [ > { > suggestible => 1, > facet => 1, > marc21 => '245a', > unimarc => '200a', > normarc => '245a', > }, > ] > }, > } > > And I have some questions :) > - Do you agree with the idea? Well... I don't know. Though I'm not a fan of that structure really, as it's not ideal, and is a bit more limited. Also, in this case you can't have more than one title, but that's not really the big issue. Mostly it's just a very denormalised view of the data. Better for manually editing an SQL file, but not really so good for a computer to use. This is why the SQL file has the data in that form and then normalises it in the database. > - Don't you think the index_name should be a column of the search_fields > table? Yes, it should be kept with search_field.name as it's effectively more information needed to describe where something gets stored. > - Some of the fields don't have a type, should we assign "string" as the > default value? I'd like to not just because that implies that they've consciously been made strings. Ideally as time goes on, people will decide that this is a date, and this is a ... IP address or something, and add those as types while putting the logic in to handle it. So, if a type is unspecified, then it gets treated like a string by default, but it really means "we haven't decided yet." > - wordings: 'sortable' and 'facetable' sounds more appropriate than 'sort' > and 'facet' hmm. I don't really mind either way. My thinking was that "facet" and "sort" were easier to type. But I broke the consistency because "suggest" seemed weird. I don't object to any of them changing. > - (/me is clearing his throat) I think that all the mappings of a field > should be removed if the field is removed. In other word, there is a 1-n > relationship beetwen search_field and search_marc_map, which means that the > join table (search_marc_to_field) is not needed and we could simplify the > structure removing it. I had a good reason for doing many-to-many. Let me see if I can remember it... Oh wait, I documented it: -- This joins the two search tables together. We can have any combination: -- one marc field could have many search fields (maybe you want one value -- to go to 'author' and 'corporate-author) and many marc fields could go -- to one search field (e.g. all the various author fields going into -- 'author'.) If you remove the many-to-many relationship then you end up with duplication/denormalisation. My thinking behind the UI is that you might have, say, a list of all the fields and under them, a set of all the MARC fields that map to it. Or perhaps the inverse. I hadn't really thought about it too much, but a properly denormalised relational structure means that we have the maximum amount of flexibility. The only improvement to the structure in this respect is that the sort, facet, suggest things should really be at the join level. I have a feeling I considered that, then decided it risked crossing the line into too fiddly, but it would get more power out of it. At the moment you'd have to duplicate the MARC details if you want different values for those three which isn't ideal. (In reply to Jonathan Druart from comment #144) > Ha, something else: the biblionumber should be a field of the biblios index. Oh, it's embedded as the ID on the ES record. There's no point duplicating it as its own field, but it's reasonable to copy it out as a post-process step and put it into a biblionumber field. We don't have a reliable
(In reply to Robin Sheat from comment #145) > We don't have a reliable I was going to say we don't have a reliable biblionumber source in the marc, but we do as I force it to be correct. But that's not relevant, we should just use the ID.
(In reply to Robin Sheat from comment #145) > (In reply to Jonathan Druart from comment #143) > > Maybe I have not drunk enough tea this morning but... > > I am trying to improve the mappings area to have a consistent interface to > > manage them. > > The idea is to 1) move the elastic_mapping.sql to a elastic_mapping.json > > file (easier to modify and read), 2) provide methods to > > serialize/unserialize mappings and then 3) introduce a backup/import/reset > > mappings feature and finally 4) make easier the mapping progression to get a > > good basis to use ES. > > > > I have managed to create a json file from the sql file, the structure is > > something like: > > > > { > > biblio => { > > title => { > > label => 'Title', > > type => 'string', > > mappings => [ > > { > > suggestible => 1, > > facet => 1, > > marc21 => '245a', > > unimarc => '200a', > > normarc => '245a', > > }, > > ] > > }, > > } > > > > And I have some questions :) > > - Do you agree with the idea? > > Well... I don't know. Though I'm not a fan of that structure really, as it's > not ideal, and is a bit more limited. Also, in this case you can't have more > than one title, but that's not really the big issue. Mostly it's just a very > denormalised view of the data. Better for manually editing an SQL file, but > not really so good for a computer to use. This is why the SQL file has the > data in that form and then normalises it in the database. I don't understand the problem with the structure, you could have several mappings (it's an arrayref of hashrefs). With this structure I could insert exactly the same data in the tables (except if I missed something...). > > - Some of the fields don't have a type, should we assign "string" as the > > default value? > > I'd like to not just because that implies that they've consciously been made > strings. Ideally as time goes on, people will decide that this is a date, > and this is a ... IP address or something, and add those as types while > putting the logic in to handle it. So, if a type is unspecified, then it > gets treated like a string by default, but it really means "we haven't > decided yet." So todo later :) > > - wordings: 'sortable' and 'facetable' sounds more appropriate than 'sort' > > and 'facet' > > hmm. I don't really mind either way. My thinking was that "facet" and "sort" > were easier to type. But I broke the consistency because "suggest" seemed > weird. I don't object to any of them changing. Not a big deal but better sooner than later. > > - (/me is clearing his throat) I think that all the mappings of a field > > should be removed if the field is removed. In other word, there is a 1-n > > relationship beetwen search_field and search_marc_map, which means that the > > join table (search_marc_to_field) is not needed and we could simplify the > > structure removing it. > > I had a good reason for doing many-to-many. Let me see if I can remember > it... > > Oh wait, I documented it: > > -- This joins the two search tables together. We can have any combination: > -- one marc field could have many search fields (maybe you want one value > -- to go to 'author' and 'corporate-author) and many marc fields could go > -- to one search field (e.g. all the various author fields going into > -- 'author'.) > > If you remove the many-to-many relationship then you end up with > duplication/denormalisation. My thinking behind the UI is that you might > have, say, a list of all the fields and under them, a set of all the MARC > fields that map to it. Or perhaps the inverse. I hadn't really thought about > it too much, but a properly denormalised relational structure means that we > have the maximum amount of flexibility. The only improvement to the > structure in this respect is that the sort, facet, suggest things should > really be at the join level. I have a feeling I considered that, then > decided it risked crossing the line into too fiddly, but it would get more > power out of it. At the moment you'd have to duplicate the MARC details if > you want different values for those three which isn't ideal. Yes, it's related to the index_name unique key discussion we had last week. We should either move sort, facet, suggest to the join table or remove it, but not keep the current structure. I am not sure about the gain of having the three tables, we could still know what fields are mapped with this MARC field or the inverse (same marc_field values). Anyway, the current structure force us to duplicate the MARC details, because of the sort, facet, suggest, which could differ. > (In reply to Jonathan Druart from comment #144) > > Ha, something else: the biblionumber should be a field of the biblios index. > > Oh, it's embedded as the ID on the ES record. There's no point duplicating > it as its own field, but it's reasonable to copy it out as a post-process > step and put it into a biblionumber field. We don't have a reliable Indeed we could add it later, but don't you think it's worth to let the librarians (and devs) search something like "biblionumber:42", which is a more familiar term than "ID"?
(In reply to Jonathan Druart from comment #147) > I don't understand the problem with the structure, you could have several > mappings (it's an arrayref of hashrefs). Oh, yes, you're right. I missed that bit. > With this structure I could insert exactly the same data in the tables > (except if I missed something...). It's just that it's not a natural representation from a database theory point of view. Why are the various MARC forms lumped together like that? Why should a schema change be necessary when (god forbid) a new flavour is added? Actually, that could be important as if we want to add support for different things into ES, say importing a feed from a journal provider or whatever, you could use a flavour to mark how the source is mapped into elasticsearch. Additionally, it promotes redundancy as you are forced to repeat things to have all the required combinations, which always causes an icky feeling :) Essentially, having a proper relational many-to-many is the most expressive and flexible system that is feasible. I think I'm OK with moving the attributes (sortable, etc) into the join though. It probably makes the most sense. > Yes, it's related to the index_name unique key discussion we had last week. > We should either move sort, facet, suggest to the join table or remove it, > but not keep the current structure. > > I am not sure about the gain of having the three tables, we could still know > what fields are mapped with this MARC field or the inverse (same marc_field > values). > Anyway, the current structure force us to duplicate the MARC details, > because of the sort, facet, suggest, which could differ. Right, but removing the M-to-M still means we have to duplicate things. And that'll be why the index was there but problematic, because the attributes are in the wrong place. The index will (I think) be fine if they get removed. > Indeed we could add it later, but don't you think it's worth to let the > librarians (and devs) search something like "biblionumber:42", which is a > more familiar term than "ID"? Oh, good point. Yes, that should be done.
(In reply to Robin Sheat from comment #91) > > The item is a "Visual Materials" which has a itemtype.notforloan flag set. > > Good to know, I've not tested that case yet. Well, I spent far too long staring at that, only to find it's been working all along. You can set it on the itemtype and it should have always worked (as that's done in post-processing, which is the same between zebra and es), and all it needed to work at the item level was a mapping from the notforloan field into es to be added.
(In reply to Robin Sheat from comment #148) > (In reply to Jonathan Druart from comment #147) > > I don't understand the problem with the structure, you could have several > > mappings (it's an arrayref of hashrefs). > > Oh, yes, you're right. I missed that bit. > > > With this structure I could insert exactly the same data in the tables > > (except if I missed something...). > > It's just that it's not a natural representation from a database theory > point of view. Why are the various MARC forms lumped together like that? Why > should a schema change be necessary when (god forbid) a new flavour is > added? Actually, that could be important as if we want to add support for > different things into ES, say importing a feed from a journal provider or > whatever, you could use a flavour to mark how the source is mapped into > elasticsearch. Additionally, it promotes redundancy as you are forced to > repeat things to have all the required combinations, which always causes an > icky feeling :) Erk, of course! I did not c/p the correct structure I have in my document! It was: { biblio => { title => { # name label => 'Title', type => 'string', mappings => [ { suggestible => 1, facet => 1, sort => 0, marc_type => 'marc21', marc_field => '245a', }, { suggestible => 1, facet => 1, sort => 0, marc_type => 'unimarc', marc_field => '200a', }, [ and normarc of course :) ] ] }, }
Found something else: In the sql file you have Heading-main vs Heading-Main. record-source vs Record-Source. For the first one, it's a typo (both for authorities). For the second one (it's a typo too, but), and for Local-number, you have the search_field.name which will be mapped with biblios and authorities fields. Does it make sense in your mind? In mine it's a bit weird (but maybe I need a pint).
(In reply to Jonathan Druart from comment #150) > Erk, of course! > I did not c/p the correct structure I have in my document! OK, yeah, that's a fair bit better :) But you still have an amount of duplication, for example when two search fields come from one marc field, you'll have to repeat the marc field. I don't know if that's good or bad really, but from a database normalisation point of view, it's bad (not seriously so though.) But between what's there now (including having moved the attributes appropriately), and what you're suggesting, I don't have particularly strong feelings. If you think it'll make other parts easier, then go with it. Changing the schema is pretty easy really, there's only a few places it touches. (In reply to Jonathan Druart from comment #151) > Found something else: > In the sql file you have Heading-main vs Heading-Main. record-source vs > Record-Source. > For the first one, it's a typo (both for authorities). > For the second one (it's a typo too, but), and for Local-number, you have > the search_field.name which will be mapped with biblios and authorities > fields. > > Does it make sense in your mind? In mine it's a bit weird (but maybe I need > a pint). It's OK to have the same field for authorities and biblios, they're totally different indices. As for the capitalisation stuff, there was a period when I wanted everything to be lower case before I discovered that would cause many things to break. So there may be some leftovers from that. Or just typos. What really ought to happen is a big go-over where the mapping is generated from scratch from the zebra files. Probably in an automated fashion. Mostly these mappings are there to get something functional out of it, it'll be a bit more work to make them actually correct :)
I've created a zebra-running version to help shake out the bugs that have accumulated in there. It's at: http://zebra.koha.catalystdemo.net.nz/ and is exactly the same code as the elasticsearch version, but with the SearchEngine preference set to 'Zebra'. It isn't working right now, I've probably broken something. That's what I'll be fixing for the next little while.
Created attachment 43345 [details] [review] Bug 12478: [DO NOT PUSH] script to generate the mappings yaml file
Created attachment 43346 [details] [review] Bug 12478: Add the yaml mappings file This should facilitate the modification of the mappings. The yaml format is more readable than sql.
Created attachment 43347 [details] [review] Bug 12478: Add Koha::SearchField[s] and Koha::SearchMarcMap[s] classes This was initially submited on bug 14899, but we will need it here.
Created attachment 43348 [details] [review] Bug 12478: Move mapping attributes to the join table To make things consistent, the attributes of the mappings (sort, suggestible, facet) should be moved to the join tables (search_marc_to_field). Indeed the search_marc_map could represent a mapping for several search fields with different attributes. Before this change it was not possible.
Created attachment 43349 [details] [review] Bug 12478: update the schema These files have been generated with the misc/devel/update_dbix_class_files.pl script.
Created attachment 43350 [details] [review] Bug 12478: Manually add the many_to_many relationships This will create the add_to_* accessors for us.
Created attachment 43351 [details] [review] Bug 12478: Remove previous mappings file (sql)
Created attachment 43352 [details] [review] Bug 14899: Add a link to the new page in the admin
Hi Robin, I think I did it! :) Last patch set move the mapping attributes to the join table and provide a yaml file to manage the default mappings. I will test it a bit more tomorrow, to be sure I have not introduced regression. Let me know if it goes in the direction of what you had in mind! I am also attaching patches on bug 14899, the interface is now much more consistent.
well, s/much more/a bit more
Jonathan, you're not putting correct copyright headers into the files you're creating/modifying. You need to.
(In reply to Jonathan Druart from comment #162) > Let me know if it goes in the direction of what you had in mind! Just reading the patches, this all looks good to me. I'll pull them in (excluding the one you don't want pulled :) My only thought is that doing scripted (e.g. in vim) updates of the yaml will be harder, as it's not each entry on one like the SQL. Perhaps the best way to deal with that is something that will dump the database out to YAML, so you can do batch changes in SQL and then just rexport the YAML. > I am also attaching patches on bug 14899, the interface is now much more > consistent. Brilliant, I need to try that out some time soon. I'm kinda thinking of moving this somewhere else, maybe gitlab, so that you don't have to wait for me to merge things in for you. What do you think?
Commit cd4905c2 (Display facet terms ordered by number of occurrences) introduces a bug when searching with zebra (i.e. it makes it break.) I will roll it back until we get a better solution. Feel free to unroll it back if you get the chance to make one.
Created attachment 43373 [details] [review] Bug 12478: Display facet terms ordered by number of occurrences By default ES returns the facet terms ordered by most used, which makes sense. This patch removes resort done in the scripts (catalogue/search.pl and opac/opac-search.pl) and moves it to the module. For Zebra it's now done in C4::Search::getRecords, and there is no change to expect (still alphabetically). On the Elastic search side, we could imagine to let the library define the order of the facets. The facet terms are now sorted by most used. To test easily this change, turn on the displayFacetCount pref.
(In reply to Robin Sheat from comment #166) > Commit cd4905c2 (Display facet terms ordered by number of occurrences) > introduces a bug when searching with zebra (i.e. it makes it break.) I will > roll it back until we get a better solution. Feel free to unroll it back if > you get the chance to make one. Oops, sorry about that! The last patch should work now.
(In reply to Jonathan Druart from comment #168) > Oops, sorry about that! > The last patch should work now. It does, thanks!
Dear folk, I git clone git://git.catalyst.net.nz/koha.git . os - debian 7.9 wheezy , x86_64 Linux debian 3.2.0-4-amd64 #1 SMP Debian 3.2.68-1+deb7u4 x86_64 GNU/Linux koha version - 3.21.00.030 perl - 5.014002 mysql - mysql Ver 14.14 Distrib 5.5.44, for debian-linux-gnu (x86_64) using readline 6.2 Elasticsearch version: 1.7.2 JVM name: Java HotSpot(TM) 64-Bit Server VM JVM vendor: Oracle Corporation JVM version: 25.60-b23 Java version: 1.8.0_60 I use bulkmarcimport.pl to import marc data, But import 1,000 records every time , it occurred "................[NoNodes] ** No nodes are available: [http://127.0.0.1:9200], called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /usr/local/share/perl/5.14.2/Catmandu/Store/ElasticSearch.pm line 61.root@debian:/usr/share/koha/bin/migration_tools#" This job stopped. Has anyone see the same condition? BR. longshan
(In reply to kohayu from comment #171) > > I use bulkmarcimport.pl to import marc data, > But import 1,000 records every time , it occurred > > "................[NoNodes] ** No nodes are available: > [http://127.0.0.1:9200], called from sub > Search::Elasticsearch::Role::Client::Direct::__ANON__ at > /usr/local/share/perl/5.14.2/Catmandu/Store/ElasticSearch.pm line > 61.root@debian:/usr/share/koha/bin/migration_tools#" > > This job stopped. See comment 62.
(In reply to Jonathan Druart from comment #172) > (In reply to kohayu from comment #171) > > > > I use bulkmarcimport.pl to import marc data, > > But import 1,000 records every time , it occurred > > > > "................[NoNodes] ** No nodes are available: > > [http://127.0.0.1:9200], called from sub > > Search::Elasticsearch::Role::Client::Direct::__ANON__ at > > /usr/local/share/perl/5.14.2/Catmandu/Store/ElasticSearch.pm line > > 61.root@debian:/usr/share/koha/bin/migration_tools#" > > > > This job stopped. > > See comment 62. my config <elasticsearch> <server>127.0.0.1:9200</server> <!-- may be repeated to include all servers on your cluster --> <index_name>koha_robin</index_name> <!-- should be unique amongst all the indices on your cluster. _biblios and _authorities will be appended. --> </elasticsearch> elasticsearch no log, my elasticseach works very well. http://i.imgur.com/ESRj0oj.png ElasticSearch.pm sub _build_es { my ($self) = @_; my $es = Search::Elasticsearch->new($self->_es_args); # line 61 unless ($es->indices->exists(index => $self->index_name)) { $es->indices->create( index => $self->index_name, body => { settings => $self->index_settings, mappings => $self->index_mappings, }, ); } $es;
my elaticsearch index http://imgur.com/htHLWCW koha_robin_authorities koha_robin_biblios I import data by hand , koha can't import all marc data once. BR, longshan
What is the return of curl -X GET http://localhost:9200/ ?
(In reply to Jonathan Druart from comment #175) > What is the return of > curl -X GET http://localhost:9200/ > ? Dear Jonbathan, output as follow, root@debian:~# curl -X GET http://localhost:9200/ { "status" : 200, "name" : "Zero", "cluster_name" : "elasticsearch", "version" : { "number" : "1.7.2", "build_hash" : "e43676b1385b8125d647f593f7202acbd816e8ec", "build_timestamp" : "2015-09-14T09:49:53Z", "build_snapshot" : false, "lucene_version" : "4.10.4" }, "tagline" : "You Know, for Search" }
Sorry, the answer was in your first post: try using jessie instead of wheezy.
(In reply to Jonathan Druart from comment #177) > Sorry, the answer was in your first post: try using jessie instead of wheezy. At least it's what I am using, maybe it works under wheezy, I don't know!
(In reply to Jonathan Druart from comment #178) > (In reply to Jonathan Druart from comment #177) > > Sorry, the answer was in your first post: try using jessie instead of wheezy. > > At least it's what I am using, maybe it works under wheezy, I don't know! ok, i try jessie , thanks a lot,
(In reply to Jonathan Druart from comment #178) > At least it's what I am using, maybe it works under wheezy, I don't know! I forget the exact details, but there were some issues with wheezy. Additionally, when debugging, always add: <trace_to>Stderr</trace_to> to the <elasticsearch> block in koha-conf.xml, it will trace all the traffic between elasticsearch and Koha to stderr.
(In reply to Robin Sheat from comment #180) > (In reply to Jonathan Druart from comment #178) > > At least it's what I am using, maybe it works under wheezy, I don't know! > > I forget the exact details, but there were some issues with wheezy. > > Additionally, when debugging, always add: > > <trace_to>Stderr</trace_to> > > to the <elasticsearch> block in koha-conf.xml, it will trace all the traffic > between elasticsearch and Koha to stderr. Dear Robin, Jonbathan I change jessie, it can import marc data to koha successfully. I also find some es plugin to debug koha. https://github.com/lukas-vlcek/bigdesk https://github.com/mobz/elasticsearch-head https://github.com/karmi/elasticsearch-paramedic https://github.com/NLPchina/elasticsearch-sql/ opac http://i.imgur.com/orvED9C.png thanks a lot, BR, longshan
I've started writing some documentation on how the Elasticsearch stuff works internally here: http://wiki.koha-community.org/wiki/Elasticsearch/Implementation Right now it's just the high-level stuff, but I'll get finer detail on it soon.
FYI, this is my last week at Catalyst and hence working on this. I'm going to get the docs finished off to make it easy for everyone, but if you've got any questions, best get them in soon.
(In reply to Robin Sheat from comment #183) > FYI, this is my last week at Catalyst and hence working on this. I'm going > to get the docs finished off to make it easy for everyone, but if you've got > any questions, best get them in soon. On that note, where are all the patches for this development? All I see is "Bug 12478: Display facet terms ordered by number of occurrences". Were the rest accidentally marked as obsolete or are they on different bugs?
(In reply to David Cook from comment #184) > On that note, where are all the patches for this development? All I see is > "Bug 12478: Display facet terms ordered by number of occurrences". Were the > rest accidentally marked as obsolete or are they on different bugs? They've all been pulled into the branch on the catalyst repo. It was my plan to break them up into functional patches afterwards.
(In reply to Robin Sheat from comment #185) > > They've all been pulled into the branch on the catalyst repo. It was my plan > to break them up into functional patches afterwards. Ahhh sounds good. I'll look there if/when I get to checking out. Thanks!
Es can not "edit record" or "edit items" in UNIMARC structure. (But it works well in USMARC.) It can show results, but when I click "Edit record", it shows "Add MARC record "--blank MARC. http://127.0.1.1:8080/cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber= If I click "Edit items" , it shows http://127.0.1.1:8080/cgi-bin/koha/cataloguing/additem.pl?biblionumber= Software error: Can't call method "fields" on an undefined value at /home/koha/kohaclone/cataloguing/additem.pl line 703. For help, please send mail to the webmaster (webmaster@koha), giving this error message and the time and date of the error. I think it can not get the " biblionumber= ".
(In reply to Peter Zhao from comment #187) > Es can not "edit record" or "edit items" in UNIMARC structure. (But it > works well in USMARC.) > > It can show results, but when I click "Edit record", it shows "Add MARC > record "--blank MARC. > http://127.0.1.1:8080/cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber= > > > > If I click "Edit items" , it shows > > http://127.0.1.1:8080/cgi-bin/koha/cataloguing/additem.pl?biblionumber= > > Software error: > > Can't call method "fields" on an undefined value at > /home/koha/kohaclone/cataloguing/additem.pl line 703. > > For help, please send mail to the webmaster (webmaster@koha), giving this > error message and the time and date of the error. > > I think it can not get the " biblionumber= ". I suspect that it comes from the fact that biblionumber is not indexed in a specific field (see comment 148). Moreover I don't think the following comment is correct http://git.catalyst.net.nz/gw?p=koha.git;a=blob;f=Koha/ElasticSearch/Indexer.pm;h=06dcb1d3e0b57633652d4cb70ed46ac59f0f802e;hb=refs/heads/elastic_search#l176 167 sub _sanitise_records { 168 my ($self, $biblionums, $records) = @_; 169 170 confess "Unequal number of values in \$biblionums and \$records." if (@$biblionums != @$records); 171 172 my $c = @$biblionums; 173 for (my $i=0; $i<$c; $i++) { 174 my $bibnum = $biblionums->[$i]; 175 my $rec = $records->[$i]; 176 # I've seen things you people wouldn't believe. Attack ships on fire 177 # off the shoulder of Orion. I watched C-beams glitter in the dark near 178 # the Tannhauser gate. MARC records where 999$c doesn't match the 179 # biblionumber column. All those moments will be lost in time... like 180 # tears in rain... 181 $rec->delete_fields($rec->field('999')); 182 $rec->append_fields(MARC::Field->new('999','','','c' => $bibnum, 'd' => $bibnum)); 183 } 184 } The biblionumber is stored in 001, isn't it?
(In reply to Jonathan Druart from comment #188) > The biblionumber is stored in 001, isn't it? No. For imported records, the 001 is not touched. For original cataloguing (starting from scratch) 001 has to be filled manually. The one place where you can find the biblionumber in our MARC records is in 999 $c and $d.
I think UNIMARC uses 001 if I am not mistaken.
Hi all, I just got the elastic branch setup on kohadevbox and am eager to do some testing, I am just wondering if there is anywhere specific we are poking yet, or just a general 'try to break it'/'figure out what it doesn't do' Searching definitely feels faster, even on a 100 record db. I do same to be able to switch between zebra and elastic with an issue once they are up and running which is very useful In just poking for now I spot 2 things: 1 - I don't seem to able to edit/add records. I get an error: Can't locate object method "field" via package "<?xml version="1.0" encoding="UTF-8"?>..folowed by full marcxml for record 2 - Searching doesn't automatically truncate/stem. i.e. to search 'Char' and get 'Charles' I have to search for 'Char*' and when I do highligthing doesn't work I am excited to do more testing, just want to know what the goals are to get this in shape to make it in to Koha so I can focus on hitting those pieces. -Nick
Created attachment 47767 [details] Bug12748 - Fixes duplicate serials with an "expected" status bug Added a new sub to Serials.pm to be able to get serials with their statuses. Now the sub ModSerialStatus checks for other serials with an "expected" status before doing anything. Also modified Serials.t to be able to test those changes. Test Plan 1) Apply patch 2) Run ./t/db_dependent/Serials.t 3) Validate that there are no errors 4) Go on "Serial collection information" page for a serial of your choice 5) Click on "Generate next" 6) Change the status of the original serial from "late" to "expected" 7) Change the newly generated serial from "expected" to "delete" 8) Validate that there are no new serials created by instruction 7 and that the serial was deleted 9) Run ./t/db_dependent/Serials.t With QA Fixes - Use the constant instead of the code (1 vs EXPECTED) - Avoid interpolation in query - use selectall_arrayref instead of fetchall_arrayref Signed-off-by: Liz Rea <liz@catalyst.net.nz>
Created attachment 47768 [details] Bug 12748 - Add test for step 7 and 8 and rename findSerialByStatus
> I am excited to do more testing, just want to know what the goals are to get > this in shape to make it in to Koha so I can focus on hitting those pieces. > > -Nick Hi Nick From my mind, the main thing to test hard at this point, as that if you have this code, and you turn off the syspref .. that it has no effect. IE, this breaks nothing existing. If so then I reckon it could go in as experimental, and we can get more people testing and new bugs can be opened for missing features/bugs related to it. Does that makes sense? (just my 2 cents)
I have let some works to be done in comments 100-102 (see also Robin's kanban). It's not ready for production at all and people are waiting for this for 5 years now. It will be very weird not to provide something a bit more complete. The development is inactive for 6 months, and Robin has left the Koha project. We need a plan... The "let's push that and wait for people to open bug reports and enhancement requests" approach sounds a bit risky.
Not as risky as making one massive bug that will never get pushed because it is too big to test properly.
I think we should try to get it in and explicitly mark it as experimental. That would give interested (non-dev) parties the chance to try it and give feedback. I have spoken to a lot of people about Koha recently, and being able to test ES during their evaluation of the system would likely help some of them to make a decision in favour of Koha.
We should avoid to repeat the Solr problem - I think part of the problem was that it was not production ready when included in Koha and I'd hate to see us get stuck in a similar way with ES. I think some good information/documentation about what works and what works not (yet) would be good. I am thinking about a wiki page - with more end user oriented information - which system preferences are supported, why does a search x result in y...(FAQ?), how to configure in the GUI, and fill in any gaps while testing. I could start a page, but will need help filling in the information and also to get ES running on my computer.
I would like to get other opinions on the discussion on comment 38 and comment 39 about the idea to reuse existing search scripts and templates. IMO it will be a mess in a near future: 1/ we will want to implement things for ES, not supported by Zebra 2/ at some point the features already implemented for Zebra won't be available for ES => We will have to write switch (if se=='es' {} elsif se == 'zebra') everywhere.
And I would also like to know who has already looked at the ES implementation and get their feedbacks.
(In reply to Jonathan Druart from comment #199) > I would like to get other opinions on the discussion on comment 38 and > comment 39 about the idea to reuse existing search scripts and templates. > IMO it will be a mess in a near future: > 1/ we will want to implement things for ES, not supported by Zebra > 2/ at some point the features already implemented for Zebra won't be > available for ES > => We will have to write switch (if se=='es' {} elsif se == 'zebra') > everywhere. A sort of road map that I have in mind (and yes I'll back that up with funding etc.) 1. Get Elastic into Koha (marked as experimental) - making sure that we aren't breaking anything with Zebra! (that's the main testing we need now) 2. Elasticsearch talks JSON 3. As we want to develop new features that are ES centric and not Zebra - we split those off 4. Meaning we can build a new frontend for it, that uses elasticsearch to search, and the restful api to get records. 5. New Frontend for it - that is not C4::Search (no one really wants to work that anymore). 6. introducing the Catmandu libraries into Koha should be a good thing - we can start to explore RDF etc.
(In reply to Katrin Fischer from comment #198) > We should avoid to repeat the Solr problem - I think part of the problem was > that it was not production ready when included in Koha and I'd hate to see > us get stuck in a similar way with ES. I think the problem was that Solr integration ran out of steam ( i.e. development funding ). That is not going to happen with Elastic. I think it's safe to say that we are fully committed to getting Elastic to work with Koha. It's something of a chicken and egg problem. We can't let past failures cause hesitation on future opportunities, that will only lead to stagnation. As an aside, I'm looking forward to Elastic as a way to greatly speed up our patron search. With Elastic we'll be able to index our patron data as well as record data! In fact, we'll be able to use it for any type of data where we search on large data sets! Once Elastic is part of Koha I plan on starting to work on an (optional) way for the patron search to take advantage of Elastic.
Hi Kyle, I should try and clarify my comment :) I am not against including this as an experimental feature at all! - I just think we need to be clear and upfront in documentation on what is working and what not - so people know what to expect. Right now I am not quite sure what to expect of the work available - what should be working? What is intended to work? It would help me to understand that better and give me a starting point.
(In reply to Katrin Fischer from comment #203) > Hi Kyle, I should try and clarify my comment :) I am not against including > this as an experimental feature at all! - I just think we need to be clear > and upfront in documentation on what is working and what not - so people > know what to expect. Right now I am not quite sure what to expect of the > work available - what should be working? What is intended to work? It would > help me to understand that better and give me a starting point. Agreed! I think we are all on the same page. Nick is going to begin testing these patches for sign-off. I imagine we will be able to help fill in the gaps where documentation is lacking and be able to tell us if it is lacking in any functionality.
So far most things have worked nicely, problems I have found: Authority search fails when sorting is enabled (Software error:Unable to understand your search query, please rephrase and try again.) and lots of this in logs: Parse Failure [No mapping found for [Heading.phrase] in order to sort on] Authority searching did not throw an error, but wouldn't return results with no sorting. I tried doing a rebuild to clear above but I got errors about AUTOLOAD inheritance for StripNonXmlChars in Authority.pm. I tried adding "use C4::Charset;" but then got an object error. Authority searching from a record via popup gave: Can't locate object method "field" via package "<?xml version="1.0" encoding="UTF-8"?>..followed by full marcxml for record Location (Branch) facet returns no results when combined with search Title-series limit often returns no results when combined with search The only problem I hit so far in switching back to zebrawas when I selected zebra as the search engine the cataloging search fails: Can't locate object method "simple_search_compat" via package "Koha::SearchEngine::Zebra::Search" I am going to keep plugging but authorities seems to be the biggest issue for now
The branch is 500 commits behind master, is someone in charge of rebasing it?
(In reply to Jonathan Druart from comment #206) > The branch is 500 commits behind master, is someone in charge of rebasing it? Your wish is my command, rebased now :)
_sanitise_records makes it impossible to create or import authority records and to use Z39.50 authority search. (staff client, Z39.50 and bulkmarcimport.pl) It will fail related to > Can't call method "field" on an undefined value at /home/koha/koha/Koha/ElasticSearch/Indexer.pm line 181. Without 181 and 182 I am able to create a record in staff client manually and with Z39.50. > # $rec->delete_fields($rec->field('999')); > # $rec->append_fields(MARC::Field->new('999','','','c' => $bibnum, 'd' => $bibnum)); It looks like _sanitise_records is biblio-specific and should probably not be used with auth files, or should be changed to handle them correctly. There are still errors with bulkmarcimport with these lines commented out.
Auth search in "entire record" does not work. Software error: Invalid marclist field provided: all at /home/koha/koha/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm line 417. Koha::SearchEngine::Elasticsearch::QueryBuilder::build_authorities_query_compat(Koha::SearchEngine::Elasticsearch::QueryBuilder=HASH(0x867c5d8), ARRAY(0x8649338), ARRAY(0x8649308), ARRAY(0x867c4b8), ARRAY(0x8689490), ARRAY(0x9b55800), "CORPO_NAME", "HeadingAsc") called at /home/koha/koha/authorities/authorities-home.pl line 86
Auth search for anything else than "search entire record" throws no error, but returns no results. I can't find any authority functionality at all. Am I missing something?
Aaaaand, auth search is broken for Zebra too. Software error: Can't locate object method "simple_search_compat" via package "Koha::SearchEngine::Zebra::Search" at /home/koha/koha/C4/AuthoritiesMarc.pm line 357.
(In reply to Mirko Tietgen from comment #211) > Aaaaand, auth search is broken for Zebra too. > > Software error: > > Can't locate object method "simple_search_compat" via package > "Koha::SearchEngine::Zebra::Search" at /home/koha/koha/C4/AuthoritiesMarc.pm > line 357. Sweet thanks for finding that, I'm mostly concerned in finding things that are broken when not using ElasticSearch. Because those are the blockers. I'll try to get a fix for this, sometime this week.
I have rebased the branch (and push it on github https://github.com/joubu/Koha/commits/elastic_search). I have also submitted a patch to fix the authority search.
Created attachment 50115 [details] [review] Bug 12478: Define simple_search_compat for Zebra When Zebra is defined as the search engine, the authority search is broken (Can't locate object method "simple_search_compat" via package "Koha::SearchEngine::Zebra::Search" at /home/koha/koha/C4/AuthoritiesMarc.pm line 357.) simple_search_compat is not defined in Koha::SearchEngine::Zebra::Search, it should be a wrapper for C4::Search::SimpleSearch
Note that the tests are completely out of sync and do not pass.
new_12478_elasticsearch is now up to date, including the latest patch. Ready for testing for regressions tomorrow
Can't index authorities in ES. koha-koha@jessie:/home/koha/koha$ perl misc/search_tools/rebuild_elastic_search.pl -v -a Indexing authorities Can't locate object method "get_all_authorities_iterator" via package "Koha::Authority" at misc/search_tools/rebuild_elastic_search.pl line 151. Not sure why that is the case, there is a sub get_all_authorities_iterator in Koha/Authority.pm
Created attachment 50159 [details] [review] Bug 12478: (QA followup) Koha::SearchEngine should fallback to Zebra Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>
Branch is up to date again with the latest patches
Branch updated with 2 fixes for installing/upgrading
I have tested all the things I can think of (authorities,staff/opac, facets, acquisitions searching, importing) from fresh db and upgrade and found no regression with these patches for using zebra
The new_12478_elasticsearch now has all the patches signed off
(In reply to Jonathan Druart from comment #215) > Note that the tests are completely out of sync and do not pass. That will certainly be a blocker for QA.
Have you tried with bug 16249!
Heh that was meant to be a ? Not a !
(In reply to Chris Cormack from comment #224) > Have you tried with bug 16249! I was talking about t/Koha_ElasticSearch_Indexer.t, t/Koha_ElasticSearch_Search.t and t/Koha_ElasticSearch.t
Will work on the tests today and attach them here
Created attachment 50211 [details] [review] Bug 12478 : Fixing the tests in t/Koha_ElasticSearch_Indexer.t
Created attachment 50212 [details] [review] Bug 12478 : Fixing the tests for Koha::SearchEngine::Elasticsearch::Search perl t/Koha_SearchEngine_Elasticsearch_Search.t
Right the ElasticSearch related tests should pass now. If you checkout the branch and apply the 2 patches on here.
Created attachment 50213 [details] [review] Bug 12478 : Fixing the tests in t/Koha_ElasticSearch_Indexer.t Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Created attachment 50214 [details] [review] Bug 12478 : Fixing the tests for Koha::SearchEngine::Elasticsearch::Search perl t/Koha_SearchEngine_Elasticsearch_Search.t Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comments about the last 2 patches: 1/ The tests are db dependent, the files should be moved to t/db_dependent 2/ The license part is missing, you have put a custom block instead I think we should have a better test coverage before pushing this work.
Created attachment 50245 [details] [review] Bug 12478 Shifting tests and adding copyright headers
With the 3 test files we have Koha::ElasticSearch 61.3% statement coverage Koha::ElasticSearch::Indexer 56.9% statement coverage Koha::SearchEngine::Elasticsearch::Search 24.6% statement coverage I'll work now on increasing the coverage of them.
Created attachment 50246 [details] [review] Bug 12478 Increasing test Coverage for Koha::SearchEngine::Elasticsearch::Search Before this patch it was 24.6%, after it is 64%
Created attachment 50291 [details] [review] Bug 12478 Shifting tests and adding copyright headers Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Created attachment 50292 [details] [review] Bug 12478 Increasing test Coverage for Koha::SearchEngine::Elasticsearch::Search Before this patch it was 24.6%, after it is 64% Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
I have been testing the ElasticSearch patch set, focusing on ensuring that it does not alter the search results when using Zebra. May testing has verified that the results look identical using master vs master + elastic. The only noteworthy difference is that with Elastic, the facets are now in alphabetical order. I do not think this is an issue. If a library would like to customize the order of the facets, it can always be done via javascript. This can be considered my sign-off! Kyle
(In reply to Kyle M Hall from comment #239) > The only noteworthy difference is that with Elastic, the facets are now in > alphabetical order. I do not think this is an issue. You mean using Zebra on the master+ES branch? Looking at the commit msg of Bug 12478: Display facet terms ordered by number of occurrences For Zebra it's now done in C4::Search::getRecords, and there is no change to expect (still alphabetically). We should not expect any changes, the facets should already been sorted alphabetically. > If a library would like > to customize the order of the facets, it can always be done via javascript. Not sure it will be an easy one :)
Comment on attachment 50292 [details] [review] Bug 12478 Increasing test Coverage for Koha::SearchEngine::Elasticsearch::Search Branch is now up to date
Branch has been updated with signoffs and two followups to fix QA/test issues.
Pushed to Master - Should be in the May 2016 Release. Thanks! (will run the Schema update in a bit)
I had a problem with the database patch. I had to do: diff --git a/installer/data/mysql/updatedatabase.pl b/installer/data/mysql/updatedatabase.pl index 9e6d390..e092704 100755 --- a/installer/data/mysql/updatedatabase.pl +++ b/installer/data/mysql/updatedatabase.pl @@ -12332,13 +12332,13 @@ my $indexes = LoadFile( $mappings_yaml ); while ( my ( $index_name, $fields ) = each %$indexes ) { while ( my ( $field_name, $data ) = each %$fields ) { - my $field_type = $data->{type}; + my $field_type = $data->{type} || 'string'; my $field_label = $data->{label}; my $mappings = $data->{mappings}; my $search_field = Koha::SearchFields->find_or_create({ name => $field_name, label => $field_label, type => $field_type }, { key => 'name' }); for my $mapping ( @$mappings ) { my $marc_field = Koha::SearchMarcMaps->find_or_create({ index_name => $index_name, marc_type => $mapping->{marc_type}, marc_field => $mapping->{marc_field} }); - $search_field->add_to_search_marc_maps($marc_field, { facet => $mapping->{facet}, suggestible => $mapping->{suggestible}, sort => $mapping->{sort} } ); + $search_field->add_to_search_marc_maps($marc_field, { facet => $mapping->{facet} || 0, suggestible => $mapping->{suggestible} || 0, sort => $mapping->{sort} || undef } ); } } } Was it just my mysql?
Also, with my version of elastic, I had to do following changes: diff --git a/Koha/ElasticSearch.pm b/Koha/ElasticSearch.pm index d56c344..22bc064 100644 --- a/Koha/ElasticSearch.pm +++ b/Koha/ElasticSearch.pm @@ -171,11 +171,6 @@ sub get_elasticsearch_mappings { include_in_all => JSON::false, type => "string", }, - '_all.phrase' => { - search_analyzer => "analyser_phrase", - index_analyzer => "analyser_phrase", - type => "string", - }, } } }; @@ -195,15 +190,12 @@ sub get_elasticsearch_mappings { ? 'boolean' : 'string'; $mappings->{data}{properties}{$name} = { - search_analyzer => "analyser_standard", - index_analyzer => "analyser_standard", + analyzer => "analyser_standard", type => $es_type, fields => { phrase => { - search_analyzer => "analyser_phrase", - index_analyzer => "analyser_phrase", + analyzer => "analyser_phrase", type => "string", - copy_to => "_all.phrase", }, raw => { "type" => "string", @@ -222,22 +214,19 @@ sub get_elasticsearch_mappings { if ($suggestible) { $mappings->{data}{properties}{ $name . '__suggestion' } = { type => 'completion', - index_analyzer => 'simple', - search_analyzer => 'simple', + analyzer => 'simple', }; } # Sort may be true, false, or undef. Here we care if it's # anything other than undef. if (defined $sort) { $mappings->{data}{properties}{ $name . '__sort' } = { - search_analyzer => "analyser_phrase", - index_analyzer => "analyser_phrase", + analyzer => "analyser_phrase", type => "string", include_in_all => JSON::false, fields => { phrase => { - search_analyzer => "analyser_phrase", - index_analyzer => "analyser_phrase", + analyzer => "analyser_phrase", type => "string", }, }, diff --git a/installer/data/mysql/updatedatabase.pl b/installer/data/mysql/updatedatabase.pl index 9e6d390..e092704 100755 --- a/installer/data/mysql/updatedatabase.pl +++ b/installer/data/mysql/updatedatabase.pl @@ -12332,13 +12332,13 @@ my $indexes = LoadFile( $mappings_yaml ); while ( my ( $index_name, $fields ) = each %$indexes ) { while ( my ( $field_name, $data ) = each %$fields ) { - my $field_type = $data->{type}; + my $field_type = $data->{type} || 'string'; my $field_label = $data->{label}; my $mappings = $data->{mappings}; my $search_field = Koha::SearchFields->find_or_create({ name => $field_name, label => $field_label, type => $field_type }, { key => 'name' }); for my $mapping ( @$mappings ) { my $marc_field = Koha::SearchMarcMaps->find_or_create({ index_name => $index_name, marc_type => $mapping->{marc_type}, marc_field => $mapping->{marc_field} }); - $search_field->add_to_search_marc_maps($marc_field, { facet => $mapping->{facet}, suggestible => $mapping->{suggestible}, sort => $mapping->{sort} } ); + $search_field->add_to_search_marc_maps($marc_field, { facet => $mapping->{facet} || 0, suggestible => $mapping->{suggestible} || 0, sort => $mapping->{sort} || undef } ); } } }
I can't reindex Authorities with rebuild_elastic_search.pl, as Nick Clemens said. I added C4::Charset and now I got this error: DBIx::Class::ResultSet::new_result(): Result object instantiation requires a hashref as argument at Koha/Object.pm line 59 When I edit an Authority and save it, it goes to ES indexed, but full reindex can not be done.
Testing at the KohaCon hackfest I also couldn't reindex authorities - biblios worked fine.
I am investigating the authority indexing issue. It seems to be caused by a bad merge fix conflict.
(In reply to Jonathan Druart from comment #248) > I am investigating the authority indexing issue. > It seems to be caused by a bad merge fix conflict. See bug 16708.
Thank you Jonathan, the patch in bug 16708 applies correctly and the authorities reindex fine
(In reply to Juan Romay Sieira from comment #250) > Thank you Jonathan, the patch in bug 16708 applies correctly and the > authorities reindex fine So go and sign it off :)
Trying to test implementation of ElasticSearch. I don't understand why it does not work... If someone has an idea.. https://semestriel.framapad.org/p/jTiBzc32B7
I had something similar with mismatching libcatmandu-perl, libcatmandu-store-elasticsearch-perl and libcatmandu-marc-perl. Which versions do you have please?
(In reply to Srdjan Jankovic from comment #253) > I had something similar with mismatching libcatmandu-perl, > libcatmandu-store-elasticsearch-perl and libcatmandu-marc-perl. Which > versions do you have please? Hi Srdjan, the pad Claire posted contains all the infos ;) I c/p it here not to loose it: $ misc/search_tools/rebuild_elastic_search.pl -v -bn=1 Indexing biblios 1 must be hashref, arrayref, coderef or iterable object Trace begun at /usr/local/share/perl/5.22.1/Catmandu/Fix.pm line 171 Catmandu::Fix::fix('Catmandu::Fix=HASH(0x6c3ba40)', 'Catmandu::Importer::MARC=HASH(0x6b44988)') called at /home/koha/src/Koha/SearchEngine/Elasticsearch/Indexer.pm line 193 Koha::SearchEngine::Elasticsearch::Indexer::_convert_marc_to_json('Koha::SearchEngine::Elasticsearch::Indexer=HASH(0xab9fa0)', 'ARRAY(0x4cd6600)') called at /home/koha/src/Koha/SearchEngine/Elasticsearch/Indexer.pm line 69 Koha::SearchEngine::Elasticsearch::Indexer::update_index('Koha::SearchEngine::Elasticsearch::Indexer=HASH(0xab9fa0)', 'ARRAY(0x4cd6738)', 'ARRAY(0x4cd6600)') called at misc/search_tools/rebuild_elastic_search.pl line 191 main::do_reindex('CODE(0x8b3ba0)', 'biblios') called at misc/search_tools/rebuild_elastic_search.pl line 139 $ git log -1 --oneline --decorate a8376f9 (HEAD -> master, origin/master, origin/HEAD) Bug 17432: Remove minification ii elasticsearch 1.7.5 all Open Source, Distributed, RESTful Search Engine ii libcatmandu-store-elasticsearch 0.0304-1 all A searchable store backed by Elasticsearch ii libsearch-elasticsearch-perl 1.19-1 all The official client for Elasticsearch ii libcatmandu-marc-perl 0.214-1 all modules for working with MARC data within the Catmandu framework ii libcatmandu-perl 0.9505-1 all metadata toolkit ii libcatmandu-store-elasticsearch 0.0304-1 all A searchable store backed by Elasticsearch ii liblog-any-adapter-perl 0.11-2 all <short description; defaults to some wise words> ii liblog-any-perl 1.03-1 all Perl module to log messages safely and efficiently
End of the week, last chance, I dropped my environnement and did a fresh reinstall and it works \o/ ii elasticsearch 1.7.5 all Open Source, Distributed, RESTful Search Engine ii libcatmandu-store-elasticsearch 0.0304-1 all A searchable store backed by Elasticsearch ii libsearch-elasticsearch-perl 1.19-1 all The official client for Elasticsearch ii libcatmandu-marc-perl 0.214-1 all modules for working with MARC data within the Catmandu framework ii libcatmandu-perl 0.9505-1 all metadata toolkit ii liblog-any-adapter-perl 0.11-2 all <short description; defaults to some wise words> ii liblog-any-perl 1.03-1 all Perl module to log messages safely and efficiently Versions seems the same... but it works !
I am trying to test Elasticsearch in Koha and I keep getting an error: No 'elasticsearch' block is defined in koha-conf.xml. I am running a fresh install of Koha 16.11.03.000 and Debian 8 and imported our data from 3.16.04.000 I have enabled Memcached and Plack. Installed openjdk-8 and elastic search. I first got the error while trying run rebuild_elastic_search.pl I not sure why but changing <memcached_server> to the static IP allowed the index to run but when I try to do a search on the staff side I get the error again. If this is not the place to ask please let me know. Jorge de Cardenas
Hi Jorge, You will need to manual add a block like the below into the config section of your koha-conf.xml: <elasticsearch> <server>es-server:9200</server> <index_name>koha_instance</index_name> </elasticsearch> The best way to reach the community is probably via Probably using the listservs: https://koha-community.org/support/koha-mailing-lists/ Or IRC: https://koha-community.org/get-involved/irc/ Good luck!