Bug 17500 - Use Elasticsearch to search for patrons
Summary: Use Elasticsearch to search for patrons
Status: In Discussion
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: unspecified
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Jonathan Druart
QA Contact:
URL:
Keywords:
Depends on: 12478
Blocks:
  Show dependency treegraph
 
Reported: 2016-10-26 12:56 UTC by Jonathan Druart
Modified: 2017-12-05 15:07 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 17500: Implement a very basic patron search (7.25 KB, patch)
2016-10-26 13:23 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 17500: Add a script to reindex patrons (2.30 KB, patch)
2016-10-26 13:23 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 17500: Display branchname and category description (1.47 KB, patch)
2016-10-26 13:23 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 17500: Index category_type to make searchable and display from the ES results (1.26 KB, patch)
2016-10-26 13:23 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 17500: Display correct total values (940 bytes, patch)
2016-10-26 13:23 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 17500: implement order by (2.94 KB, patch)
2016-10-26 13:23 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 17500: Implement a very basic patron search (7.23 KB, patch)
2017-03-22 16:43 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 17500: Add a script to reindex patrons (2.30 KB, patch)
2017-03-22 16:43 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 17500: Display branchname and category description (1.47 KB, patch)
2017-03-22 16:43 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 17500: Index category_type to make searchable and display from the ES results (1.26 KB, patch)
2017-03-22 16:43 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 17500: Display correct total values (940 bytes, patch)
2017-03-22 16:43 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 17500: implement order by (2.94 KB, patch)
2017-03-22 16:43 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 17500 - updates for ES 5.1 (3.11 KB, patch)
2017-03-22 16:44 UTC, Nick Clemens
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Jonathan Druart 2016-10-26 12:56:52 UTC
Adding and keeping up to date a new Elasticsearch index for patrons will allow us to use it to search for patrons.
Comment 1 Jonathan Druart 2016-10-26 13:23:04 UTC
Created attachment 56867 [details] [review]
Bug 17500: Implement a very basic patron search
Comment 2 Jonathan Druart 2016-10-26 13:23:07 UTC
Created attachment 56868 [details] [review]
Bug 17500: Add a script to reindex patrons
Comment 3 Jonathan Druart 2016-10-26 13:23:10 UTC
Created attachment 56869 [details] [review]
Bug 17500: Display branchname and category description

TODO display category_type
Comment 4 Jonathan Druart 2016-10-26 13:23:13 UTC
Created attachment 56870 [details] [review]
Bug 17500: Index category_type to make searchable and display from the ES results
Comment 5 Jonathan Druart 2016-10-26 13:23:16 UTC
Created attachment 56871 [details] [review]
Bug 17500: Display correct total values
Comment 6 Jonathan Druart 2016-10-26 13:23:19 UTC
Created attachment 56872 [details] [review]
Bug 17500: implement order by
Comment 7 Jonathan Druart 2016-10-26 13:25:21 UTC
This patchset is a POC and cannot be pushed as it for several reasons:

- we will need to rethink the existing code as we would need to index/search something else than bibliographic records. And the code is too bib-centric at the moment
- the patron attributes are not indexed and so not searchable
- the index is not maintained when a patron record is updated
- it makes ES mandatory as it removes the existing MySQL search. We will need to let the ability to choose/switch between both (at least for a given period).

It highlights the fact that we need to abstract our Elasticsearch code to make it more powerful and extensive.
Moreover there is something hard to fix: at the moment it's possible to sort the table by library name or patron category name. But as we only index the code it is possible to order the results by these 2 fields, and it won't be possible unless we index them.
But ofc if we index them, we will have to reindex everything if the description is updated in the admin config. Note that we will have this problematic for bibliographic records very soon as well and we will have to decide what to do.
Comment 8 Nick Clemens 2017-03-22 16:43:30 UTC
Created attachment 61481 [details] [review]
Bug 17500: Implement a very basic patron search
Comment 9 Nick Clemens 2017-03-22 16:43:40 UTC
Created attachment 61482 [details] [review]
Bug 17500: Add a script to reindex patrons
Comment 10 Nick Clemens 2017-03-22 16:43:45 UTC
Created attachment 61483 [details] [review]
Bug 17500: Display branchname and category description

TODO display category_type
Comment 11 Nick Clemens 2017-03-22 16:43:50 UTC
Created attachment 61484 [details] [review]
Bug 17500: Index category_type to make searchable and display from the ES results
Comment 12 Nick Clemens 2017-03-22 16:43:54 UTC
Created attachment 61485 [details] [review]
Bug 17500: Display correct total values
Comment 13 Nick Clemens 2017-03-22 16:43:59 UTC
Created attachment 61486 [details] [review]
Bug 17500: implement order by
Comment 14 Nick Clemens 2017-03-22 16:44:04 UTC
Created attachment 61487 [details] [review]
Bug 17500 - updates for ES 5.1
Comment 15 Nick Clemens 2017-12-05 15:07:09 UTC
(In reply to Jonathan Druart from comment #7)
> - we will need to rethink the existing code as we would need to index/search
> something else than bibliographic records. And the code is too bib-centric
> at the moment

I think this can be done - I think is a good opportunity to consider accessing ElasticSearch via perl modules from ES and not through catmandu

> - the patron attributes are not indexed and so not searchable

Yup

> - the index is not maintained when a patron record is updated

Yup

> - it makes ES mandatory as it removes the existing MySQL search. We will
> need to let the ability to choose/switch between both (at least for a given
> period).

If it replicates close enough I think this is a good opportunity to turn ES on by default, then search testing is possible for all

> 
> It highlights the fact that we need to abstract our Elasticsearch code to
> make it more powerful and extensive.
> Moreover there is something hard to fix: at the moment it's possible to sort
> the table by library name or patron category name. But as we only index the
> code it is possible to order the results by these 2 fields, and it won't be
> possible unless we index them.
> But ofc if we index them, we will have to reindex everything if the
> description is updated in the admin config. Note that we will have this
> problematic for bibliographic records very soon as well and we will have to
> decide what to do.

I think the descriptions don't change very often - a warning that things need to be reindexed I think suffices.  Bug 18950 proposed allowing user to reindex, however, I think safer is to allow user to request reindex, then handle on server by it staff to check if reindex requested and schedule.

Bug 18948 also serves to ease reindexing by doing it in the background and preserving searching while done. Paves the way for allowing changes to mappings/descriptions etc and only committing on reindex