Bug 18514 - Allow library to search their Koha and Overdrive collections using ElasticSearch
Summary: Allow library to search their Koha and Overdrive collections using ElasticSearch
Status: Patch doesn't apply
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Chris Cormack
QA Contact:
URL:
Keywords:
Depends on: 16034
Blocks:
  Show dependency treegraph
 
Reported: 2017-04-30 23:59 UTC by Chris Cormack
Modified: 2019-10-04 08:15 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 18514 : Script to fetch records from Overdrive (4.96 KB, patch)
2017-05-01 00:17 UTC, Chris Cormack
Details | Diff | Splinter Review
Bug 18514 : Adding search capabilities for Overdrive results in Koha (1.97 KB, patch)
2017-05-01 00:18 UTC, Chris Cormack
Details | Diff | Splinter Review
Bug 18514 Fix link on results page, and show overdrive data on detail page (5.07 KB, patch)
2017-05-01 00:18 UTC, Chris Cormack
Details | Diff | Splinter Review
Bug 18514 - Followup (1.71 KB, patch)
2017-07-12 10:53 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 18514 - Followup (1.77 KB, patch)
2017-08-10 20:43 UTC, Chris Cormack
Details | Diff | Splinter Review
Bug 18514 : Script to fetch records from Overdrive (5.03 KB, patch)
2017-08-10 20:47 UTC, Chris Cormack
Details | Diff | Splinter Review
Bug 18514 : Adding search capabilities for Overdrive results in Koha (1.97 KB, patch)
2017-08-10 20:47 UTC, Chris Cormack
Details | Diff | Splinter Review
Bug 18514 Fix link on results page, and show overdrive data on detail page (5.07 KB, patch)
2017-08-10 20:48 UTC, Chris Cormack
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Chris Cormack 2017-04-30 23:59:04 UTC
Building on the Elasticsearch and Overdrive work, this feature will allow users to run a cronjob to populate their elastic index with collection information from Overdrive.

This makes both local Koha records, and Overdrive ones searchable.

This bug will add the code to fetch and index the records, and to search and display them in the OPAC

Subsequent bugs will add the detail display to the staff side
Comment 1 Chris Cormack 2017-05-01 00:17:53 UTC
Created attachment 62902 [details] [review]
Bug 18514 : Script to fetch records from Overdrive

To Test:

You must be running Elasticsearch with Koha
1/ Fill in the Overdrive system preferences
2/ run this script
3/ Records should be in ElasticSearch
Comment 2 Chris Cormack 2017-05-01 00:18:17 UTC
Created attachment 62903 [details] [review]
Bug 18514 : Adding search capabilities for Overdrive results in Koha

To test
1/ Apply the first patch and run the script in it
2/ Search, notice no Overdrive results showing
3/ Search again, now you should get results in the OPAC results page
Comment 3 Chris Cormack 2017-05-01 00:18:36 UTC
Created attachment 62904 [details] [review]
Bug 18514 Fix link on results page, and show overdrive data on detail page

To test
1/ Apply the first 2 patches, and run the fetcher script (in patch one)
2/ Search, notice the overdrive results result in a 404 if you click on
them
3/ Apply this patch
4/ Search again
5/ Click on a result, you should now be at a detail page
Comment 4 Nick Clemens 2017-05-25 14:52:59 UTC
I think there are some updates on a git branch that need to be imported here, current code doesn't run
Comment 5 Chris Cormack 2017-05-25 16:00:04 UTC
Actually no there aren't. Have you not been seeing my updates in our WR system Nick?

Basically there isn't missing code here, rebuilding elastic search indexes is busted in master. 

[Request] ** http://localhost:9200 -[400] [mapper_parsing_exception] Failed to parse mapping [data]: The [string] type is removed in 5.0 and automatic upgrade failed because parameters [index_analyzer] are not supported for automatic upgrades. You should now use either a [text] or [keyword] field instead for field [ff7-01-02]

This is nothing to do with the overdrive code. 

As it dies for misc/search_tools/rebuild_elastic_search.pl too. Which is not touched by this. That dies for me on current master, with or without these patches. 

(As an aside we need to track down why you didnt see the updates)
Comment 6 Nick Clemens 2017-05-25 16:04:21 UTC
I will double check, I may have just missed those and jumped here.

A few lines do need fixed though:
Koha::ElasticSearch::Indexer;
should be 
Koha::SearchEngine::Elasticsearch::Indexer;

Do you have a bug for ES indexing broken? I haven't hit a problem, but maybe I need to refresh my mappings.

-Nick
Comment 7 Nick Clemens 2017-07-12 10:53:41 UTC
Created attachment 65004 [details] [review]
Bug 18514 - Followup

Adjust ES paths
Alter fix to retain ID (so records overwrite on re-run)
Commit the records
Comment 8 Nick Clemens 2017-07-12 10:57:12 UTC
With a few minor adjustments things are working and records get indexed.
Comment 9 Chris Cormack 2017-08-10 20:43:14 UTC
Created attachment 65858 [details] [review]
Bug 18514 - Followup

Adjust ES paths
Alter fix to retain ID (so records overwrite on re-run)
Commit the records

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Comment 10 Chris Cormack 2017-08-10 20:47:02 UTC
Created attachment 65859 [details] [review]
Bug 18514 : Script to fetch records from Overdrive

To Test:

You must be running Elasticsearch with Koha
1/ Fill in the Overdrive system preferences
2/ run this script
3/ Records should be in ElasticSearch
Comment 11 Chris Cormack 2017-08-10 20:47:46 UTC
Created attachment 65860 [details] [review]
Bug 18514 : Adding search capabilities for Overdrive results in Koha

To test
1/ Apply the first patch and run the script in it
2/ Search, notice no Overdrive results showing
3/ Search again, now you should get results in the OPAC results page
Comment 12 Chris Cormack 2017-08-10 20:48:25 UTC
Created attachment 65861 [details] [review]
Bug 18514 Fix link on results page, and show overdrive data on detail page

To test
1/ Apply the first 2 patches, and run the fetcher script (in patch one)
2/ Search, notice the overdrive results result in a 404 if you click on
them
3/ Apply this patch
4/ Search again
5/ Click on a result, you should now be at a detail page
Comment 13 Brendan Gallagher 2019-10-04 08:15:59 UTC
If you do get a chance to rebase this.  Please ping me I'd be really happy to test it immediately so we don't have to go through another rebase :) The code does look pretty cool though.