Bug 23173

Summary: ILL should be able to search third party sources prior to request creation
Product: Koha Reporter: Andrew Isherwood <bugzilla>
Component: ILLAssignee: Andrew Isherwood <bugzilla>
Status: CLOSED FIXED QA Contact: Kyle M Hall (khall) <kyle>
Severity: enhancement    
Priority: P5 - low CC: 1joynelson, black23, cslone, janet.mcgowan, jeremy.evans, jonathan.field, kyle, magnus, martin.renvoize, tomascohen
Version: Main   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=14973
Change sponsored?: Sponsored Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
This feature adds the required infrastructure to enable ILL availability plugins to intercept the request creation process and, using the supplied metadata, search for and display possible relevant items from whichever availability plugins are installed. **New system preference**: `ILLCheckAvailability` defaults to disabled.
Version(s) released in:
20.05.00
Circulation function:
Attachments: Bug 23173: Add ILLCheckAvailability syspref
Bug 23173: Provide core infrastructure
Bug 23173: Add ILLCheckAvailability syspref
Bug 23173: Provide core infrastructure
Bug 23173: Add ILLCheckAvailability syspref
Bug 23173: Provide core infrastructure
Bug 23173: Add ILLCheckAvailability syspref
Bug 23173: Provide core infrastructure
Bug 23173: Add ILLCheckAvailability syspref
Bug 23173: Provide core infrastructure
Bug 23173: Add ILLCheckAvailability syspref
Bug 23173: Provide core infrastructure
Bug 23173: Add unit tests
Bug 23173: Add ILLCheckAvailability syspref
Bug 23173: Provide core infrastructure
Bug 23173: Add unit tests
Bug 23173: Add ILLCheckAvailability syspref
Bug 23173: Provide core infrastructure
Bug 23173: Add unit tests
Bug 23173: Add ILLCheckAvailability syspref
Bug 23173: Provide core infrastructure
Bug 23173: Add unit tests
Bug 23173: Add ILLCheckAvailability syspref
Bug 23173: Provide core infrastructure
Bug 23173: Add unit tests
Bug 23173: (follow-up) Allow for wide characters
Bug 23173: (follow-up) Allow for customisable display name
Bug 23173: (follow-up) Allow for customisable display name
Bug 23173: (follow-up) Allow for customisable display name
Bug 23173: (follow-up) Display available partners
Bug 23173: Add ILLCheckAvailability syspref
Bug 23173: Provide core infrastructure
Bug 23173: Add unit tests
Bug 23173: (follow-up) Allow for wide characters
Bug 23173: (follow-up) Allow for customisable display name
Bug 23173: (follow-up) Display available partners
Bug 23173: Add ILLCheckAvailability syspref
Bug 23173: Provide core infrastructure
Bug 23173: Add unit tests
Bug 23173: (follow-up) Allow for wide characters
Bug 23173: (follow-up) Allow for customisable display name
Bug 23173: (follow-up) Display available partners
Bug 23173: (follow-up) Small improvements in efficiency
Bug 23173: (follow-up) Fix QA script errors
Bug 23173: (follow-up) Fix QA script errors

Description Andrew Isherwood 2019-06-21 07:55:55 UTC
Currently, there is no integrated way of knowing if an item being requested is already available.

If a user creating a request wants to know if the item they are requesting is already available, it is necessary to search the, possibly, multiple sources the library has available (catalogue, discovery interface, open access provider etc.) first. The problem is further compounded if the institution has the unmediated ILL workflow enabled where a request can be placed with no oversight, even though the institution may already have access to it.

I propose a mechanism by which, during request creation, one or more "sources" are searched with the provided metadata to try and determine if the item being requested is already available and, if so, links to that item are provided. This behaviour will be optional, via a syspref.

"Sources" will be provided as Koha plugins and will provide the ability to accept metadata conforming to the established ILL schema, search the source with which they're concerned, and return results in a prescribed format. This will be performed via an API common to all "source" plugins. All plugins performing this task will be in a common plugin category and therefore enumerable.

The workflow for two backends:

Freeform:
- User fills in Freeform form, clicks submit button
- User is taken to an "Availability" page
- ILL enumerates source plugins that can service this type of request (mostly determined by material type) and makes search requests to each using provided metadata
- Displays results from each source, with links to matching items
- User can then either choose a result and be taken to it, or choose none and continue with the request as normal

BLDSS:
- Identical to above except the user will choose a BLDSS result rather than filling in the Freeform form

As part of this work, a source plugin, which will serve as a reference, will be provided.
Comment 1 Andrew Isherwood 2019-06-21 08:08:27 UTC
Just to clarify, in the above workflows, at the stage:

- User can then either choose a result and be taken to it, or choose none and continue with the request as normal

If no sources return any results, then the user would continue with the request as normal
Comment 2 Andrew Isherwood 2019-07-11 13:24:59 UTC Comment hidden (obsolete)
Comment 3 Andrew Isherwood 2019-07-11 13:25:03 UTC Comment hidden (obsolete)
Comment 4 Andrew Isherwood 2019-08-07 09:11:10 UTC
Test plan:

- Apply bug
- Enable the "ILLCheckAvailability" syspref
- Install the Unpaywall plugin and configure using your email addresss
- Ensure you have a work ILL setup with the FreeForm backend
- Create an article FreeForm request containing a valid DOI (ideally one that you know Unpaywall is aware of)
- Click "Submit" on the FreeForm request form
=> TEST: Observe that ILL now displays the availability of the requested item at Unpaywall
=> TEST: If an item is displayed, observe that clicking the item link takes you directly to the item
- Click the "continue creating your request" button
=> TEST: Observe that the request continues as normal
Comment 5 Andrew Isherwood 2019-08-09 08:58:36 UTC Comment hidden (obsolete)
Comment 6 Andrew Isherwood 2019-08-09 08:58:40 UTC Comment hidden (obsolete)
Comment 7 Andrew Isherwood 2019-08-09 08:59:05 UTC
Rebased on top of latest master and merge conflicts resolved
Comment 8 Andrew Isherwood 2019-09-03 09:40:56 UTC Comment hidden (obsolete)
Comment 9 Andrew Isherwood 2019-09-03 09:40:59 UTC Comment hidden (obsolete)
Comment 10 Andrew Isherwood 2019-09-03 13:55:29 UTC Comment hidden (obsolete)
Comment 11 Andrew Isherwood 2019-09-03 13:55:33 UTC Comment hidden (obsolete)
Comment 12 Andrew Isherwood 2019-09-06 12:39:50 UTC Comment hidden (obsolete)
Comment 13 Andrew Isherwood 2019-09-06 12:39:53 UTC Comment hidden (obsolete)
Comment 14 Andrew Isherwood 2019-09-09 12:33:07 UTC
Updated test plan:

- Apply bug
- Rebuild CSS
- Enable the "ILLCheckAvailability" syspref
- Ensure you have a working ILL setup with the FreeForm backend
- Ensure you have at least one ILL request partner set up. By default, this is patrons in the ILLLIBS category, who have a primary email defined.
- Install the Unpaywall availability plugin (https://github.com/PTFS-Europe/koha-plugin-ill-avail-unpaywall) and configure using your email addresss
- Install the z39.50 availability plugin (https://github.com/PTFS-Europe/koha-plugin-ill-avail-z3950) and configure at least one target. Ensure you specify the ID of at least one of your request partners, this creates a mapping between that target and the request partner

== Main availability searching
- Create an article FreeForm request containing a valid DOI (ideally one that you know Unpaywall is aware of)
- Click "Submit" on the FreeForm request form
=> TEST: Observe that ILL now displays the availability of the requested item at Unpaywall
=> TEST: If an item is displayed, observe that clicking the item link takes you directly to the item
- Click the "continue creating your request" button
=> TEST: Observe that the request continues as normal

== Request from partners availability searching
- On a request's "Manage request" screen, select "Place request with partners"
- On the "Place request with partners" screen, select at least one of the partners with which you associated a Z target in the plugin config
=> TEST: Observe that when a partner is selected, the "Search selected partners" button appears
- Click "Search selected partners"
=> TEST: Observe that the "Search partners" modal appears and upon the search completing, displays matching results for each all partners who are attached to a Z target.
Comment 15 Andrew Isherwood 2019-09-10 15:33:44 UTC
Created attachment 92693 [details] [review]
Bug 23173: Add ILLCheckAvailability syspref

This patch adds the new syspref "ILLCheckAvailability"
Comment 16 Andrew Isherwood 2019-09-10 15:33:48 UTC Comment hidden (obsolete)
Comment 17 Andrew Isherwood 2019-09-10 15:33:51 UTC Comment hidden (obsolete)
Comment 18 Andrew Isherwood 2019-09-12 11:32:38 UTC Comment hidden (obsolete)
Comment 19 Andrew Isherwood 2019-09-12 11:32:42 UTC Comment hidden (obsolete)
Comment 20 Andrew Isherwood 2019-09-12 11:32:45 UTC Comment hidden (obsolete)
Comment 21 Andrew Isherwood 2019-09-25 09:55:05 UTC Comment hidden (obsolete)
Comment 22 Andrew Isherwood 2019-09-25 09:55:08 UTC Comment hidden (obsolete)
Comment 23 Andrew Isherwood 2019-09-25 09:55:11 UTC Comment hidden (obsolete)
Comment 24 Andrew Isherwood 2019-10-09 10:26:17 UTC Comment hidden (obsolete)
Comment 25 Andrew Isherwood 2019-10-09 10:26:20 UTC Comment hidden (obsolete)
Comment 26 Andrew Isherwood 2019-10-09 10:26:23 UTC Comment hidden (obsolete)
Comment 27 Andrew Isherwood 2019-11-11 14:46:47 UTC
Created attachment 95268 [details] [review]
Bug 23173: Add ILLCheckAvailability syspref

This patch adds the new syspref "ILLCheckAvailability"

Sponsored-by: Public Health England
Comment 28 Andrew Isherwood 2019-11-11 14:46:53 UTC
Created attachment 95269 [details] [review]
Bug 23173: Provide core infrastructure

This patch adds the required infrastructure to enable ILL availability
plugins to intercept the request creation process and, using the
supplied metadata, search for and display possible relevant items from
whichever availability plugins are installed.

Currently three availability plugins exist:

z39.50 - Searches any number of the Koha instance's configured Z targets
https://github.com/PTFS-Europe/koha-plugin-ill-avail-z3950

EDS - Searches the EBSCO Discovery Service
https://github.com/PTFS-Europe/koha-plugin-ill-avail-eds

Unpaywall - Searches the Unpaywall API for possible open access versions
of the requested item
https://github.com/PTFS-Europe/koha-plugin-ill-avail-unpaywall

The Unpaywall plugin is intended to serve as a "reference" plugin as the
API it deals with is extremely simple
Comment 29 Andrew Isherwood 2019-11-11 14:46:59 UTC
Created attachment 95270 [details] [review]
Bug 23173: Add unit tests

Test Koha::Illrequests::Availability
Comment 30 Andrew Isherwood 2019-11-11 14:47:35 UTC
Needed a rebase
Comment 31 Andrew Isherwood 2020-01-28 11:54:01 UTC
The test plans says "Create an article FreeForm request containing a valid DOI (ideally one that you know Unpaywall is aware of)", it turns out finding a DOI that Unpaywall is aware of is harder than you think, so:

10.1038/nature12373
Comment 32 Andrew Isherwood 2020-02-06 11:10:47 UTC
Created attachment 98521 [details] [review]
Bug 23173: (follow-up) Allow for wide characters

The conversion of the metadata into Base64 for passing in the API call
to the plugin API routes was failing if the metadata contained a wide
character, so we now encode it into UTF-8 first
Comment 33 Andrew Isherwood 2020-02-17 12:31:17 UTC
Created attachment 99083 [details] [review]
Bug 23173: (follow-up) Allow for customisable display name

We now allow for the service to return a name that can be specified in
the plugin config. This patch switches to using that, rather than just
the plugin name defined in the plugin's metadata hashref
Comment 34 Andrew Isherwood 2020-03-16 09:12:58 UTC
Created attachment 100724 [details] [review]
Bug 23173: (follow-up) Allow for customisable display name

We now allow for the service to return a name that can be specified in
the plugin config. This patch switches to using that, rather than just
the plugin name defined in the plugin's metadata hashref
Comment 35 Andrew Isherwood 2020-03-16 09:15:42 UTC
Created attachment 100725 [details] [review]
Bug 23173: (follow-up) Allow for customisable display name

We now allow for the service to return a name that can be specified in
the plugin config. This patch switches to using that, rather than just
the plugin name defined in the plugin's metadata hashref
Comment 36 Andrew Isherwood 2020-03-16 12:07:36 UTC
Created attachment 100740 [details] [review]
Bug 23173: (follow-up) Display available partners

When selecting partners for search during the "Place request with
partner libraries" workflow, there was no indication of which partners
could actually be searched (i.e. are attached to a Z target and enabled
for partner search). We now list them
Comment 37 Andrew Isherwood 2020-04-01 09:15:53 UTC
Created attachment 102179 [details] [review]
Bug 23173: Add ILLCheckAvailability syspref

This patch adds the new syspref "ILLCheckAvailability"

Sponsored-by: Public Health England
Signed-off-by: Niamh Walker-Headon <Niamh.Walker-Headon@tudublin.ie>
Comment 38 Andrew Isherwood 2020-04-01 09:15:57 UTC
Created attachment 102180 [details] [review]
Bug 23173: Provide core infrastructure

This patch adds the required infrastructure to enable ILL availability
plugins to intercept the request creation process and, using the
supplied metadata, search for and display possible relevant items from
whichever availability plugins are installed.

Currently three availability plugins exist:

z39.50 - Searches any number of the Koha instance's configured Z targets
https://github.com/PTFS-Europe/koha-plugin-ill-avail-z3950

EDS - Searches the EBSCO Discovery Service
https://github.com/PTFS-Europe/koha-plugin-ill-avail-eds

Unpaywall - Searches the Unpaywall API for possible open access versions
of the requested item
https://github.com/PTFS-Europe/koha-plugin-ill-avail-unpaywall

The Unpaywall plugin is intended to serve as a "reference" plugin as the
API it deals with is extremely simple

Signed-off-by: Niamh Walker-Headon <Niamh.Walker-Headon@tudublin.ie>
Comment 39 Andrew Isherwood 2020-04-01 09:16:00 UTC
Created attachment 102181 [details] [review]
Bug 23173: Add unit tests

Test Koha::Illrequests::Availability

Signed-off-by: Niamh Walker-Headon <Niamh.Walker-Headon@tudublin.ie>
Comment 40 Andrew Isherwood 2020-04-01 09:16:04 UTC
Created attachment 102182 [details] [review]
Bug 23173: (follow-up) Allow for wide characters

The conversion of the metadata into Base64 for passing in the API call
to the plugin API routes was failing if the metadata contained a wide
character, so we now encode it into UTF-8 first

Signed-off-by: Niamh Walker-Headon <Niamh.Walker-Headon@tudublin.ie>
Comment 41 Andrew Isherwood 2020-04-01 09:16:07 UTC
Created attachment 102183 [details] [review]
Bug 23173: (follow-up) Allow for customisable display name

We now allow for the service to return a name that can be specified in
the plugin config. This patch switches to using that, rather than just
the plugin name defined in the plugin's metadata hashref

Signed-off-by: Niamh Walker-Headon <Niamh.Walker-Headon@tudublin.ie>
Comment 42 Andrew Isherwood 2020-04-01 09:16:11 UTC
Created attachment 102184 [details] [review]
Bug 23173: (follow-up) Display available partners

When selecting partners for search during the "Place request with
partner libraries" workflow, there was no indication of which partners
could actually be searched (i.e. are attached to a Z target and enabled
for partner search). We now list them

Signed-off-by: Niamh Walker-Headon <Niamh.Walker-Headon@tudublin.ie>
Comment 43 Kyle M Hall (khall) 2020-04-02 11:58:49 UTC
Created attachment 102286 [details] [review]
Bug 23173: Add ILLCheckAvailability syspref

This patch adds the new syspref "ILLCheckAvailability"

Sponsored-by: Public Health England
Signed-off-by: Niamh Walker-Headon <Niamh.Walker-Headon@tudublin.ie>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 44 Kyle M Hall (khall) 2020-04-02 11:59:00 UTC
Created attachment 102287 [details] [review]
Bug 23173: Provide core infrastructure

This patch adds the required infrastructure to enable ILL availability
plugins to intercept the request creation process and, using the
supplied metadata, search for and display possible relevant items from
whichever availability plugins are installed.

Currently three availability plugins exist:

z39.50 - Searches any number of the Koha instance's configured Z targets
https://github.com/PTFS-Europe/koha-plugin-ill-avail-z3950

EDS - Searches the EBSCO Discovery Service
https://github.com/PTFS-Europe/koha-plugin-ill-avail-eds

Unpaywall - Searches the Unpaywall API for possible open access versions
of the requested item
https://github.com/PTFS-Europe/koha-plugin-ill-avail-unpaywall

The Unpaywall plugin is intended to serve as a "reference" plugin as the
API it deals with is extremely simple

Signed-off-by: Niamh Walker-Headon <Niamh.Walker-Headon@tudublin.ie>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 45 Kyle M Hall (khall) 2020-04-02 11:59:03 UTC
Created attachment 102288 [details] [review]
Bug 23173: Add unit tests

Test Koha::Illrequests::Availability

Signed-off-by: Niamh Walker-Headon <Niamh.Walker-Headon@tudublin.ie>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 46 Kyle M Hall (khall) 2020-04-02 11:59:07 UTC
Created attachment 102289 [details] [review]
Bug 23173: (follow-up) Allow for wide characters

The conversion of the metadata into Base64 for passing in the API call
to the plugin API routes was failing if the metadata contained a wide
character, so we now encode it into UTF-8 first

Signed-off-by: Niamh Walker-Headon <Niamh.Walker-Headon@tudublin.ie>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 47 Kyle M Hall (khall) 2020-04-02 11:59:10 UTC
Created attachment 102290 [details] [review]
Bug 23173: (follow-up) Allow for customisable display name

We now allow for the service to return a name that can be specified in
the plugin config. This patch switches to using that, rather than just
the plugin name defined in the plugin's metadata hashref

Signed-off-by: Niamh Walker-Headon <Niamh.Walker-Headon@tudublin.ie>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 48 Kyle M Hall (khall) 2020-04-02 11:59:13 UTC
Created attachment 102291 [details] [review]
Bug 23173: (follow-up) Display available partners

When selecting partners for search during the "Place request with
partner libraries" workflow, there was no indication of which partners
could actually be searched (i.e. are attached to a Z target and enabled
for partner search). We now list them

Signed-off-by: Niamh Walker-Headon <Niamh.Walker-Headon@tudublin.ie>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 49 Kyle M Hall (khall) 2020-04-02 11:59:16 UTC
Created attachment 102292 [details] [review]
Bug 23173: (follow-up) Small improvements in efficiency

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 50 Kyle M Hall (khall) 2020-04-02 11:59:20 UTC
Created attachment 102293 [details] [review]
Bug 23173: (follow-up) Fix QA script errors

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 51 Kyle M Hall (khall) 2020-04-02 12:14:51 UTC
Created attachment 102294 [details] [review]
Bug 23173: (follow-up) Fix QA script errors

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 52 Martin Renvoize (ashimema) 2020-04-06 11:22:36 UTC
Nice work everyone!

Pushed to master for 20.05
Comment 53 Joy Nelson 2020-04-29 20:17:47 UTC
enhancement not backported to 19.11.x