Bug 26578

Summary: OverDrive results can return false positives when searches contain CCL syntax
Product: Koha Reporter: Nick Clemens <nick>
Component: OPACAssignee: Nick Clemens <nick>
Status: Pushed to stable --- QA Contact: Marcel de Rooy <m.de.rooy>
Severity: normal    
Priority: P5 - low CC: andrew, caroline.cyr-la-rose, fridolin.somers, m.de.rooy
Version: master   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: Small patch
Text to go in the release notes:
Version(s) released in:
21.05.00,20.11.03
Attachments: Bug 26578: Clean overdrive searches before passing to OD
Bug 26578: Clean overdrive searches before passing to OD
Bug 26578: Clean overdrive searches before passing to OD
Bug 26578: Clean overdrive searches before passing to OD
Bug 26578: Clean overdrive searches before passing to OD

Description Nick Clemens 2020-09-30 12:57:31 UTC
This problem is from the ':' and '=' signs, along with field codes, that can be passed to overdrive when forming queries or clicking links. Depending on the overdrive catalog, these can match records causing inaccurate results

To recreate:
1 - Have OverDrive enabled and configured
2 - set UseAuthoritiesForTracings to "Don't use"
3 - Search in the catalog for a record
4 - From the details page for a record click on an author link
5 - Note the query in catalog is like 'au:&quot;Whitford, Bradley.&quot;'
6 - Some overdrive catalogs will return records matching 'au:'
Comment 1 Nick Clemens 2020-09-30 13:03:34 UTC Comment hidden (obsolete)
Comment 2 Nick Clemens 2020-11-23 13:21:16 UTC Comment hidden (obsolete)
Comment 3 Nick Clemens 2020-11-30 14:28:33 UTC
Created attachment 114058 [details] [review]
Bug 26578: Clean overdrive searches before passing to OD

This patch copies regex used in buildQuery to detect ccl code in queries, but here we adapt
it to remove any ccl signifiers to ensure correct results from OD

To test:
1 - Have OverDrive enabled and configured
2 - set UseAuthoritiesForTracings to "Don't use"
3 - Search in the catalog for a record
4 - From the details page for a record click on an author link
5 - Note the query in catalog is like 'au:&quot;Whitford, Bradley.&quot;'
6 - Some overdrive catalogs will return records matching 'au:'
7 - Apply patch
8 - Reload the page, if there were spurious results before, they should now be gone
9 - Perform a search that returns overdrive results
10 - For a title in overdrive search koha with:
    au:{Author} AND ti:{Title}
11 - Confirm overdrive results are returned
12 - Check the link to overdrive results and note query section is:
    q=Author AND Title


Current status: Needs Signoff
Comment 4 Kelly McElligott 2020-12-02 14:38:03 UTC
Created attachment 114098 [details] [review]
Bug 26578: Clean overdrive searches before passing to OD

This patch copies regex used in buildQuery to detect ccl code in queries, but here we adapt
it to remove any ccl signifiers to ensure correct results from OD

To test:
1 - Have OverDrive enabled and configured
2 - set UseAuthoritiesForTracings to "Don't use"
3 - Search in the catalog for a record
4 - From the details page for a record click on an author link
5 - Note the query in catalog is like 'au:&quot;Whitford, Bradley.&quot;'
6 - Some overdrive catalogs will return records matching 'au:'
7 - Apply patch
8 - Reload the page, if there were spurious results before, they should now be gone
9 - Perform a search that returns overdrive results
10 - For a title in overdrive search koha with:
    au:{Author} AND ti:{Title}
11 - Confirm overdrive results are returned
12 - Check the link to overdrive results and note query section is:
    q=Author AND Title

Signed-off-by: Kelly McElligott <kelly@bywatersolutions.com>
Comment 5 Marcel de Rooy 2021-01-29 10:21:32 UTC
Created attachment 116014 [details] [review]
Bug 26578: Clean overdrive searches before passing to OD

This patch copies regex used in buildQuery to detect ccl code in queries, but here we adapt
it to remove any ccl signifiers to ensure correct results from OD

To test:
1 - Have OverDrive enabled and configured
2 - set UseAuthoritiesForTracings to "Don't use"
3 - Search in the catalog for a record
4 - From the details page for a record click on an author link
5 - Note the query in catalog is like 'au:&quot;Whitford, Bradley.&quot;'
6 - Some overdrive catalogs will return records matching 'au:'
7 - Apply patch
8 - Reload the page, if there were spurious results before, they should now be gone
9 - Perform a search that returns overdrive results
10 - For a title in overdrive search koha with:
    au:{Author} AND ti:{Title}
11 - Confirm overdrive results are returned
12 - Check the link to overdrive results and note query section is:
    q=Author AND Title

Signed-off-by: Kelly McElligott <kelly@bywatersolutions.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 6 Marcel de Rooy 2021-01-29 10:23:50 UTC
Looks good to me.
Cannot actually test OverDrive.
Decoding html entities with DomParser is a recommended approach.
This change will only affect people with OD enabled. No harm expected.
Comment 7 Jonathan Druart 2021-02-01 16:26:28 UTC
Pushed to master for 21.05, thanks to everybody involved!
Comment 8 Fridolin Somers 2021-02-05 15:46:11 UTC
Pushed to 20.11.x for 20.11.03
Comment 9 Andrew Fuerste-Henry 2021-02-09 12:56:25 UTC
Doesn't apply cleanly to 20.05, please rebase if needed