Bug 25532 - Add a wikipedia-style search suggestion feature to the OPAC
Summary: Add a wikipedia-style search suggestion feature to the OPAC
Status: RESOLVED MOVED
Alias: None
Product: Koha
Classification: Unclassified
Component: OPAC (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement (vote)
Assignee: David Roberts
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-05-18 15:56 UTC by David Roberts
Modified: 2022-12-06 23:56 UTC (History)
7 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 25532: Add a wikipedia-style search suggestion feature to the OPAC (6.18 KB, patch)
2020-05-20 08:35 UTC, David Roberts
Details | Diff | Splinter Review
Bug 25532: Add a wikipedia-style search suggestion feature to the OPAC (6.43 KB, patch)
2020-05-21 15:23 UTC, David Roberts
Details | Diff | Splinter Review
Bug 25532: Add a wikipedia-style search suggestion feature to the OPAC (6.48 KB, patch)
2020-06-02 13:59 UTC, ByWater Sandboxes
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description David Roberts 2020-05-18 15:56:33 UTC
Some of our (PTFS-E) customer libraries like to have a "Did you mean" feature in the OPAC to help users who mis-spell their search terms. We have developed this using jQuery and Wikipedia, but it would be nice to have this in the core code, behind a system preference for those who don't want it.
Comment 1 David Cook 2020-05-20 02:43:37 UTC
We created a local package for this called Koha::SuggestionEngine::Plugin::CommonMisspellings which our clients use, but I don't think it's designed well enough for master.

I'd be interested in testing out something else though that meets this need.
Comment 2 David Roberts 2020-05-20 08:35:41 UTC
Created attachment 105138 [details] [review]
Bug 25532: Add a wikipedia-style search suggestion feature to the OPAC

This patch adds an auto-complete function to the OPAC search and a new
system preference 'OPACSearchAutoComplete'

To test:

1) Install the patch
2) Go to the OPAC and start typing a search term, e.g. Harry Potter
3) You should see a drop down will appear suggesting likely search terms
which can be selected
4) Repeat the search, but mis-spell your search term, e.g. Hrry Potter.
5) You should still see the correctly spelled search term being offered
for selection
6) Check that the search is unaffected, and still works correctly, both
when logged in and not logged in.
Comment 3 David Roberts 2020-05-20 08:36:53 UTC
(In reply to David Cook from comment #1)
> We created a local package for this called
> Koha::SuggestionEngine::Plugin::CommonMisspellings which our clients use,
> but I don't think it's designed well enough for master.
> 
> I'd be interested in testing out something else though that meets this need.

Thanks David - our customers really like this feature, but obviously it does depend on there being a relevant Wikipedia page in order to be able to correct the spelling, or make a suggestion!
Comment 4 Owen Leonard 2020-05-20 13:37:49 UTC
This is really interesting, and works well. However, I'm concerned about the possibility of confusion on the part of the user.

My assumption is that if a user is familiar with how autocomplete works, they will assume that autocomplete is pulling from a list of existing values. I think it will be confusing for a patron to select from the suggested Wikipedia terms only to find that nothing in the catalog matches.

It's nonsensical to see in the suggestions an entry like "Harry Potter and the Deathly Hallows – Part 1 (disambiguation)."

A partial solution might be to add more explicit labeling to the autocomplete. Something like, "Search terms by Wikipedia" or something like that. I would suggest that the preference be renamed as well to something like "OPACWikipediaSearchSuggestions."
Comment 5 David Roberts 2020-05-20 15:10:33 UTC
(In reply to Owen Leonard from comment #4)
> This is really interesting, and works well. However, I'm concerned about the
> possibility of confusion on the part of the user.
> 
> My assumption is that if a user is familiar with how autocomplete works,
> they will assume that autocomplete is pulling from a list of existing
> values. I think it will be confusing for a patron to select from the
> suggested Wikipedia terms only to find that nothing in the catalog matches.
> 
> It's nonsensical to see in the suggestions an entry like "Harry Potter and
> the Deathly Hallows – Part 1 (disambiguation)."
> 
> A partial solution might be to add more explicit labeling to the
> autocomplete. Something like, "Search terms by Wikipedia" or something like
> that. I would suggest that the preference be renamed as well to something
> like "OPACWikipediaSearchSuggestions."

I'd go along with that - I had trouble thinking what to call this feature when I was writing it! I'll change the naming.
Comment 6 David Cook 2020-05-20 23:32:19 UTC
Ahhh I thought it was a "Did you mean" feature for after you've done a search, but it's a search suggestion before the search is done. Neato. 

I agree with what Owen has said.
Comment 7 David Roberts 2020-05-21 15:23:53 UTC
Created attachment 105220 [details] [review]
Bug 25532: Add a wikipedia-style search suggestion feature to the OPAC

This patch adds an auto-complete function to the OPAC search and a new
system preference 'OPACSearchAutoComplete'

To test:

1) Install the patch
2) Go to the OPAC and start typing a search term, e.g. Harry Potter
3) You should see a drop down will appear suggesting likely search terms
which can be selected
4) Repeat the search, but mis-spell your search term, e.g. Hrry Potter.
5) You should still see the correctly spelled search term being offered
for selection
6) Check that the search is unaffected, and still works correctly, both
when logged in and not logged in.
Comment 8 David Roberts 2020-05-21 15:29:01 UTC
I've implemented the suggestions made so far - you will now see the sys pref has been renamed to OPACWikipediaSearchSuggestions, and you will now see a "Powered by Wikipedia" strap line on the drop down of search suggestions.

I've also just realised that I neglected to suggest in my test plan that you actually switch the feature on! So the test plan is:

1) Install the patch and enable the new OPACWikipediaSearchSuggestions system preference.
2) Go to the OPAC and start typing a search term, e.g. Harry Potter
3) You should see a drop down will appear suggesting likely search terms
which can be selected
4) Repeat the search, but mis-spell your search term, e.g. Hrry Potter.
5) You should still see the correctly spelled search term being offered
for selection
6) Check that the search is unaffected, and still works correctly, both
when logged in and not logged in.
Comment 9 ByWater Sandboxes 2020-06-02 13:59:37 UTC
Created attachment 105488 [details] [review]
Bug 25532: Add a wikipedia-style search suggestion feature to the OPAC

This patch adds an auto-complete function to the OPAC search and a new
system preference 'OPACSearchAutoComplete'

To test:

1) Install the patch
2) Go to the OPAC and start typing a search term, e.g. Harry Potter
3) You should see a drop down will appear suggesting likely search terms
which can be selected
4) Repeat the search, but mis-spell your search term, e.g. Hrry Potter.
5) You should still see the correctly spelled search term being offered
for selection
6) Check that the search is unaffected, and still works correctly, both
when logged in and not logged in.

Signed-off-by: Sally <sally.healey@cheshiresharedservices.gov.uk>
Comment 10 Joonas Kylmälä 2020-07-31 13:34:46 UTC
Instead of depending on Wikipedia being online and transferring users' queries there I think we should use instead one of the search engines Koha supports to do this, Elasticsearch seems to have support for auto completion and correction: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-suggesters.html
Comment 11 Joonas Kylmälä 2020-08-24 06:24:53 UTC
Just a gentle reminder I would like to hear others' (especially people using this feature) thoughts on the Elasticsearch vs Wikipedia approach to this. Also have you thought about having this JavaScript code in OPACUserJS? I tried to put it there and it worked flawlessly. Would it be OK solution if we just documented the steps in wiki for example on how to add this feature to the syspref?

Some potential pros & cons:

Elasticsearch:
+ user queries stay within Koha instance, no privacy leak
+ tailored suggestions for the Koha database
- more difficult to implement

Wikipedia:
+ simple to implement
- suggestion feature availability depends on wikipedia
- search queries go to third party server
+/- suggestions might provide no results / not tailored to the Koha database
Comment 12 Katrin Fischer 2020-10-19 15:28:10 UTC
I am torn here, this works actually really nicely just by putting it into OpacUserJs. And we could document it on https://wiki.koha-community.org/wiki/JQuery_Library but I believe without further "noise" not a lot of libraries will find it there. Adding it to Koha would ensure that and make it really easy to use.

> Some potential pros & cons:
> 
> Elasticsearch:
> + user queries stay within Koha instance, no privacy leak
> + tailored suggestions for the Koha database
> - more difficult to implement

I think it's not an either or - this can work for libraries until we implement something using Elasticsearch. So yes, having Elasticsearch suggestions woudl be great, but I think that's a separate discussion to have.

> Wikipedia:
> + simple to implement
> - suggestion feature availability depends on wikipedia
> - search queries go to third party server
> +/- suggestions might provide no results / not tailored to the Koha database

The third party is a valid argument (thinking of data privavy, GDPR), but could be remedied by a clear hint and link to terms of use or similar in the pref description. We offer other third party services too.
Comment 13 Henry Bolshaw 2020-10-21 12:07:10 UTC
I think it would be good to have this functionality in Koha itself rather than OpacUserJS because this would:

- make the functionality more visible to users and therefore increase the number of people using and benefitting from the functionality
- improve the feature documentation e.g. we can add info to the manual
- increase the potential number of users and developers working with the feature making it more likely that it will iterate and improve (if we add the code to the wiki then it will probably remain there unchanged indefinitely)

We probably wouldn't use this feature in our library currently - the wikipedia search suggestions don't match the titles in our collection very well although I'm sure they do work for some other libraries - so an elasticsearch version based on the koha database would be preferable. However, I agree with Katrin that we don't have to choose one: we can have both this version and an elasticsearch version at a later date.
Comment 14 Joonas Kylmälä 2020-10-21 12:24:20 UTC
Ok, so we all agree having auto-completion would be a great and much wanted feature to have. I still think we should not rush implementing this using the Wikipedia API because it seems like it would not be the ideal solution for many users, if not for all them. Instead the database customized search results using for example Elasticsearch's features seems to be the ideal solution here. Why I think not to rush this, and not to go with Wikipedia suggestions:

 - Double the maintenance work when Wikipedia and Elasticsearch based implementations are both implemented
 - The deprecation of Wikipedia implementation will take probably few releases to make sure everybody has stopped using it and switches to Elasticsearch implementation. Meaning the feature still has to be maintained all this time.

On a side note: I couldn't find any guidelines for deprecating features in Koha. There are already some cases I think doing just that would help the Koha users because for example the local hold groups feature with correct circulation rules could probably replace the system preference OPACHoldsIfAvailableAtPickup.
Comment 15 Katrin Fischer 2021-01-31 16:52:50 UTC
In OpacUserJs:

$(document).ready(function(){
	$("#translControl1").autocomplete({
        source: function(request, response) {
            $.ajax({
                url: "https://en.wikipedia.org/w/api.php",
                dataType: "jsonp",
                data: {
                    'action': "opensearch",
                    'format': "json",
                    'search': request.term
                },
                success: function(data) {
                    response(data[1]);
                }
            });
        },
        open: function() {
            $('.ui-autocomplete').append('<li style="text-align:right;font-size:80%;padding: 5px 10px;background:rgba(0,0,0,0.1)">Powered by Wikipedia</li>');
       }
    });
});
Comment 16 Lucas Gass 2022-12-05 17:15:23 UTC
As of 22.05 the jQuery Katrin provided will no longer work in Koha since we've removed jQueryUI and 'autocomplete'. 

Maybe it is time to revisit this as an ES implementation? Some libraries will miss the autocompete feature provided via jQuery.
Comment 17 Katrin Fischer 2022-12-06 23:56:30 UTC
(In reply to Lucas Gass from comment #16)
> As of 22.05 the jQuery Katrin provided will no longer work in Koha since
> we've removed jQueryUI and 'autocomplete'. 
> 
> Maybe it is time to revisit this as an ES implementation? Some libraries
> will miss the autocompete feature provided via jQuery.

There is already some work on that - I'd love to see comments and get it moving: bug 27113.