Bug 8726

Summary:	Did you mean? plugin for broader/narrower/related terms
Product:	Koha	Reporter:	Jared Camins-Esakov <jcamins>
Component:	Searching	Assignee:	Jared Camins-Esakov <jcamins>
Status:	CLOSED FIXED	QA Contact:	Jonathan Druart <jonathan.druart>
Severity:	enhancement
Priority:	P5 - low	CC:	jonathan.druart, matted-34813, paul.poulain
Version:	3.10
Hardware:	All
OS:	All
Change sponsored?:	---	Patch complexity:	---
Documentation contact:		Documentation submission:
Text to go in the release notes:		Version(s) released in:
Circulation function:
Bug Depends on:	8209, 8211
Bug Blocks:
Attachments:	Bug 8726: ExplodedTerms suggestion plugin Bug 8726: ExplodedTerms suggestion plugin Bug 8726: ExplodedTerms suggestion plugin SIGNED-OFF-Bug-8726-ExplodedTerms-suggestion-plugin. Bug 8726: ExplodedTerms suggestion plugin overflow Bug 8726: ExplodedTerms suggestion plugin (functionality) Bug 8726: ExplodedTerms suggestion plugin (jQueryUI) Bug 8726 follow-up: fix perlcritic issues Bug 8726 follow-up: fix QA issues Bug 8726 follow-up: fix QA issues Bug 8726: Followup FIX save didyoumean syspref Bug 8726 follow-up: fix QA issues Bug 8726: Followup FIX save didyoumean syspref OPAC screenshot Bug 8726: add CSS to CCSR theme

Description Jared Camins-Esakov 2012-09-04 22:02:54 UTC

Building on the work in bug 8211, we propose adding a "Did you mean?" plugin which will suggest that users review broader, narrower, and related terms when they do subject searches, thus making the functionality usable to the general public.

Comment 1 Jared Camins-Esakov 2012-09-06 13:13:22 UTC Comment hidden (obsolete)

Created attachment 12008 [details] [review]
Bug 8726: ExplodedTerms suggestion plugin

When working with hierarchical subject headings, it is sometimes helpful
to do a search for all records with a specific subject, plus
broader/narrower/related subjects. This patch adds a suggestion plugin for
these "exploded" subject searches to Koha. Note that this patch depends on
both bug 8211 AND bug 8209.

To test (NOTE: this test plan covers both 8211 and 8726):
1) Make sure you have a bunch of hierarchical subjects. I created
   geographical subjects for "Arizona," "United States," and "Phoenix,"
   and linked them together using 551s, and made sure I had a half
   dozen records linking to each (but not all to all three).
2) Do a search for su-br:Arizona (or choose "Subject and broader terms"
   on the advanced search screen with "more options" displayed), and
   check that you get the records with the subject "Arizona" and the
   records with the subject "United States"
3) Do a search for su-na:Arizona (or choose "Subject and narrower terms"
   on the advanced search screen with "more options" displayed), and
   check that you get the records with the subject "Arizona" and the
   records with the subject "Phoenix"
4) Do a search for su-rl:Arizona (or choose "Subject and related terms"
   on the advanced search screen with "more options" displayed), and
   check that you get the records with the subject "Arizona," the
   records with the subject "United States," and the records with the
   subject "Phoenix"
5) Ensure that other searches still work (keyword, subject, ccl,
   whatever)
6) Use "Did you mean?" page in admin section to enable ExplodedTerms
   plugin
7) Do a keyword search on the OPAC, confirm that searching for exploded
   terms is suggested.
8) Do a subject search on the OPAC, confirm that searching for exploded
   terms is suggested.
9) Do a non-keyword, non-subject search on the OPAC, confirm that
   searching for exploded terms is NOT suggested.
10) Disable ExplodedTerms plugin and enable AuthorityFile plugin.
11) Do search on OPAC, confirm suggestions are made from authority file.
12) Sign off

Comment 2 Jared Camins-Esakov 2012-09-09 21:26:21 UTC Comment hidden (obsolete)

Created attachment 12091 [details] [review]
Bug 8726: ExplodedTerms suggestion plugin

When working with hierarchical subject headings, it is sometimes helpful
to do a search for all records with a specific subject, plus
broader/narrower/related subjects. This patch adds a suggestion plugin for
these "exploded" subject searches to Koha. Note that this patch depends on
both bug 8211 AND bug 8209.

To test (NOTE: this test plan covers both 8211 and 8726):
1) Make sure you have a bunch of hierarchical subjects. I created
   geographical subjects for "Arizona," "United States," and "Phoenix,"
   and linked them together using 551s, and made sure I had a half
   dozen records linking to each (but not all to all three).
2) Do a search for su-br:Arizona (or choose "Subject and broader terms"
   on the advanced search screen with "more options" displayed), and
   check that you get the records with the subject "Arizona" and the
   records with the subject "United States"
3) Do a search for su-na:Arizona (or choose "Subject and narrower terms"
   on the advanced search screen with "more options" displayed), and
   check that you get the records with the subject "Arizona" and the
   records with the subject "Phoenix"
4) Do a search for su-rl:Arizona (or choose "Subject and related terms"
   on the advanced search screen with "more options" displayed), and
   check that you get the records with the subject "Arizona," the
   records with the subject "United States," and the records with the
   subject "Phoenix"
5) Ensure that other searches still work (keyword, subject, ccl,
   whatever)
6) Use "Did you mean?" page in admin section to enable ExplodedTerms
   plugin
7) Do a keyword search on the OPAC, confirm that searching for exploded
   terms is suggested.
8) Do a subject search on the OPAC, confirm that searching for exploded
   terms is suggested.
9) Do a non-keyword, non-subject search on the OPAC, confirm that
   searching for exploded terms is NOT suggested.
10) Disable ExplodedTerms plugin and enable AuthorityFile plugin.
11) Do search on OPAC, confirm suggestions are made from authority file.
12) Sign off

Comment 3 Jared Camins-Esakov 2012-09-10 07:55:21 UTC Comment hidden (obsolete)

Created attachment 12099 [details] [review]
Bug 8726: ExplodedTerms suggestion plugin

When working with hierarchical subject headings, it is sometimes helpful
to do a search for all records with a specific subject, plus
broader/narrower/related subjects. This patch adds a suggestion plugin for
these "exploded" subject searches to Koha. Note that this patch depends on
both bug 8211 AND bug 8209.

To test (NOTE: this test plan covers both 8211 and 8726):
1) Make sure you have a bunch of hierarchical subjects. I created
   geographical subjects for "Arizona," "United States," and "Phoenix,"
   and linked them together using 551s, and made sure I had a half
   dozen records linking to each (but not all to all three).
2) Do a search for su-br:Arizona (or choose "Subject and broader terms"
   on the advanced search screen with "more options" displayed), and
   check that you get the records with the subject "Arizona" and the
   records with the subject "United States"
3) Do a search for su-na:Arizona (or choose "Subject and narrower terms"
   on the advanced search screen with "more options" displayed), and
   check that you get the records with the subject "Arizona" and the
   records with the subject "Phoenix"
4) Do a search for su-rl:Arizona (or choose "Subject and related terms"
   on the advanced search screen with "more options" displayed), and
   check that you get the records with the subject "Arizona," the
   records with the subject "United States," and the records with the
   subject "Phoenix"
5) Ensure that other searches still work (keyword, subject, ccl,
   whatever)
6) Use "Did you mean?" page in admin section to enable ExplodedTerms
   plugin
7) Do a keyword search on the OPAC, confirm that searching for exploded
   terms is suggested.
8) Do a subject search on the OPAC, confirm that searching for exploded
   terms is suggested.
9) Do a non-keyword, non-subject search on the OPAC, confirm that
   searching for exploded terms is NOT suggested.
10) Disable ExplodedTerms plugin and enable AuthorityFile plugin.
11) Do search on OPAC, confirm suggestions are made from authority file.
12) Sign off

Comment 4 wajasu 2012-09-10 18:10:05 UTC Comment hidden (obsolete)

Created attachment 12117 [details] [review]
SIGNED-OFF-Bug-8726-ExplodedTerms-suggestion-plugin.

I applied:
0001-Bug-8744-Thesaurus-in-authorities-should-not-lock-fi.patch
0002-Bug-8649-Add-unit-test-for-C4-Search.patch
0003-bug-8649-make-sure-C4-Context-is-fully-mocked-up-bef.patch
0004-bug-8649-make-sure-we-can-exit-if-a-test-fails.patch
0005-bug-8649-create-a-temp-directory-for-Zebra-for-C4-Se.patch
0006-bug-8649-quiet-a-variable-use-warning.patch
0007-Bug-8211-Add-exploded-search-options.patch
0008-SIGNED-OFF-Bug-8209-Did-you-mean-from-authorities.patch
0009-Bug-8726-ExplodedTerms-suggestion-plugin.patch

As mentioned, the work done in 8211 testing is duplicated here,
so I leveraged the authority linked geographical names i had in my db.

I applied the updatedatabase.pl
I configured the syspref  DidYouMean... to Suggest.  AND
on the main Administration web page, in the bottom right corner, there
is a new link "Did you mean?" to configure the suggestion plugins,
of which I checked (Exploded Terms, Auhtority file, Both) and did my searches.

Suggestion links showed up in the results page, to offer alternate searches,
(su-br, su-na, su-rl  accordingly) and I got the same results as 8211 provided.
The authority file link would show up using my search term as a link, and I had to pull down some biblios for "Smith, John" and use the authority search plugin to link the authority to the biblio item.

wajasu


Note: You can change the order of the suggestion plugins by using drag n drop of the control in the DidyYouMean configuration page.

Comment 5 Jonathan Druart 2012-09-13 09:36:42 UTC

Hi Jared,

The patch does not apply anymore, please could you provide a new patch ?

Comment 6 Jared Camins-Esakov 2012-09-13 11:30:54 UTC Comment hidden (obsolete)

Created attachment 12186 [details] [review]
Bug 8726: ExplodedTerms suggestion plugin

When working with hierarchical subject headings, it is sometimes helpful
to do a search for all records with a specific subject, plus
broader/narrower/related subjects. This patch adds a suggestion plugin for
these "exploded" subject searches to Koha. Note that this patch depends on
both bug 8211 AND bug 8209.

To test (NOTE: this test plan covers both 8211 and 8726):
1) Make sure you have a bunch of hierarchical subjects. I created
   geographical subjects for "Arizona," "United States," and "Phoenix,"
   and linked them together using 551s, and made sure I had a half
   dozen records linking to each (but not all to all three).
2) Do a search for su-br:Arizona (or choose "Subject and broader terms"
   on the advanced search screen with "more options" displayed), and
   check that you get the records with the subject "Arizona" and the
   records with the subject "United States"
3) Do a search for su-na:Arizona (or choose "Subject and narrower terms"
   on the advanced search screen with "more options" displayed), and
   check that you get the records with the subject "Arizona" and the
   records with the subject "Phoenix"
4) Do a search for su-rl:Arizona (or choose "Subject and related terms"
   on the advanced search screen with "more options" displayed), and
   check that you get the records with the subject "Arizona," the
   records with the subject "United States," and the records with the
   subject "Phoenix"
5) Ensure that other searches still work (keyword, subject, ccl,
   whatever)
6) Use "Did you mean?" page in admin section to enable ExplodedTerms
   plugin
7) Do a keyword search on the OPAC, confirm that searching for exploded
   terms is suggested.
8) Do a subject search on the OPAC, confirm that searching for exploded
   terms is suggested.
9) Do a non-keyword, non-subject search on the OPAC, confirm that
   searching for exploded terms is NOT suggested.
10) Disable ExplodedTerms plugin and enable AuthorityFile plugin.
11) Do search on OPAC, confirm suggestions are made from authority file.
12) Sign off

Signed-off-by: wajasu <matted-34813@mypacks.net>

Note that you will need to shift-refresh in order to get the new Javascript
after the latest update to jQueryUI.

Comment 7 Jonathan Druart 2012-09-14 10:34:55 UTC

Created attachment 12214 [details]
overflow

I get an overflow issue on wide screen (24"), see screenshot.

Moreover when I click on the Intranet AuthorityFile *label*, the checkbox of Opac AuthorityFile become checked or unchecked :) (same for ExplodedTerms)

Comment 8 Jonathan Druart 2012-09-14 12:50:43 UTC

QA comments:

* 3 perlcritics errors on Koha/SuggestionEngine/Base.pm:
  Expression form of "eval" at line 137, column 12.  See page 161 of PBP.  (Severity: 5)
  Expression form of "eval" at line 151, column 12.  See page 161 of PBP.  (Severity: 5)
  Expression form of "eval" at line 165, column 12.  See page 161 of PBP.  (Severity: 5)
But I don't know how to fix them!

* The 2 sysprefs don't appear in a yaml file. I understand why (there is a specific config page) but they appear in the "Local Use" tab :-/
It could be considered as an issue.

* It would be great to provide 2 patchs. 1 for the feature and 1 for the jQuery changes. Like that if a problem appears on it, it will be easy to revert the commit.

I don't find anything else :)
It works great !

I Failed QA for the overflow issue (see previous comment). After that I will be happy to mark it as passed qa.

Comment 9 Jared Camins-Esakov 2012-09-14 13:03:01 UTC

(In reply to comment #8)
> QA comments:
> 
> * 3 perlcritics errors on Koha/SuggestionEngine/Base.pm:
>   Expression form of "eval" at line 137, column 12.  See page 161 of PBP. 
> (Severity: 5)
>   Expression form of "eval" at line 151, column 12.  See page 161 of PBP. 
> (Severity: 5)
>   Expression form of "eval" at line 165, column 12.  See page 161 of PBP. 
> (Severity: 5)
> But I don't know how to fix them!

I'll see if I can figure something out.

> * The 2 sysprefs don't appear in a yaml file. I understand why (there is a
> specific config page) but they appear in the "Local Use" tab :-/
> It could be considered as an issue.

I think this is a feature, because that way it is possible to work around not having javascript. My warning makes it very clear that setting those sysprefs directly is a bad idea, though.

> * It would be great to provide 2 patchs. 1 for the feature and 1 for the
> jQuery changes. Like that if a problem appears on it, it will be easy to
> revert the commit.

I will break the existing patch into two, and do a follow-up for the overflow.

> I don't find anything else :)
> It works great !
> 
> I Failed QA for the overflow issue (see previous comment). After that I will
> be happy to mark it as passed qa.

Comment 10 Jared Camins-Esakov 2012-09-20 11:43:38 UTC

Created attachment 12379 [details] [review]
Bug 8726: ExplodedTerms suggestion plugin (functionality)

When working with hierarchical subject headings, it is sometimes helpful
to do a search for all records with a specific subject, plus
broader/narrower/related subjects. This patch adds a suggestion plugin for
these "exploded" subject searches to Koha. Note that this patch depends on
both bug 8211 AND bug 8209.

To test (NOTE: this test plan covers both 8211 and 8726):
1) Make sure you have a bunch of hierarchical subjects. I created
   geographical subjects for "Arizona," "United States," and "Phoenix,"
   and linked them together using 551s, and made sure I had a half
   dozen records linking to each (but not all to all three).
2) Do a search for su-br:Arizona (or choose "Subject and broader terms"
   on the advanced search screen with "more options" displayed), and
   check that you get the records with the subject "Arizona" and the
   records with the subject "United States"
3) Do a search for su-na:Arizona (or choose "Subject and narrower terms"
   on the advanced search screen with "more options" displayed), and
   check that you get the records with the subject "Arizona" and the
   records with the subject "Phoenix"
4) Do a search for su-rl:Arizona (or choose "Subject and related terms"
   on the advanced search screen with "more options" displayed), and
   check that you get the records with the subject "Arizona," the
   records with the subject "United States," and the records with the
   subject "Phoenix"
5) Ensure that other searches still work (keyword, subject, ccl,
   whatever)
6) Use "Did you mean?" page in admin section to enable ExplodedTerms
   plugin
7) Do a keyword search on the OPAC, confirm that searching for exploded
   terms is suggested.
8) Do a subject search on the OPAC, confirm that searching for exploded
   terms is suggested.
9) Do a non-keyword, non-subject search on the OPAC, confirm that
   searching for exploded terms is NOT suggested.
10) Disable ExplodedTerms plugin and enable AuthorityFile plugin.
11) Do search on OPAC, confirm suggestions are made from authority file.
12) Sign off

Signed-off-by: wajasu <matted-34813@mypacks.net>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Split into two patches. This patch includes only the functionality.

Comment 11 Jared Camins-Esakov 2012-09-20 11:43:51 UTC

Created attachment 12380 [details] [review]
Bug 8726: ExplodedTerms suggestion plugin (jQueryUI)

When working with hierarchical subject headings, it is sometimes helpful
to do a search for all records with a specific subject, plus
broader/narrower/related subjects. This patch adds a suggestion plugin for
these "exploded" subject searches to Koha. Note that this patch depends on
both bug 8211 AND bug 8209.

To test (NOTE: this test plan covers both 8211 and 8726):
1) Make sure you have a bunch of hierarchical subjects. I created
   geographical subjects for "Arizona," "United States," and "Phoenix,"
   and linked them together using 551s, and made sure I had a half
   dozen records linking to each (but not all to all three).
2) Do a search for su-br:Arizona (or choose "Subject and broader terms"
   on the advanced search screen with "more options" displayed), and
   check that you get the records with the subject "Arizona" and the
   records with the subject "United States"
3) Do a search for su-na:Arizona (or choose "Subject and narrower terms"
   on the advanced search screen with "more options" displayed), and
   check that you get the records with the subject "Arizona" and the
   records with the subject "Phoenix"
4) Do a search for su-rl:Arizona (or choose "Subject and related terms"
   on the advanced search screen with "more options" displayed), and
   check that you get the records with the subject "Arizona," the
   records with the subject "United States," and the records with the
   subject "Phoenix"
5) Ensure that other searches still work (keyword, subject, ccl,
   whatever)
6) Use "Did you mean?" page in admin section to enable ExplodedTerms
   plugin
7) Do a keyword search on the OPAC, confirm that searching for exploded
   terms is suggested.
8) Do a subject search on the OPAC, confirm that searching for exploded
   terms is suggested.
9) Do a non-keyword, non-subject search on the OPAC, confirm that
   searching for exploded terms is NOT suggested.
10) Disable ExplodedTerms plugin and enable AuthorityFile plugin.
11) Do search on OPAC, confirm suggestions are made from authority file.
12) Sign off

Signed-off-by: wajasu <matted-34813@mypacks.net>

Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Split into two patches. This patch includes only the jQueryUI upgrade

Comment 12 Jared Camins-Esakov 2012-09-20 11:44:06 UTC Comment hidden (obsolete)

Created attachment 12381 [details] [review]
Bug 8726 follow-up: fix perlcritic issues

Comment 13 Jared Camins-Esakov 2012-09-20 11:50:56 UTC Comment hidden (obsolete)

Created attachment 12382 [details] [review]
Bug 8726 follow-up: fix QA issues

Fixes several perlcritic complaints, and some overflow in the
configuration page. Also notes that Did You Mean is not yet
enabled on the staff client.

Comment 14 Jared Camins-Esakov 2012-09-20 11:53:43 UTC Comment hidden (obsolete)

Created attachment 12383 [details] [review]
Bug 8726 follow-up: fix QA issues

Fixes several perlcritic complaints, and some overflow in the
configuration page. Also notes that Did You Mean is not yet
enabled on the staff client.

Comment 15 Jared Camins-Esakov 2012-09-20 11:54:34 UTC

Apologies for the long delay (and the number of patches I just obsoleted). Fixing the perlcritic warnings was much harder than anticipated.

Comment 16 Jonathan Druart 2012-09-20 12:21:49 UTC Comment hidden (obsolete)

Created attachment 12384 [details] [review]
Bug 8726: Followup FIX save didyoumean syspref

Comment 17 Jonathan Druart 2012-09-20 12:32:29 UTC

Unit tests failed now :-/
Marked as Failed QA.

Can't use string ("AuthorityFile") as an ARRAY ref while "strict refs" in use at /home/koha/Koha/SuggestionEngine.pm line 91.

Can't use string ("ExplodedTerms") as an ARRAY ref while "strict refs" in use at /home/koha/Koha/SuggestionEngine.pm line 91.

Can't use string ("ABCD::EFGH::IJKL") as an ARRAY ref while "strict refs" in use at /home/koha/Koha/SuggestionEngine.pm line 91.

Comment 18 Jared Camins-Esakov 2012-09-20 13:45:35 UTC

Created attachment 12390 [details] [review]
Bug 8726 follow-up: fix QA issues

Fixes several perlcritic complaints, and some overflow in the
configuration page. Also notes that Did You Mean is not yet
enabled on the staff client.

Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Sorry about the problematic last attempt at uploading. I missed a
patch when squashing. It is now included.

Comment 19 Jared Camins-Esakov 2012-09-20 13:45:51 UTC

Created attachment 12391 [details] [review]
Bug 8726: Followup FIX save didyoumean syspref

Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>

Comment 20 Jonathan Druart 2012-09-20 14:42:19 UTC

Now all seem good to me. Marked as Passed QA.

Comment 21 Paul Poulain 2012-09-24 17:25:00 UTC

Created attachment 12482 [details]
OPAC screenshot

Question: I'm testing this feature, and see the following results on my screen.
Is it what I should expect ?

( Note i'm using the cssr theme )

QA comment: you remove the DidYouMeanFromAuthorities from the sysprefs.sql, shouldn't we remove it during updatedatabase also ?

Comment 22 Jared Camins-Esakov 2012-09-24 17:28:12 UTC

Comment on attachment 12482 [details]
OPAC screenshot

It looks like the CSS is missing for the opac.css in the ccsr theme. I will do a follow-up. Other than that, yes, that is correct.

The DidYouMeanFromAuthorities should probably be taken out of updatedatabase.pl as well. Shall I let you do that?

Comment 23 Jared Camins-Esakov 2012-09-24 17:32:04 UTC

Created attachment 12483 [details] [review]
Bug 8726: add CSS to CCSR theme

Comment 24 Paul Poulain 2012-09-28 15:23:07 UTC

Patch pushed to master

Comment 25 Jared Camins-Esakov 2012-12-03 13:22:00 UTC

This feature is available in Master.