Bug 15048 - Genre/Form (655) searches fail on searches with $x 'General subdivision' subfield values
Summary: Genre/Form (655) searches fail on searches with $x 'General subdivision' subf...
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal (vote)
Assignee: Mason James
QA Contact: Marcel de Rooy
URL:
Keywords:
Depends on: 15047
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-22 03:18 UTC by Mason James
Modified: 2023-12-28 20:42 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
22.11.00, 22.05.08


Attachments
successful search on 655$a - 'speedy reads' (174.66 KB, image/png)
2015-10-22 23:54 UTC, Mason James
Details
FAILED search on 655$a and 655$x - 'Family -- Fiction' (146.22 KB, image/png)
2015-10-22 23:58 UTC, Mason James
Details
Bug 15048 - subject/genre (650/655) searches fail on searches with $x 'General subdivision' subfield values (2.73 KB, patch)
2015-10-27 01:02 UTC, Mason James
Details | Diff | Splinter Review
SUCCESSFUL search on 655$a and 655$x - 'Family -- Fiction' (87.22 KB, image/png)
2015-10-27 01:06 UTC, Mason James
Details
Bug 15048 - subject/genre (650/655) searches fail on searches with $x 'General subdivision' subfield values (2.74 KB, patch)
2016-03-09 16:27 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 15048: Index all possible searched subfields for index-term-genre (3.27 KB, patch)
2022-06-23 11:45 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 15048: Index all possible searched subfields for index-term-genre (3.33 KB, patch)
2022-07-18 19:55 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 15048: Index all possible searched subfields for index-term-genre (3.42 KB, patch)
2022-10-21 06:33 UTC, Marcel de Rooy
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Mason James 2015-10-22 03:18:58 UTC
more info to come...
Comment 1 Mason James 2015-10-22 23:48:00 UTC
to test bug...

1/ define a 655$a value in a bib,  save, reindex

2/ in opac-detail.pl click on its genre/form url, the search should successfully return a result (good!)


3/ next, define both a 655$a *and* 655$x value in a bib, save, reindex

4/ in opac-detail.pl click its genre/form url, the search should FAIL to return a result (BAD!) :/
Comment 2 Mason James 2015-10-22 23:51:20 UTC Comment hidden (obsolete)
Comment 3 Mason James 2015-10-22 23:54:46 UTC
Created attachment 43794 [details]
successful search on 655$a - 'speedy reads'
Comment 4 Mason James 2015-10-22 23:58:24 UTC
Created attachment 43795 [details]
FAILED search on 655$a and 655$x - 'Family -- Fiction'
Comment 5 Mason James 2015-10-23 01:07:11 UTC
my hunch is that the CCL syntax for the search query is incorrect

my example returns no results..
 (su,complete-subfield:"Family") AND (su,complete-subfield:"Fiction")
Comment 6 Mason James 2015-10-23 02:37:43 UTC
(In reply to Mason James from comment #5)
> my hunch is that the CCL syntax for the search query is incorrect
> 
> my example returns no results..
>  (su,complete-subfield:"Family") AND (su,complete-subfield:"Fiction")

the other issue is that the 650$x and/or 655$x subfields don't seem to be configured to be indexed by Zebra in..
 /etc/koha/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml (and others)

..so a query for (su,complete-subfield:"Fiction") would be expected to fail, as that info is located in the unindexed 655$x subfield
Comment 7 Mason James 2015-10-23 04:58:06 UTC
(In reply to Mason James from comment #6)
> (In reply to Mason James from comment #5)
> > my hunch is that the CCL syntax for the search query is incorrect
> > 
> > my example returns no results..
> >  (su,complete-subfield:"Family") AND (su,complete-subfield:"Fiction")
> 
> the other issue is that the 650$x and/or 655$x subfields don't seem to be
> configured to be indexed by Zebra in..
>  /etc/koha/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml (and
> others)
> 
> ..so a query for (su,complete-subfield:"Fiction") would be expected to fail,
> as that info is located in the unindexed 655$x subfield

good news, i have a fix for this...

i'll attach a patch soon
Comment 8 Mason James 2015-10-27 01:02:39 UTC Comment hidden (obsolete)
Comment 9 Mason James 2015-10-27 01:06:59 UTC
Created attachment 44041 [details]
SUCCESSFUL search on 655$a and 655$x - 'Family -- Fiction'
Comment 10 Mason James 2015-10-27 01:08:02 UTC
(In reply to Mason James from comment #9)
> Created attachment 44041 [details]
> SUCCESSFUL search on 655$a and 655$x - 'Family -- Fiction'

pic of successful result attached ^
Comment 11 Mason James 2015-10-27 01:39:33 UTC
(In reply to Mason James from comment #6)
> (In reply to Mason James from comment #5)
> > my hunch is that the CCL syntax for the search query is incorrect
> > 
> > my example returns no results..
> >  (su,complete-subfield:"Family") AND (su,complete-subfield:"Fiction")
> 
> the other issue is that the 650$x and/or 655$x subfields don't seem to be
> configured to be indexed by Zebra 


NOTE: this patch only fixes Genre/Form (655) searches - not subject/650 searches

The patch to fix the subject/650 equivalent of this issue will be moved to a separate bug
Comment 12 Nick Clemens 2015-11-24 20:12:32 UTC
I can't trigger the error, though searches don't seem to include the $x either before or after the patch.  Are there any sysprefs to check?
Comment 13 Mason James 2015-12-12 06:57:39 UTC
(In reply to Nick Clemens from comment #12)
> I can't trigger the error, though searches don't seem to include the $x
> either before or after the patch.  Are there any sysprefs to check?

Nope, no sysprefs to check. The 'error' is a failed search result. 
So, if you get a failed search result, then you have triggered the error

You may/will need to manually add a 655$x value to your test bib. 
If you add one, and can't see it - you have an unrelated problem.
Comment 14 Nick Clemens 2016-03-09 16:27:23 UTC
Created attachment 48887 [details] [review]
Bug 15048 - subject/genre (650/655) searches fail on searches with $x 'General subdivision' subfield values

to test...

1/ define both a 655$a *and* 655$x value in a bib, save, reindex

2/ in opac-detail.pl click bib's genre/form url, the search should FAIL to return a result (BAD!) :/

3/ apply patch

4/ copy updated files to your live zebradb dir (or build/install a debian package)

$ sudo cp ./etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml \
  /etc/koha/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml

$ sudo cp etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl \
  /etc/koha/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl

5/ run a full zebra reindex

6/ in opac-detail.pl click bib's genre/form url, the search should return a result (GOOD!) :0)

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 15 Nick Clemens 2016-03-09 16:27:50 UTC
It was TraceCompleteSubfields I was missing  :-)
Comment 16 Marcel de Rooy 2016-04-15 07:44:14 UTC
QA Comment:
Thanks for your patch, Mason.
I think it still needs some attention:
[1] You add $x when indexing 655. Please note that the whole field 655 is already indexed under term Subject. (So you would only add to Index-term-genre.) But the XSLT just looks at Subject (and only with some specific settings for UseAuthoritiesForTracings and TraceSubjectSubdivisions). Normally you would search for the authority number (an).
So, it does not seem to be necessary to add it.
Please explain why you need it.

[2] If it should be necessary, what about $v $y and $z ?
Comment 17 Nick Clemens 2022-06-23 11:45:10 UTC
Created attachment 136449 [details] [review]
Bug 15048: Index all possible searched subfields for index-term-genre

Currently we only index a - but we can setup the system such that avxyz are searched

To test:
 1 - define both a 655$a *and* 655$x value in a bib, save, reindex
 2 - Set system preferences:
      TraceSubjectSubdivisions: Include
      TraceCompleteSubfields: Force
 3 - View the record edited above in the opac
 4 - Click on the subject heading
 5 - No results found
 6 - Copy zebra files:
  sudo cp ./etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml \
  /etc/koha/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml
  sudo cp etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl \
  /etc/koha/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl
 7 - restart all and reindex
 8 - Click on the subject heading in OPAC
 9 - Sucess!
10 - Repeat with other fields (vyz)
11 - Repeat under ES, reindexing and resetting mappings
Comment 18 Nick Clemens 2022-06-23 11:46:14 UTC
(In reply to Marcel de Rooy from comment #16)
> QA Comment:
> Thanks for your patch, Mason.
> I think it still needs some attention:
> [1] You add $x when indexing 655. Please note that the whole field 655 is
> already indexed under term Subject. (So you would only add to
> Index-term-genre.) But the XSLT just looks at Subject (and only with some
> specific settings for UseAuthoritiesForTracings and
> TraceSubjectSubdivisions). Normally you would search for the authority
> number (an).
> So, it does not seem to be necessary to add it.
> Please explain why you need it.

See updated test plan - with those settings we are specifically searching genre index and fields are needed

> [2] If it should be necessary, what about $v $y and $z ?

Indeed - added
Comment 19 Katrin Fischer 2022-07-16 12:53:23 UTC
Can we please have a rebase?

Apply? [(y)es, (n)o, (i)nteractive] y
Applying: Bug 15048 - subject/genre (650/655) searches fail on searches with $x 'General subdivision' subfield values
Applying: Bug 15048: Index all possible searched subfields for index-term-genre
Using index info to reconstruct a base tree...
M	etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml
M	etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl
Falling back to patching base and 3-way merge...
Auto-merging etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl
CONFLICT (content): Merge conflict in etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl
Auto-merging etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml
CONFLICT (content): Merge conflict in etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml
error: Failed to merge in the changes.
Patch failed at 0001 Bug 15048: Index all possible searched subfields for index-term-genre
hint: Use 'git am --show-current-patch=diff' to see the failed patch
When you have resolved this problem run "git bz apply --continue".
If you would prefer to skip this patch, instead run "git bz apply --skip".
To restore the original branch and stop patching run "git bz apply --abort".
Patch left in /tmp/Bug-15048-Index-all-possible-searched-subfields-fo-UEZYrg.patch
Comment 20 Nick Clemens 2022-07-18 19:21:01 UTC
(In reply to Katrin Fischer from comment #19)
> Can we please have a rebase?
> 

I just forgot to obsolete the older patch, should work now
Comment 21 Katrin Fischer 2022-07-18 19:55:41 UTC
Created attachment 137842 [details] [review]
Bug 15048: Index all possible searched subfields for index-term-genre

Currently we only index a - but we can setup the system such that avxyz are searched

To test:
 1 - define both a 655$a *and* 655$x value in a bib, save, reindex
 2 - Set system preferences:
      TraceSubjectSubdivisions: Include
      TraceCompleteSubfields: Force
 3 - View the record edited above in the opac
 4 - Click on the subject heading
 5 - No results found
 6 - Copy zebra files:
  sudo cp ./etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml \
  /etc/koha/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml
  sudo cp etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl \
  /etc/koha/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl
 7 - restart all and reindex
 8 - Click on the subject heading in OPAC
 9 - Sucess!
10 - Repeat with other fields (vyz)
11 - Repeat under ES, reindexing and resetting mappings

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 22 Katrin Fischer 2022-07-18 19:58:14 UTC
I feel like we can't expect people to constantly reset their Elasticsearch mappings and redo their own. As this fixed a bug, could we update mappings with a database update?
Comment 23 Nick Clemens 2022-07-20 19:38:15 UTC
(In reply to Katrin Fischer from comment #22)
> I feel like we can't expect people to constantly reset their Elasticsearch
> mappings and redo their own. As this fixed a bug, could we update mappings
> with a database update?

I don't think it is that simple. We have created the ability for librarians to manage their own ES configuration - we cannot know what thy have done and whether our fixes are desired

bug 30836 needs to address this as a larger issue, but I don't think this should be held up by that
Comment 24 Katrin Fischer 2022-08-27 13:05:26 UTC
> bug 30836 needs to address this as a larger issue, but I don't think this
> should be held up by that

Agreed. So looking for another set of QA eyes or another sign-off here!
Comment 25 Marcel de Rooy 2022-10-21 06:31:34 UTC
(In reply to Marcel de Rooy from comment #16)
> QA Comment:
> Thanks for your patch, Mason.
> I think it still needs some attention:
> [1] You add $x when indexing 655. Please note that the whole field 655 is
> already indexed under term Subject. (So you would only add to
> Index-term-genre.) But the XSLT just looks at Subject (and only with some
> specific settings for UseAuthoritiesForTracings and
> TraceSubjectSubdivisions). Normally you would search for the authority
> number (an).
> So, it does not seem to be necessary to add it.
> Please explain why you need it.
> 
> [2] If it should be necessary, what about $v $y and $z ?

I am revisiting this report now. And just notice my older comment. Mason did not answer it and Nick picked up this patch 6 years later..
Comment 26 Marcel de Rooy 2022-10-21 06:33:07 UTC
Created attachment 142290 [details] [review]
Bug 15048: Index all possible searched subfields for index-term-genre

Currently we only index a - but we can setup the system such that avxyz are searched

To test:
 1 - define both a 655$a *and* 655$x value in a bib, save, reindex
 2 - Set system preferences:
      TraceSubjectSubdivisions: Include
      TraceCompleteSubfields: Force
 3 - View the record edited above in the opac
 4 - Click on the subject heading
 5 - No results found
 6 - Copy zebra files:
  sudo cp ./etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml \
  /etc/koha/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml
  sudo cp etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl \
  /etc/koha/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl
 7 - restart all and reindex
 8 - Click on the subject heading in OPAC
 9 - Sucess!
10 - Repeat with other fields (vyz)
11 - Repeat under ES, reindexing and resetting mappings

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 27 Tomás Cohen Arazi 2022-10-24 17:41:09 UTC
Pushed to master for 22.11.

Nice work everyone, thanks!
Comment 28 Lucas Gass 2022-12-05 22:43:19 UTC
Backported to 22.05.x for upcoming 22.05.08
Comment 29 wainuiwitikapark 2023-03-15 01:05:01 UTC
Not backported to 21.05.x