Bug 14277 - Search index 'lex' does not honor MARC indicator "ind1"
Summary: Search index 'lex' does not honor MARC indicator "ind1"
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Barton Chittenden
QA Contact: Testopia
URL:
Keywords:
: 7377 (view as bug list)
Depends on: 14217
Blocks:
  Show dependency treegraph
 
Reported: 2015-05-26 15:07 UTC by Barton Chittenden
Modified: 2017-06-14 22:16 UTC (History)
9 users (show)

See Also:
Change sponsored?: Sponsored
Patch complexity: ---
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 14277 add zebra indexes for lexile that respect 521 indicator 1. (8.28 KB, patch)
2015-12-17 18:59 UTC, Barton Chittenden
Details | Diff | Splinter Review
Test data containing 521 8 $b Lexile $a 530 (1.63 KB, text/plain)
2015-12-28 13:43 UTC, Barton Chittenden
Details
Test data (21.41 KB, text/plain)
2015-12-28 14:44 UTC, Barton Chittenden
Details
Bug 14277 add zebra indexes for lexile that respect 521 indicator 1. (8.32 KB, patch)
2015-12-30 18:55 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 14277: add zebra indexes for lexile that respect 521 indicator 1. (8.40 KB, patch)
2016-03-03 19:50 UTC, Héctor Eduardo Castro Avalos
Details | Diff | Splinter Review
Bug 14277 - (QA followup) (1.09 KB, patch)
2016-04-07 15:35 UTC, Barton Chittenden
Details | Diff | Splinter Review
Bug 14277: add zebra indexes for lexile that respect 521 indicator 1. (8.45 KB, patch)
2016-04-07 16:11 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 14277: (QA followup) Silent GRS-1 tests (1.16 KB, patch)
2016-04-07 16:11 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Barton Chittenden 2015-05-26 15:07:44 UTC
The current definition in 
etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml for the 'lex' index for lexile scores looks like this:

  <!--record.abs line 153: melm 521$a      lex:n-->
  <index_subfields tag="521" subfields="a">
    <target_index>lex:w</target_index>
    <target_index>lex:p</target_index>
    <target_index>lex:n</target_index>
  </index_subfields>

This uses MARC 521 -- "Target Audience Note" as the marc field containing lexile score... however, the contents of the various subfields MARC 521 are dependent on the value of indicator 1, which encodes "audience"

Full details can be found at http://www.loc.gov/marc/bibliographic/bd521.html

Here are the possible values of indicator 1:

0 - Reading grade level
1 - Interest age level
2 - Interest grade level
3 - Special audience characteristics
4 - Motivation/interest level
8 - No display constant generated

Lexile score would be encoded under indicator 8.

Trying to index other indicators as "lex" will create indexes which are inconsistent at best, and which may return incorrect search results.
Comment 1 Barton Chittenden 2015-12-17 18:59:45 UTC Comment hidden (obsolete)
Comment 2 Kyle M Hall 2015-12-28 11:21:52 UTC
Please provide a test plan!
Comment 3 Barton Chittenden 2015-12-28 13:43:08 UTC Comment hidden (obsolete)
Comment 4 Barton Chittenden 2015-12-28 14:44:51 UTC
Created attachment 46005 [details]
Test data

Query for test data:

select
        biblionumber,
        title,
        ExtractValue( marcxml, '//datafield[@tag=521][@ind1=0]/subfield[@code="a"]' ) as 'Reading grade level',
        ExtractValue( marcxml, '//datafield[@tag=521][@ind1=1]/subfield[@code="a"]' ) as 'Interest age level',
        ExtractValue( marcxml, '//datafield[@tag=521][@ind1=2]/subfield[@code="a"]' ) as 'Interest grade level',
        ExtractValue( marcxml, '//datafield[@tag=521][@ind1=8]/subfield[@code="a"]' ) as 'Lexile number'
from
        biblioitems
        inner join biblio using (biblionumber)
where
        biblionumber in ( 86147 , 88090 , 89886 , 90198 , 90669 , 221316 , 221317 , 221318 , 221319 , 221320 )


+--------------+--------------------------+---------------------+--------------------+----------------------+----------------+
| biblionumber | title                    | Reading grade level | Interest age level | Interest grade level | Lexile number  |
+--------------+--------------------------+---------------------+--------------------+----------------------+----------------+
|        86147 | Laura Ingalls Wilder     | 5.0 4. 6.1          | 7-11.              | 5-8 2-6.             | Lexile 780 780 |
|        88090 | Poppleton forever        | 1.                  | 4-7.               | P-3.                 | Lexile 410 410 |
|        89886 | The music of dolphins    | 6.                  | 11-13.             | 5-9.                 | 560            |
|        90198 | Worth                    | 4.5.                | 9-12.              | 4-7.                 | T 830          |
|        90669 | Monday with a mad genius | 3.8.                | 6-9.               | 1-4.                 | 590 590        |
|       221316 | Persian cats /           | 0.5.                | 4-8.               | K-3 0-2.             | 580            |
|       221317 | Maine coon cats /        | 0.5.                | 4-8.               | K-3 0-2.             | 600            |
|       221318 | Calico cats /            | 0.5.                | 4-8.               | K-3 0-2.             | 510            |
|       221319 | Scottish fold cats /     | 0.5.                | 4-8.               | K-3 0-2.             | 550            |
|       221320 | Siamese cats /           | 0.5.                | 4-8.               | K-3 0-2.             | 480            |
+--------------+--------------------------+---------------------+--------------------+----------------------+----------------+
Comment 5 Barton Chittenden 2015-12-28 15:26:38 UTC
== TO TEST ==

Set-up:

Add the following branch codes:  CES, EBE, PSM, SES, TCE, UES.
Add itemtype BOOK

Import data from attachment 46005 [details].

Re-index, search for 'title Laura Ingalls Wilder'. This should return biblionumber 86147.

Add the following to intranetuserjs

    $(document).ready(function(){
        //add lexile to search pull downs
        $("select[name='idx']").append("<option value='lexile-number,st-numeric'>Lexile score (e.g. 600 or 550-650 )</option>");
        $("select[name='idx']").append("<option value='Interest-age-level,wrdl'>Interest Age Level</option>");
        $("select[name='idx']").append("<option value='Interest-grade-level,wrdl'>Interest Grade Level</option>");
        $("select[name='idx']").append("<option value='Reading-grade-level,wrdl'>+Reading Grade Level</option>");
    });

Test plan:

1/ Go to advanced search, select 'Lexile score'. Search for 600 . Search will fail
2/ Go to advanced search, select 'Lexile score'. Search for 500-599 . Search will fail
3/ Go to advanced search, select 'Reading grade level'. Search for 1 . Search will fail
4/ Go to advanced search, select 'Interest age level'. Search for 4-8 . Search will fail
5/ Go to advanced search, select 'Interest grade level'. Search for 1-4 . Search will fail
6/ Apply patch.
7/ Go to advanced search, select 'Lexile score'. Search for 600 . 
   Search will return
   Title: 'Maine coon cats'
8/ Go to advanced search, select 'Lexile score'. Search for 500-599 . 
   Search will return
   Title: 'The music of dolphins', 'Monday with a mad genius', 'Persian cats', 'Calico cats', ' Scottish fold cats'
9/ Go to advanced search, select 'Reading grade level'. Search for 1 . 
   Search will return
   Title: 'Poppleton forever'
10/ Go to advanced search, select 'Interest age level'. Search for 4-8 . 
   Search will return
   Title: 'Maine coon cats', 'Siamese cats', 'Persian cats', 'Calico cats', 'Scottish fold cats'
11/ Go to advanced search, select 'Interest grade level'. Search for 1-4 . 
   Search will return 
   Title: 'Monday with a mad genius'
Comment 6 Nick Clemens 2015-12-30 18:55:19 UTC Comment hidden (obsolete)
Comment 7 Nick Clemens 2015-12-30 18:56:42 UTC
Internal sign-off - still needs community sign-off
Comment 8 Héctor Eduardo Castro Avalos 2016-03-03 19:50:53 UTC Comment hidden (obsolete)
Comment 9 Héctor Eduardo Castro Avalos 2016-03-03 19:51:32 UTC
Hi Nick and Barton
Community sign-off
Comment 10 Tomás Cohen Arazi 2016-03-14 19:01:47 UTC
Barton, can u take a look at the new warnings raised by t/db_dependent/Search.t with your patch?
Comment 11 Jesse Weaver 2016-03-29 20:09:31 UTC
Tomás, this looks to be entirely due to the "lex" definition in GRS-1, which I think is deprecated. We can choose to either ignore the warning or remove the index, IMO.
Comment 12 Tomás Cohen Arazi 2016-03-29 20:41:50 UTC
I think we should just remove the GRS-1 portion, and set it back to signed-off
Comment 13 Barton Chittenden 2016-04-07 15:35:55 UTC Comment hidden (obsolete)
Comment 14 Tomás Cohen Arazi 2016-04-07 16:11:18 UTC
Created attachment 50036 [details] [review]
Bug 14277: add zebra indexes for lexile that respect 521 indicator 1.

Added the following indexes:

Interest-age-level | 591$a ind1=1
Interest-grade-level | 591$a ind1=2
lexile-number | 591$a ind1=8
Reading-grade-level | 591$a ind1=0

Moved 'lex' from a zebra index to a ccl alias to lexile-number.

Changed the handling of st-numeric in C4/Search.pm to allow for search ranges.

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Hector Castro <hector.hecaxmmx@gmail.com>
Works as advertised
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>
Comment 15 Tomás Cohen Arazi 2016-04-07 16:11:27 UTC
Created attachment 50037 [details] [review]
Bug 14277: (QA followup) Silent GRS-1 tests

changed ocurrences of 'lex' to 'lexile-number' in record.abs

Edits were made to the deprecated file record.abs *solely* to quiet
warnings in tests -- this makes sense until GRS-1 code is removed
from Koha.

Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>
Comment 16 Jesse Weaver 2016-04-07 16:35:50 UTC
Pushed, will be in the May 2016 release. Thanks for your work.
Comment 17 Barton Chittenden 2016-11-27 19:14:25 UTC
*** Bug 7377 has been marked as a duplicate of this bug. ***