Bug 9620 - IncludeSeeFromInSearches ON and data in $9 subfields can break indexing
Summary: IncludeSeeFromInSearches ON and data in $9 subfields can break indexing
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: Main
Hardware: All All
: P5 - low major (vote)
Assignee: Katrin Fischer
QA Contact:
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-02-15 15:48 UTC by Katrin Fischer
Modified: 2014-05-26 21:04 UTC (History)
3 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 9620: IncludeSeeFromInSearches and date in $9 subfields can break indexing (2.64 KB, patch)
2013-02-15 17:07 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 9620: IncludeSeeFromInSearches and date in $9 subfields can break indexing (2.36 KB, patch)
2013-02-15 17:09 UTC, Katrin Fischer
Details | Diff | Splinter Review
[SIGNED-OFF] Bug 9620: IncludeSeeFromInSearches and date in $9 subfields can break indexing (3.50 KB, patch)
2013-02-16 00:07 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 9620: IncludeSeeFromInSearches and date in $9 subfields can break indexing (3.56 KB, patch)
2013-03-06 21:15 UTC, Elliott Davis
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Katrin Fischer 2013-02-15 15:48:34 UTC
In Germany the ISBN with hyphens is stored in 020$9.
If you turn on the system preference IncludeSeeFromInSearches this can break zebra indexing process.

There are potentially other fields where this problem can occour. 

Working on a test plan to reproduce the error consistently.
Comment 1 Katrin Fischer 2013-02-15 16:35:46 UTC
The problem is, that only sometimes the ISBN will falsely match an authority record. So the problem is a bit hard to reproduce this way, as I have not found out how to make sure that ISBN and Koha's authority number match. We had this occuring twice for a dataset of about 50.000 records.

So this is a sure way to reproduce the problem:

1) Change your frameworks: 
  * Make sure 084 $a is visible
  * Create a 084 $9 that you link to PERSO_NAME authorities.
2) Create a record or edit an existing one:
  * Use the plugin on 084 to link the record to an authority.
  * Make sure 400 is filled in your authority record.

Preparations all done, now test indexing:

1) Make sure IncludeSeeFromInSearches is OFF
> Reindexing should work without any problems.

2) Activate IncludeSeeFromInSearches
> Reindexing should throw an error and stop:
201.Subfields are only for data fields (generally, just tags >= 010 at /home/koha/Koha/Filter/MARC/EmbedSee FromHEadings.pm line 94

So what happened: The system found the $9, found the matching authority, found the 400, wanted to add the content of 400 to the bibliographic record for indexing, constructed a field that was below >010 to store the data in - and exploded.
Because controlfields have no subfields. (Naturally)
Comment 2 Katrin Fischer 2013-02-15 17:07:57 UTC Comment hidden (obsolete)
Comment 3 Katrin Fischer 2013-02-15 17:09:33 UTC Comment hidden (obsolete)
Comment 4 Bernardo Gonzalez Kriegel 2013-02-16 00:07:35 UTC Comment hidden (obsolete)
Comment 5 Elliott Davis 2013-03-06 21:15:48 UTC
Created attachment 15924 [details] [review]
Bug 9620: IncludeSeeFromInSearches and date in $9 subfields can break indexing

In Germany 020$9 is used to store the ISBN with hyphens. In rare cases this
can break indexing in combination with IncludeSeeFromInSearches.

To reproduce:
1) Change your frameworks:
  * Make sure 084 $a is visible
  * Create a 084 $9 that you link to PERSO_NAME authorities.
2) Create a record or edit an existing one:
  * Use the plugin on 084 to link the record to an authority.
  * Make sure 400 is filled in your authority record.

Preparations all done, now test indexing:

3) Make sure IncludeSeeFromInSearches is OFF
  * Reindexing should work without any problems.

4) Activate IncludeSeeFromInSearches
  * Reindexing should throw an error and stop:
    201.Subfields are only for data fields (generally,
    just tags >= 010 at /home/koha/Koha/Filter/MARC/EmbedSee
    FromHEadings.pm line 94

5) Apply patch and reindex.
  * No error should occur.
  * Test 4xx fields from other tags were properly indexed.

What causes the problem is the fact, that Koha tries to store
the date from the 400 in the bibliographic record for indexing.
For this it calculates a tag in the bibliographic record from the
tag it found the authority link in and the 4xx tags used in the
authority record. If you have an authority linked to a 0xx field
and the matching authority has a 4xx where xx < 10, the calculated
tag will be a control field. The attempt to add subfields to this
control field, is causing the error and breaks the indexing process.

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>

Comment: Work as described, and is a long description :)
Test:
1) Enabled 084 (all subfields visible), added 084_9 (missing from
default framework) linked to PERSO_NAME
2) Edited authority record and added some data to 400
3) Edited bib record, linked 084 to previous auth record
4) With IncludeSeeFromInSearches OFF, re-index without problem
5) With IncludeSeeFromInSearches ON, re-index problems!!!
====================
exporting biblio
====================
1...Subfields are only for data fields (generally, just tags >= 010) at...
Abort
6) Applied patch
7) With IncludeSeeFromInSearches ON, re-index without problem
...
====================
5001....................................................................................................
Records exported: 5060
====================
...
====================
2001....................................................................................................
Records exported: 2024
====================
CLEANING
====================
Good!

*) That's "works as described", don't know if other tests are needed.

No errors from koha-qa

Signed-off-by: Elliott Davis <elliott@bywatersolions.com>
Comment 6 Jared Camins-Esakov 2013-03-07 14:31:40 UTC
This patch has been pushed to master.
Comment 7 Chris Cormack 2013-03-11 09:36:54 UTC
Pushed to 3.10.x will be in 3.10.4