Bug 6789 - Biblios with many items can result in broken search results links
Summary: Biblios with many items can result in broken search results links
Alias: None
Product: Koha
Classification: Unclassified
Component: Z39.50 / SRU / OpenSearch Servers (show other bugs)
Version: 3.6
Hardware: All All
: PATCH-Sent (DO NOT USE) minor (vote)
Assignee: Ian Walls
QA Contact: Bugs List
Depends on:
Blocks: 2453
  Show dependency treegraph
Reported: 2011-08-25 23:31 UTC by Ian Walls
Modified: 2013-12-05 19:53 UTC (History)
3 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Text to go in the release notes:
Version(s) released in:

Proposed Patch (4.72 KB, patch)
2011-09-30 14:34 UTC, Ian Walls
Details | Diff | Splinter Review
Signed-off patch (4.79 KB, patch)
2011-10-03 15:09 UTC, D Ruth Holloway
Details | Diff | Splinter Review
Bug 6789: biblios with many items can result in broken search results link (4.83 KB, patch)
2011-10-14 09:59 UTC, Paul Poulain
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Ian Walls 2011-08-25 23:31:28 UTC
If a title has enough item record information to push it's MARCXML record size past 1MB, any information beyond that will be ignored.  The record will be indexed, but the 999$c, which contains the biblionumber and is by its numerical nature the last field in the record, could be lost, and thus the link in the search results will be broken (no biblionumber filled in, so 404 error)

The solution, ideally, would be to fix Net::Z3950::ZOOM to allow for the specification of a larger maximum record size, but that's impractical at this time.  Koha can be altered to put all the 952 information AFTER the 999, practically guaranteeing that the biblionumber will be in the first 1MB worth of the MARCXML.  If a few items records are lost to the indexer, that's unfortunate, but bearable.
Comment 1 Ian Walls 2011-09-30 14:34:38 UTC Comment hidden (obsolete)
Comment 2 Ian Walls 2011-09-30 14:46:06 UTC
Example record (cannot attach due to current attachment size limits): http://bywatersolutions.com/wp-content/uploads/2011/09/bug6789-test.xml_.txt

Had to rename the extension to make the website happy
Comment 3 D Ruth Holloway 2011-10-03 15:09:27 UTC Comment hidden (obsolete)
Comment 4 D Ruth Holloway 2011-10-03 15:10:30 UTC
Ian's solution here works like a charm!  Tested with a real-world known-broken instance of this, and it's now working.
Comment 5 Paul Poulain 2011-10-14 09:59:11 UTC
Created attachment 5886 [details] [review]
Bug 6789: biblios with many items can result in broken search results link

This patch fixes an issue whereby biblios with many items (often > 500) would index,
but not the biblionumber itself, resulting in search results with a) inaccurate item counts
and b) no biblionumber to use in the link to the details page.  This is due to Net::Z3950::ZOOM  not providing
a mechanism for specifying different connection attributes; the maximumRecordSize ZOOM connection attribute,
if not specified, defaults to 1MB, which is less than the size of a MARC record with many, many 952 fields.  Since
it is unlikely we can fix Net::Z3950::ZOOM in a timely fashion, this patch aims to build a workaround on the Koha end.

This patch changes EmbedItemsInMarcBiblio to use append_fields instead of insert_ordered_fields,
so the 999$c will come before the item records.  It's VERY unlikely we will encounter more than 1MB of biblio-level MARC
content, as this would break the ISO-2709 standard by a large factor.

To this end, it also moves the fix_biblio_ids portion of get_corrected_marc_record out of rebuild_zebra.pl,
and makes it a part of GetMarcBiblio (right before EmbedItemsInMarcBiblio, so the 952s still come last).  fix_biblio_ids
is kept as a subroutine for the deletion portion of rebuild_zebra.pl, which still uses it.

It also uses the subroutine parameter in GetMarcBiblio to do the EmbedItemsInMarcBiblio action, rather than having
rebuild_zebra.pl perform it on the itemless record returned from GetMarcBiblio.  Simpler and cleaner that way.

To verify bug issue:
1. Find a biblio with over 700 items (or enough that the resulting MARCXML is greater than 1MB)
2. search for this biblio (in a search that would return multiple results, not just this title).  You should get the title in
the results list
3. attempt to click the link to this biblio's details page; the biblionumber should be blank, leading to a 404

To test solution:
1. Apply patch
2. modify the biblio slightly (click the 005 for example) and save
   OR manually add the biblio to zebraqueue for reindexing
3. after rebuild_zebra.pl -z -b -x runs, use the same search as above. The title should still appear.
4. click the link, and find yourself on the biblio detail page as desired

Signed-off-by: D Ruth Bavousett <ruth@bywatersolutions.com>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Comment 6 Paul Poulain 2011-10-14 10:07:03 UTC
QA comment :
  1- full test plan
  2- change made in the core of the code but only a few lines, no side-effect I can see, except a light loss of performance caused by the _koha_marc_update_bib_ids call in GetMarcBiblio
  3- MARC-independant, so works in UNIMARC as well (note in UNIMARC, the biblionumber is in 001, so the insert_ordered was not a problem)
  4- Moving fix_biblio_ids into GetMarcBiblio is a good idea, and should/could be done for authorities as well : move the fix_authority_id into GetAuthority sub as well

marked passed QA as the last comment is small compared to the problem this patch fixes. I suggest keeping the bug open until a fix is provided for my point 4
Comment 7 Ian Walls 2011-10-14 12:29:19 UTC

Thanks for the QA!  I was very uncomfortable pushing this as a ByWater-only venture, so having an independent analysis from another community member (and one who can speak to UNIMARC support at that) eases my mind muchly.
Comment 8 Chris Cormack 2011-10-15 00:48:00 UTC
Pushed please test.
Comment 9 Jared Camins-Esakov 2012-05-23 12:31:34 UTC
This appears to be fixed. Closing.