Bug 9511 - DOM indexing problem
Summary: DOM indexing problem
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: 3.8
Hardware: All All
: P5 - low critical (vote)
Assignee: Galen Charlton
QA Contact:
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-01-30 19:49 UTC by Melia Meggs
Modified: 2019-06-27 09:24 UTC (History)
4 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Melia Meggs 2013-01-30 19:49:01 UTC
On sites using DOM, when you delete a bib record, it doesn't go away immediately. When the fast indexer runs, the edit appeared in the search results, but the deleted record did not go away. It can take 3 or 4 days but does eventually disappear.  

This also causes problems when trying to import records because it's matching things that they've already deleted that really should be gone.
Comment 1 Galen Charlton 2013-01-30 21:04:46 UTC
Could you add the exact cronjob invocation to this bug?  I tried deleting bibs in a tiny test database that's using DOM indexing, both with and without the patch for bug 9496, and the records were deleted from the Zebra index when I ran rebuild_zebra.pl -b -z or rebuild_zebra.pl -b -x -z.
Comment 2 Larry Baerveldt 2013-01-30 21:11:47 UTC
Galen,

What we typically use on all sites is 'rebuild_zebra.pl -x -b -z'.

- Larry
Comment 3 Jorge de Cardenas 2013-02-04 19:36:47 UTC
I am seeing the same issue.

Example 1: 
delete all items
run ./rebuild_zebra -b -z
rerun search 
biblio is found with no items
delete record
run ./rebuild_zebra -b -z
rerun search
biblio with no items still shows up

Example 2:
delete all items and delete record
run ./rebuild_zebra -b -z
rerun search
biblio with items will show up

As a note: if "adelete" is replace with "delete" in  you get
"[warn] cannot delete record above (seems new)"
for both cases above.

The only way to correct the search results is to run ./rebuild_zebra -b -r

It appears that the biblio record cannot be found by zebraidx to be removed.
Comment 4 Jorge de Cardenas 2013-02-05 20:22:07 UTC
I hope this might help: This record has been deleted. It still shows up on searches and when you click on it, you get

"The record you requested does not exist (81828)."

I found this using yaz-client (not easy).

Z> show
Sent presentRequest (20+1).
Records: 1
[biblios]Record type: USmarc
01187nam a2200313Ia 4500
001 ocm56473895
003 OSt
005 20111031143426.0
008 040909s2004    nyua   j      000 0 eng d
020    $a 0439455243  $c $3.50
040    $a OCO $c OCO $d STD $d OSt $d OhAtLVL
049    $a STDN
100 1  $a McCann, Jesse Leon. $9 1001550701
245 10 $a Scooby-doo and the werewolf / $c by Jesse Leon McCann.
246 3  $a Scooby-doo & the werewolf
260    $a New York : $b Scholastic, $c c2004.
300    $a 24 p. : $b ill. ; $c 22 cm.
440 00 $a Scooby-Doo! $9 3000254601
500    $a At head of title: Cartoon Network.
521    $a Level M (red) ; $b Scholastic Leveled Reading.
526    $a Accelerated reader ; $c 2.7 ; $d 0.5 ;
650  0 $a Scooby-Doo (Fictitious character) $v Juvenile fiction $9 2001577542
650  0 $a Leveled reader $9 100098602612
650  0 $a Accelerated Reader $9 100098597294
710 2  $a Cartoon Network (Television network) $9 1001407241
910    $a 267343
942    $c GE $k Scooby- Doo  $2 ddc
999    $c 81828 $d 81828
952    $w 2007-02-25 $p 33529002654449 $v 3.50 $r 2012-12-01 $4 0 $0 0 $6 SCOOBY__DOO $9 95084 $b MAIN $m 12 $1 0 $o Scooby- Doo $d 2004-09-14 $7 0 $c JE $y GE $s 2012-11-10 $l 124 $a MAIN
Comment 5 Jared Camins-Esakov 2013-02-05 20:27:04 UTC
What version of Koha are you using, Jorge?
Comment 6 Jorge de Cardenas 2013-02-05 21:03:26 UTC
Our production Koha is 3.10 but I reproduced the problem and took index data from a test server on master (3.11.00.014).
Comment 7 Paul A 2013-02-10 16:22:21 UTC
I can confirm (3.8.5) that rebuild_zebra with the incremental option -z, the biblio deletion does not occur; a full rebuild -b (with or without -x) is fully functional but is a *necessity*.

This bug is [seriously?] compounded by overwriting a biblio with a Z39.50 import. This creates a second record in Zebra (not MySQL), identical biblionumber, callnumber, intemnumber: see <http://opac.navalmarinearchive.com/cgi-bin/koha/opac-search.pl?idx=callnum&q=GRLK-GREE-10>

If a staff cataloguer is unaware of this bug and deletes the apparently incorrect (old) record, this deletes the good MySQL record.

I therefore suggest upgrading this bug to "critical."
Comment 8 Galen Charlton 2013-02-28 01:56:54 UTC
For those who are seeing this bug, could you paste the results of running the command 

zebraidx -V
Comment 9 Paul A 2013-02-28 15:43:51 UTC
(In reply to comment #8)
> For those who are seeing this bug, could you paste the results of running
> the command 
> 
> zebraidx -V

Zebra 2.0.44
(C) 1994-2010, Index Data ApS
Zebra is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
SHA1 ID: 419ad759807269fdfa379799a051ed3a551c6541
Using ICU
Comment 10 Galen Charlton 2013-02-28 17:16:53 UTC
A comparison of the Zebra config of a database experiencing this problem shows that it lacked the patch for bug 8665.  In particular, the Zebra record ID must be taken (for MARC21) from the 999$c, but prior to the patch for 8665, it was taken from the 001.

This also explains why I couldn't reproduce the problem myself -- I was using master.

Try updating biblio-koha-indexdefs.xml and biblio-zebra-indexdefs.xml, then do a full reindex of the bibs.  After that is done, I suspect that rebuild_zebra.pl -z will start removing deleted bibs from the Zebra index.
Comment 11 Galen Charlton 2013-02-28 17:24:44 UTC
For those on 3.8.x, the fix for 8665 was first released in 3.8.7.
Comment 12 Melia Meggs 2013-03-05 19:41:40 UTC
We've got two libraries that were previously experiencing this problem on 3.8.5.  We have upgraded them to 3.8.8, and both are now reporting that this problem looks to be resolved based on their testing so far.

Thanks, Galen!

Melia
Comment 13 Paul A 2013-03-07 16:27:32 UTC
3.8.10 tarball on Ubuntu 12.04.2 LTS, db imported from 3.8.5; using an incremental CRON '$KOHAPATH/bin/migration_tools/rebuild_zebra.pl -a -b -z -x' (with or without the '-x') does not remove the deleted record; a complete rebuild_zebra.pl -b -r -v -x is required.
Comment 14 Paul A 2013-03-09 14:52:49 UTC
(In reply to comment #13)
> 3.8.10 tarball on Ubuntu 12.04.2 LTS, db imported from 3.8.5; using an
> incremental CRON '$KOHAPATH/bin/migration_tools/rebuild_zebra.pl -a -b -z
> -x' (with or without the '-x') does not remove the deleted record; a
> complete rebuild_zebra.pl -b -r -v -x is required.

Illustrated example of problem:

Starting point: type='BK', title='The Hudson through the years', author='Adams, Arthur G.' 

Before intervention, we had *three* biblios, 1 for publisher='Fordham...', two for publisher='Lind...', each biblio with a single, valid 'item.' 

We did the following:
     - Transfer (copy barcode and "attach") a 'Lind' item from one biblio to the other and delete the "empty" biblio.
     - Overwrite the 'Lind' biblio now containing two items with a Z39.50 biblio from LoC (DLC.)

After this, we had *four* [relevant] biblios in OPAC, see <http://navalmarinearchive.com/ims/opac_hudson_before_full_rebuild.jpg>. #1 as expected, untouched, #2 gave a 404 error (the deleted biblio), #3 was the *old* biblio with both items, #4 was the desired result (Z39.50 biblio with both items.)

The "incremental" user=koha CRON ($KOHAPATH/bin/migration_tools/rebuild_zebra.pl -a -b -z) ran every minute overnight and did not correct this. Errors are redirected  >> /tmp/cron_koha.log 2>&1 and no errors were reported.

Running a ./bin/migration_tools/rebuild_zebra.pl -b -r -v -x this morning instantly corrected OPAC, see <http://navalmarinearchive.com/ims/opac_hudson_after_full_rebuild.jpg>.
Comment 15 Katrin Fischer 2014-03-15 10:04:47 UTC
Reading through the comments, this seems to be resolved since 3.8.7. Please reopen if I am mistaken and you still see this problem in current versions.