This applies only to things catalogued since upgrading to 3.4 The search results for these new records show no items in the hit list. But if you click on the title to bring up the normal view, the items are there, and are fully functional. The OPAC seems unaffected. The records created before we moved to 3.4 are also unaffected.
I cant replicate this, both with and without xslt turned on
Just to be sure, is it save to rerun remove_items_from_biblioitems.pl? I can rerun to make sure this has been done. (and then rebuild the zebra index)
To clarify: Records added before the switch display in the hit list with three columns: Results, Location, and an unlabeled one that contains a list of items for that biblio. The biblios added after the upgrade follow after the ones added before so it's easy to spot the change in our catalogue. The biblios added after the switch all say No items in the third column. But the items are there when you click on the title and even in the hit list, you have a link to Edit items, even when it says no items. The inconsistency is curious. But I cannot find a difference between the records created before and after the switch that would cause this.
I just experienced the same problem on a freshly installed test server. All mandatory and optional data was imported at the time of the installation. No settings had been touched as far as I am aware. However, any items that were catalogued were showing in the results as "no items", but the record details listed the items just fine. I re-indexed the catalogue with "misc/migration_tools/rebuild_zebra.pl -b -r -v" and now everything shows up just fine. When I set this installation up, I used the koha zebraqueue daemon to manage the zebraqueue. Could this be the source of the problem? It must be something zebra related for a re-index to fix the problem for me.
We have now upgraded to 3.4.2 and the problem persists. We can fix it by running './rebuild_zebra.pl -b', but this only fixes it until we update the record and then a search that returns the biblio will again show "No items". This is definitely some sort of Koha to Zebra update issue. The only errors I find recorded in /var/log/koha around that time are: [Wed Jul 06 22:54:29 2011] [error] [client 92.234.0.16] [Wed Jul 6 22:54:29 2011] search.pl: Use of uninitialized value $key in hash element at /usr/share/koha/lib/C4/Templates.pm line 227., referer: http://catalogue-admin.efdss.org/cgi-bin/koha/catalogue/detail.pl?biblionumber=14329 [Wed Jul 06 22:54:31 2011] [error] [client 92.234.0.16] [Wed Jul 6 22:54:31 2011] search.pl: Use of uninitialized value $key in hash element at /usr/share/koha/lib/C4/Templates.pm line 227., referer: http://catalogue-admin.efdss.org/cgi-bin/koha/catalogue/search.pl?q=evelyn+sharp [Wed Jul 06 22:55:39 2011] [error] [client 92.234.0.16] [Wed Jul 6 22:55:39 2011] addbiblio.pl: Subroutine plugin_parameters redefined at /usr/share/koha/intranet/cgi-bin/cataloguing/value_builder/callnumber.pl line 39., referer: http://catalogue-admin.efdss.org/cgi-bin/koha/catalogue/search.pl?q=evelyn+sharp [Wed Jul 06 22:55:39 2011] [error] [client 92.234.0.16] [Wed Jul 6 22:55:39 2011] addbiblio.pl: Subroutine plugin_javascript redefined at /usr/share/koha/intranet/cgi-bin/cataloguing/value_builder/callnumber.pl line 42., referer: http://catalogue-admin.efdss.org/cgi-bin/koha/catalogue/search.pl?q=evelyn+sharp [Wed Jul 06 22:55:39 2011] [error] [client 92.234.0.16] [Wed Jul 6 22:55:39 2011] addbiblio.pl: Subroutine plugin redefined at /usr/share/koha/intranet/cgi-bin/cataloguing/value_builder/callnumber.pl line 73., referer: http://catalogue-admin.efdss.org/cgi-bin/koha/catalogue/search.pl?q=evelyn+sharp [Wed Jul 06 22:55:59 2011] [error] [client 92.234.0.16] [Wed Jul 6 22:55:59 2011] search.pl: Use of uninitialized value $key in hash element at /usr/share/koha/lib/C4/Templates.pm line 227., referer: http://catalogue-admin.efdss.org/cgi-bin/koha/catalogue/detail.pl?biblionumber=14329 [Wed Jul 06 22:56:01 2011] [error] [client 92.234.0.16] [Wed Jul 6 22:56:01 2011] search.pl: Use of uninitialized value $key in hash element at /usr/share/koha/lib/C4/Templates.pm line 227., referer: http://catalogue-admin.efdss.org/cgi-bin/koha/catalogue/search.pl?q=evelyn+sharp Zebraqueue daemon is running and appears to be happy. Syslog show the following almost immediately when I update a record: Jul 6 23:12:49 www Koha Zebraqueue [6696]: 1 operations waiting to be run Jul 6 23:12:49 www Koha Zebraqueue [6696]: Executing zebra operations Further ideas are most welcome.
So it looks like some problem with the zebra incremental update managed by the zebraqueue daemon. Rebuilds of the index (rebuild_zebra.pl -b) resolves the issue temporarily (until the next record update). It appears to be an issue induced by the upgrade we did from v3.2.6 to v3.4.1. We are now on v3.4.2 and the issue persists.
I can confirm this issue: we have the same (running 3.4.2), and it was noted after transition from 3.2.x to 3.4.x As workaround I have set cronjobs for launch regulary 'rebuild_zebra.pl -b -a -z'.
I think this bug is a consequence of removing items form MARC. When using rebuild_zebra.pl, in get_raw_marc_record() : - without "--noxml" option the items are added to MARC::Record in GetMarcBiblio() call. - with "--noxml" option, MARC::Record is build from biblioitems.marc and items are not added. This might be corrected by Bug 10482.
This is an interesting observation Fridolyn. We are running 3.10.6. BIB_INDEX_MODE=dom. We have recently started seeing problems again with item records disappearing when the biblio record is updated. If you do a search, you will get back the matching biblios, and the modified one will show "No items.", but if you drill into the result, the item *is* displayed, so its in the Koha system. This happens with biblios that have a single item record. If I reindex with "rebuild_zebra.pl -b -v -x" then the item reappears in catalog searchs. Could it be that after the biblio change, an incomplete MARC record is exported for update? We are using the koha-index-daemon.
You mean zebraqueue-daemon.pl ?
I was using koha-index-daemon (http://search.cpan.org/~fredericd/Koha-Contrib-Tamil-0.029/bin/koha-index-daemon) but clearly this does not correctly export all records. So I have stopped and reverted back to the less elegant (but functioning) koha-zebraqueue-daemon (zebraqueue-daemon.pl). We should either get this other daemon fixed or removed. I forget how I got pointed to it originally. Our problem is now resolved.
zebraqueue_daemon is broken and was removed from Koha several versions ago. Do not use it. Your best bet would be to either use rebuild_zebra and teach your staff to deal with the delay, or to find someone who can troubleshoot the koha-index-daemon from Tamil.
OK, I have gone to the cron job. I will look into an alternative wrapper daemon that just checks the mysql queue and calls rebuild only when necessary.
Created attachment 19267 [details] [review] Patch to add daemon mode to rebuild_zebra.pl This diff adds a daemon mode to rebuild_zebra.pl. By polling the queue every few seconds from a running process, we can achieve very fast indexing response with low system overhead.
Your patch contains: sleep 5; This must be configurable. 5 seconds is too small for large catalogs.
Created attachment 19710 [details] [review] Improved patch to add daemon mode to rebuild_zebra.pl This improved patch adds a variable to set the sleep time between checks for new records to index. The check is very cheap and not dependent on the size of the catalog so we make it every 5 seconds. Any records found (probably just one), are indexed by zebra incrementally. This will make new records nearly instantly available for search.
(In reply to Doug Kingston from comment #16) > Created attachment 19710 [details] [review] [review] Your patch is very odd. Did you create it with git format-patch ?
Hi Doug, I would like to test this patch, however it appears to be formatted incorrectly - please see http://wiki.koha-community.org/wiki/Version_Control_Using_Git for help on setting up git and sending in patches. If you are having trouble with any of that, do say and I'll see about making a git formatted patch for you - but please have a try at doing it with git first. :) Thanks, Liz
Created attachment 20574 [details] [review] My attempt to generate a useful patch from Doug's bizarre diff file. I wanted to accomplish something today. :) The one problem with his patch that I noticed thus far would include: -- Failed to update the HELP to give information regarding the parameters added. test: $ ./misc/migration_tools/rebuild_zebra.pl --help Take this patch, and fix it up, Doug. :)
Created attachment 20575 [details] [review] Convert Doug's diff into a patch. Oops! Had a space problem that would have triggered a QA issue. :)
Created attachment 21097 [details] [review] Proper GIT patch for the daemon mode addition to rebuild_zebra.pl This patch is GIT generated against HEAD and includes updates to the help as requested. Once this patch has been vetted and tested by others, we should probably update the guidance in the various INSTALL files to move away from periodic cron jobs.
"Seconcds to sleep between checks" typo in the patch.
Created attachment 21099 [details] [review] Proper GIT patch for the daemon mode addition to rebuild_zebra.pl (w/typo fixed) Trivial update to prior patch.
Changed status, because Doug forgot. :)
Applying: Bug 6435 Add daemon mode to rebuild_zebra.pl (w/typo fix) Repository lacks necessary blobs to fall back on 3-way merge. Cannot fall back to three-way merge. Patch failed at 0001 Bug 6435 Add daemon mode to rebuild_zebra.pl (w/typo fix) The copy of the patch that failed is found in: /home/christopher/git/koha/.git/rebase-apply/patch When you have resolved this problem, run "git am --continue". If you prefer to skip this patch, run "git am --skip" instead. To restore the original branch and stop patching, run "git am --abort".
Ahh I see the problem, the first patch has been obsoleted, there should be 2 patches on this bug, will restore the first one and test again. Yay for the bot catching this!
Created attachment 21390 [details] [review] Bug 6435 Add daemon mode to rebuild_zebra.pl This change adds code to check the zebraqueue table with a cheap SQL query and a daemon loop that checks for new entries and processes them incremantally before sleeping for a controllable number of seconds. The default is 5 seconds which provides a near realtime search index update. This is desirable particularly for libraries that are doing active catalogue updating. Help text updated. Tested against a live 3.12 system. Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz> Does what it says on the box, I wonder if it leaks ram at all, would have to run for a long time. But I'm happy to sign off as it doesn't break the existing functionality, while adding new.
Created attachment 21391 [details] [review] Bug 6435 Add daemon mode to rebuild_zebra.pl (w/typo fix) This change adds code to check the zebraqueue table with a cheap SQL query and a daemon loop that checks for new entries and processes them incremantally before sleeping for a controllable number of seconds. The default is 5 seconds which provides a near realtime search index update. This is desirable particularly for libraries that are doing active catalogue updating. Help text updated. Tested against a live 3.12 system. Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Please provide a step by step test plan to make this easier to QA. Thx!
Apply both patches above. One is the real code and the small one fixes a typo. Suggested test plan: Invoke the daemon exercising all the options I added and observe it doing the intended updates in line with the update interval. 1. Set up a test koha configuration (populate with data if you wish) 2. Ensure that the zebra database is up to date (rebuild_zebra.pl -b -a) 3. Stop any cronjob that may be updating zebra with rebuild_zebra.pl 4. Start rebuild_zebra in daemon mode (rebuild_zebra.pl -daemon -sleep 3 -b -a v 5. TEST: observe the periodic message "queued records: 0" every 3 seconds. this will test the use of -sleep option. 6. Add an authority. 7. TEST: observe the daemon pick up the authority change and update zebra (which should be within 3 seconds of completing the add operation) 8. TEST: check that you can search for the new authority from the admin interface 9. Add a biblio. 10. TEST: observe teh daemon pickup up the biblio change and update zebra (which should be within 3 seconds of completing the add operation) 11. TEST: check that you can search for the new biblio in either OPAC or admin interface. Possibly repeat steps 6-11 with authority and biblio modification and deletion.
Doug, you wrote the patch, so you are assignee :)
Created attachment 22039 [details] [review] Add locking to rebuild_zebra to prevent races This patch adds flock based locking for rebuild_zebra.pl on a per-instance basis. This prevents races between full rebuilds and background incremental updates from the zebraqueue table in the database. The race condition exists whether you are doing incremental updates with a periodic cronjob or with the new daemon mode. Suppose you start a full rebuild at time T0 which will take until T20 to extract the records. Suppose also at T10, a biblio or auth is updated and processed through the zebraqueue by T15. In this situation the updated record in zebra will be overwritten when the full rebuild records are uploaded to zebra after T20. We prevent this by only allowing one rebuild_zebra per koha instance to be running at one time. When running in daemon mode, incremental updates will be skipped while a full rebuild is running, and resume afterwards. A full rebuild or other adhoc request will wait for any previous lock to clear. Tested by flocking the lock file while invoking rebuild_zebra.pl in various modes (daemon, adhoc zebraqueue task, and full rebuild) using flock program I will attach to bug.
Created attachment 22040 [details] Small perl program to flock files and run commands This program can be used to test the flock functionality introduced in patch 0003 also attached to this bug. Test I performed that QA can replicate (modify for your test instance name): flock the file, and make sure a full rebuild waits for the lock to clear before proceeding window 1: flock /var/lock/koha_rebuild_zebra_koha_test/lock sleep 15 window 2: ./rebuild_zebra.pl -b -a -v window 2 should wait for window 1's sleep to time out before proceeding. test that an flock from another process with cause the daemon mode to pause updates and resume when the flock is removed window 1: ./rebuild_zebra.pl -daemon -sleep 2 -v -z -b -a window 2: flock /var/lock/koha_rebuild_zebra_koha_test/lock sleep 15 start window 2 after window 1 is printing a message every 2 seconds. when you launch the sleep, it will stop the updates which will resume when the sleep times out. remove /var/log/koha_rebuild_zebra_koha_test and make sure its recreated on the next invocation of rebuild_zebra.pl.
Any chance we can move the 'Add locking to rebuild_zebra to prevent races' followup patch to a dependant bug instead of adding it here.. That way QA can go on with the deamon mode patch without being held in waiting for a further signoff. (PS, if you do that, I'll test and sign off on it).
I have created separate bug 11078 to track the race condition and its locking fix. 11078 depends on this bug. 6435 does not create any race condition that did not already exist. The patch for locking and the flock test program are attached to 11078.
Created attachment 22080 [details] empty file to allow obsoleting other files. Deleting unneeded files which have moved to a new bug 11078.
Comment on attachment 22080 [details] empty file to allow obsoleting other files. There is no need for this file. To obsolete files: Click on "Details" beside the attachment name. Click on "Edit Details" beside the name at the top. Check the obsolete box near the middle/bottom right. Click the appropriate save/submit button. To un-obsolete files: Click on link to show all the obsoleted items Click on "Details" beside the attachment name. Click on "Edit Details" beside the name at the top. Un-check the obsolete box near the middle/bottom right. Click the appropriate save/submit button.
Created attachment 22201 [details] [review] Bug 6345 Add locking to rebuild_zebra This patch adds flock based locking for rebuild_zebra.pl on a per-instance basis. This prevents races between full rebuilds and background incremental updates from the zebraqueue table in the database. The race condition exists whether you are doing incremental updates with a periodic cronjob or with the new daemon mode. Suppose you start a full rebuild at time T0 which will take until T20 to extract the records. Suppose also at T10, a biblio or auth is updated and processed through the zebraqueue by T15. In this situation the updated record in zebra will be overwritten when the full rebuild records are uploaded to zebra after T20. We prevent this by only allowing one rebuild_zebra per koha instance to be running at one time. When running in daemon mode, incremental updates will be skipped while a full rebuild is running, and resume afterwards. A full rebuild or other adhoc request will wait for any previous lock to clear. Tested by flocking the lock file while invoking rebuild_zebra.pl in various modes (daemon, adhoc zebraqueue task, and full rebuild) using flock program I will attach to bug. http://bugs.koha-community.org/show_bug.cgi?id=11078 Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment on attachment 22201 [details] [review] Bug 6345 Add locking to rebuild_zebra Oops.. wrong bug.
I think this doesn't work quite right if you use the daemon mode only for -b not for -a. The command I ran was: ./misc/migration_tools/rebuild_zebra.pl -daemon -sleep 4 -b -v -z Then I added a new authority, it keeps repeating the following lines: queued records: 1 skipping authorities ==================== exporting biblio ==================== Records exported: 0 Records exported: 0 ==================== REINDEXING zebra ==================== The record is not reindexed, but Koha keeps trying. I think this is because the SQL statement doesn't differentiate between authority and biblio records. Doug, could you take a look?
Created attachment 22457 [details] [review] [ENH] Bug 6435 Add daemon mode to rebuild_zebra.pl (revised 10/27) (combines previous patches and addresses recent QA issue) This change adds code to check the zebraqueue table with a cheap SQL query and a daemon loop that checks for new entries and processes them incremantally before sleeping for a controllable number of seconds. The default is 5 seconds which provides a near realtime search index update. This is desirable particularly for libraries that are doing active catalogue updating. The query is adjusted based on whether -a, -b, or -a -b are specified. Help text updated. Tested against a live 3.12 system. Note that this fix will benefit from the fix to lack of locking (bug 11078)
I'm just curious if you would want to run daemon mode for only biblios or only authorities. If this process is going to run in daemon mode, would you not want both to be picked up? Otherwise, you'd have to run two daemons (one for bibs and one for auths). Changing the status to needs sign off, since this question doesn't really affect whether the code does what it is supposed to do, but I haven't tested it.
Yes, you'd probably want both, but for example if your library don't uses authorities and you don't turn it on and then someone catalogs an authority for testing... it shouldn't lead to trouble. It's just something that could go wrong that we might want to avoid either by documentation or by catching it in the code.
I addressed the recent QA issue on 10/28 with the updated patch. Can this be re-reviewed?
Created attachment 22851 [details] [review] Bug 6435 Add daemon mode to rebuild_zebra.pl This change adds code to check the zebraqueue table with a cheap SQL query and a daemon loop that checks for new entries and processes them incremantally before sleeping for a controllable number of seconds. The default is 5 seconds which provides a near realtime search index update. This is desirable particularly for libraries that are doing active catalogue updating. The query is adjusted based on whether -a, -b, or -a -b are specified. Help text updated. Tested against a live 3.12 system. Note that this fix will benefit from the fix to lack of locking (bug 11078) Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Created attachment 22859 [details] [review] Bug 6435 Add daemon mode to rebuild_zebra.pl This change adds code to check the zebraqueue table with a cheap SQL query and a daemon loop that checks for new entries and processes them incremantally before sleeping for a controllable number of seconds. The default is 5 seconds which provides a near realtime search index update. This is desirable particularly for libraries that are doing active catalogue updating. The query is adjusted based on whether -a, -b, or -a -b are specified. Help text updated. Tested against a live 3.12 system. Note that this fix will benefit from the fix to lack of locking (bug 11078) Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
I'm up and down on passing this one.. discussed it with Katrin.. 1. Passes QA script (after I fixed some minor white space errors - Doug, please take a look at http://blog.bigballofwax.co.nz/2011/12/15/fixing-whitespace-when-apply-patches-with-git/ to stop these in their tracks before they hit QA with future patches ;) ) 2. Tested with various combinations of options: -a -daemon, -b -daemon, -a -b -daemon, -a -b -x -daemon, all worked as expected along with not breaking existing options. One niggle worth discussing: The intention of this patch is clear, to allow regular rebuilds as a daemon.. However, the patch allows all/any params to be passed along with the daemon mode. This slightly scares me as it becomes rather easy to break the system using the wrong combination of options. (for instance, running with -r would mean that every time a catalogue update happens, the zebra index disappears moments later whilst a full rebuild is happening) I don't think it's a QA fail, but it may be worth documenting that certain options don't work well with a daemon model.
Follow-up note 1: I also tested the -sleep option and it worked well. Follow-up note 2: As an afterthought, should the -z be implied by daemon mode?
(In reply to Martin Renvoize from comment #48) > Follow-up note 1: I also tested the -sleep option and it worked well. > Follow-up note 2: As an afterthought, should the -z be implied by daemon > mode? In response to this and the previous comment, my view is that --daemon should imply (and be) "here's how you keep your Zebra index up to date." IOW, I think --daemon should imply the following options: -z -a -b It should exclude (by refusing to start or by simply ignoring) the following options: -r -s -k -h -y -where -length -offset -I | -skip-index I wouldn't consider this a blocker for putting the patch in master, but lean towards considering a it blocker for putting in a stable branch.
I can try to pull that patch together this weekend. -Doug-
(In reply to Doug Kingston from comment #50) > I can try to pull that patch together this weekend. Great, thanks!
Created attachment 22986 [details] [review] QA Patch - Make daemon mode imply -z -a -b and abort on incompatible flags
Pushed to master. Thanks, Doug! Also, this appears to be your first patch accepted into Koha. Congratulation!
Nice work Doug, excellent follow through.
This patch has been pushed to 3.14.x, will be in 3.14.1 Contains 3 patches : 08c8bea Bug 6435: (follow-up) make -daemon really imply -a and -b ae9e02a Bug 6435: (follow-up) rebuild_zebra -daemon option now smarter e7482ec Bug 6435: Add daemon mode to rebuild_zebra.pl