For using with the zebraqueue daemon script I propose refactoring rebuild_zebra.pl script so we can avoid spawning a new perl interpreter each run. Server load in mind. This is an initial submit so we discuss this topic with something concrete to talk about. Another option would be to just use a shell script.
Created attachment 2565 [details] [review] Initial patch
What's the interface to this code to be?
Created attachment 3239 [details] [review] THE patch
For testing follow the old install steps: {{{ $ sudo ln -s ${SCRIPT_DIR}/koha-zebraqueue-ctl.sh \ /etc/init.d/koha-zebraqueue-daemon $ sudo update-rc.d koha-zebraqueue-daemon defaults }}}
If you'd like to test on a dev install, run daemon like this: KOHA_DEV_SRC=/home/tomas/Work/koha/devel/koha-community-src KOHA_DEV_ETC=/home/tomas/koha-dev/etc/koha-conf.xml MYUSER=<user that runs processes> su - $MYUSER PERL5LIB=$KOHA_DEV_SRC KOHA_CONF=$KOHA_DEV_ETC $KOHA_DEV_SRC/misc/bin/zebraqueue_daemon.pl
Patch does not apply. -- Applying: Refactor rebuild_zebra.pl into a library /home/julian/koha/versions/koha-community-git/.git/rebase-apply/patch:86: trailing whitespace. returns the number of updated+deleted authority records /home/julian/koha/versions/koha-community-git/.git/rebase-apply/patch:100: trailing whitespace. returns the number of updated+deleted biblio records /home/julian/koha/versions/koha-community-git/.git/rebase-apply/patch:114: trailing whitespace. returns the number of updated+deleted authority and biblio records /home/julian/koha/versions/koha-community-git/.git/rebase-apply/patch:134: trailing whitespace. returns the number of updated+deleted $record_type records /home/julian/koha/versions/koha-community-git/.git/rebase-apply/patch:161: trailing whitespace. returns the number of updated+deleted $record_type records error: patch failed: installer/data/mysql/de-DE/mandatory/sysprefs.sql:285 error: installer/data/mysql/de-DE/mandatory/sysprefs.sql: patch does not apply error: patch failed: installer/data/mysql/en/mandatory/sysprefs.sql:285 error: installer/data/mysql/en/mandatory/sysprefs.sql: patch does not apply error: patch failed: installer/data/mysql/fr-FR/1-Obligatoire/unimarc_standard_systemprefs.sql:286 error: installer/data/mysql/fr-FR/1-Obligatoire/unimarc_standard_systemprefs.sql: patch does not apply error: patch failed: installer/data/mysql/it-IT/necessari/sysprefs.sql:271 error: installer/data/mysql/it-IT/necessari/sysprefs.sql: patch does not apply error: patch failed: installer/data/mysql/pl-PL/mandatory/sysprefs.sql:283 error: installer/data/mysql/pl-PL/mandatory/sysprefs.sql: patch does not apply error: patch failed: installer/data/mysql/ru-RU/mandatory/system_preferences_full_optimal_for_install_only.sql:338 error: installer/data/mysql/ru-RU/mandatory/system_preferences_full_optimal_for_install_only.sql: patch does not apply error: patch failed: installer/data/mysql/uk-UA/mandatory/system_preferences_full_optimal_for_install_only.sql:363 error: installer/data/mysql/uk-UA/mandatory/system_preferences_full_optimal_for_install_only.sql: patch does not apply Patch failed at 0001 Refactor rebuild_zebra.pl into a library
Created attachment 3380 [details] [review] Initial patch, refactored (libs)
Created attachment 3381 [details] [review] Sysprefs changes
Created attachment 3382 [details] [review] zebraqueue daemon scripts
Created attachment 3383 [details] [review] updatedatabase.pl fix
*** Bug 5165 has been marked as a duplicate of this bug. ***
Correct me if im wrong, but this doesnt seem to touch rebuild_zebra.pl at all As such the options in the systempreferences only apply if you are running the new, (and hopefully not memory leaking zebraqueue_daemon). Before I can merge this into master, we would need to first change the wording in the sys prefs to make it clear that if you are running rebuild_zebra.pl as a cron job these will have no effect. Then we need to test the daemon does not memory leak (functionality was never a problem with it, leaking memory and taking the system down was) The branch is in new/enh/bug_5166 please could others take a look.
As-is, this code doesn't hurt any previous functionality. That's what I wanted, and why I didn't touch rebuild_zebra.pl (which is trivial to change to use the sysprefs as default, I can do it if preferred). Also I didn't port fullindex and other rebuild_zebra.pl functionalities to avoid making this a big commit, but those functions are trivial to implement too.
Ah no, i think we are miscommunicating :) I don't want it ported to rebuild_zebra.pl, it's too close to the release to do something like that. What I do want is the words in the preferences screen to make it very clear, these system preferences are only valid if you are using the zebraqueue daemon. So that people dont change them, and wonder why they are having no effect. Also Id like them to make it clear that the daemon is still expiremental, until we are sure it still isn't memory leaking. So what I would like is a follow up patch to change the wording of the preferences page. If you need help with the words I can help out with that.
Created attachment 3643 [details] [review] Sysprefs make 'experimental' nature explicit Also updates updatedatabase.pl
Hi Tomás What branch did you base your new patch on, it will not apply on new/enh/bug_5166 Chris
Created attachment 3664 [details] [review] Sysprefs make 'experimental' nature explicit - again
shouldn't this patch be set to "Needs signoff" again ?
This bug is mentioned in: [Bug 5166] Refactor rebuild_zebra.pl into a library http://lists.koha-community.org/pipermail/koha-patches/2011-March/013969.html [Bug 5166] Libraries and zebraqueue daemon scripts http://lists.koha-community.org/pipermail/koha-patches/2011-March/014163.html [Bug 5166] Sysprefs http://lists.koha-community.org/pipermail/koha-patches/2011-March/014164.html [Bug 5166] zebraqueue daemon scripts http://lists.koha-community.org/pipermail/koha-patches/2011-March/014165.html [Bug 5166] updatedatabase.pl fix http://lists.koha-community.org/pipermail/koha-patches/2011-March/014166.html Bug 5166: Libraries and zebraqueue daemon scripts http://lists.koha-community.org/pipermail/koha-patches/2011-March/014175.html Bug 5166: Sysprefs http://lists.koha-community.org/pipermail/koha-patches/2011-March/014176.html Bug 5166: zebraqueue daemon scripts http://lists.koha-community.org/pipermail/koha-patches/2011-March/014177.html Bug 5166: updatedatabase.pl fix http://lists.koha-community.org/pipermail/koha-patches/2011-March/014178.html
Pushed up into new/enh/bug_5166 and master merged into it, please test. It doesn't touch any main functionality so if it doesnt cause any regressions it should be ok for 3.4.0 Please update the bug with your testing report
Added 2 records. Added 2 authority records. Modified 1 record (added an item). Deleted 1 authority record. Deleted 1 biblio record. Operations completed successfully. Just got a warning when updating authority. It actually belongs to AuthoritiesMarc.pm: Argument "a" isn't numeric in numeric ne (!=) at /home/tomas/Work/koha/devel/koha-community-src/C4/AuthoritiesMarc.pm line 1462. so not relevant here.
I forgot to mention that I had to change several lines (between #320 and #332) for the search to work in staff interface as the conversion to TT seems broken in this branch.
In Comment 22 I meant to say I had to change several lines IN FILE 'catalogue/results.tt' to be able to test indexing was working.
The results.tt file has been changed in master, new/enh/bug_5166 needs to have master merged into it. It is now too close to the release for this to make it into 3.4.0, it also still needs 3rd party sign off. I have merged master in, and pushed, if you can find someone to sign off on it that would be what would be needed for it to go into master.
Julian Maurice signed-off on this bug. The only change it had was some changes in the wording of the sysprefs. Anyway, I'll ask someone on the irc to sign-off.
In that case, if you can get someone to signoff on the branch now, just even noting it in the bugreport is fine. That the new/enh/bug_5166 branch works fine, with no changes needed. I will merge it for 3.4.0 as it was signed off before the string freeze.
Bug found with the patch
Created attachment 3958 [details] [review] Fix syspref, for applying to new/enh/bug_5166
Created attachment 4031 [details] [review] Take note of Bug 5724 Adjust the order in which updated/deleted records are processed so updates occur before deleted records.
Tested and everything went fine.
Sorry, but the updatedatabase patch does not apply anymore, could you rebase please ? Note also i'm confused by the patch names. I have : 0001-Bug-5166-Fix-searching.pref-also-rephrase.patch 0001-Bug-5166-Libraries-and-zebraqueue-daemon-scripts.patch 0002-Bug-5166-Fix-as-of-Bug-5724.patch 0002-Bug-5166-Sysprefs.patch 0003-Bug-5166-zebraqueue-daemon-scripts.patch 0004-Bug-5166-updatedatabase.pl-fix.patch 0005-Bug-5166-Sysprefs-make-experimental-nature-explicit-.patch The 0001-Bug-5166-Fix-searching.pref-also-rephrase.patch and 0002-Bug-5166-Fix-as-of-Bug-5724.patch being number 6 and 7 in the patch list. In which order should we apply them ? However, I like the patch and the idea. But unless i've missed something it's not related to "refactor rebuild_zebra.pl" because there's nothing to do with rebuild_zebra. It's more "reintroduce a zebra daemon, and provide tools that will enable us later to refactor rebuild_zebra.pl". Note i'm happy with this idea, maybe we/you can just update the title of your bug ?
Created attachment 4831 [details] [review] New zebraqueue daemon and libs Rebased against origin/master. It was already signed-off and in new/enh/bug_5166 branch.
Please note the adition of: installer/data/mysql/atomicupdate/bug_5166-zebraqueuedaemon_is_back.pl as noted on IRC this could help QA and RM to avoid rebasing so frecquently.
Updating Version : This ENH will be for Koha 3.8
Bug versionned for master. entries will be made against rel_3_8 once the patch has been applied (see thread about that on koha-devel yesterday)
The system preference management has changed since this patch was written, only one file has to be changed now. Please fix and resubmit.
Created attachment 6902 [details] [review] Rebased against origin/master Please let me know if there are problems with this patch as soon as possible. End of the year is approaching and my bosses have many many things for me to finish before that.
I have an alternative implementation of a patch for this bug. I've packaged it as a CPAN module for easy testing. It contains two scripts: one for punctual indexing, and one for queued records indexing. What is specific to this implementation: - No use of POE, but AnyEvent: http://search.cpan.org/dist/AnyEvent/ - Moose OO classes - Export records for Zebra only in XML - No records content modification (leader, authority id, unimarc 100 tag). As it is, it works with a lot of UNIMARC/MARC21 Koha instances loaded with various records types. How to test it: - Install the module: sudo cpan Koha::Contrib::Tamil - You have two scripts installed: koha-index koha-index-daemon - You can access their respective man pages: man koha-index man koha-index-daemon - To reindex a whole DB, standard env variable being set: koha-index --verbose koha-index --source authority --verbose - To start the indexing daemon, with indexing occurring every minute: koha-index-daemon - To start the same daemon, with indexing occuring every 30 seconds: koha-index-daemon --timeout 30
Testing notes: Install: * One *must* have their Koha environment variables set (KOHA_CONF and PERL5LIB) for the CPAN module to install. Questions: * How does this mingle with the packages scripts that do similar things? I understand the daemon will be a different process - what about a normal reindex? Is the koha-index --verbose process in some way superior? Testing notes: I did a batch import of about 30k records - I set the timeout to 5. The daemon picked up between 80 and 200 records a tick, and never seemed to get overwhelmed. This was on Frederic's cpan module. I think it's good - we should include it. How do you want the signoff as this is a cpan module? Is there a reason to choose one of these implementations over the other? Should I test the other version as well?
The script devel/misc/testbugzillapatches.pl has detected that this bug does not apply anymore, so i'm marking this bug as "does not apply". The script is supposed to handle conflicts related to updatedatabase, but no other cases of merge conflict. So the conflict may be trivial to fix. If it happens that the patch attached does apply cleanly, please send me an email, there's something wrong in the script. It's possible, as it's a recent script. If the patch applies does not apply as I think, please rebase and resubmit, thanks
I'd like to note that I initiated this bug when we implemented here (UNC) a way (call it wrapper / new daemon / whatever) to achieve the sub-minute record index update our librarians required. We've been using this in production since early 2010. We just used the logic already present in rebuild_zebra.pl and thought it would be a fast cool feature everyone could enjoy until that zerba stuff was rewritten. I rebased this patches for a year and a half, also adjusting some bits to mimic a diverging rebuild_zebra.pl script everytime smth changed. We don't need this inside koha per-se, and I don't have the time to keep struggling with this patch. Ence, I retire from this bug, and would like to ask if I should close it to clean bugzilla. Regards To+
My question still remains - should we pick one of these implementations over the other? I've been using the cpan module version for several months and let me tell you the freedom to set 5 second zebraqueue updates is awesome. This should be in Koha. Can we make it happen? What needs to happen?
(In reply to comment #42) > I've been using the cpan module version for several months and let me tell > you the freedom to set 5 second zebraqueue updates is awesome. This should > be in Koha. Yeah, that's why this bug was submitted for the first time. > My question still remains - should we pick one of these implementations over > the other? I think Frederic's rewrite of this is a good one. I personally dont like the idea of it being an external lib, as I think that libs should be tied to specific versions of Koha (think of db changes, we realize smth should be done other way, etc). So a reworked version, more daemon-friendly (so you can do 'update-rc.d koha-zeebraqueue-daemon default' to have it at startup), etc could make it. What I propose is to close this bug and maybe start a new one on integrating Tamil's lib into Koha. I just abandoned my patches for this bug.
I agree with this idea, integrating the Tamil code into koha
Hi we are working to commit this patch and after that patch 7430 So we will put all in specific places. But we need to start with this. We are at HackFest in Marseille
Created attachment 8357 [details] [review] Last rebase of the patch
I sugget to close this bug with a 'Wont fix' situation. For who want background indexing, I will insert instruction in documentation. See Bug 7759