Here is another approach for excluding a limited number of fields from Zebra DOM indexing. (Check the see also reports too.) I rename index_all to index_most and add a few lines to that named template. This will not win beauty prices, but imo it is easier than adding any or anywhere to a large number of fields. What do you think?
Created attachment 31366 [details] [review] Bug 12872: Exclude limited number of fields from Zebra DOM indexing This is a pragmatic approach to exclude some fields from the Any index. The index_all routine in koha-indexdefs-to-zebra.xsl is renamed to index_most and includes a choose-when construct to exclude some fields from indexing. This seems a simpler approach than removing this routine and adding Any almost anywhere :) Test plan: Put some uncommon string like jh234gj23 into 952$e somewhere. Copy the xsl file to your Zebra indexing folders. Run the xsl to produce a new biblio-zebra-indexdefs.xsl. Restart Zebra and reindex. Check if you can still find the uncommon word. Now uncomment the line in koha-indexdefs-to-zebra.xsl. Rerun the xsl, restart and reindex. You should not find the uncommon word in the added 952$e.
Please signoff if this patch does what it promises in your setup, or please give suggestions how to handle this more elegantly. Thanks.
Created attachment 31367 [details] [review] Bug 12872: Exclude limited number of fields from Zebra DOM indexing This is a pragmatic approach to exclude some fields from the Any index. The index_all routine in koha-indexdefs-to-zebra.xsl is renamed to index_most and includes a choose-when construct to exclude some fields from indexing. This seems a simpler approach than removing this routine and adding Any almost anywhere :) Test plan: Put some uncommon string like jh234gj23 into 952$e somewhere. Copy the xsl file to your Zebra indexing folders. Run the xsl to produce a new biblio-zebra-indexdefs.xsl. Restart Zebra and reindex. Check if you can still find the uncommon word. Now uncomment the line in koha-indexdefs-to-zebra.xsl. Rerun the xsl, restart and reindex. You should not find the uncommon word in the added 952$e.
Oops, still the wrong version.
Created attachment 31368 [details] [review] Bug 12872: Exclude limited number of fields from Zebra DOM indexing This is a pragmatic approach to exclude some fields from the Any index. The index_all routine in koha-indexdefs-to-zebra.xsl is renamed to index_most and includes a choose-when construct to exclude some fields from indexing. This seems a simpler approach than removing this routine and adding Any almost anywhere :) Test plan: Put some uncommon string like jh234gj23 into 952$e somewhere. Copy the xsl file to your Zebra indexing folders. Run the xsl to produce a new biblio-zebra-indexdefs.xsl. Restart Zebra and reindex. Check if you can still find the uncommon word. Now uncomment the line in koha-indexdefs-to-zebra.xsl. Rerun the xsl, restart and reindex. You should not find the uncommon word in the added 952$e.
Comment on attachment 31368 [details] [review] Bug 12872: Exclude limited number of fields from Zebra DOM indexing This approach would work, but I would rather opt for something different.
On second thought, I am widening the scope of this report.
Created attachment 31527 [details] [review] Bug 12872: Exclude limited number of fields from Zebra indexing This patch adds the use of preference ZebraExclude to rebuild_zebra.pl. If that pref does not exist or is empty, behavior is unchanged. The pref allows you to exclude fields, specific subfields, and even field groups like 9XX from passing them to Zebra. The magic field XXX allows you to skip a whole record while indexing; this is of course only practical with a preceding specific condition. The pref may contain multiple lines with their own conditions. A line may start with a regex like /regex/:fieldlist. The regex is applied to the marcxml. If it matches, the field excludes are processed. Test plan: The plan includes a few rebuilds. It may be practical to run this test on a smaller test database. [1a] Edit ZebraExclude syspref. Set it to 590, 952e [1b] Edit a few records/items for these fields. [1c] Rebuild the zebra index. [1d] Check that info in 590 and 952e cannot be found in Zebra. [2a] Edit ZebraExclude syspref. Set it to /_skip_/: XXX [2b] Enter the _skip_ string into a few biblios (not in the same field). [2c] Rebuild the zebra index. [2d] Check that you cannot find these records in Zebra. [3a] Edit ZebraExclude syspref. Set it to 9XX [3b] Edit a few 9XX fields including some items (952). [3c] Rebuild the zebra index. [3d] Check that you cannot find 9XX info in Zebra. [4 ] Enter some other value in the pref. Edit, rebuild and check.
Created attachment 31528 [details] [review] Bug 12872: Unit tests for ZebraExclude preference Test plan: Run ZebraExclude.t
Created attachment 31529 [details] [review] Bug 12872: Db revision for ZebraExclude pref Test plan: Run install or upgrade. Check insertion of pref into database. Check editing the value via interface.
Will still add some documentation in online help later.
(In reply to M. de Rooy from comment #11) > Will still add some documentation in online help later. And still some POD in a follow-up for the module.
I've applied against master 3.17.00.027 Two patches need trivial rebasing. I pass the patch to "Patch doesn't apply" status. ------------------------------------ Some problems about "any" index (searching by keyword index). For example if the subfields 999c/999d exists in a record. In intranet, if I searched the biblionumber (=value of the number in the 2 subfields) by keyword index (on advanced search) or entering search keywords on the simple search, I found that record even if the "ZebraExclude" syspref has been set to 999. I found that record by sn index too, even if the "ZebraExclude" syspref has been set to 999. ------------------------------------
(In reply to Paola Rossi from comment #13) > I've applied against master 3.17.00.027 > > Two patches need trivial rebasing. > > I pass the patch to "Patch doesn't apply" status. > > ------------------------------------ > Some problems about "any" index (searching by keyword index). > > For example if the subfields 999c/999d exists in a record. > In intranet, if I searched the biblionumber (=value of the number in the 2 > subfields) by keyword index (on advanced search) or entering search keywords > on the simple search, I found that record even if the "ZebraExclude" > syspref has been set to 999. > I found that record by sn index too, even if the "ZebraExclude" syspref has > been set to 999. > ------------------------------------ Thanks for testing! The thing that you mentio about field 999 is by design. Please have a look at sub load_zebra_exclude. It uses GetMarcFromKohaField to get the tag that Koha uses to store biblionumber and biblioitemnumber. If that is indeed 999 (as it is by default), this tag will not be excluded since Zebra needs that information somewhere. I would not recommend to use the same tag where we internally store biblionumber for other data. So I did not see this as a problem. Could you retry with another field? Do not forget to reindex. Will rebase the last patch.
Created attachment 32578 [details] [review] Bug 12872: Db revision for ZebraExclude pref Test plan: Run install or upgrade. Check insertion of pref into database. Check editing the value via interface.
Created attachment 32584 [details] [review] Bug 12872: Exclude limited number of fields from Zebra indexing I've applied the patches against master 3.17.00.029 The case about 999 was the only problem I tried. Thanks, M. de Rooy, for the answer in comment 14. So everything is OK. And I pass the patch to "Signed Off" status.
Created attachment 32585 [details] Bug 12872: Unit tests for ZebraExclude preference
Created attachment 32586 [details] [review] Bug 12872: Db revision for ZebraExclude pref
Created attachment 32892 [details] [review] Bug 12872: Exclude limited number of fields from Zebra indexing This patch adds the use of preference ZebraExclude to rebuild_zebra.pl. If that pref does not exist or is empty, behavior is unchanged. The pref allows you to exclude fields, specific subfields, and even field groups like 9XX from passing them to Zebra. The magic field XXX allows you to skip a whole record while indexing; this is of course only practical with a preceding specific condition. The pref may contain multiple lines with their own conditions. A line may start with a regex like /regex/:fieldlist. The regex is applied to the marcxml. If it matches, the field excludes are processed. Test plan: The plan includes a few rebuilds. It may be practical to run this test on a smaller test database. [1a] Edit ZebraExclude syspref. Set it to 590, 952e [1b] Edit a few records/items for these fields. [1c] Rebuild the zebra index. [1d] Check that info in 590 and 952e cannot be found in Zebra. [2a] Edit ZebraExclude syspref. Set it to /_skip_/: XXX [2b] Enter the _skip_ string into a few biblios (not in the same field). [2c] Rebuild the zebra index. [2d] Check that you cannot find these records in Zebra. [3a] Edit ZebraExclude syspref. Set it to 9XX [3b] Edit a few 9XX fields including some items (952). [3c] Rebuild the zebra index. [3d] Check that you cannot find 9XX info in Zebra. [4 ] Enter some other value in the pref. Edit, rebuild and check. Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Created attachment 32893 [details] [review] Bug 12872: Unit tests for ZebraExclude preference Test plan: Run ZebraExclude.t Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Created attachment 32894 [details] [review] Bug 12872: Db revision for ZebraExclude pref Test plan: Run install or upgrade. Check insertion of pref into database. Check editing the value via interface. Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Trivial rebase, had to fight with the mbox attachment that should have been a patch..
This needs some end-user documentation. There really is nothing telling the user what to put in the system preference specifically. On the other hand, would it be better to use command line options for rebuild_zebra.pl for this? Considering any changes require a full rebuild, it kind of feels that way to me. Perhaps the settings could be stored in a file in etc if it's too complicated for the command line.
Why is this feature needed? Cant you just change the Zebra index definitions?
(In reply to Olli-Antti Kivilahti from comment #24) > Why is this feature needed? > Cant you just change the Zebra index definitions? Have a look at some of the obsoleted patches too. I used another approach there.
Hi Marcel, would this apply to keyword searches (any index)?
(In reply to Katrin Fischer from comment #26) > Hi Marcel, would this apply to keyword searches (any index)? Not sure what you exactly mean, but the idea was that you do not want any to include all fields whatsoever. You may want to exclude some fields from a general full-text search.
That's what I meant actually - thx! This is definitely interesting. I generally feel that command line options are not the way to go with more and more multi instance setups. I'd prefer the system preferences or another way to define this per instance easily.
Hi Marcel and Kyle, what's needed to get this moving again? Would love to be able to exclude the internal notes from being indexed (bug 15050)
(In reply to Katrin Fischer from comment #29) > Hi Marcel and Kyle, what's needed to get this moving again? > Would love to be able to exclude the internal notes from being indexed (bug > 15050) Hard to say. Another question is: If we exclude field A from Zebra, do we need to exclude it from ElasticSearch too ? If we add a pref or a config var, it is on the instance level. We probably don't want it to be on search engine level ? Nick: Could you please report the current status of Elastic in this regard? How can I exclude a specific field from indexing per INSTANCE in Elastic now ?
(In reply to Marcel de Rooy from comment #30) > (In reply to Katrin Fischer from comment #29) > > Hi Marcel and Kyle, what's needed to get this moving again? > > Would love to be able to exclude the internal notes from being indexed (bug > > 15050) > Hard to say. > > Another question is: If we exclude field A from Zebra, do we need to exclude > it from ElasticSearch too ? If we add a pref or a config var, it is on the > instance level. We probably don't want it to be on search engine level ? > > Nick: Could you please report the current status of Elastic in this regard? > How can I exclude a specific field from indexing per INSTANCE in Elastic now > ? I would say yes, if excluding from Zebra we should exclude from Elastic, I am having a hard time thinking of a reason you would only want to exclude form one search engine. Currently ES implementation doesn't support this, we push the entire record into the '_all' field. Catmandu fixes do allow exclusion of fields I believe, so we should be able to add the ability to exclude fields into the configuration section, or in a syspref, to allow this per instance
(In reply to Marcel de Rooy from comment #30) > (In reply to Katrin Fischer from comment #29) > > Hi Marcel and Kyle, what's needed to get this moving again? > > Would love to be able to exclude the internal notes from being indexed (bug > > 15050) > Hard to say. > > Another question is: If we exclude field A from Zebra, do we need to exclude > it from ElasticSearch too ? If we add a pref or a config var, it is on the > instance level. We probably don't want it to be on search engine level ? > > Nick: Could you please report the current status of Elastic in this regard? > How can I exclude a specific field from indexing per INSTANCE in Elastic now > ? Definitely worth re thinking on ES as we could have separate indexes for intranet and OPAC. Filtering should be achieved using the record processor filter over MARC records we already have, before sending for indexing.
I think with Zebra separate complete indexes would need twice as much space, that might be an issue for bigger installations. Also needs separate reindexing etc. Using the record filter would require that. Having some indexes separate for intranet/staff might be less resource intensive.