Bug 12872 - Exclude some fields from Zebra indexing
Summary: Exclude some fields from Zebra indexing
Status: CLOSED WONTFIX
Alias: None
Product: Koha
Classification: Unclassified
Component: Architecture, internals, and plumbing (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal (vote)
Assignee: Marcel de Rooy
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-09-04 11:18 UTC by Marcel de Rooy
Modified: 2022-12-12 21:25 UTC (History)
7 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 12872: Exclude limited number of fields from Zebra DOM indexing (3.22 KB, patch)
2014-09-04 11:47 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 12872: Exclude limited number of fields from Zebra DOM indexing (3.23 KB, patch)
2014-09-04 12:10 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 12872: Exclude limited number of fields from Zebra DOM indexing (3.22 KB, patch)
2014-09-04 12:32 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 12872: Exclude limited number of fields from Zebra indexing (9.06 KB, patch)
2014-09-11 13:28 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 12872: Unit tests for ZebraExclude preference (6.07 KB, patch)
2014-09-11 13:28 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 12872: Db revision for ZebraExclude pref (3.10 KB, patch)
2014-09-11 13:28 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 12872: Db revision for ZebraExclude pref (3.13 KB, patch)
2014-10-22 14:03 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 12872: Exclude limited number of fields from Zebra indexing (9.33 KB, patch)
2014-10-22 15:52 UTC, Paola Rossi
Details | Diff | Splinter Review
Bug 12872: Unit tests for ZebraExclude preference (6.25 KB, application/mbox)
2014-10-22 15:52 UTC, Paola Rossi
Details
Bug 12872: Db revision for ZebraExclude pref (3.25 KB, patch)
2014-10-22 15:53 UTC, Paola Rossi
Details | Diff | Splinter Review
Bug 12872: Exclude limited number of fields from Zebra indexing (9.12 KB, patch)
2014-10-29 12:36 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 12872: Unit tests for ZebraExclude preference (6.12 KB, patch)
2014-10-29 12:38 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 12872: Db revision for ZebraExclude pref (3.33 KB, patch)
2014-10-29 12:46 UTC, Marcel de Rooy
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Marcel de Rooy 2014-09-04 11:18:40 UTC
Here is another approach for excluding a limited number of fields from Zebra DOM indexing. (Check the see also reports too.)

I rename index_all to index_most and add a few lines to that named template.

This will not win beauty prices, but imo it is easier than adding any or anywhere to a large number of fields.

What do you think?
Comment 1 Marcel de Rooy 2014-09-04 11:47:07 UTC Comment hidden (obsolete)
Comment 2 Marcel de Rooy 2014-09-04 11:54:16 UTC
Please signoff if this patch does what it promises in your setup, or please give suggestions how to handle this more elegantly. Thanks.
Comment 3 Marcel de Rooy 2014-09-04 12:10:54 UTC Comment hidden (obsolete)
Comment 4 Marcel de Rooy 2014-09-04 12:23:47 UTC
Oops, still the wrong version.
Comment 5 Marcel de Rooy 2014-09-04 12:32:24 UTC Comment hidden (obsolete)
Comment 6 Marcel de Rooy 2014-09-08 12:57:06 UTC
Comment on attachment 31368 [details] [review]
Bug 12872: Exclude limited number of fields from Zebra DOM indexing

This approach would work, but I would rather opt for something different.
Comment 7 Marcel de Rooy 2014-09-11 12:53:55 UTC
On second thought, I am widening the scope of this report.
Comment 8 Marcel de Rooy 2014-09-11 13:28:13 UTC Comment hidden (obsolete)
Comment 9 Marcel de Rooy 2014-09-11 13:28:16 UTC Comment hidden (obsolete)
Comment 10 Marcel de Rooy 2014-09-11 13:28:20 UTC Comment hidden (obsolete)
Comment 11 Marcel de Rooy 2014-09-11 13:29:03 UTC
Will still add some documentation in online help later.
Comment 12 Marcel de Rooy 2014-09-11 13:31:38 UTC
(In reply to M. de Rooy from comment #11)
> Will still add some documentation in online help later.

And still some POD in a follow-up for the module.
Comment 13 Paola Rossi 2014-10-17 16:10:21 UTC
I've applied against master 3.17.00.027

Two patches need trivial rebasing.

I pass the patch to "Patch doesn't apply" status.

------------------------------------
Some problems about "any" index (searching by keyword index).

For example if the subfields 999c/999d exists in a record.
In intranet, if I searched the biblionumber (=value of the number in the 2 subfields) by keyword index (on advanced search) or entering search keywords on  the simple search, I found that record even if the "ZebraExclude" syspref has been set to 999.
I found that record by sn index too, even if the "ZebraExclude" syspref has been set to 999.
------------------------------------
Comment 14 Marcel de Rooy 2014-10-22 14:02:00 UTC
(In reply to Paola Rossi from comment #13)
> I've applied against master 3.17.00.027
> 
> Two patches need trivial rebasing.
> 
> I pass the patch to "Patch doesn't apply" status.
> 
> ------------------------------------
> Some problems about "any" index (searching by keyword index).
> 
> For example if the subfields 999c/999d exists in a record.
> In intranet, if I searched the biblionumber (=value of the number in the 2
> subfields) by keyword index (on advanced search) or entering search keywords
> on  the simple search, I found that record even if the "ZebraExclude"
> syspref has been set to 999.
> I found that record by sn index too, even if the "ZebraExclude" syspref has
> been set to 999.
> ------------------------------------

Thanks for testing!
The thing that you mentio about field 999 is by design. Please have a look at sub load_zebra_exclude. It uses GetMarcFromKohaField to get the tag that Koha uses to store biblionumber and biblioitemnumber. If that is indeed 999 (as it is by default), this tag will not be excluded since Zebra needs that information somewhere.
I would not recommend to use the same tag where we internally store biblionumber for other data. So I did not see this as a problem.
Could you retry with another field? Do not forget to reindex.
Will rebase the last patch.
Comment 15 Marcel de Rooy 2014-10-22 14:03:50 UTC Comment hidden (obsolete)
Comment 16 Paola Rossi 2014-10-22 15:52:12 UTC Comment hidden (obsolete)
Comment 17 Paola Rossi 2014-10-22 15:52:51 UTC Comment hidden (obsolete)
Comment 18 Paola Rossi 2014-10-22 15:53:23 UTC Comment hidden (obsolete)
Comment 19 Marcel de Rooy 2014-10-29 12:36:46 UTC
Created attachment 32892 [details] [review]
Bug 12872: Exclude limited number of fields from Zebra indexing

This patch adds the use of preference ZebraExclude to rebuild_zebra.pl.
If that pref does not exist or is empty, behavior is unchanged.

The pref allows you to exclude fields, specific subfields, and even
field groups like 9XX from passing them to Zebra. The magic field XXX allows
you to skip a whole record while indexing; this is of course only
practical with a preceding specific condition.

The pref may contain multiple lines with their own conditions. A line may
start with a regex like /regex/:fieldlist. The regex is applied to the
marcxml. If it matches, the field excludes are processed.

Test plan:
The plan includes a few rebuilds. It may be practical to run this test on
a smaller test database.

[1a] Edit ZebraExclude syspref. Set it to 590, 952e
[1b] Edit a few records/items for these fields.
[1c] Rebuild the zebra index.
[1d] Check that info in 590 and 952e cannot be found in Zebra.

[2a] Edit ZebraExclude syspref. Set it to /_skip_/: XXX
[2b] Enter the _skip_ string into a few biblios (not in the same field).
[2c] Rebuild the zebra index.
[2d] Check that you cannot find these records in Zebra.

[3a] Edit ZebraExclude syspref. Set it to 9XX
[3b] Edit a few 9XX fields including some items (952).
[3c] Rebuild the zebra index.
[3d] Check that you cannot find 9XX info in Zebra.

[4 ] Enter some other value in the pref. Edit, rebuild and check.

Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Comment 20 Marcel de Rooy 2014-10-29 12:38:29 UTC
Created attachment 32893 [details] [review]
Bug 12872: Unit tests for ZebraExclude preference

Test plan:
Run ZebraExclude.t

Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Comment 21 Marcel de Rooy 2014-10-29 12:46:04 UTC
Created attachment 32894 [details] [review]
Bug 12872: Db revision for ZebraExclude pref

Test plan:
Run install or upgrade. Check insertion of pref into database.
Check editing the value via interface.

Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Comment 22 Marcel de Rooy 2014-10-29 12:48:04 UTC
Trivial rebase, had to fight with the mbox attachment that should have been a patch..
Comment 23 Kyle M Hall 2014-10-31 15:15:36 UTC
This needs some end-user documentation. There really is nothing telling the user what to put in the system preference specifically.

On the other hand, would it be better to use command line options for rebuild_zebra.pl for this? Considering any changes require a full rebuild, it kind of feels that way to me. Perhaps the settings could be stored in a file in  etc if it's too complicated for the command line.
Comment 24 Olli-Antti Kivilahti 2014-11-24 09:37:06 UTC
Why is this feature needed?
Cant you just change the Zebra index definitions?
Comment 25 Marcel de Rooy 2014-11-24 09:45:56 UTC
(In reply to Olli-Antti Kivilahti from comment #24)
> Why is this feature needed?
> Cant you just change the Zebra index definitions?

Have a look at some of the obsoleted patches too. I used another approach there.
Comment 26 Katrin Fischer 2017-04-18 15:21:21 UTC
Hi Marcel, would this apply to keyword searches (any index)?
Comment 27 Marcel de Rooy 2017-04-19 11:17:54 UTC
(In reply to Katrin Fischer from comment #26)
> Hi Marcel, would this apply to keyword searches (any index)?

Not sure what you exactly mean, but the idea was that you do not want any to include all fields whatsoever. You may want to exclude some fields from a general full-text search.
Comment 28 Katrin Fischer 2017-04-19 11:52:41 UTC
That's what I meant actually - thx! This is definitely interesting.

I generally feel that command line options are not the way to go with more and more multi instance setups. I'd prefer the system preferences or another way to define this per instance easily.
Comment 29 Katrin Fischer 2017-08-17 22:26:15 UTC
Hi Marcel and Kyle, what's needed to get this moving again? 
Would love to be able to exclude the internal notes from being indexed (bug 15050)
Comment 30 Marcel de Rooy 2017-08-21 12:28:53 UTC
(In reply to Katrin Fischer from comment #29)
> Hi Marcel and Kyle, what's needed to get this moving again? 
> Would love to be able to exclude the internal notes from being indexed (bug
> 15050)
Hard to say.

Another question is: If we exclude field A from Zebra, do we need to exclude it from ElasticSearch too ? If we add a pref or a config var, it is on the instance level. We probably don't want it to be on search engine level ?

Nick: Could you please report the current status of Elastic in this regard? How can I exclude a specific field from indexing per INSTANCE in Elastic now ?
Comment 31 Nick Clemens 2017-08-21 12:42:19 UTC
(In reply to Marcel de Rooy from comment #30)
> (In reply to Katrin Fischer from comment #29)
> > Hi Marcel and Kyle, what's needed to get this moving again? 
> > Would love to be able to exclude the internal notes from being indexed (bug
> > 15050)
> Hard to say.
> 
> Another question is: If we exclude field A from Zebra, do we need to exclude
> it from ElasticSearch too ? If we add a pref or a config var, it is on the
> instance level. We probably don't want it to be on search engine level ?
> 
> Nick: Could you please report the current status of Elastic in this regard?
> How can I exclude a specific field from indexing per INSTANCE in Elastic now
> ?

I would say yes, if excluding from Zebra we should exclude from Elastic, I am having a hard time thinking of a reason you would only want to exclude form one search engine.

Currently ES implementation doesn't support this, we push the entire record into the '_all' field.

Catmandu fixes do allow exclusion of fields I believe, so we should be able to add the ability to exclude fields into the configuration section, or in a syspref, to allow this per instance
Comment 32 Tomás Cohen Arazi 2017-08-21 12:57:56 UTC
(In reply to Marcel de Rooy from comment #30)
> (In reply to Katrin Fischer from comment #29)
> > Hi Marcel and Kyle, what's needed to get this moving again? 
> > Would love to be able to exclude the internal notes from being indexed (bug
> > 15050)
> Hard to say.
> 
> Another question is: If we exclude field A from Zebra, do we need to exclude
> it from ElasticSearch too ? If we add a pref or a config var, it is on the
> instance level. We probably don't want it to be on search engine level ?
> 
> Nick: Could you please report the current status of Elastic in this regard?
> How can I exclude a specific field from indexing per INSTANCE in Elastic now
> ?

Definitely worth re thinking on ES as we could have separate indexes for intranet and OPAC. Filtering should be achieved using the record processor filter over MARC records we already have, before sending for indexing.
Comment 33 Katrin Fischer 2017-08-21 13:17:44 UTC
I think with Zebra separate complete indexes would need twice as much space, that might be an issue for bigger installations. Also needs separate reindexing etc. Using the record filter would require that. 
Having some indexes separate for intranet/staff might be less resource intensive.