Bug 8962 - Restore Any index behavior and add Anywhere index
Summary: Restore Any index behavior and add Anywhere index
Status: CLOSED WONTFIX
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: Main
Hardware: All All
: P4 enhancement (vote)
Assignee: Fridolin Somers
QA Contact:
URL:
Keywords:
: 11291 (view as bug list)
Depends on: 11202
Blocks: 9234
  Show dependency treegraph
 
Reported: 2012-10-24 18:07 UTC by Mathieu Saby
Modified: 2019-10-14 19:57 UTC (History)
18 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
[PATCH] Bug 8962 : Create a new anywhere index (draft patch) (32.36 KB, patch)
2012-11-12 19:05 UTC, Mathieu Saby
Details | Diff | Splinter Review
[PATCH 1/3] Bug 8962: Changes to CCL and templates (3.03 KB, patch)
2013-01-26 19:45 UTC, Mathieu Saby
Details | Diff | Splinter Review
[PATCH 3/3] Bug 8962: Marc21GRS1 (29.23 KB, patch)
2013-01-26 19:48 UTC, Mathieu Saby
Details | Diff | Splinter Review
[PATCH 2/3] Bug 8962: UnimarcGRS1 (41.09 KB, patch)
2013-01-26 19:49 UTC, Mathieu Saby
Details | Diff | Splinter Review
[PATCH 1/3] Bug 8962: Changes to CCL and templates (3.91 KB, patch)
2013-10-14 21:42 UTC, Mathieu Saby
Details | Diff | Splinter Review
[PATCH 3/3] Bug 8962: Unimarc biblios and authorithies (GRS1) (44.45 KB, patch)
2013-10-14 21:43 UTC, Mathieu Saby
Details | Diff | Splinter Review
[PATCH 2/3] Bug 8962: Marc21 biblios and authorithies (GRS1) (45.18 KB, patch)
2013-10-14 21:48 UTC, Mathieu Saby
Details | Diff | Splinter Review
[PATCH 3/3] Bug 8962: Unimarc biblios and authorithies (GRS1) (44.83 KB, patch)
2013-11-03 18:16 UTC, Mathieu Saby
Details | Diff | Splinter Review
[PATCH 3/3] Bug 8962: Unimarc biblios and authorithies (GRS1) (46.09 KB, patch)
2013-11-22 21:59 UTC, Mathieu Saby
Details | Diff | Splinter Review
[PATCH 3/3] Bug 8962: Unimarc biblios and authorithies (GRS1) (45.97 KB, patch)
2013-11-23 00:25 UTC, Mathieu Saby
Details | Diff | Splinter Review
[PATCH 1/3] Bug 8962: Changes to CCL.properties, bib1 for authorities, and templates (4.36 KB, patch)
2013-11-27 23:51 UTC, Mathieu Saby
Details | Diff | Splinter Review
[SIGNED OFF] Bug 8962: Changes to CCL.properties, bib1 for authorities, and templates (4.87 KB, patch)
2014-02-26 17:31 UTC, Martin Renvoize
Details | Diff | Splinter Review
[SIGNED OFF] Bug 8962: Changes to CCL.properties, bib1 for authorities, and templates (4.87 KB, patch)
2014-02-26 17:32 UTC, Martin Renvoize
Details | Diff | Splinter Review
[SIGNED OFF] Bug 8962: Unimarc biblios and authorithies (GRS1) (46.13 KB, patch)
2014-02-26 17:35 UTC, Martin Renvoize
Details | Diff | Splinter Review
[SIGNED OFF] Bug 8962: Marc21 biblios and authorithies (GRS1) (45.27 KB, patch)
2014-02-26 17:35 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 8962: Marc21 biblios (DOM) (148.98 KB, patch)
2014-02-26 17:36 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 8962: Restore Any index behavior and add Anywhere index (2.55 KB, patch)
2017-12-22 15:24 UTC, Fridolin Somers
Details | Diff | Splinter Review
Bug 8962: Changes to CCL.properties, bib1 for authorities, and templates (4.83 KB, patch)
2018-04-04 12:35 UTC, Fridolin Somers
Details | Diff | Splinter Review
Bug 8962: Restore Any index behavior and add Anywhere index (2.55 KB, patch)
2018-04-04 12:35 UTC, Fridolin Somers
Details | Diff | Splinter Review
Bug 8962: Changes to CCL.properties, bib1 for authorities, and templates (4.89 KB, patch)
2018-05-25 19:30 UTC, Charles Farmer
Details | Diff | Splinter Review
Bug 8962: Restore Any index behavior and add Anywhere index (2.61 KB, patch)
2018-05-25 19:30 UTC, Charles Farmer
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Mathieu Saby 2012-10-24 18:07:51 UTC
In the current record.abs, the index "any" is defined as "all any". So, all the fields indexed in other indexes are also indexed in "any".
So, for example, in my unimarc catalogue, if I search for "fre" in the opac, I get 342825 results, because it is the code for french language in 101a field, index in "ln".
Idem if I search for "lc" or "rameau" (equivalent to LCSH in France), or "texte"
And with fuzzy or autotruncate activated, it is, very very annoying.

My idea is :
- to keep the "all any"
- to create a more specific index, matching only important field, but in any event NOT matching coded fields and some boring notes...
- use this index in opac simple search (and maybe in staff interface simple search but this could be discussed...)

Todo : 
- check bib1.att & ccl.properties to see if some index not used in Koha today could be used. If nothing usable, create completely new index.
- edit record.abs for unimarc, marc21 and normarc biblios
- edit DOM config files for unimarc, marc21 and normarc biblios
- check if it is necessary to edit zebra config files for authorities
- edit the templates
- maybe edit some perl files

Mathieu Saby
Rennes 2 University
Comment 1 Jared Camins-Esakov 2012-10-26 12:44:52 UTC
It may be preferable to use Any for the new index, and convert the old index to Anywhere. From the Attribute Set Bib-1 Semantics document (http://www.loc.gov/z3950/agency/bib1.html):
Any                  1016  The record is selected if there
                           exists a Use attribute that the
                           target supports (and considers
                           appropriate - see note 1) such
                           that the record would be
                           selected if the target were to
                           substitute that attribute.
       Notes:
        (1) When the origin uses 'any' the intent is that the target
            locate records via commonly used access points. The target
            may define 'any' to refer to a selected set of Use
            attributes corresponding to its commonly used access points.
        (2) In set terminology: when Any is the Use attribute, the set
            of records selected is the union of the sets of records
            selected by each of the (appropriate) Use attributes that
            the target supports.

Anywhere             1035  The record is selected if the
                           term value (as qualified by the
                           other attributes) occurs anywhere
                           in the record.

            Note: A target might choose to support 'Anywhere' only in
            combination with specific (non-Use) attributes. For example, a
            target might support 'Anywhere' only in combination with the
            Relation attribute 'AlwaysMatches' (see below), to locate all
            records in a database.

       Notes on relationship of Any and Anywhere:
        (1) A target may support Any but not Anywhere, or vice versa, or
            both.  However, if a target supports both, then it should
            exclude 'Anywhere' from the list of Use attributes
            corresponding to 'Any' (if it does not do so, then the set
            of records located by 'Any' will be a superset of those
            located by 'Anywhere').
        (2) A distinction between the two attributes may be informally
            expressed as follows: 'anywhere' might result in more
            expensive searching than 'any'; if the target (and origin)
            support both 'any' and 'anywhere', if the origin uses 'Any'
            (rather than 'Anywhere') it is asking the target to locate
            the term only if it can do so relatively inexpensively.
Comment 2 Mathieu Saby 2012-11-12 19:05:47 UTC Comment hidden (obsolete)
Comment 3 Kyle M Hall 2012-12-07 16:43:04 UTC
I'm getting no results with "Any word (all indexes)". I copied the new ccl.properties and record.abs over my old ones in koha-dev, and reindexed zebra. What am I missing?
Comment 4 Mathieu Saby 2012-12-09 16:55:13 UTC
Hello. I will rebase the patch once bz 8984 is pushed (it will break this one), and I will see why it is not working.
M. Saby
Comment 5 Mathieu Saby 2012-12-12 23:04:22 UTC
I tried to apply my patch. With unimarc flavor, it seems to work : in advanced search in staff interface, "Keyword" index gives me less results as the new "Any word (all indexes)".

Maybe it is broken for marc21...
Kyle, did you notice some unusual warns when you rebuild zebra ?

Mathieu Saby
Comment 6 Kyle M Hall 2012-12-13 15:24:43 UTC
I don't recall seeing anything unusual when I ran rebuild_zebra.pl

(In reply to comment #5)
> I tried to apply my patch. With unimarc flavor, it seems to work : in
> advanced search in staff interface, "Keyword" index gives me less results as
> the new "Any word (all indexes)".
> 
> Maybe it is broken for marc21...
> Kyle, did you notice some unusual warns when you rebuild zebra ?
> 
> Mathieu Saby
Comment 7 Mathieu Saby 2012-12-13 15:27:54 UTC
And you tested on a Marc21 base I suppose?
M. saby
Comment 8 Kyle M Hall 2012-12-13 16:06:21 UTC
That is correct.

(In reply to comment #7)
> And you tested on a Marc21 base I suppose?
> M. saby
Comment 9 Mathieu Saby 2013-01-26 19:42:38 UTC
It is still a draft, but I have rebased and completed my patch, and splitted it in 3 patches (more easy to test and to rewrite).
It is still only for GRS1 indexing.
1st patch contains CCL changes and templates changes/
2st patch contains Marc21 changes (biblio and authorities)
3rd patch contains Unimarc changes (biblio and authorities)

What should be different from before ?
This patch adds a new index Anywhere in ccl.properties and add "any word" option in intranet search for using this index.
Default OPAC and intranet search in still using CCL "Any" index.
In record.abs for biblio and authorities, for the 2 marc flavor : 
- Defines a new index "anywhere" for indexing all biblio and authorities subfields.
- Old "any" index is slightly more selective than before

Details of changes for Unimarc : 
- for authorities : all fields indexed in any except 001 and notes
- for biblios  : index 01X, 07X, 7XX except $9, 200, 205, 210c, 225, 328, 330, 4XX, 5XX except $9, 6XX except $9 and 67/68X, 995fkr

For Marc21 : 
- for authorities : all fields indexed in any except 001
- for biblios : fields indexed in any : 02X, 030, 110, 111, 130, 21X, 22X, 24X, 260b, 4XX, 505, 520, 6XX, 7XX, 8XX, 952p

=> so, for example, "any" does NOT indexes
* notes (except 328/330 in Unimarc and 505/520 in Marc21)
* date of publication
* place of publication (town of the publisher)
* extent
* item notes
* biblionumber

To test :
- apply the 3 patches
- rebuild zebra biblios index
- rebuild zebra authorities index
- make some searches : for example, check if place of edition or notes are indexed...

If it works, we need to 
- discuss the choice of fields that must remain in "any"
- do the same for DOM indexing
- do the same for Normarc

M. Saby
Rennes 2 University
Comment 10 Mathieu Saby 2013-01-26 19:45:53 UTC Comment hidden (obsolete)
Comment 11 Mathieu Saby 2013-01-26 19:48:26 UTC Comment hidden (obsolete)
Comment 12 Mathieu Saby 2013-01-26 19:49:19 UTC Comment hidden (obsolete)
Comment 13 Mathieu Saby 2013-03-22 17:27:35 UTC
you can test it, but please do not sign this patch before other patches regarding Unimarc Zebra are pushed : 9830, 9352, 9828 (there will be a lot of conflicts, and the logic of these patchs are diffrent from this one).

Mathieu Saby
Comment 14 Magnus Enger 2013-08-09 11:23:16 UTC
Adding some bugs to "Depends on", based on earlier comments.
Comment 15 Magnus Enger 2013-08-09 11:24:16 UTC
Setting status t o(In reply to mathieu saby from comment #13)
> you can test it, but please do not sign this patch before other patches
> regarding Unimarc Zebra are pushed : 9830, 9352, 9828 (there will be a lot
> of conflicts, and the logic of these patchs are diffrent from this one).

Setting the status to BLOCKED since thos bugs are still "needs signoff".
Comment 16 Mathieu Saby 2013-08-19 08:32:45 UTC
Thank you Magnus.
But even if some conflicts could occur for UNIMARC (and maybe for Marc21 : I did not check, and the patch is rather old now...), you can try to  do a 1st test anyway, or just give your opinion about the logic of the patch ;-)

Mathieu
Comment 17 Mathieu Saby 2013-10-14 21:42:59 UTC Comment hidden (obsolete)
Comment 18 Mathieu Saby 2013-10-14 21:43:49 UTC Comment hidden (obsolete)
Comment 19 Mathieu Saby 2013-10-14 21:48:05 UTC Comment hidden (obsolete)
Comment 20 Mathieu Saby 2013-10-14 21:54:56 UTC
I rewrote the patchs and add test plan.
Only for GRS1 for the moment.
My Koha VM is a UNIMARC DOM, so I cannot test it myself, but I hope it will work.

I made 3 patchs so that somebody can test and sign Marc21 patch and somebody else UNIMARC patch.
The list of fields I kept in "any" can be discussed, but once the patch is pushed, each library will have the ability of deleting or adding fields by modifying its zebra conf files.
When we all agree on the list of fields to keep, I will make patchs for DOM.

M. Saby
Comment 21 Mathieu Saby 2013-10-14 21:56:28 UTC
I add Galen, as we had elsewhere some discussion about this patch.

M. Saby
Comment 22 Mathieu Saby 2013-10-14 22:00:18 UTC
Of course, in test plan of UNIMARC patch, read "In a UNIMARC GRS-1 Koha" (and not Marc21!)

M. Saby
Comment 23 Mathieu Saby 2013-10-15 07:23:48 UTC
Maybe I could remove the changes made for authorities, as they are not really needed, and create an other bug for improving auth indexing (index coded fields, marc21 notes etc)

M. saby
Comment 24 I'm just a bot 2013-10-30 07:14:14 UTC
Applying: Bug 8962: Unimarc biblios and authorithies (GRS1)
Using index info to reconstruct a base tree...
M	etc/zebradb/marc_defs/unimarc/biblios/record.abs
Falling back to patching base and 3-way merge...
Auto-merging etc/zebradb/marc_defs/unimarc/biblios/record.abs
CONFLICT (content): Merge conflict in etc/zebradb/marc_defs/unimarc/biblios/record.abs
Patch failed at 0001 Bug 8962: Unimarc biblios and authorithies (GRS1)
The copy of the patch that failed is found in:
   /home/christopher/git/koha/.git/rebase-apply/patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
Comment 25 Mathieu Saby 2013-11-03 18:16:35 UTC Comment hidden (obsolete)
Comment 26 Mathieu Saby 2013-11-22 21:59:25 UTC Comment hidden (obsolete)
Comment 27 I'm just a bot 2013-11-22 22:45:09 UTC
Patch applied cleanly, go forth and signoff
Comment 28 I'm just a bot 2013-11-22 22:45:52 UTC
Applying: Bug 8962: Unimarc biblios and authorithies (GRS1)
Repository lacks necessary blobs to fall back on 3-way merge.
Cannot fall back to three-way merge.
Patch failed at 0001 Bug 8962: Unimarc biblios and authorithies (GRS1)
The copy of the patch that failed is found in:
   /home/christopher/git/koha/.git/rebase-apply/patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
Comment 29 Mathieu Saby 2013-11-23 00:19:39 UTC
For DOM, (not written yet) it will depend on bug 11291
Comment 30 Mathieu Saby 2013-11-23 00:25:04 UTC Comment hidden (obsolete)
Comment 31 Mathieu Saby 2013-11-27 23:02:39 UTC
I've removed the dependency on bug 11291. It was not a "real" dependancy. I just want to wait for bugs 11291 and 11202 being pushed to master before writing the DOM followup for this patch.

I also wait for this patch to be tested and signed off with the current indexing style (GRS1) before porting it for DOM.

Mathieu
Comment 32 Mathieu Saby 2013-11-27 23:51:00 UTC Comment hidden (obsolete)
Comment 33 Martin Renvoize 2014-02-26 17:31:36 UTC Comment hidden (obsolete)
Comment 34 Martin Renvoize 2014-02-26 17:32:27 UTC
Created attachment 25672 [details] [review]
[SIGNED OFF] Bug 8962: Changes to CCL.properties, bib1 for authorities, and templates

This patch adds a new index Anywhere in ccl.properties in bib1 for authorities, and add "any word" option in intranet search for using this index.
Default OPAC and intranet search in still using CCL "Any" index.

Testing plan :
In a Marc21 GRS-1 Koha :
1/ make some biblio searches without selecting any index (Koha will use "any" index) and note the results. Ex : search "fre", "eng", "book", "history", "history book"
2/ apply the 3 patches
3/ copy the 4 new record.abs files (Marc21 auth, Marc21 bib, Unimarc auth, Unimarc bib) from your sources to the directory used by Zebra configuration in your Koha instance
4/ launch rebuild_zebra.pl -b -r -x -v
5/ check that a new choice "Any word (all indexes)" is visible in advanced search.
6/ make the same searches as 1/ : you should see less results (detailed test plan in patchs 2 and 3)
7/ in Staff interface adavanced search, select "Any word (all indexes)" and make the same searches as 1/ : you should see the same results (maybe with a slight differences, because more notes fields are indexed in Marc21)

In a UNIMARC GRS-1 Koha :
same test

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 35 Martin Renvoize 2014-02-26 17:35:55 UTC
Created attachment 25673 [details] [review]
[SIGNED OFF] Bug 8962: Unimarc biblios and authorithies (GRS1)

This patch depends on Bug 11202

This patch makes some changes UNIMARC GRS1 indexing, for indexing only some specific fields in "any" index.
The patch is fully testable, and can be signed off if it works.
But the list of fields I kept in "any" is rather subjective, and based as my experience as a library user:
It can be discussed in followups.

1. Defines a new index "anywhere" for indexing all biblio and authorities subfields.
2. Make "any" index more selective than before :
- for authorities : all fields indexed in any except 001 and notes (3XX)
- for biblios : fields indexed in any :
* biblio fields indexed in "any" :
** the more "user friendly" control numbers : 01X, 07X
** authors : 7XX except $9, 200$f$g
** titles : 4XX except $9, 5XX except $9, 200$a$c$d$e$h$i
** edition statement : 205
** series : 225, 410, 411
** subjects : 6XX except $9 and 67X/68X
** notes : only 327, 328, 314, 330
* items fields indexed in "any" :
** barcode : 995f
** callnumber : 995k

Test plan :
In a Marc21 GRS-1 Koha :
1/ apply patch 1 and this one
2/ in Staff interface, make some searches on authorities (choose "Keyword search" tab), including searches of words only present in 3XX fields.
3/ copy the 2 new record.abs files (UNIMARC auth, UNIMARC bib) from your sources to the directory used by Zebra configuration in your Koha instance
4/ launch rebuild_zebra.pl -b -r -x -v
5/ make the same authorities searches as 2/ : you should get the same results, except for searches based on words only present in 3XX fields : in that case, you should not see the auth records.
6/ in default search (without selecting any index), search for some record by
- its isbn
- its issn
- its title
- its author(s)
- its series name
- its publisher
- its summary (330 field)
- several subjects
- the barcode of one of its items
- the callnumber of one of its items
=> you should find the record each time
7/ in default search (without selecting any index), search for some record by
- the content of any 3XX fields (except 327,328,314,330)
- the number of pages or volumes (ex : "510 p." or "1 vol")
- its language code ("eng", "fre"...)
- the value of 200$b (like "Texte imprimé")
- its publication year
- its town of publication
- the content of a 670-680 field
- the content of a 995 subfield (except barcode and callnumber)
- the content of a 099 subfield
=> you should not find the record each time
8/ regression test : check that specific indexes are working as usual (title, subject, callnum, barcode, pubdate etc)

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 36 Martin Renvoize 2014-02-26 17:35:59 UTC
Created attachment 25674 [details] [review]
[SIGNED OFF] Bug 8962: Marc21 biblios and authorithies (GRS1)

This patch makes some changes MARC21 GRS1 indexing, for indexing only some specific fields in "any" index.
The patch is fully testable, and can be signed off if it works.
But the list of fields I kept in "any" is rather subjective, and based as my experience as a library user:
It can be discussed in followups.

1. Defines a new index "anywhere" for indexing all biblio and authorities subfields.
2. Make "any" index more selective than before :
- for authorities : all fields indexed in any except 001
- for biblios : fields indexed in any :
* biblio fields indexed in "any" :
** the more "user friendly" control numbers : 020->030
** main entries (authors and uniform title) : 1XX
**  title and Title-Related Fields : 20X-24X
** publisher : 260$b
** series : 4XX
** some notes : 502 (dissertation), 520 (abstract), 505 (formated content), 511 (participant)
** subjects : 6XX
** added Entry Fields : 70X-75X
** linking Entries-General Information : 76X-78X
** series Added Entry Fields : 80X-83X
* item fields indexed in "any" :
** callnumber : 952$o
** barcode : 952$p
3. Also index all the Marc21 notes (5XX) in "Notes" index (same logic as current 3XX fields indexing in UNIMARC)
- new fields not indexed before : 504,506,507,508,511,513,514,515,516,518,521,522,524,525,530,533,534,535, 536,538,540,541,542,544,545,547,550,552,555,556,561,562,563,567,580,581,583,584,585,586,588
- fields indexed in "Notes" in addition to a more specific index : 502,505,521,520,510,526
4. Also index all the Marc21 notes (5XX) in "Notes" index (same logic as current 3XX fields indexing in UNIMARC)
5. Adds a lot of comments to make easier further changes and error checking

Test plan :
In a Marc21 GRS-1 Koha :
1/ apply patch 1 and this one
2/ in Staff interface, make some searches on authorities (choose "Keyword search" tab)
3/ copy the 2 new record.abs files (Marc21 auth, Marc21 bib) from your sources to the directory used by Zebra configuration in your Koha instance
4/ launch rebuild_zebra.pl -b -r -x -v
5/ make the same authorities searches as 2/ : you should get the same results
5/ edit a record to put the value "xxxx" in newly indexed "5XX" field (for example in 530)
6/ search "note:xxxx" : you should find the record
7/ edit a record to put the value "xxxx" in a "5XX" field index in note in addition to a more specifi index (for example in 520)
8/ search "note:xxxx" : you should find the record
9/ in default search (without selecting any index), search for some record by
- its isbn
- its issn
- its title
- its author(s)
- its series name
- its publisher
- its summary (520 field)
- several subjects
- the barcode of one of its items
- the callnumber of one of its items
=> you should find the record each time
10/ in default search (without selecting any index), search for some record by
- the content of any 5XX fields (except 502,505,511,520)
- the number of pages (ex : "510p")
- its language code ("eng", "fre"...)
- its publication year
- its town of publication
- the content of a 05X-08X field
- the content of a 010-018 field
- the content of a 942 subfield
- the content of a 952 subfield (except barcode and callnumber)
=> you should not find the record each time
11/ regression test : check that specific indexes are working as usual (title, subject, callnum, barcode, pubdate etc)

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 37 Martin Renvoize 2014-02-26 17:36:06 UTC
Created attachment 25675 [details] [review]
Bug 8962: Marc21 biblios (DOM)

This followup adds DOM support (MARC21) to this patch series. The
same set of fields are indexed in the 'Any' index as were chosen
by Mathieu Saby for GRS-1 setups. An 'Anywhere' index was also
added to maintain backward compatability as per Mathieu's patches.

Test Plan: As for GRS-1, but with DOM enabled as aposed to GRS-1
Comment 38 David Cook 2014-06-19 05:07:56 UTC
Comment on attachment 25675 [details] [review]
Bug 8962: Marc21 biblios (DOM)

Review of attachment 25675 [details] [review]:
-----------------------------------------------------------------

This looks really good at a glance. I'll have to test this one sometime in the future.

I think some of the choices for the new "any" index might be a bit arbitrary, but we can fine tune that over time (or people can customize it as needed).

I noticed some Notes fields have been added as well...I assume this is just to bring DOM in line with GRS-1?

::: etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml
@@ +874,5 @@
>      <target_index>Subject:w</target_index>
>      <target_index>Subject:p</target_index>
> +    <target_index>any:w</target_index>
> +    <target_index>any:p</target_index>
> +    <target_index>any:w</target_index>

Looks like you accidentally doubled-up here.

@@ +1605,5 @@
>      <target_index>Local-classification:p</target_index>
>      <target_index>Local-classification:s</target_index>
> +    <target_index>any:w</target_index>
> +    <target_index>any:p</target_index>
> +    <target_index>any:s</target_index>

I actually don't think that we need to use the 'sort' register here, since we don't sort on the "any" index.

We need Local-classification:s, because we have 'Call number' sort on the OPAC, but that's the only reason. Probably safe to nix it here.

That said, no harm done by including it. It's probably just redundant.

http://www.indexdata.com/zebra/doc/administration-ranking.html
Comment 39 David Cook 2014-06-19 06:03:35 UTC
Martin, I noticed that you've only done biblios, but I believe that authorities are needed as well.

Hmm, the more I look at this...the more I think that it's sub-optimal to pepper "any:w" and "any:p" throughout "biblio-koha-indexdefs.xml".

It might make more sense to add <index_any> elements to "biblio-koha-indexdefs.xml" and "authority-koha-indexdefs.xml", and create a new template in "koha-indexdefs-to-zebra.xsl" to create the appropriate XSLTs as necessary. 

Galen, as it looks like you did a lot of work on "koha-indexdefs-to-zebra.xsl", what are your thoughts on doing it that way?

(As a sidebar, I think we could optimize "koha-indexdefs-to-zebra.xsl" a bit more in terms of how templates are applied, but also of the mind "if it ain't broke, don't fix it".)
Comment 40 Martin Renvoize 2014-06-19 15:57:28 UTC
Hi David, 

It's really great someone else is starting to look at this, we found it made a huge difference to search for our customers.

My patch simply took mathieu's work through the conversion to DOM process (which jcamins hates me doing ;) ).

I'd love to see it 'done right' but at the time, and still now unfortunately, I haven't had the time to really dead the dom configs and understand them enough to write them manually.

I also agree that this should probably be done for authorities to, but I hadn't gotten around to it either, as biblios we're my highest priority.

I'm all in favour of the plan to nuke the GRS-1 configs completely in favour of using DOM everywhere and therefore only having the one config style to learn.

Let me know if I can help with testing any of it.. but at he moment I'm a tad too snowed under to do any real work on coding it myself.
Comment 41 Mathieu Saby 2014-06-19 20:52:03 UTC
I thought I had done something for authorities, or tried. But I can't find bakc (except bug 11191 and bug 11053, but not directly related).
Anyway, biblio records indexing can be improved separately, and it is probably more important than authorities.

Mathieu
Comment 42 David Cook 2014-06-19 23:58:35 UTC
(In reply to Martin Renvoize from comment #40)
> Hi David, 
> 
> It's really great someone else is starting to look at this, we found it made
> a huge difference to search for our customers.
> 
> My patch simply took mathieu's work through the conversion to DOM process
> (which jcamins hates me doing ;) ).
> 
> I'd love to see it 'done right' but at the time, and still now
> unfortunately, I haven't had the time to really dead the dom configs and
> understand them enough to write them manually.
> 
> I also agree that this should probably be done for authorities to, but I
> hadn't gotten around to it either, as biblios we're my highest priority.
> 
> I'm all in favour of the plan to nuke the GRS-1 configs completely in favour
> of using DOM everywhere and therefore only having the one config style to
> learn.
> 
> Let me know if I can help with testing any of it.. but at he moment I'm a
> tad too snowed under to do any real work on coding it myself.

I just stumbled across it as I was looking for all "Search" bugs to test (since I've been looking at Search so much lately). That said, it does seem like it would be useful!

Ahhh, I gotcha. Fair enough about time. I'm about to work on other projects for the day, but I would be willing to look into this one next week probably.
Comment 43 Marcel de Rooy 2014-09-04 12:55:01 UTC
Martin: The first three patches are signed off already, the fourth patch is DOM only. Could we move your fourth patch to a new report and promote this report to Signed off?
Note also that patch 2 and 3 are GRS1, to be obsoleted in near future. I am not sure if it still makes sense to put this in master..

Spotted a typo in the first patch: Any world (all indexes) --> word perhaps :)

A question on the DOM patch: Any is now a subset of Anywhere, which is good. But I am not sure if everybody wants Anywhere by default (in terms of space, performance or privacy). Could we make it an installation choice somehow? If Anywhere should not be there, should we still do something to let Any be its fallback?
Comment 44 Mathieu Saby 2014-09-04 17:45:42 UTC
Hello

If you are interested in improving indexing, you can see Bug 9352 (the 3rd patch introduces a new logic for grouping subfields), and bug 9828

Mathieu
Comment 45 David Cook 2014-09-05 00:41:37 UTC
(In reply to M. de Rooy from comment #43)
> Martin: The first three patches are signed off already, the fourth patch is
> DOM only. Could we move your fourth patch to a new report and promote this
> report to Signed off?
> Note also that patch 2 and 3 are GRS1, to be obsoleted in near future. I am
> not sure if it still makes sense to put this in master..
> 
> Spotted a typo in the first patch: Any world (all indexes) --> word perhaps
> :)
> 
> A question on the DOM patch: Any is now a subset of Anywhere, which is good.
> But I am not sure if everybody wants Anywhere by default (in terms of space,
> performance or privacy). Could we make it an installation choice somehow? If
> Anywhere should not be there, should we still do something to let Any be its
> fallback?

Those are all really good questions.

After reviewing the patches a bit more, I don't think it actually makes sense to even add "anywhere" to GRS1, as it doesn't agree with the Bib-1 semantics (http://www.loc.gov/z3950/agency/bib1.html). 

The "Any" index/register should contains all the data contained in all the other indexes/registers (e.g. Title, Author, Subject, etc). 

[Note: this is how it currently is set up to work.]

The "Anywhere" index/register should contain the entire record including fields/subfields that aren't indexed anywhere else. 

[Note: while this is possible using DOM, I'm not sure that it is possible with GRS1. Even if it is, since we're deprecating it, it becomes unnecessary.]

It probably would be a good idea to make inclusion of Anywhere configurable (in terms of space, performance, and privacy). That's a really really really good point.

How to do that though... that could be trickier. When using DOM, we could disable Anywhere by default, and then say to run something like the following:

"xsltproc koha-indexdefs-to-zebra.xsl {biblio,authority}-koha-indexdefs.xml >
{biblio,authority}-zebra-indexdefs.xsl --stringparam anywhere index"

We would just need to add a xsl:param for "anywhere" into the koha-indexdefs-to-zebra.xsl. 

As for having "Any" be the fallback of "Anywhere"... you might be able to do something tricky in ccl.properties using aliases, but I'm not sure.
Comment 46 Martin Renvoize 2014-09-05 07:45:26 UTC
Wow there suddenly been allot of movement on this and related bugs!

It was my feeling that, at the time of writing the DOM follow-up, all indexing related bugs should be written to account for both DOM and GRS1 (We had real issues with the two configs falling out of sync very often).

However, with the imminent complete deprecation of GRS1 I'd actually prefer to see Patch 1 and 4 (and a patch 5 with Unimarc for DOM) end up as the final set.

Anywho, that's enough history. As for Davids comments, I'm not Zebra spec expert, but it was my belief at the time that the Any Index was in fact a wholly inclusive index and was therefore not especially useful, hence being very interested in a narrowing down of the fields present within it.

As for the Anywhere index, it sort of made sense at the time, basically to prevent a regression. I personally don't think it's a useful index, but that's just an opinion ;)

I'm gonna go read Mathieu's related bugs now.. see what they do and try to work out whether they're supplementary or as a posed to this bug.
Comment 47 Bernardo Gonzalez Kriegel 2015-05-14 11:44:15 UTC
This no longer applies

CONFLICT (content): Merge conflict in koha-tmpl/intranet-tmpl/prog/en/includes/search_indexes.inc


Also GRS-1 is deprecated
Comment 48 Mathieu Saby 2015-05-14 21:58:16 UTC
Hello
I won't be able to work on that bug anymore. Feel free to take my patches as a base for further work if you think they can be useful.

Mathieu
Comment 49 Fridolin Somers 2017-12-22 14:43:18 UTC
*** Bug 11291 has been marked as a duplicate of this bug. ***
Comment 50 Fridolin Somers 2017-12-22 14:53:18 UTC
I found this exact bug and found an easy way to propose a solution.

I set GRS1 patches has invalid.
Comment 51 Fridolin Somers 2017-12-22 15:03:56 UTC
> 
> The "Any" index/register should contains all the data contained in all the
> other indexes/registers (e.g. Title, Author, Subject, etc). 
> 
> The "Anywhere" index/register should contain the entire record including
> fields/subfields that aren't indexed anywhere else. 
> 

This is what I want to do so I changed the bug description.
Comment 52 Fridolin Somers 2017-12-22 15:24:35 UTC
Created attachment 70127 [details] [review]
Bug 8962: Restore Any index behavior and add Anywhere index

The "Any" index/register should contains all the data contained in all the other indexes/registers (e.g. Title, Author, Subject, etc).

The "Anywhere" index/register should contain the entire record including fields/subfields that aren't indexed anywhere else.

In GRS1 indexing the Any index was indeed only on fields defined as indexed.
This has changed in DOM where all data in MARC is indexed with Any.
This generates a lot of results when no index is defined in search.

This patch changes the DOM xsl config generation to add Any to each indexed data and add Anywhere to keep a way of searching on all data.

Test plan :
- Add a data on a field that is not indexed : for example "toto" in 998$z
- Add same data as the title of another record
- Index the records
- Search on toto without index (uses Any) => you get the 2 records
- Search on toto with title index => you get the record with "toto" as title
- Apply patch
- Update DOM xsl config files
- Reindex
- Search on toto without index (uses Any) => you get the record with "toto" as title
- Search on toto with title index => you get the record with "toto" as title
- Search on toto with Anywhere index => you get the 2 records
Comment 53 Fridolin Somers 2017-12-22 15:27:56 UTC
Once validated, we will patch the DOM XSL config files.

Maybe first patch needs some changes, let me know.
Comment 54 Magnus Enger 2017-12-28 10:28:42 UTC
(In reply to Fridolin SOMERS from comment #51)
> > 
> > The "Any" index/register should contains all the data contained in all the
> > other indexes/registers (e.g. Title, Author, Subject, etc). 
> > 
> > The "Anywhere" index/register should contain the entire record including
> > fields/subfields that aren't indexed anywhere else. 
> > 
> 
> This is what I want to do so I changed the bug description.

Which index will be used for the default/simple search from the OPAC front page? 

One of my least favourite "features" of search in Koha is that the default/simple search includes "Place of publication" (MARC21: 260$a). If you have a Norwegian catalogue, a lot of the books in it will have Oslo as Place of publication. This means that if you do a simple search for "oslo", because you want to find books about Oslo, you will get *a lot* of noise, because you not only find the books about Oslo, but also all the books published in Oslo. 

Any chance of fixing this as part of this bug?
Comment 55 Katrin Fischer 2017-12-28 21:55:48 UTC
Hi Magnus, I think this is out of scope if this aims to restore the former behaviour - you'd have to remove the place of publication from indexes.
Maybe a next step could be a configurable Any index?
Comment 56 Katrin Fischer 2017-12-28 21:57:44 UTC
As this would be a change of behavior now after a long time of dom indexing... should we make it possible to switch the one that is used in the GUI when selecting 'keyword'?
Comment 57 Magnus Enger 2017-12-29 10:29:23 UTC
(In reply to Katrin Fischer from comment #55)
> Hi Magnus, I think this is out of scope if this aims to restore the former
> behaviour - you'd have to remove the place of publication from indexes.
> Maybe a next step could be a configurable Any index?

Sounds good!
Comment 58 Fridolin Somers 2018-01-10 09:12:05 UTC
(In reply to Katrin Fischer from comment #56)
> As this would be a change of behavior now after a long time of dom
> indexing... should we make it possible to switch the one that is used in the
> GUI when selecting 'keyword'?

Ineed, maybe we could create a system pref that defines which index is used when no index is specified : any, anywhere or other custom index maybe.
Comment 59 Magnus Enger 2018-01-10 09:32:01 UTC
(In reply to Fridolin SOMERS from comment #58)
> (In reply to Katrin Fischer from comment #56)
> > As this would be a change of behavior now after a long time of dom
> > indexing... should we make it possible to switch the one that is used in the
> > GUI when selecting 'keyword'?
> 
> Ineed, maybe we could create a system pref that defines which index is used
> when no index is specified : any, anywhere or other custom index maybe.

Yups, that sounds like an excellent idea. But probably something for a new bug. I have reported it here: "Bug 19944 - Make the default index configurable"
Comment 60 Katrin Fischer 2018-01-10 12:28:06 UTC
I think for this bug an option to switch between Any and Anywhere would be good. Then for existing installations keep the current behaviour and for new installations (default setting) use the 'smaller' index. The custom one is definitely for a new bug.
Comment 61 Charles Farmer 2018-03-30 21:23:11 UTC
Unless I'm mistaken, this patch doesn't apply anymore:

> CONFLICT (content): Merge conflict in koha-tmpl/intranet-tmpl/prog/en/includes/search_indexes.inc
Comment 62 Fridolin Somers 2018-04-04 12:35:24 UTC
Created attachment 73617 [details] [review]
Bug 8962: Changes to CCL.properties, bib1 for authorities, and templates

This patch adds a new index Anywhere in ccl.properties in bib1 for authorities, and add "any word" option in intranet search for using this index.
Default OPAC and intranet search in still using CCL "Any" index.

Testing plan :
In a Marc21 GRS-1 Koha :
1/ make some biblio searches without selecting any index (Koha will use "any" index) and note the results. Ex : search "fre", "eng", "book", "history", "history book"
2/ apply the 3 patches
3/ copy the 4 new record.abs files (Marc21 auth, Marc21 bib, Unimarc auth, Unimarc bib) from your sources to the directory used by Zebra configuration in your Koha instance
4/ launch rebuild_zebra.pl -b -r -x -v
5/ check that a new choice "Any word (all indexes)" is visible in advanced search.
6/ make the same searches as 1/ : you should see less results (detailed test plan in patchs 2 and 3)
7/ in Staff interface adavanced search, select "Any word (all indexes)" and make the same searches as 1/ : you should see the same results (maybe with a slight differences, because more notes fields are indexed in Marc21)

In a UNIMARC GRS-1 Koha :
same test

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 63 Fridolin Somers 2018-04-04 12:35:42 UTC
Created attachment 73618 [details] [review]
Bug 8962: Restore Any index behavior and add Anywhere index

The "Any" index/register should contains all the data contained in all the other indexes/registers (e.g. Title, Author, Subject, etc).

The "Anywhere" index/register should contain the entire record including fields/subfields that aren't indexed anywhere else.

In GRS1 indexing the Any index was indeed only on fields defined as indexed.
This has changed in DOM where all data in MARC is indexed with Any.
This generates a lot of results when no index is defined in search.

This patch changes the DOM xsl config generation to add Any to each indexed data and add Anywhere to keep a way of searching on all data.

Test plan :
- Add a data on a field that is not indexed : for example "toto" in 998$z
- Add same data as the title of another record
- Index the records
- Search on toto without index (uses Any) => you get the 2 records
- Search on toto with title index => you get the record with "toto" as title
- Apply patch
- Update DOM xsl config files
- Reindex
- Search on toto without index (uses Any) => you get the record with "toto" as title
- Search on toto with title index => you get the record with "toto" as title
- Search on toto with Anywhere index => you get the 2 records
Comment 64 Fridolin Somers 2018-04-04 12:36:39 UTC
(In reply to Charles Farmer from comment #61)
> Unless I'm mistaken, this patch doesn't apply anymore:
> 
> > CONFLICT (content): Merge conflict in koha-tmpl/intranet-tmpl/prog/en/includes/search_indexes.inc

Indeed.
Easy rebase.
Comment 65 Charles Farmer 2018-05-25 19:30:44 UTC
Created attachment 75569 [details] [review]
Bug 8962: Changes to CCL.properties, bib1 for authorities, and templates

This patch adds a new index Anywhere in ccl.properties in bib1 for authorities, and add "any word" option in intranet search for using this index.
Default OPAC and intranet search in still using CCL "Any" index.

Testing plan :
In a Marc21 GRS-1 Koha :
1/ make some biblio searches without selecting any index (Koha will use "any" index) and note the results. Ex : search "fre", "eng", "book", "history", "history book"
2/ apply the 3 patches
3/ copy the 4 new record.abs files (Marc21 auth, Marc21 bib, Unimarc auth, Unimarc bib) from your sources to the directory used by Zebra configuration in your Koha instance
4/ launch rebuild_zebra.pl -b -r -x -v
5/ check that a new choice "Any word (all indexes)" is visible in advanced search.
6/ make the same searches as 1/ : you should see less results (detailed test plan in patchs 2 and 3)
7/ in Staff interface adavanced search, select "Any word (all indexes)" and make the same searches as 1/ : you should see the same results (maybe with a slight differences, because more notes fields are indexed in Marc21)

In a UNIMARC GRS-1 Koha :
same test

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Charles Farmer <charles.farmer@inLibro.com>
Comment 66 Charles Farmer 2018-05-25 19:30:51 UTC
Created attachment 75570 [details] [review]
Bug 8962: Restore Any index behavior and add Anywhere index

The "Any" index/register should contains all the data contained in all the other indexes/registers (e.g. Title, Author, Subject, etc).

The "Anywhere" index/register should contain the entire record including fields/subfields that aren't indexed anywhere else.

In GRS1 indexing the Any index was indeed only on fields defined as indexed.
This has changed in DOM where all data in MARC is indexed with Any.
This generates a lot of results when no index is defined in search.

This patch changes the DOM xsl config generation to add Any to each indexed data and add Anywhere to keep a way of searching on all data.

Test plan :
- Add a data on a field that is not indexed : for example "toto" in 998$z
- Add same data as the title of another record
- Index the records
- Search on toto without index (uses Any) => you get the 2 records
- Search on toto with title index => you get the record with "toto" as title
- Apply patch
- Update DOM xsl config files
- Reindex
- Search on toto without index (uses Any) => you get the record with "toto" as title
- Search on toto with title index => you get the record with "toto" as title
- Search on toto with Anywhere index => you get the 2 records

Signed-off-by: Charles Farmer <charles.farmer@inLibro.com>
Comment 67 Charles Farmer 2018-05-25 19:33:35 UTC
Had to remove the firstmost patch named "[SIGNED OFF] Changes to CCL.properties ...". Nothing lost, however, since it looked like a patch that was rebased but not obsoleted.
Comment 68 Jonathan Druart 2018-05-30 16:42:03 UTC
Is this only for GRS-1?

Bug 14302 - Remove GRS1 indexing related code
Comment 69 Fridolin Somers 2018-06-18 13:58:04 UTC
(In reply to Jonathan Druart from comment #68)
> Is this only for GRS-1?
> 
> Bug 14302 - Remove GRS1 indexing related code

Nope it has started for GRS-1 but its now for DOM only.
Comment 70 Katrin Fischer 2018-08-29 19:39:42 UTC
Some notes:

1) Typo here, worlds would be nice :) Also, why are both descriptions the same?

+                            [% IF ( ms_anywhere ) %]<option selected="selected" value="anywhere">Any world (all indexes)</option>
+                            [% ELSE %]<option value="anywhere">Any word (all indexes)</option>[% END %]

2) QA script complains about tabs in Search.pm (L 1090)

3) The commit message still talks about GRS-1, please update to have a consistent test plan. At the omment I am a little confused as to what index is what and where will it be used.

4) While I like the idea of this, I think it needs some more thought.
I am not happy with the change of behaviour. It's true that we changed it when introducing DOM, but I think that's been too far ago to step back from now. 

What I'd like to see is a configuration option on what should be the default index to use with keyword. 

Setting to FQA for commenting, but feel free to ask for more opinions!
Comment 71 Fridolin Somers 2018-09-16 22:08:54 UTC
I've tested the behavior with ElasticSearch with a local field 987$a, not present in the mapping.
From my tests any MARC data is indexed.

So in the end I'd say we have to be consistent between our two search engines.
Comment 72 Katrin Fischer 2018-10-28 17:12:55 UTC
Hi Frido, does this mean you are no longer working on this patch? Should we move it to WONTFIX?
Comment 73 Fridolin Somers 2018-11-05 11:43:58 UTC
Indeed, I think this is not worth continuing on it
Comment 74 Marcel de Rooy 2018-11-05 12:04:15 UTC
(In reply to Fridolin SOMERS from comment #71)
> I've tested the behavior with ElasticSearch with a local field 987$a, not
> present in the mapping.
> From my tests any MARC data is indexed.
> 
> So in the end I'd say we have to be consistent between our two search
> engines.

So you are saying that in ElasticSearch currently any==anywhere==really all data.
But I suppose that it is just a matter of configuration?

Does ES facilitate an any index in the sense of all user defined indexes ? In that case we would have found an easy 'any'. If not, we should somehow define it.
As you say, anywhere is already there..

I would recommend to reopen this report (although you might want to obsolete some patches) but it seems to be imo an interesting option to explore in ES.
Comment 75 Fridolin Somers 2018-11-05 12:13:25 UTC
(In reply to Marcel de Rooy from comment #74)
> 
> I would recommend to reopen this report (although you might want to obsolete
> some patches) but it seems to be imo an interesting option to explore in ES.

I suggest you open a new bug report dedicated to ES
Comment 76 Marcel de Rooy 2018-11-05 12:16:51 UTC
(In reply to Fridolin SOMERS from comment #75)
> (In reply to Marcel de Rooy from comment #74)
> > 
> > I would recommend to reopen this report (although you might want to obsolete
> > some patches) but it seems to be imo an interesting option to explore in ES.
> 
> I suggest you open a new bug report dedicated to ES

21762