Bug 7818 - support DOM mode for Zebra indexing of bibliographic records
Summary: support DOM mode for Zebra indexing of bibliographic records
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Architecture, internals, and plumbing (show other bugs)
Version: 3.8
Hardware: All All
: P1 - high critical (vote)
Assignee: Robin Sheat
QA Contact: Mason James
URL:
Keywords:
Depends on: 8252
Blocks: 8115 7417 8211 8212 8665 8805 8958
  Show dependency treegraph
 
Reported: 2012-03-23 21:07 UTC by Galen Charlton
Modified: 2019-06-27 09:24 UTC (History)
7 users (show)

See Also:
Change sponsored?: Sponsored
Patch complexity: ---
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 7818: update installer for biblio DOM indexing (17.17 KB, patch)
2012-04-06 16:27 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
Bug 7818: update miscellaneous installer scripts (1.40 KB, patch)
2012-04-06 16:27 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
Bug 7818: utility to generate DOM indexing configs (11.69 KB, patch)
2012-04-06 16:27 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
Bug 7818: move koha-indexdefs-to-zebra.xsl (999 bytes, patch)
2012-04-06 16:27 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
Bug 7818: add index_data_field option to DOM indexing repertoire (5.00 KB, patch)
2012-04-06 16:28 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
Bug 7818: Zebra DOM filter index definitions for MARC21 bibs (162.79 KB, patch)
2012-04-06 16:28 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
Bug 7818: -x option of rebuild_zebra.pl now works with DOM filter (9.47 KB, patch)
2012-04-06 16:28 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
Bug 7818: use Title-cover for title phrase weighting (1.44 KB, patch)
2012-04-06 16:28 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
Bug 7818: update comment to clarify availability of DOM index mode (1.65 KB, patch)
2012-04-06 16:28 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
Bug 7818: add warning about not editing record.abs when using DOM filter (5.81 KB, patch)
2012-04-06 16:28 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
UNIMARC DOM (156.37 KB, patch)
2012-04-06 17:58 UTC, Frédéric Demians
Details | Diff | Splinter Review
Bug 7818 Add UNIMARC biblio records zebra DOM def files (157.37 KB, patch)
2012-05-26 16:14 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
Bug 7818 Fix rebuild_zebra.pl --nosanitize (996 bytes, patch)
2012-06-07 08:01 UTC, Frédéric Demians
Details | Diff | Splinter Review
Bug 7818 : Following, fixing so the packages build Still need to update koha-create to make use of the new file (2.97 KB, patch)
2012-06-29 10:25 UTC, Chris Cormack
Details | Diff | Splinter Review
Bug 7818 : Following, fixing so the packages build Still need to update koha-create to make use of the new file (3.09 KB, patch)
2012-07-02 18:05 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Galen Charlton 2012-03-23 21:07:47 UTC
Koha DOM Indexing for bibliographic records
-------------------------------------------

Overview
========
Koha uses the Zebra search engine to implement bibliographic and authority record search.  Zebra comes with several filters that support the indexing of various document types, including MARC and XML.  Koha currently uses Zebra’s GRS-1 filter for indexing MARC and Zebra’s DOM filter for indexing authority records (represented as MARCXML).

To oversimplify a bit, the GRS-1 filter allows for constructing the following kinds of search indexes for MARC records:
Keyword and phrase indexes of the contents of a MARC subfield
Keyword and phrase indexes of the contents of a fixed field

What the GRS-1 index cannot provide, however, is the ability to index a phrase that spans subfield boundaries.  For example, giving the subject heading:

650 $a Cats $x pharmacology $x catnip

it is not possible to construct a search that will reliably match all instances of the phrase “cats pharmacology catnip”.  This in turn means that it is not possible to reliably search on a bibliographic or authority heading with out extensive post-search pruning of the result set.

The DOM filter, in contrast, takes an XML document (in this case, a MARC XML record) and applies a stylesheet to generate another XML document that contains all of the strings to be indexed by Zebra.  This allows for much more flexibility, and in particular, allows for retrieving records on normalized heading.

Furthermore, since the indexing definitions are ultimately expressed as an XML stylesheet, the DOM filter allows for an arbitrary amount of manipulation of tokens and phrases to be made during indexing.

Goals
=====

The goals of this development project are to:

* Add support for the Zebra DOM filter for indexing bibliographic records in Koha, similar to existing support for DOM indexing of authority records.
* Make use of DOM indexing to allow for complete and partial searches of headings fields.
* Provide indexing definitions to allow for searching of title phrases that span subfield boundaries in the 245.

Components
==========

Indexing Preparation
====================
A new class (Koha::Indexer) hierarchy will be created to manage preparation of biblio and authority XML records for indexing.  In the case of a bib record, 

biblionumber 
→ MARC::Record representation of bib
→ one or more Koha::Indexer::MARC::RecordNormalizer routines that perform operations on the MARC::Record object.  The one normalizer to be added for this project will be Koha::Indexer::MARC::RecordNormalizer::EmbedItems.
→ MARCXML representation of bib
→ Zebra indexing pipeline

Two tables will be defined to allow for specifying pre-indexing record normalizer profiles

record_norm_profile
  code varchar2(10) PK
  name varchar2(100)
  metadata_type varchar2(10) -- currently only ‘MARC’
  record_type varchar2(10) -- ‘bilbio’ or ‘authority’

record_norm_profile_normalizer
  seq int -- order in which normalizer is to be applied
  record_norm_profile FK references record_norm_profile.code
  normalizer varchar2(100) -- references available 

Koha::Indexer::$metadata_type::RecordNormalizer classes

The normalization pipeline will be used by rebuild_zebra.pl.
DOM indexing

The koha-indexdefs-to-zebra.xsl stylesheets currently used for DOM indexing of authorities will be made available to biblio as well.  In addition XSLT functions will be added that can

-- trim nonfiling characters
-- calculate the applicable thesaurus of a bibliographic subject heading

A script (misc/maintenance/convert_record_abs_to_dom.pl) will be written that will convert indexing definitions from record.abs to DOM format.  For example:

melm 022$a      ISSN:w,Identifier-standard:w

would be come something like this:
  <kohaidx:index_subfields tag="022" subfields="a">
    <kohaidx:target_index>ISSN:w</kohaidx:target_index>
    <kohaidx:target_index>Identifier-standard:w</kohaidx:target_index>
  </kohaidx:index_subfields>

In addition to handling converting existing record.abs config files, additional new standard indexing definitions will be added to index authority-controlled headings fields the same way as they are currently indexed in authority records.  For example, topical subject headings will use:

  <kohaidx:index_heading tag="650" subfields="abvxyz" subdivisions="vxyz">
    <kohaidx:target_index>Subject-topical-heading:w</kohaidx:target_index>
    <kohaidx:target_index>Subject-topical-heading:p</kohaidx:target_index>
    <kohaidx:target_index>Subject-topical-heading:s</kohaidx:target_index>
    <kohaidx:target_index>Heading:w</kohaidx:target_index>
    <kohaidx:target_index>Heading:p</kohaidx:target_index>
    <kohaidx:target_index>Heading:s</kohaidx:target_index>
  </kohaidx:index_heading>

Headings index definitions will be provided in at least two variants:
heading as a phrase
heading as a phrase with insertions to include subdivision markers and thesaurus -- this type is meant to be used for future work for linking bib records to authority headings

Searching enhancements
======================
The new standard indexing definitions will include two whole-245-as-phrase indexes, with one variant including nonfiling characters and the other excluding them.

OPAC display enhancements
=========================
The OPAC display XSLT (MARC21slim2OPACDetail.xsl) will be changed so that subject heading links will construct phase searches instead of searches on the subfield $9:

The Koha interface should be updated to allow for user to select partial subject headings from the OPAC and staff client Details pages.
for example, on a record with subject heading “History -- United States -- 18th Century -- Periodicals”, clicking:
History would search for “History”
United States would search for “History -- United States”
18th Century would search for “History -- United States -- 18th Century”
Periodicals would search for “History -- United States -- 18th Century -- Periodicals”

The advanced search page template will include an exact-title search that will perform an whole-phrase search on the new whole-245-as-phrase indexes.

The subject search in the masthead and advanced search templates will be changed to use the new subject heading index, allowing for partial heading searches.

Installation and upgrade considerations
=======================================
During installation, a new query will be adding to Makefile.PL to specify the biblio indexing mode.  The default value will be DOM.
During upgrade, an option will be presented to convert the indexing definitions to DOM and reindex the biblio records.  The specific upgrade steps will include:

-- running misc/maintenance/convert_record_abs_to_dom.pl to create a biblio-koha-indexdefs.xml that includes the existing indexing definitions as well as the new standard bib headings indexes.
-- using koha-indexdefs-to-zebra.xsl and an XSLT processor to create an biblio-zebra-indexdefs.xml to be used as the stylesheet that Zebra uses directly to convert incoming MARCXML records to their indexed phrases.
-- running rebuild_zebra.pl -b -r on the entire database
Comment 1 Galen Charlton 2012-03-26 05:29:13 UTC
As of now, http://git.koha-community.org/gitweb/?p=wip/koha-equinox.git;a=shortlog;h=refs/heads/dom-biblio contains code to add support for DOM indexing equivalent to the current GRS-1 filter's support for MARC21 with the improvement that indexed phrases can cross subfield boundaries.

Although it's a little early, given timing I'm marking the branch as of today as needs signoff to encourage testing.
Comment 2 Jared Camins-Esakov 2012-03-28 20:18:18 UTC
I tested it, and ran into an issue with the relevancy ranking. The translation of the following lines is problematic:

melm 243$n      Thematic-number
melm 243$r      Music-key
melm 243        Title:w,Title:p,Title-collective
melm 245$a      Title-cover:w,Title-cover:p,Title-cover:s,Title:w,Title:p,Title:s
melm 245$c      Author,Author-in-order:w,Author-in-order:p,Author-in-order:s
melm 245$9      Cross-Reference:w,Koha-Auth-Number
melm 245        Title:w,Title:p
(I kept the 243 lines for comparison sake, the issue is really with 245, as explained below)

In particular, note that 245$c in the GRS-1 configuration does *not* get put into the Title index. In C4::Search, hits in the Title index are marked as r3, which (under GRS-1) ignores the statement of responsibility in 245$c. With the conversion of record.abs to DOM indexes, the Title:w, Title:p indexes now include the statement of responsibility, which means that authors' names will be given r3 ranking, *plus* the additional weighting for appearing additional times elsewhere in the record.
Comment 3 Jared Camins-Esakov 2012-03-28 21:00:28 UTC
Also, XML reindexing still doesn't work with DOM:


Entity: line 87: parser error : XML declaration allowed only at the start of the document
<?xml version="1.0" encoding="UTF-8"?>
     ^
Entity: line 88: parser error : Extra content at the end of the document
<record
^
16:42:05-28/03 zebraidx(8685) [warn] extract error: generic
Comment 4 Galen Charlton 2012-03-29 12:31:48 UTC
(In reply to comment #2)
> In particular, note that 245$c in the GRS-1 configuration does *not* get put
> into the Title index. In C4::Search, hits in the Title index are marked as
> r3, which (under GRS-1) ignores the statement of responsibility in 245$c.
> With the conversion of record.abs to DOM indexes, the Title:w, Title:p
> indexes now include the statement of responsibility, which means that
> authors' names will be given r3 ranking, *plus* the additional weighting for
> appearing additional times elsewhere in the record.

My inclination would be to add a "title-rel" index to the index config that includes (say) the 245$a and $b and have that be used for the hard-coded relevance ranking, with suitable shimming to leave GRS-1-based configs unaffected.  Thoughts?
Comment 5 Jared Camins-Esakov 2012-03-29 12:34:30 UTC
(In reply to comment #4)
> (In reply to comment #2)
> > In particular, note that 245$c in the GRS-1 configuration does *not* get put
> > into the Title index. In C4::Search, hits in the Title index are marked as
> > r3, which (under GRS-1) ignores the statement of responsibility in 245$c.
> > With the conversion of record.abs to DOM indexes, the Title:w, Title:p
> > indexes now include the statement of responsibility, which means that
> > authors' names will be given r3 ranking, *plus* the additional weighting for
> > appearing additional times elsewhere in the record.
> 
> My inclination would be to add a "title-rel" index to the index config that
> includes (say) the 245$a and $b and have that be used for the hard-coded
> relevance ranking, with suitable shimming to leave GRS-1-based configs
> unaffected.  Thoughts?

+1 from me. I'd suggest adding title-rel to record.abs, and just adjust the r3 configuration for all installations.
Comment 6 Galen Charlton 2012-04-06 04:09:15 UTC
(In reply to comment #3)
> Also, XML reindexing still doesn't work with DOM:
> 
> 
> Entity: line 87: parser error : XML declaration allowed only at the start of
> the document
> <?xml version="1.0" encoding="UTF-8"?>
>      ^
> Entity: line 88: parser error : Extra content at the end of the document
> <record
> ^
> 16:42:05-28/03 zebraidx(8685) [warn] extract error: generic

I've pushed an update to the dom-biblio branch to now make this work.
Comment 7 Galen Charlton 2012-04-06 04:53:43 UTC
(In reply to comment #5)
> +1 from me. I'd suggest adding title-rel to record.abs, and just adjust the
> r3 configuration for all installations.

I decided to back off from doing that to avoid the potential for forcing *all* users to reindex their bibs upon upgrade to 3.8.  I've pushed a compromise of using Title-cover,phr as the r3 weighted part of the default relevance search.
Comment 8 Galen Charlton 2012-04-06 15:30:21 UTC
After testing (thanks Jared!), a revised branch for signoff is now available in the wip/koha-equinox.git repository, branch bug7818-dom-biblio-final.
Comment 9 Jared Camins-Esakov 2012-04-06 16:27:33 UTC Comment hidden (obsolete)
Comment 10 Jared Camins-Esakov 2012-04-06 16:27:41 UTC Comment hidden (obsolete)
Comment 11 Jared Camins-Esakov 2012-04-06 16:27:48 UTC Comment hidden (obsolete)
Comment 12 Jared Camins-Esakov 2012-04-06 16:27:57 UTC Comment hidden (obsolete)
Comment 13 Jared Camins-Esakov 2012-04-06 16:28:06 UTC Comment hidden (obsolete)
Comment 14 Jared Camins-Esakov 2012-04-06 16:28:15 UTC Comment hidden (obsolete)
Comment 15 Jared Camins-Esakov 2012-04-06 16:28:22 UTC Comment hidden (obsolete)
Comment 16 Jared Camins-Esakov 2012-04-06 16:28:32 UTC Comment hidden (obsolete)
Comment 17 Jared Camins-Esakov 2012-04-06 16:28:39 UTC Comment hidden (obsolete)
Comment 18 Jared Camins-Esakov 2012-04-06 16:28:47 UTC Comment hidden (obsolete)
Comment 19 Jared Camins-Esakov 2012-04-06 16:30:33 UTC
Both "traditional" GRS-1-mode indexing and DOM-mode indexing work perfectly with these patches, so I recommend the feature for inclusion in 3.8, as it does not change existing behavior.
Comment 20 Galen Charlton 2012-04-06 16:39:36 UTC
Here is a summary for documentation purposes of the patches that have been signed off:

This is a series of patches implementing DOM-filter indexing for 
bibliographic records.  Currently this series of patches does the
following:

[1] Adds a Makefile.PL option to enable DOM filter indexing
[2] Adds a utility for converting a record.abs index definitions
    to DOM-style definitions
[3] Includes DOM-style index definitions for MARC21 bibs that are
    equivalent to the legacy record.abs, with the exception that
    phrases can span subfield boundaries.

To test this on an existing database, some variation of the following
procedure should work:

[1] perl Makefile.PL and choose to enable DOM indexing of bibs
[2] If the index defintions have been customized, use make_zebra_dom_cfg_from_record_abs
    and the koha-indexdefs-to-zebra.xsl stylesheet to generate a
    an updated biblio-zebra-indexdefs.xsl
[3] make
[4] make update_zebra_conf

This patch series also includes the following related changes:

[1] The -a and -x options to rebuild_zebra.pl are no longer mutually exclusive.
[2] The title phrase rank 3 field weight has been replaced by Title-cover.  This prevents a side effect of DOM indexing (where the DOM-filter equivalent of record.abs melm XXX indexes the entire field as a unit, not each subfield separately) from given the statement of responsibility in the 245$c undue weight in relevance ranking.
[3] Some comments were added regarding the potential future deprecation of the use of Zebra's GRS-1 filter.
Comment 21 Frédéric Demians 2012-04-06 17:35:22 UTC
QA: It works as described for a MARC21 DB.

I will try now with UNIMARC DB after generating DOM biblios def.
Comment 22 Frédéric Demians 2012-04-06 17:58:24 UTC Comment hidden (obsolete)
Comment 23 Mason James 2012-05-01 07:08:38 UTC
(In reply to comment #22)
> Created attachment 8982 [details] [review]
> UNIMARC DOM
> 
> It works with UNIMARC also

one more sign-off on Frédéric's patch with UNIMARC, and i'll pass-QA on it :)

ps: Galen, it looks very good!
Comment 24 Jared Camins-Esakov 2012-05-26 16:14:02 UTC Comment hidden (obsolete)
Comment 25 Katrin Fischer 2012-05-26 16:16:31 UTC
Please QA fast - waiting for this to get in! :)
Comment 26 Mason James 2012-05-28 15:07:15 UTC
>> one more sign-off on Frédéric's patch with UNIMARC, and i'll pass-QA on it :)

> Bug 7818 Add UNIMARC biblio records zebra DOM def files
> 
> Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>

sign-off for the last patch has been completed 
all patches pass 'perlcritic' and 'prove xt' tests

i think this awesome feature just passed-QA! :)
Comment 27 Mason James 2012-05-28 15:38:17 UTC
> all patches pass 'perlcritic' and 'prove xt' tests

fyi: all patches applied clean to 722ca127dd0aaf0646001594ebbece18c75d12b1
Comment 28 Paul Poulain 2012-06-06 16:47:02 UTC
I've tested this patch, and after some pain understanding which files I must put where, I succeded.

Just a comment, I get a nasty error when using the -x flag:

18:22 ~/koha.dev/installs/lecannetdesmaures/etc$ ~/koha.dev/koha-community/misc/migration_tools/rebuild_zebra.pl -b -v -x --offset=100 --length=10 -r
Entity: line 84: parser error : Extra content at the end of the document
<record
^
18:23:08-06/06 zebraidx(11130) [warn] extract error: generic


When indexing without -x, things are OK (~/koha.dev/koha-community/misc/migration_tools/rebuild_zebra.pl -b -v --offset=100 --length=10 -r )
Comment 29 Paul Poulain 2012-06-06 17:45:07 UTC
I can't make it work properly, so I need to understand what I made wrong...

Here are the steps I followed on an existing (UNIMARC) setup:
 * Edit etc/zebradb/zebra-biblios.cfg file:
    - update profilePath: line by appending xsl path: 
        :$KOHA/etc/zebradb/xsl
    - comment the following line:
         marcxml.recordType:grs.sgml
    - add after the commented line:
         recordtype: dom.$KOHA/etc/zebradb/biblios/etc/dom-config.xml
         marcxml.recordtype: dom.$KOHA/etc/zebradb/biblios/etc/dom-config.xml
         iso2709.recordtype: dom.$KOHA/etc/zebradb/biblios/etc/dom-config-marc.xml
    - if your $KOHA setup is not where your koha code is, you must copy 2 new files: cp $KOHAGIT/etc/zebradb/biblios/etc/*.xml $KOHA/etc/zebradb/biblios/etc/
    - generate xsl configuration: 
        $KOHA/misc/maintenance/make_zebra_dom_cfg_from_record_abs --input $ZEBRADIR/zebradb/marc_defs/unimarc/biblios/record.abs --output $ZEBRADIR/zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xml
        xsltproc $ZEBRADIR/etc/zebradb/xsl/koha-indexdefs-to-zebra.xsl $ZEBRADIR/zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xml >zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl

Once I've made that, if I try to reindex:
19:41 ~/koha.dev/installs/koha-community/etc$ ~/koha.dev/koha-community/misc/migration_tools/rebuild_zebra.pl -b -v --offset=0 --length=10 -r -x -k
Zebra configuration information
================================
Zebra biblio directory      = /home/paul/koha.dev/installs/koha-community/var/lib/zebradb/biblios
Zebra authorities directory = /home/paul/koha.dev/installs/koha-community/var/lib/zebradb/authorities
Koha directory              = /home/paul/koha.dev/koha-community
BIBLIONUMBER in :     001$@
BIBLIOITEMNUMBER in : 090$a
================================
skipping authorities
====================
exporting biblio
====================
1.........
Records exported: 10
====================
REINDEXING zebra
====================
Entity: line 741: parser error : Extra content at the end of the document
<record
^
19:43:10-06/06 zebraidx(12791) [warn] extract error: generic
====================
CLEANING
====================


I've checked that the XML file looks good, it does. If I remove the -x flag, it works.

Any help/hint welcomed, I spent a lot of time on this setup, and will continue
Comment 30 Paul Poulain 2012-06-06 17:56:40 UTC
Trying the -x and --nosanitize flags, I get another error:
19:55 ~/koha.dev/installs/koha-community/etc$ ~/koha.dev/koha-community/misc/migration_tools/rebuild_zebra.pl -b -v --offset=0 --length=10 -r -x -k --nosanitize
Zebra configuration information
<snope>
====================
REINDEXING zebra
====================
Entity: line 742: parser error : XML declaration allowed only at the start of the document
<?xml version="1.0" encoding="UTF-8"?>
     ^
19:55:27-06/06 zebraidx(13014) [warn] extract error: generic
Comment 31 Frédéric Demians 2012-06-06 18:08:01 UTC
> 18:22 ~/koha.dev/installs/lecannetdesmaures/etc$
> ~/koha.dev/koha-community/misc/migration_tools/rebuild_zebra.pl -b -v
> -x --offset=100 --length=10 -r Entity: line 84: parser error : Extra
> content at the end of the document

I think it means that the file containing the exported marcxml biblio is
not a valid XML file. <collection> </collection> is missing. Try to add
it by hand, and reindex directly with zebraidx.
Comment 32 Frédéric Demians 2012-06-07 08:01:34 UTC Comment hidden (obsolete)
Comment 33 Paul Poulain 2012-06-07 08:51:14 UTC
(In reply to comment #31)
> > 18:22 ~/koha.dev/installs/lecannetdesmaures/etc$
> > ~/koha.dev/koha-community/misc/migration_tools/rebuild_zebra.pl -b -v
> > -x --offset=100 --length=10 -r Entity: line 84: parser error : Extra
> > content at the end of the document
> 
> I think it means that the file containing the exported marcxml biblio is
> not a valid XML file. <collection> </collection> is missing. Try to add
> it by hand, and reindex directly with zebraidx.

OK, seems you're right Frederic:
zebraidx -c /home/paul/koha.dev/installs/koha-community/etc/zebradb/zebra-biblios.cfg -g marcxml -d biblios update /tmp/kUzN0iLpkp/biblio/

=> without changing anything to /tmp/kUzN0iLpkp/biblio/ I get
Entity: line 741: parser error : Extra content at the end of the document
<record
^
20:18:38-06/06 zebraidx(13191) [warn] extract error: generic


=> after adding <collection> and </collection> I get:
20:20:26-06/06 zebraidx(13208) [log] update dom./home/paul/koha.dev/installs/koha-community/etc/zebradb/biblios/etc/dom-config.xml /tmp/kUzN0iLpkp/biblio/exported_records 102404
20:20:26-06/06 zebraidx(13208) [log] Iterations: isam/dict 1808/1803
20:20:26-06/06 zebraidx(13208) [log] Dict: inserts/updates/deletions: 901/0/902
20:20:27-06/06 zebraidx(13208) [log] Records: 10 i/u/d 0/10/0


So the NEXT QUESTION is = why is'nt the <collection> included automatically (as it used to be)

Looking at the code, it's clearly related to sub include_xml_wrapper {}

This sub should return 1 and return 0.
It should return 1 from this line:
    return 1 if $record_type eq 'biblio' and $bib_index_mode eq 'dom';

The $bib_index_mode is set by:
my $bib_index_mode = C4::Context->config('zebra_bib_index_mode') || 'grs1';

looking at my configuration file...
There is no <zebra_bib_index_mode>__BIB_INDEX_MODE__</zebra_bib_index_mode>

adding it....

and it works, with the exception of a warning i'll investigate later, but if you have an idea, let me know !
20:27:55-06/06 zebraidx(13358) [warn] Unknown register type: 
(nothing after the "register type:")
Comment 34 Paul Poulain 2012-06-07 13:28:05 UTC
Trying to search, I get no results, but a nasty

Can't call method "data" on an undefined value at /home/paul/koha.dev/koha-community/C4/Search.pm line 1496.

Adding a warning in the code, I see:
DUMP$VAR1 = bless( {
                  '_fields' => [],
                  '_leader' => '                        ',
                  '_warnings' => [
                                   'Record length "" is not numeric in record 1'
                                 ]
               }, 'MARC::Record' );

Adding the following lines that are missing in the koha-conf.xml:
<!-- <maximumrecordsize>2000000</maximumrecordsize> -->
<xi:include href="/home/paul/koha.dev/installs/lecannetdesmaures/etc/zebradb/retrieval-info-bib-dom.xml"
            xmlns:xi="http://www.w3.org/2001/XInclude">
  <xi:fallback>
<retrievalinfo>
and
 </xi:fallback>
 </xi:include>
after the </retrievalinfo>


The result is the same:
zebra log file says:
12:27:16-07/06 zebrasrv(1) [session] Session - OK 1 unix:/home/paul/koha.dev/installs/lecannetdesmaures/var/run/zebradb/bibliosocket 19584
12:27:16-07/06 zebrasrv(1) [request] Auth idPass kohauser -
12:27:16-07/06 zebrasrv(1) [request] Init OK - ID:81 Name:ZOOM-C/YAZ Version:4.2.18 cc3abee6f861d0166bef5333b1fc72fbdc8bd94c
12:27:16-07/06 zebrasrv(1) [warn] Missing attribute 'type' in attribute info
12:27:16-07/06 zebrasrv(1) [warn] Missing attribute 'type' in attribute info
12:27:16-07/06 zebrasrv(1) [warn] Missing attribute 'type' in attribute info
12:27:16-07/06 zebrasrv(1) [warn] Missing attribute 'type' in attribute info
12:27:16-07/06 zebrasrv(1) [warn] Missing attribute 'type' in attribute info
12:27:16-07/06 zebrasrv(1) [warn] Missing attribute 'type' in attribute info
12:27:16-07/06 zebrasrv(1) [log] dict_lookup_grep: (\x01\x02)(j(e|\xC3\xA9|\xC3\xA8|\xC3\xAA|\xE1\xBA\xBD|\xC4\x95|\xC4\x99|\xC4\x97|\xC4\x9B|\xC8\x85|\xC8\x87)(a|\xC3\xA1|\xC3\xA0|\xC3\xA3|\xC3\xA5|\xC3\xA2|\xC4\x83|\xC4\x85|\xC8\xA7|\xC7\x8E|\xC8\x81|\xC8\x83)n)
12:27:16-07/06 zebrasrv(1) [request] Search biblios OK 1 1 1+0 RPN @attrset Bib-1 @attr 1=1016 @attr 4=6 jean
12:27:16-07/06 zebrasrv(1) [log] /home/paul/koha.dev/koha-community/etc/zebradb/biblios/etc/dom-config.xml dom filter: loading config file /home/paul/koha.dev/koha-community/etc/zebradb/biblios/etc/dom-config.xml
12:27:16-07/06 zebrasrv(1) [request] Present OK -   1 1+1 
12:27:16-07/06 zebrasrv(1) [session] Connection closed by client

Checking what zebrasrv says when you launch it, I see:
I/O warning : failed to load external entity "/home/paul/koha.dev/installs/lecannetdesmaures/etc/zebradb/retrieval-info-bib-dom.xml"

the file is not here, copying it at the right place (from $KOHA/etc/zebradb/)

it works !

will summarize all my tests in a wiki page: "switching to DOM indexing when you have Koha already running" !
Comment 35 Paul Poulain 2012-06-09 09:45:56 UTC
PAtch pushed, and http://wiki.koha-community.org/wiki/Switching_to_dom_indexing created. It's a big & major patch, I'll call for more testers on koha-devel immediatly
Comment 36 Chris Cormack 2012-06-25 20:59:07 UTC
I'm going to pull this back into 3.8.x now, it is a no risk change, that does nothing unless you specifically enable it. 

It's bordering on more of a bug fix than an enhancement as it fixes some fairly serious bugs in searching. So I want to let libraries take advantage of it if they wish.
Comment 37 Chris Cormack 2012-06-25 21:23:06 UTC
Pushed to 3.8.x, contains no database changes and makes no changes to anything on upgrade. But allows new libraries to use DOM indexing for biblios, or existing libraries to choose to upgrade to it.

Searching works well, indexing is fast. Will be in 3.8.3
Comment 38 Chris Cormack 2012-06-26 01:55:10 UTC
We now need a patch to fix the packaging.

dh_install: etc/koha/zebradb/zebra-biblios-dom.cfg exists in debian/tmp but is not installed to anywhere
Comment 39 Chris Cormack 2012-06-26 20:19:34 UTC
git/catalyst-koha/debian% ls templates 
apache-shared.conf	     apache-site.conf.in		zebra-authorities-site.cfg.in
apache-shared-disable.conf   koha-conf-site.xml.in		zebra-biblios-site.cfg.in
apache-shared-intranet.conf  SIPconfig.xml			zebra.passwd.in
apache-shared-opac.conf      zebra-authorities-dom-site.cfg.in
Comment 40 Chris Cormack 2012-06-29 10:25:47 UTC Comment hidden (obsolete)
Comment 41 Chris Cormack 2012-06-29 10:27:06 UTC
Patch to allow the packages to build. 

Still needs another followup to make use of the dom config, but at least now the packages can build
Comment 42 Jared Camins-Esakov 2012-07-02 18:05:37 UTC
Created attachment 10617 [details] [review]
Bug 7818 : Following, fixing so the packages build Still need to update koha-create to make use of the new file

Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Comment 43 Mason James 2012-07-05 22:53:55 UTC
(In reply to comment #42)
> Created attachment 10617 [details] [review]
> Bug 7818 : Following, fixing so the packages build Still need to update
> koha-create to make use of the new file
> 
> Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>


Passing-QA on this patch

applies clean to master 'ce138ac96f3a5a729568f6d7db2e50ce3653b7bb'
Comment 44 Paul Poulain 2012-07-06 09:30:52 UTC
Follow-up for packages pushed
Comment 45 Chris Cormack 2012-07-06 21:26:24 UTC
Follow up pushed to 3.8.x
Comment 46 Marcel de Rooy 2012-08-30 07:16:19 UTC
An idea from bug 3087:
> Actually, I would recommend to just delete the retrieval fallback 
> section in koha-conf.xml.  Now we have a confusing situation; fallback 
> within the same etc directory does not make much sense to me. 

The retrieval sections are now in separate files in etc/zebradb. Fallback is in etc/koha-conf.xml. Can we simplify this and eliminate confusion?
Comment 47 Marcel de Rooy 2012-08-30 07:30:35 UTC
Another dumb question:
When I reindex zebra (marc21 dom) on current master, I have this warning:
14:54:01-29/08 zebraidx(23260) [warn] Unknown register type:

The register type is apparently empty. This does not happen on current master with grs1.
Any idea? A problem with the definitions xml?