Oh yes, coded fields are very bad indexed in unimarc.
One pbm is some index in unimarc don't match exaclty indexs in marc21.
I have prepared a big patch for that (defining new indexs in CCL etc). I will send it to you by email, maybe it could help you.

M. Saby

Of course, in my description, I mean "In DOM indexing, positions of biblio CODED FIELDS are not taken into account."

M. Saby

Patch contains cosmetic changes (in templates). I will re-write it as soon as possible.

Mathieu Saby

I am making 2 patches from my former patch  : one for fixing some bugs in GRS-1 indexing, the other for DOM.

I think we also may need an other patch later for indexdefs-to-zebra.xsl :
I had to edit biblio-zebra-indexdefs.xsl by hand, because etc/zebdradb/xml/koha-indexdefs-to-zebra.xsl does only support "subtring" in handle-one-index-control-field template.
It is good for MARC21, but not for UNIMARC : in MARC21, indexing subtrings is needed for controled field (001-009, with no subfields)
But in UNIMARC it is needed for subfields of 1XX fields.
So if DOM indexing is working with these new files, we may need to change koha-indexdefs-to-zebra.xsl.


M. Saby

In 1st patch, changes made to templates files are maybe too important (no absolute need to changes some indexes names), but I can rewrite it.
M. Saby

I forgot one step in testing : in order to force Zebra to take into account the new config files, I had to copy them from sources to main koha directory, before launching rebuild_zebra -b -x -r -v.

For example, from
home/msaby/koha-dev/src/etc/zebra_db/marc_defs/unimarc/biblios
to 
home/msaby/koha-dev/etc/zebra_db/marc_defs/unimarc/biblios

M. Saby

hi Mathieu,

Trying to investigate this patch, because it's a useful improvement

I haven't tested it yet, but i'm QAing first.

(disclaimer : when I say "MARC21", I mean MARC21 and NORMAC, that are very close)

I find some strange things:

* Country of publication
+                    'Country-publication',
=> you add this entry, just for UNIMARC. Isn't it relevant for MARC21 ?
The LoC use Country-publication for 1053, so your choice is OK (http://www.loc.gov/z3950/agency/defns/bib1.html)

* in MARC21, there's a use of Date/time-last-modified, not Date-time-last-modified :
etc/zebradb/marc_defs/normarc/biblios/record.abs:# NiN melm 005        Date/time-last-modified
etc/zebradb/marc_defs/marc21/biblios/record.abs:melm 005        Date/time-last-modified
etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml:  <!--record.abs line 44: melm 005        Date/time-last-modified-->
etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml:    <target_index>Date/time-last-modified:w</target_index>
etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl:    <z:index name="Date/time-last-modified:w">

=> WHY this change ? 

* music publisher: you change
-att 51    Number-music-publisher
+att 51    Music-number

but number-music-publisher is used by MARC21, so why do you remove it ?
I think we could/should use Identifier-publisher-for-music everywhere, as it's bib1 attribute name for LoC (http://www.loc.gov/z3950/agency/bib1.html)

I have no other comments, except a general one = it would have been much easier to test this patch if it had been splitted in smaller ones.

Hello Paul

For 
1. Country of publication

In UNIMARC, country of publication is stored in 102$a
In MARC21 in 044a and 044c in 2 different formats (http://www.loc.gov/marc/bibliographic/bd044.html)

In current UNIMARC record.abs, there is 
melm 102$a     Country-heading
In current MARC21 record.abs, there is nothing for 044

So countries are supposed to be indexed in UNIMARC Koha.

BUT, this is not working, because : 
In Bib1, there is this attribute "att 1053    Country-publication", and no attribute "Country-heading"
In current ccl.properties, there is neither Country-heading nor Country-publication

I don't know how to index countries in MARC21. It could be done by some MARC21 guy in an other patch if they find it useful.

For 
2. music publisher

I took this page as a reference the name of the index in UNIMARC record.abs, but you are true, we would better use the name defined in bib1.

For 
3. "Date/time-last-modified"

I found some cases where the "/" was changed to a "-", so I supposed there were some reason for that.
for ex : 
the "official" bib1 1011 is "Date/time added to db "
but in Koha it is "date-entered-on-file"

But in Koha we have other att names with "/"
att 1013    Authority/format-id
and
att 1075    Subject-genre/form

Can you confirm me the "/" in an attribute name is not breaking anything ?
If it not the case, my change was not useful, and that we can keep the "/".


I will make a followup for 2 and 3.
Mathieu

I added some precisions to test plans for patch 1 and 2 (for example to test patch 2 you need to apply patch 1 before)

patch 3 is a followup reverting the change of Date/time-last-modified index, and renaming Music-number index.

M. Saby

There is a warn when you rebuild zebra, but I think it is not linked with my patches
(22:03:02-06/05 zebraidx(13748) [warn] Unknown register type: )

M. Saby

Long to test, but works.

But I would be very happy if we have a sample of UNIMARC bib record with its authorities/auth types for testing ready to be imported on standar dev install.

A small but known dataset could made search testing easier.

Good work!

Long to test, but quite long to write too ;-)

I would prefer a 2d signoff, but if the QA is made by a UNIMARC QAer (Biblibre?), they will have a set of records in UNIMARC, so I think they could confirm if it works or not.

Thank you for testing

M. Saby

By the way, if tests and QA is OK, further work will be needed to change the behavior of koha-indexdefs-to-zebra.xsl (as explained in the commit message of attachment 21058 [details] [review].

M. Saby

(In reply to mathieu saby from comment #22)
> Long to test, but quite long to write too ;-)
> 
> I would prefer a 2d signoff, but if the QA is made by a UNIMARC QAer
> (Biblibre?), they will have a set of records in UNIMARC, so I think they
> could confirm if it works or not.
> 

Oh, I have UNIMARC records :)
I dumped data from BibLibre demo site and a sandbox, now I have ~15K records
But I don't have authorities correctly configured. May be in the future.

Regards,
Bernardo

OK. But DOM UNIMARC authority is the object of an other bug. So I don't think it is needed for this one.

M. saby

I don't like the idea of letting this patch entering into Koha. It means that for UNIMARC biblio-zebra-indexdefs.xsl will not be deduced automatically from biblio-koha-indexdefs.xml, like for MARC21. There is a risk that a library modify it indexes XML config file and overwrite you handcrafted XSL file.

Could you point specifically a portion of your XSL file that deal with coded fields substrings?

(In reply to Frédéric Demians from comment #26)
> I don't like the idea of letting this patch entering into Koha. 

I agree, the idea was just to know if the "diagnose" was good.
If so, as you say, we must change the koha-indexdefs-to-zebra.xsl, and check it does not break anything.



It means
> that for UNIMARC biblio-zebra-indexdefs.xsl will not be deduced
> automatically from biblio-koha-indexdefs.xml, like for MARC21. There is a
> risk that a library modify it indexes XML config file and overwrite you
> handcrafted XSL file.
> 
> Could you point specifically a portion of your XSL file that deal with coded
> fields substrings?


I wrote this patch months ago, so I don't remember well the details. I can't answer today, but I will try to investigate but maybe the people who know well koha-indexdefs-to-zebra.xsl  (who wrote it first?) could gave an answer quicker than me.

Mathieu

> (who wrote it first?) could gave an answer quicker than me.

Galen Charlton. I've added him to this patch CC List.

But explaining what kind of transformation isn't handled properly, and
giving a sample will help Galen or whoever is willing to help...

Galen, Frederic : 
Hopefully I wrote down the problem in comment 8, because I did not remember it this morning ;-)

In MARC21, indexing substrings is needed for controled field (001-009, with no subfields)
But in UNIMARC it is needed for subfields of 1XX fields too.

etc/zebdradb/xml/koha-indexdefs-to-zebra.xsl does only support indexing a position within a subfield in handle-one-index-control-field and "kohaidx:index_leader" mode="secondary" templates.
It is good for MARC21, but not for UNIMARC.
We need a template to index position within a subfield for UNIMARC 1XX fields too.

Current code for handle-one-index-control-field
xsl:template name="handle-one-index-control-field">
 214         <xsl:variable name="offset"><xsl:value-of select="@offset"/></xsl:variable>
 215         <xsl:variable name="length"><xsl:value-of select="@length"/></xsl:variable>
 216         <xsl:variable name="indexes">
 217             <xsl:call-template name="get-target-indexes"/>
 218         </xsl:variable>
 219         <z:index>
 220             <xsl:attribute name="name"><xsl:value-of select="normalize-space($indexes)"/></xsl:attribute>
 221             <xslo:value-of>
 222                 <xsl:attribute name="select">
 223                     <xsl:choose>
 224                         <xsl:when test="@length">
 225                             <xsl:text>substring(., </xsl:text>
 226                             <xsl:value-of select="$offset + 1" />
 227                             <xsl:text>, </xsl:text>
 228                             <xsl:value-of select="$length"/>
 229                             <xsl:text>)</xsl:text>
 230                         </xsl:when>
 231                         <xsl:otherwise>
 232                             <xsl:text>.</xsl:text>
 233                         </xsl:otherwise>
 234                     </xsl:choose>
 235                 </xsl:attribute>
 236             </xslo:value-of>
 237         </z:index>
 238     </xsl:template>


Mathieu

Reference :

http://www.ifla.org/files/assets/uca/unimarc_updates/BIBLIOGRAPHIC/u-b_1-_coded_information_block_update.pdf


"CODED INFORMATION BLOCK"
Definition and Scope of Fields
This block contains coded fixed

Data in these fields is defined in terms of the position of the character, counting the first character following the subfield identifier as zero. If a bibliographic agency does not supply any coded information in a given field, the
field will be omitted unless mandatory. If some data in a field is supplied but not all, the omitted data element positions will contain fill characters.

In fact, it is true for all 1XX fields EXCEPT 101 (language) and 102 (country), 124, 128,145 whose subfield are repeatable in a classic way.
For field 123 there is a mix (most subfields indexed in a classic way)
For field 125 there is a mix ($b and $c indexed in a classic way)

For the details, see http://archive.ifla.org/VI/8/unimarc-concise-bibliographic-format-2008.pdf
Most often, only $a is used. Sometimes $a and $b


Same thing for Authorities (fields 100-160)


Mathieu

For authorities : 
101, 102, 152, 160 indexed in a classical way
the other must be indexed with "positional indexing".

Mathieu

It's easy to modify koha-indexdefs-to-zebra.xsl in order to generate
automatically substring extraction from MARC subfields, like for control
field (tag < 010). But your XML and XSL index files are not in
sync.

For example, in biblio-koha-indexdefs.xml, you have:

<index_subfields xmlns="http://www.koha-community.org/schemas/index-defs"
 tag="100" subfields="a" offset="8" length="1">

And in biblio-zebra-indexdefs.xsl, you have:

<z:index name="tpubdate:s">
   <xslo:value-of select="substring(., 8, 1)"/>
</z:index>

Since the XSL is valid, in biblio-koha-indexdefs.xml, you should have
rather:

<index_subfields xmlns="http://www.koha-community.org/schemas/index-defs"
 tag="100" subfields="a" offset="7" length="1">

In the XML file, counting start at 0, but it start at 1 in XSL file. So
you must modify your XML file, and substract 1 to all you offset
arguments.

Great!
I'll change xml file tomorrow

Mathieu

In fact, xml file is correct (same values as in UNIMARC manual, which is also begining by "0" position), and xsl wrong. So I am correcting xsl. 
It was signed off by Bernardo, but as - according to logic - it should NOT have worked, I will ask for a new sign off.

Followup coming.
Mathieu

> In fact, xml file is correct (same values as in UNIMARC manual, which
> is also begining by "0" position), and xsl wrong. So I am correcting
> xsl.  It was signed off by Bernardo, but as - according to logic - it
> should NOT have worked, I will ask for a new sign off.

By 'xsl' you mean Zebra Zebra XSL file?

Could you attach to this bug a sample XML UNIMARC biblio record, so we
can easily check that the indexes are properly populated?

I applied all my patches, then yours, and I run xsltproc.
I make a diff between the resulting zebra xsl files

A lot of changes like this one :
-          <xslo:value-of select="substring(., 8, 1)"/>
+          <xslo:value-of select="substring(., 9, 1)"/>

A new template called :
+      <xslo:apply-templates mode="index_all"/>
[...]
-  <xslo:template match="*">
+  <xslo:template mode="index_all" match="text()">


So, first point, your patch is working well ;-)

For the positions : 
Ex : pubdate
In the manual (which starts counting position from 0) : position 8, length 1
http://multimedia.bnf.fr/unimarcb_trad/B100-6-2010.pdf
So I suppose it must be position 8 in koha-xml file, and position 9 in zebra-xsl file.
That's what we get.

I copied the resulting koha-xsl file into my INSTALLDIR/etc/zebradb/marc_defs/unimarc/biblios/
Then I ran rebuild_zebra.pl -x -b -v -r
I did some advanced searches with coded fields, it seems to work.
Of course, I won't sign off by myself...

So, for the moment, I sign off your patch.

Created attachment 21452 [details]
Few UNIMARC biblio records in MARCXML

I replace your sample file, which is in ISO2709, with a MARCXML file containing less records. This way Zebra indexing can be tested directly with:

  xsltproc biblio-zebra-indexdefs.xsl bib-unimarc.xml

A UNIMARC expert could try to locate any inconsistency in the new indexes.

(In reply to Frédéric Demians from comment #41)
> Created attachment 21452 [details]
> Few UNIMARC biblio records in MARCXML
> 
> I replace your sample file, which is in ISO2709, with a MARCXML file
> containing less records. This way Zebra indexing can be tested directly with:
> 
>   xsltproc biblio-zebra-indexdefs.xsl bib-unimarc.xml
> 

I did not know it was possible!

(By the way, & out of scope, importing xml records into Koha would be a nice enh)

Mathieu

For further changes after this bug is pushed : make clearer the distinction between 210$b and 995r. 210$b is a typical standard unimarc field on record level, in plain text, while 995$r is a Koha field on item level. They should not be indexed in same indexes. But I will ask the list first...

Mathieu

QA comment:

The patch works as expected. I'm not sure to have tested all cases but I don't encounter any problem.
Tested with GRS-1 and DOM, tested indexes, coded fields, etc.

I am not an UNIMARC expert and certainly not a zebra expert but I take the responsibility of passing QA on these patch relying on Frédéric and Bernardo sign off.

Note: The zebra index will have to be rebuilt.

Marked as Passed QA.

Created attachment 21945 [details] [review]
Bug 8252: Fix indexing of UNIMARC 1xx for GRS-1

Before fixing UNIMARC DOM indexing, we must fix GRS-1 indexing

1) In advanced search, some Coded fields index are not working : Print, Illustration, Content
2) Country-heading index is not working
3) Some subfields are indexed in wrong indexes :
102$a should be in Country-publication instead of Country-heading (non defined in bib1.att)
106$a, filled only for printed works, should be in ff88-23 (form of item) instead of itype
(ff88-23 is made for Marc21 008 pos 23, which contains the same data as 106a)
200$b should be in Material-type instead of (or in addition to) itype and itemtype
(Material-type :"free-form string, ... that describes the material type of the item, e.g., cassette, kit, computer database, computer file.")
100$a pos 22-24 should not be indexed as "ln" : it is the language of the record, not the language of the ressource
4) Index names are too long : if we index new positions of coded fields, with existing names it breaks zebra indexing (there must be a limit in line lenghth in record.abs?)
5) There are a lot of warns when rebuiding zebra.

This patch make some changes in bib1.att (could be used later to improve search) :
- fixing wording for att 51 and 1012
- adding comments for attributes based on MARC21 008 field (8800-8841)
- creating 8806 (tpubdate), 8838 (Modified-code), 8818 (ff8-18), 8840 (ff8-18-21), 8819 (ff8-19), 8821 (ff8-21), 8828 (ff8-28), 8830 (ff8-30), 8831 (ff8-31)
- creating attributes specific to UNIMARC : 9701-9707 (Video-mt, Graphics-type, Graphics-support, Title-page-availability, Cumulative-index-availability, script-Title, char-encoding)
- setting apart 3 blocks of attributes, so it could be easy to make further changes :
-- common to Marc21 and UNIMARC : 8806, 8822, 8838
-- slightly different in Marc21 and UNIMARC (different meanings according to the type of the record => don't match a single UNIMARC field)
-- specific to UNIMARC : 9701-9707

in ccl.properties :
- creating a new index: Country-publication 1=1053
- suppressing some warns by mapping with bib1 att: Date-time-last-modified, Name, rtype, Music-number
- defining indexes using the 3 blocks attributes defined in bib1 (common to Marc21 and UNIMARC, slightly different, specific to UNIMARC)

in record.abs :
- renaming some index for 100-105-110 fields
- correcting indexing of 102$a (country of publication)
                         106$a (ff88-23)
                         100$a pos 22-24 (language of record, no more indexed)
                         105$a pos. 0-3 (illustration code)
                         200$b (for the moment, I keep it indexed in itype and itemtype, but also Material-Type)

in C4/Search.pm :
- adding "Country-publication" index

in opac and staff interface template subtypes_unimarc.in :
- renaming indexes to take into account the changes made to Zebra config files

To test (this cannot be done with a sandbox) :
1) apply the patch in a UNIMARC GRS-1 Koha instance
2) copy the following files from the etc/zebradb of your source directory into the etc/zebradb of your main Koha directory :
-- etc/zebradb/biblios/etc/bib1.att
-- etc/zebradb/ccl.properties
-- etc/zebradb/marc_defs/unimarc/biblios/record.abs
3) reindex your data (rebuild_zebra -x -b -r -v)
4) try to use those Coded fields indexes in Advanced search, in OPAC and Staff interface (available after clicking on "More options", then on "Coded information filters"):
Audience, Print, Literary genre, Biography, Illustration, Content, Video Types, Serials, Serial Type, Periodicity, Regularity
5) try to search "Country-publication=FR" in simple search

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
No koha-qa errors.

Tests for GRS-1
Followed test plan
Search by coded fields works, but only on OPAC,
on staff there are few options
Search by Country-publication works after patch

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>

Created attachment 21946 [details] [review]
Bug 8252: Fix indexing of UNIMARC 1xx for DOM

This patch makes the same changes in UNIMARC DOM configuration as PATCH 1 made for GRS-1.
You MUST apply the first patch with this one (this one depends on some modifications made by patch 1/2)

positions of subfields are indexed that way :
In biblio-koha-indexdefs.xml :
tag="100" subfields="a" offset="17" length="1"
In biblio-zebra-indexdefs.xsl :
xslo:value-of select="substring(., 17, 1)"

I had to edit biblio-zebra-indexdefs.xsl by hand, because etc/zebdradb/xml/koha-indexdefs-to-zebra.xsl does only support "subtring" in handle-one-index-control-field template.
It is good for MARC21, but not for UNIMARC : in MARC21, indexing subtrings is needed for controled field (001-009, with no subfields)
But in UNIMARC it is needed for subfields of 1XX fields.
So if DOM indexing is working with these new files, we may need to change koha-indexdefs-to-zebra.xsl.

Test plan (not possible in a sandbox) :
1) In a Koha instance using UNIMARC and DOM indexing
2) apply Patch 1 and Patch 2 (this one)
3) copy the following files from the etc/zebradb directory of your source into the etc/zebradb directory of your main Koha directory :
-- etc/zebradb/marc_defs/unimarc/biblios/biblio-koha-indexdefs.xml
-- etc/zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl
-- etc/zebradb/ccl.properties
-- etc/zebradb/biblios/etc/bib1.att
4) rebuild zebra with -x -b -r -v options
5) check if coded filters in advanced search are usable in OPAC and Staff interface

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Works. No koha-qa errors.

Test for DOM
Apply patches
Don't forget to copy files
reindex
Search by coded fields works, also Country-publication

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>

Created attachment 21947 [details] [review]
Bug 8252 : Followup for Date/time-last-modified and Music number

This followup restores the original wording of "Date/time-last-modified" index, and change the name of "Music-number" index to "Number-music-publisher"

To test :
1. In a UNIMARC Koha instance
2. apply patchs #1, #2 and this followup
3. copy from src/etc/zebradb directory to the etc/zebradb/ in your main koha directory the following files :
-- zebradb/biblios/etc/bib1.att
-- zebradb/ccl.properties
-- zebradb/marc_defs/unimarc/biblios/record.abs
-- zebradb/marc_defs/unimarc/biblios/biblio-koha-indexdefs.xml
-- zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl
4. rebuild zebra with -b -x -v -r options
5. write a value like "test071a" in 071$a field in a record
6. check if you can find this record with this search : "ccl=Number-music-publisher:test071a"

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
No koha-qa errors.

Test
Copy files
reindex full
Modify a couple of record to add 071a with test message
Reindex -v -z -b -x
Search test message as described and found modified records.

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>

Created attachment 21948 [details] [review]
Bug 8852 XSL handling sufield substring extraction

This patch modify koha-indexdefs-to-zebra.xsl in order to add the
ability to populate indexes with subfield substring.

It's now possible to understand such construction as:

<index_subfields xmlns="http://..." tag="100" subfields="a" offset="7" length="1">
  <target_index>tpubdate:s</target_index>
</index_subfields>

Signed-off-by:Mathieu Saby <mathieu.saby@univ-rennes2.fr>
I applied the patch and run xsltproc koha-indexdefs-to-zebra.xsl ../marc_defs/unimarc/biblios/biblio-koha-indexdefs.xml > ../marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl
I lookes at the generated file. It looks nice.
Then I copied it file in my INSTALLDIR/etc/zebra.... and reindexed my records with rebuild_zebra.pl
I made some searches on coded position index and non coded position indexes, everything works.

http://bugs.koha-community.org/show_bug.cgi?id=8252

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>

Created attachment 21949 [details] [review]
Bug 8252: Fixing biblio-zebra-indexdefs.xsl

    Bug 8252: Fixing biblio-zebra-indexdefs.xsl

    This patch fixes biblio-zebra-indexdefs.xsl files.
    It was generated from biblio-koha-indexdefs.xsm with the new koha-indexdefs-to-zebra.xsl amended by F. Démians's patch.

    To test :
    - take a DOM UNIMARC Koha
    - apply all the patchs of 8252 bug, including this one
    - copy src/etc/zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl to your etc/zebradb/marc_defs/unimarc/biblios/ located in your installation directory
    - run rebuid_zebra -b -x -r -v
    - make advanced searches on staff interface and opac, on coded fields indexes (Audience, Literary genre, Biography, Illustration, Content, Video Types, Serial Type, Periodicity, Regularity, Picture)

    Signed-off-by: Frédéric Demians <f.demians@tamil.fr>

    Ok for me. This patch put in sync indexes XSL definition with
    authoritative XML definition. Subsequently, it won't be difficult to
    amend DOM UNIMARC indexes defintion if necessary. And, as it is, I don't
    see any regression, whereas I can see huge improvements. Thanks Mathieu!

    Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>

Pushed to master.  Thanks, Mathieu and Frédéric!

yippee!

Mathieu

Hi Mathieu,

I've giving a try to this patch, it fixes some of the errors that appears in zebra logs.
But the following aren't:
21:12:42-22/10 zebraidx(29295) [warn] Index 'UPC' not found in attset(s)
21:12:42-22/10 zebraidx(29295) [warn] Index 'UPC' not found in attset(s)
21:12:42-22/10 zebraidx(29295) [warn] Index 'Serials' not found in attset(s)
21:12:42-22/10 zebraidx(29295) [warn] Index 'Serials' not found in attset(s)
21:12:42-22/10 zebraidx(29295) [warn] Index 'Printed-music' not found in attset(s)
21:12:42-22/10 zebraidx(29295) [warn] Index 'Printed-music' not found in attset(s)
21:12:42-22/10 zebraidx(29295) [warn] Index 'Electronic-ressource' not found in attset(s)
21:12:42-22/10 zebraidx(29295) [warn] Index 'Title-host' not found in attset(s)
21:12:42-22/10 zebraidx(29295) [warn] Index 'title-host' not found in attset(s)
21:12:42-22/10 zebraidx(29295) [warn] Index 'Title-host' not found in attset(s)
21:12:42-22/10 zebraidx(29295) [warn] Index 'title-host' not found in attset(s)
21:12:42-22/10 zebraidx(29295) [warn] Index 'lcn' not found in attset(s)
21:12:42-22/10 zebraidx(29295) [warn] Index 'lcn' not found in attset(s)

(note that title-host should be spelled Title-host in 2 places)

Some questions:
 * is there any reason why you haven't fixed those attributes as well ?
 * how to fix them ? Add another UNIMARC specific attribute into bib1.att/ccl.properties, change index name in record.abs or remove it ?

Thanks for your feedback !
(I may provide a patch, in another bug)

(In reply to Paul Poulain from comment #53)
> Hi Mathieu,
> 
> I've giving a try to this patch, it fixes some of the errors that appears in
> zebra logs.
> But the following aren't:
> 21:12:42-22/10 zebraidx(29295) [warn] Index 'UPC' not found in attset(s)
> 21:12:42-22/10 zebraidx(29295) [warn] Index 'UPC' not found in attset(s)
> 21:12:42-22/10 zebraidx(29295) [warn] Index 'Serials' not found in attset(s)
> 21:12:42-22/10 zebraidx(29295) [warn] Index 'Serials' not found in attset(s)
> 21:12:42-22/10 zebraidx(29295) [warn] Index 'Printed-music' not found in
> attset(s)
> 21:12:42-22/10 zebraidx(29295) [warn] Index 'Printed-music' not found in
> attset(s)
> 21:12:42-22/10 zebraidx(29295) [warn] Index 'Electronic-ressource' not found
> in attset(s)
> 21:12:42-22/10 zebraidx(29295) [warn] Index 'Title-host' not found in
> attset(s)
> 21:12:42-22/10 zebraidx(29295) [warn] Index 'title-host' not found in
> attset(s)
> 21:12:42-22/10 zebraidx(29295) [warn] Index 'Title-host' not found in
> attset(s)
> 21:12:42-22/10 zebraidx(29295) [warn] Index 'title-host' not found in
> attset(s)
> 21:12:42-22/10 zebraidx(29295) [warn] Index 'lcn' not found in attset(s)
> 21:12:42-22/10 zebraidx(29295) [warn] Index 'lcn' not found in attset(s)
> 
> (note that title-host should be spelled Title-host in 2 places)
> 
> Some questions:
>  * is there any reason why you haven't fixed those attributes as well ?
>  * how to fix them ? Add another UNIMARC specific attribute into
> bib1.att/ccl.properties, change index name in record.abs or remove it ?
> 
> Thanks for your feedback !
> (I may provide a patch, in another bug)



Hi
1/ It was already a complex patch to write and to test ;-)
2/ It is out of the main scope, which was, fixing coded fieds. Those warns are not linked with coded fields

So it could be fixed in other patches.

Mathieu

> >  * how to fix them ? Add another UNIMARC specific attribute into
> > bib1.att/ccl.properties, change index name in record.abs or remove it ?

I don't know it right know, but I'm pretty certain the same logic could be apply for them : check if these indexes can be mapped with existing ccl/bib1 values. And if not, create custom ccl/bib1 values for doing that.

Mathieu

2 other rules of my own would be
"check if the equivalent fields in MARC21 are already indexed, and if the MARC21 index can be reused"
and
"check if this index as a real utility"

Little enquiry : 
Here are the use for these indexes : 

melm 072$a  	UPC:w,Identifier-standard:w
melm 072$z  	UPC:w,Identifier-standard:w
melm 207    Serials,Serials:p
melm 208    Printed-music,Printed-music:p
melm 230$a      Electronic-ressource
melm 461$t    Title,Title-host:w,title-host:p
lcn is not used

I think I would try this :
UPC : 
In MARC21, it is in 024, alongside other identifiers (the indicator of the fields precise the nature of the identifier, but of course record.abs cannot take that into consideration...)
So UPC is only defined in UNIMARC, whereas it is an id kind of EAN, used mainly in the USA and Canada, MARC21 countries. That's not logical. This index is useless for UNIMARC ppl, especially if it is not working well.
=> my proposition: get rid of UPC in UNIMARC conf files and index 072 in "Identifier-standard" and in ccl.properties
If marc21 ppl want to index this in their DOM conf file, they will create the propoer custom definition in ccl and bib1.att.

207 field is equivalent to 362 field in MARC21
208 field is equivalent to 254 field in MARC21
http://www.loc.gov/marc/unimarctomarc21_2xx5xx.pdf
Both are not indexed in MARC21, so maybe the information is not really important.
BNF sites give some examples
http://multimedia.bnf.fr/unimarcb_trad/B207-6-2010.pdf
"Cette zone contient les numéros de la première et de la dernière livraison de la ressource continue et/ou les dates que couvrent ces livraisons quand ces numéros et ces dates sont connus."
Ex : "Vol. 1, no. 1 (1 Apr. 1946)-vol. 26, no. 195 (10 Oct.1972) = no. 1-no. 6943 "
=> for me, this field should NOT BE indexed in any way!

http://multimedia.bnf.fr/unimarcb_trad/B208-6-2010.pdf
"Cette zone contient une mention, transcrite à partir de la ressource, relative à la présentation particulière d'une ressource musicale imprimée ou manuscrite. "
Ex : "Partition de poche"
Maybe it can be usefull, and I did not find declarations for that in ccl or bib1, so let's create a custom one, OR maybe index it in notes.

Electronic-ressource : 230 is deprecated in UNIMARC. Maybe present in legacy records, but not sure of that.
Let's get rid of this index.


title-host : typo for Title-host

461: Title-host:
as we have 461$9 indexed in "Host-Item-Number", I guess this "Title-host" is a mistake for "Host-item".
So, 461 would behave like 773 in MARC21

(Technically, 461 is not a strict equivalent of 773. From LOC site, I get that the official equivalent of 461 in MARC21 is 774)


Mathieu

I cannot change the bug's status to "Need sign off", so I will create a new one.

M. Saby