Bug 14527 - UNIMARC: zebra error 114 when searching for items added before or after a specific date / 995$5 is not indexed
Summary: UNIMARC: zebra error 114 when searching for items added before or after a spe...
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: Label/patron card printing (show other bugs)
Version: 3.20
Hardware: PC Linux
: P5 - low normal (vote)
Assignee: Chris Nighswonger
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-07-13 13:18 UTC by Pedro Ferreira
Modified: 2020-01-08 22:58 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
printscreen of zebra error (341.40 KB, application/zip)
2015-07-13 13:18 UTC, Pedro Ferreira
Details
printscreen of zebra search by title (243.62 KB, application/zip)
2015-07-14 08:38 UTC, Pedro Ferreira
Details
Zebra re-index (147.23 KB, image/png)
2015-07-15 08:29 UTC, Pedro Ferreira
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Pedro Ferreira 2015-07-13 13:18:03 UTC
Created attachment 40967 [details]
printscreen of zebra error

Hello when i am adding a new bathc for labeling
in 
127.0.1.1:8080/cgi-bin/koha/cgi-bin/labels/label-item-search.pl

and searching on or after a specific date,
ZEBRA returns the error:

Search biblios ERROR 114 1+0 RPN @attrset Bib-1 @attr 1=32 @attr 4=5 @attr 2=4 2015-07-13 

And no data is returned.
in the attachement i send two printscreen
Comment 1 David Cook 2015-07-13 23:50:53 UTC
You might consider the mailing lists (http://koha-community.org/support/koha-mailing-lists/) for finding a resolution to this problem.

How did you install Koha?

Do any search queries return results?

You can see a fuller explanation of error codes here (http://www.loc.gov/z3950/agency/defns/bib1diag.html), but I've noticed error 114 happens when you don't have anything indexed.

My guess is that you installed Koha via git or the tarball?
Comment 2 Katrin Fischer 2015-07-14 00:04:38 UTC
I have asked him to file the bug, but haven't got around to test and confirm the problem. Maybe someone else can?
Comment 3 Katrin Fischer 2015-07-14 00:07:41 UTC
Hm, it seems t work on on master.
Comment 4 David Cook 2015-07-14 03:41:24 UTC
I would be really curious to know if Pedro is able to get search results back for any query.

It really does sound to me like an empty Zebra database or at the very least an empty index being searched.
Comment 5 Pedro Ferreira 2015-07-14 08:38:39 UTC
Created attachment 40983 [details]
printscreen of zebra search by title
Comment 6 Pedro Ferreira 2015-07-14 08:40:27 UTC
hello,
When i search by title or anything else it returns the correct search result.
I attached two printscreens.

My koha was installed in a tarball, version 3.20.

Thanks
Comment 7 David Cook 2015-07-15 08:04:12 UTC
(In reply to Pedro Ferreira from comment #6)
> hello,
> When i search by title or anything else it returns the correct search result.
> I attached two printscreens.
> 
> My koha was installed in a tarball, version 3.20.
> 
> Thanks

Mmm, thanks for those screenshots.

Well, it looks like the Zebra indexing is running or you wouldn't get any results... and it looks like there are values in "dateaccessioned" as they're showing up in the search results. 

However, when I look at the code, it looks like the item values in the search results are coming directly from the database and not from the record returned by Zebra.

I'm still thinking it's a problem of the Zebra indexes not being up-to-date. 

Pedro, how is your Zebra indexing running? A cronjob? Or did you run rebuild_zebra.pl manually in the past?

I'm guessing that maybe you indexed your bibliographic records, then added items, and the records haven't been indexed again, so you're getting these errors because the index for the item's "dateaccessioned" is empty.
Comment 8 Pedro Ferreira 2015-07-15 08:29:43 UTC
Created attachment 41006 [details]
Zebra re-index
Comment 9 Pedro Ferreira 2015-07-15 08:32:49 UTC
Hello
 i re-index zebra and preform the search by date,it returned no output.

The data error in zebra is the same, ERROR 114.
Comment 10 Pedro Ferreira 2015-07-15 08:55:53 UTC
using the search data in the screenshot i performed the same search with yaz client.

so i did 

 YAZ-CLIENT>f @attr 1=32 @attr 4=5 @attr 2=4 2015-07-05

Sent searchRequest.
Received SearchResponse.
Search was a bloomin' failure.
Number of hits: 0, setno 7
Result Set Status: none
records returned: 0
Diagnostic message(s) from database:
    [114] Unsupported Use attribute -- v2 addinfo '32'
Elapsed: 0.000181

--------
Here i can see the 114 error, but i dont know what is @attr 1=32
Comment 11 Pedro Ferreira 2015-07-16 14:55:33 UTC
Hello
 in this demos
http://koha-community.org/demo/


the marc21 demos do not suffer from the bug
but the unimarc demos do suffer from the bug.
Comment 12 Katrin Fischer 2015-07-19 21:30:40 UTC
Hm, I don't see any error in my local Unimarc installation - but I don't get any results either as soon as I add dates to the mix. Could someone else test this on a Unimarc installation maybe?
Comment 13 David Cook 2015-07-20 23:36:28 UTC
(In reply to Pedro Ferreira from comment #11)
> Hello
>  in this demos
> http://koha-community.org/demo/
> 
> 
> the marc21 demos do not suffer from the bug
> but the unimarc demos do suffer from the bug.

I should've asked you a while ago whether you were using marc21 or unimarc, as that has an enormous impact on how the indexing is done for items...

This indeed looks like an indexing configuration bug.

If you compare the Zebra indexing files for marc21 vs unimarc, you'll find that the Date-of-acquisition index (ie @attr 1=32) is filled by items in MARC21 but not by UNIMARC. In UNIMARC, the only data going into this index is from 099$c, and I'm guessing that you don't have any data in that field in any of the records you're indexing, which is why you're getting this 114 error.

etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl
etc/zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl

It might be a good idea to add Fridolyn to this bug, as I'm not very familiar with UNIMARC or how it's generally handled by Koha. 

It looks like etc/zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl might need a bit of an update. According to unimarc_framework_DEFAULT.sql, 995$5 is date acquired, so it should be indexed into "Date-of-acquisition" for this search to work...

It looks like there's some weird stuff happening in etc/zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl as well... like 995$3 which is use restrictions appears to be indexed as withdrawn. Also, lots of the item data is being indexed into an 'item' index... which seems odd to me. I suppose that's an attempt at an item-level "any" index?
Comment 14 Pedro Ferreira 2015-08-03 09:00:26 UTC
can anyone guide me to sort this problem ..?
thanks
Comment 15 David Cook 2015-08-05 05:21:20 UTC
(In reply to Pedro Ferreira from comment #14)
> can anyone guide me to sort this problem ..?
> thanks

That's a fair question. 

It looks like your etc/zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl file needs the following block added within the   <xslo:template mode="index_subfields" match="marc:datafield[@tag='995']"> element.

    <xslo:for-each select="marc:subfield">
      <xslo:if test="contains('5', @code)">
        <z:index name="Date-of-acquisition:w Date-of-acquisition:d Date-of-acquisition:s">
          <xslo:value-of select="."/>
        </z:index>
      </xslo:if>
    </xslo:for-each>

So that it looks like:

  <xslo:template mode="index_subfields" match="marc:datafield[@tag='995']">
    <xslo:for-each select="marc:subfield">
      <xslo:if test="contains('5', @code)">
        <z:index name="Date-of-acquisition:w Date-of-acquisition:d Date-of-acquisition:s">
          <xslo:value-of select="."/>
        </z:index>
      </xslo:if>
    </xslo:for-each>

After patching the file, you can re-index Zebra, and then it should work.

It would probably be prudent for a developer to add it to the Koha codebase as well. I don't use UNIMARC, so it's not really a problem for my libraries and I, but I'm sure that this problem would raise its head with others.
Comment 16 David Cook 2015-08-05 05:22:47 UTC
Typically, it's not a good idea to patch individual files, and instead wait for bug fix updates, but since this is a configuration file stored in /etc... I don't think bug fix updates will touch it. 

I think if it were fixed in master, it would only be fixed for new installs... perhaps someone else can correct me...
Comment 17 Katrin Fischer 2015-08-05 09:09:37 UTC
(In reply to David Cook from comment #16)
> Typically, it's not a good idea to patch individual files, and instead wait
> for bug fix updates, but since this is a configuration file stored in
> /etc... I don't think bug fix updates will touch it. 
> 
> I think if it were fixed in master, it would only be fixed for new
> installs... perhaps someone else can correct me...

I don't think that's quite correct and depending on how you installed it different things might happen. 

For an older standard installation, rerunning the Makefile.pl during the update process will update your index definitions as well: 
perl Makefile.PL --prev-install-log <path>
make
make test
sudo make upgrade

I am adding Robin because I think he can tell what happens for a package installation. To my knowledge the index definitions are updated there as well - we build our own packages with some additional indexes and that seems to work perfectly fine. But I think local changes might create problems and even keep the files from updating.
Comment 18 Fridolin Somers 2015-08-05 15:48:41 UTC
In UNIMARC, Date-of-acquisition is usually defined in 099$c, this is the default Zebra configuration.

One must use the script misc/maintenance/UNIMARC_sync_date_created_with_marc_biblio.pl to copy creation date from database to 099$c.

We usually set it nightly to only update daily modified/created records :
misc/maintenance/UNIMARC_sync_date_created_with_marc_biblio.pl --run --where "TO_DAYS(NOW()) - TO_DAYS(biblio.datecreated) <= 1 OR TO_DAYS(NOW()) - TO_DAYS(DATE(biblio.timestamp)) <= 1"
Comment 19 Katrin Fischer 2015-08-05 16:04:35 UTC
Hi Fridolin, I think the dateaccessioned in MARC21 is a item field related to acquisitions - does UNIMARC have no dateaccessioned or similar in 995?
Comment 20 Robin Sheat 2015-08-05 22:57:01 UTC
(In reply to Katrin Fischer from comment #17)
> I am adding Robin because I think he can tell what happens for a package
> installation. To my knowledge the index definitions are updated there as
> well - we build our own packages with some additional indexes and that seems
> to work perfectly fine. But I think local changes might create problems and
> even keep the files from updating.

It should be OK. I haven't tested this, but I would expect it would warn you that it's changed and ask you if you want to keep your own version or if you want to install the packaged version. Either way, it'll keep the other version named with the extension .dpkg-{dist,orig}
Comment 21 Fridolin Somers 2015-08-06 08:16:11 UTC
(In reply to Katrin Fischer from comment #19)
> Hi Fridolin, I think the dateaccessioned in MARC21 is a item field related
> to acquisitions - does UNIMARC have no dateaccessioned or similar in 995?

I can't find anything. I've seen it sometimes in a numeric subfield.
Maybe there is no convention.
Comment 22 Katrin Fischer 2015-08-06 09:19:47 UTC
Pedro, have you followed the discussion? It looks like there is no date_accessioned in the item in UNIMARC - can you confirm this for your installation?
Comment 23 Pedro Ferreira 2015-08-06 10:06:15 UTC
Hello , 
yes i can confirm that there is no 999$c field

doing a grep i get

xekhz@mail:/var/www/koha3.20/etc/zebradb/marc_defs/unimarc/biblios$ grep 999 biblio-zebra-indexdefs.xsl 
<xslo:stylesheet xmlns:xslo="http://www.w3.org/1999/XSL/Transform" xmlns:marc="http://www.loc.gov/MARC21/slim" xmlns:z="http://indexdata.com/zebra-2.0" xmlns:kohaidx="http://www.koha-community.org/schemas/index-defs" version="1.0">



but i think the filed 995 is what i need, because date-of-aquisition is field 995 in unimarc.


doing a grep 

xekhz@mail:/var/www/koha3.20/etc/zebradb/marc_defs/unimarc/biblios$ grep 995 biblio-zebra-indexdefs.xsl 
  <xslo:template mode="index_subfields" match="marc:datafield[@tag='995']">
  <xslo:template mode="index_data_field" match="marc:datafield[@tag='995']">
  <xslo:template mode="index_facets" match="marc:datafield[@tag='995']">
xekhz@mail:/var/www/koha3.20/etc/zebradb/marc_defs/unimarc/biblios$ 


...
Comment 24 Katrin Fischer 2015-08-09 22:21:00 UTC
Hi Pedro,

not sure you grepped for the right things. Maybe a good first step would be to see if you currently store a dateaccessioned in your items or bibliographic data - checking the Koha-to-MARC-mappings for it and also the bibliographic frameworks.
Comment 25 Pedro Ferreira 2015-08-10 09:00:47 UTC
Hello,
koha to marc mapping i have in the items tab

itemnumber 	995 	9 	Internal item number (Koha itemnumber, autogenerated) 	Alterar
barcode 	995 	f 	Barcode 	Alterar
dateaccessioned 995 	5 	Date acquired 	Alterar
homebranch 	995 	b 	Origin of item (home branch) (coded) 	Alterar



in a test koha, i created the 995$c and liked it to the database field date-of-aquisition.
Re-index the zebra and the search is empty.
Comment 26 Pedro Ferreira 2015-09-16 09:27:00 UTC
any news how to solve the problem...
Comment 27 David Cook 2016-04-27 00:34:23 UTC
(In reply to Pedro Ferreira from comment #26)
> any news how to solve the problem...

I think we've already described the steps that you need to take to diagnose and solve your problem.

Can you tell us what parts you're still confused about?
Comment 28 Pedro Ferreira 2016-04-27 14:15:26 UTC
I did not test the new koha version 3.22
But i followed all the steps and in version 3.20 did not work..
Comment 29 David Cook 2016-04-28 02:36:02 UTC
(In reply to Pedro Ferreira from comment #28)
> I did not test the new koha version 3.22
> But i followed all the steps and in version 3.20 did not work..

Can you list here the exact steps that you took? That'll help us with troubleshooting. Thanks.
Comment 30 Fridolin Somers 2016-07-13 15:34:53 UTC
(In reply to Pedro Ferreira from comment #10)
> using the search data in the screenshot i performed the same search with yaz
> client.
> 
> so i did 
> 
>  YAZ-CLIENT>f @attr 1=32 @attr 4=5 @attr 2=4 2015-07-05
> 
> Sent searchRequest.
> Received SearchResponse.
> Search was a bloomin' failure.
> Number of hits: 0, setno 7
> Result Set Status: none
> records returned: 0
> Diagnostic message(s) from database:
>     [114] Unsupported Use attribute -- v2 addinfo '32'
> Elapsed: 0.000181
> 
> --------
> Here i can see the 114 error, but i dont know what is @attr 1=32

Attribute number 32 is defined in bib1.att = Date-of-acquisition.

By default 099$c is indexed as Date-of-acquisition.
Is it ?

Note that if you have no existing 099$c, the search query will fail, not juste say 0 results.

Test : 
yaz> f @attr 1=32 @attr 2=103 ''
Comment 31 Fridolin Somers 2018-10-04 13:19:38 UTC
Is this still valid ?

Maybe it is corrected by Bug 20765
Comment 32 George Veranis 2019-11-21 21:48:08 UTC
it is still valid at version 19.05 for unimarc. 
One way to solve it is to add zebra index for 995$5 or to use the script for the maintenance. It would more easy for libraries to added to zebra index configuration in my opinion
Comment 33 David Cook 2020-01-08 22:58:15 UTC
(In reply to George Veranis from comment #32)
> it is still valid at version 19.05 for unimarc. 
> One way to solve it is to add zebra index for 995$5 or to use the script for
> the maintenance. It would more easy for libraries to added to zebra index
> configuration in my opinion

This is something I've been thinking about more recently:

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12366
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15198

If you look at https://software.indexdata.com/zebra/doc/querymodel-zebra.html, there is a "Zebra Search Attribute Extension" of value 14, which "Specifies whether un-indexed fields should be ignored. A zero value (default) throws a diagnostic when an un-indexed field is specified. A non-zero value makes it return 0 hits."

I wonder sometimes if we should just add that to all Zebra searches.

It's a shame they didn't make that configurable at the server level rather than the query level. (I suppose one option would be to request that feature from Indexdata, but that would only help people running newer versions of Zebra.)