| Summary: | Indexing problem with "&" symbol in Zebra. Some biblio-records with "&" symbol are not searching | ||
|---|---|---|---|
| Product: | Koha | Reporter: | Yuriy Kotsyuk <yuriy.kotsyuk> |
| Component: | Searching - Zebra | Assignee: | Bugs List <koha-bugs> |
| Status: | CLOSED INVALID | QA Contact: | |
| Severity: | normal | ||
| Priority: | P5 - low | CC: | dcook |
| Version: | 21.05 | ||
| Hardware: | All | ||
| OS: | Linux | ||
| GIT URL: | Initiative type: | --- | |
| Sponsorship status: | --- | Comma delimited list of Sponsors: | |
| Crowdfunding goal: | 0 | Patch complexity: | --- |
| Documentation contact: | Documentation submission: | ||
| Text to go in the release notes: | Version(s) released in: | ||
| Circulation function: | |||
Hi Yuriy, 21.05 is very old and no longer maintained. I suggest that you do a test in a newer version of Koha if the problem still remains. You could use a Koha sandbox or one of the publicly available demo installations. I can add new records without problem, even it has "&" symbol... Also, I found an “incorrect” record through the database and changed the “&” symbol to “ZZZ” (it’s a rare combination). Then I rebuilt the index. This record was indexed correctly and was found by the usual search. Next, I used the librarian interface to change “ZZZ” back to “&” symbol. After these operations, I got a really correct searchable bib-record. I’m not really sure that the problem is with the old koha version… (In reply to Katrin Fischer from comment #1) > Hi Yuriy, 21.05 is very old and no longer maintained. I suggest that you do > a test in a newer version of Koha if the problem still remains. You could > use a Koha sandbox or one of the publicly available demo installations. Well, I updated the version of koha to 23.05.06.000, but the problem still exists. (In reply to Yuriy Kotsyuk from comment #3) > I’m not really sure that the problem is with the old koha version… This has the look of a data migration issue to me... When the & is added to the MARC record via the HTML form, it should be automatically converted into the entity & When I do use the HTML form, I see the following in the database: <subfield code="b">Norton & Company</subfield> If I recall correctly, this automatic entity escaping is part of the MARC::File::XML: https://metacpan.org/dist/MARC-File-XML/source/lib/MARC/File/XML.pm#L370 -- Do you know how this record was added to Koha? The problem is that your record isn't valid XML. While it is possible for Koha to make invalid XML via the HTML form, it's very unlikely to happen with the &.
So I'd say these records are from an external source that created them without regard to XML validity.
Another noticeable flaw is also the following:
<datafield tag="852" ind1=" " ind2=" ">
<subfield code="a"></subfield>
</datafield>
That would never happen in a record created by Koha I think. If there's an empty value, the subfield and field would not have been created at all.
--
My suggestion would be to write a script to iterate through your records and check whether or not they can be parsed as XML. Flag the ones that cannot be parsed, and look at fixing the problems.
I'm going to mark this as invalid as I don't think it's a Koha bug. You can find more information on getting support with Koha here: https://koha-community.org/support/ I'm also happy to answer some follow up questions to the answers I've left here, since it seems like you're not too far away from a solution to the data problem. Well, I see. I have to correct invalid records through direct access to database. May be one more question: where can I find log with rebuilding koha-zebra index errors? (In reply to Yuriy Kotsyuk from comment #8) > Well, I see. I have to correct invalid records through direct access to > database. May be one more question: where can I find log with rebuilding > koha-zebra index errors? If you're running it manually, then it will just be on the screen. If it's a cronjob, then it'll be in your cron output, which will be emailed based on your system configuration. That's well outside the scope of Koha though. |
I find a problem in my academy koha instance. My librarians can't find some biblio-records. I check database and figured out that records looks good. So, I decided to rebuild zebra index. During rebuilding where a lot of errors. Simple analyses show, that problem was with "&" symbol. Example below: ... Wide character in warn at /usr/share/koha/lib/C4/Biblio.pm line 1199. problem with :423324 : :42: parser error : xmlParseEntityRef: no name <subfield code="b">Norton & Company</subfield> ^ <?xml version="1.0" encoding="UTF-8"?> <record xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd" xmlns="http://www.loc.gov/MARC21/slim"> <leader>00516nam a22002177a 4500</leader> <controlfield tag="003">UA-OsUOA</controlfield> <controlfield tag="001"> 133930 </controlfield> <controlfield tag="007">ta</controlfield> <controlfield tag="008">20181112b un ||||| |||| 00| 0 ukr d</controlfield> <datafield tag="090" ind1=" " ind2=" "> <subfield code="a">51</subfield> </datafield> <datafield tag="942" ind1=" " ind2=" "> <subfield code="c">BK</subfield> </datafield> <datafield tag="040" ind1=" " ind2=" "> <subfield code="a">UA-OsUOA</subfield> <subfield code="b">eng</subfield> <subfield code="c">UA-OsUOA</subfield> <subfield code="d">UA-OsUOA</subfield> </datafield> <datafield tag="245" ind1=" " ind2=" "> <subfield code="a">The Principles of Mathematics</subfield> <subfield code="c">B. Russell</subfield> </datafield> <datafield tag="300" ind1=" " ind2=" "> <subfield code="a">535 p. с.</subfield> </datafield> <datafield tag="955" ind1=" " ind2=" "> <subfield code="a">2</subfield> </datafield> <datafield tag="041" ind1=" " ind2=" "> <subfield code="a">eng</subfield> </datafield> <datafield tag="080" ind1=" " ind2=" "> <subfield code="a">51</subfield> </datafield> <datafield tag="260" ind1=" " ind2=" "> <subfield code="a">New York</subfield> <subfield code="b">Norton & Company</subfield> <subfield code="c">1951</subfield> </datafield> <datafield tag="852" ind1=" " ind2=" "> <subfield code="a"></subfield> </datafield> <datafield tag="650" ind1=" " ind2="7"> <subfield code="a">51 Математика</subfield> <subfield code="2">UDC</subfield> </datafield> <datafield tag="999" ind1=" " ind2=" "> <subfield code="c">423324</subfield> <subfield code="d">423324</subfield> </datafield> <datafield tag="901" ind1=" " ind2=" "> <subfield code="c">2</subfield> <subfield code="d">Юрій Анатолійович Коцюк</subfield> </datafield> </record> error retrieving biblio 423324 at /usr/share/koha/bin/migration_tools/rebuild_zebra.pl line 686. ... One more example: ... Wide character in warn at /usr/share/koha/lib/C4/Biblio.pm line 1199. problem with :423001 : :43: parser error : EntityRef: expecting ';' <subfield code="u">http://zt.knteu.kiev.ua/index.php?option=com_content&view ^ :43: parser error : EntityRef: expecting ';' ld code="u">http://zt.knteu.kiev.ua/index.php?option=com_content&view=article&id ^ :43: parser error : EntityRef: expecting ';' http://zt.knteu.kiev.ua/index.php?option=com_content&view=article&id=2349&Itemid ^ :43: parser error : EntityRef: expecting ';' .knteu.kiev.ua/index.php?option=com_content&view=article&id=2349&Itemid=721&lang ^ ... It's looks like script try use ampersand sequence...