Bug 15050 - Nonpublic note searchable from OPAC
Summary: Nonpublic note searchable from OPAC
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: Cataloging (show other bugs)
Version: Main
Hardware: All All
: P5 - low major with 15 votes (vote)
Assignee: Galen Charlton
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-10-22 11:54 UTC by Mirko Tietgen
Modified: 2022-06-10 06:03 UTC (History)
9 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Mirko Tietgen 2015-10-22 11:54:03 UTC
Text in the Nonpublic note can be found via OPAC. The text is not displayed, but if you know what you are looking for, you still get the information. Found in 3.20, I assume it is still valid in master. Bug 13023 does not fix it.

To reproduce, put something unique in an item's nonpublic note. Re-index. Search from the OPAC -> you will find the title your item belongs to. Delete the nonpublic note and reindex, you can't find it anymore.

Don't know if this qualifies as a security risk.
Comment 1 Marc Véron 2015-10-27 06:08:03 UTC
Reproduced on current master 3.21.00.041
Comment 2 Jonathan Druart 2015-11-10 15:31:34 UTC
How important is this one?
Should we completely remove itemnotes_nonpublic from the indexed fields?
Comment 3 Mirko Tietgen 2015-11-10 15:39:43 UTC
I think we should not index it.
Comment 4 Jonathan Druart 2015-11-11 15:33:59 UTC
Are you sure it happens with the current zebra config?
I get the problem with a UNIMARC install, but there is:

1437   <!--995       item   # just to index every subfield-->
1438   <index_data_field tag="995">
1439     <target_index>item:w</target_index>
1440   </index_data_field>

In etc/zebradb/marc_defs/unimarc/biblios/biblio-koha-indexdefs.xml

I don't see anything similar for MARC21.
Comment 5 Mirko Tietgen 2015-11-11 15:55:21 UTC
IIRC it is in items.more_subfields_xml and ends up in the index that way.
Comment 6 Jonathan Druart 2015-11-11 16:53:08 UTC
It should not if it's mapped with a DB field (bug 13023).
Comment 7 Marcel de Rooy 2015-11-14 15:26:07 UTC
(In reply to Mirko Tietgen from comment #0)
> Don't know if this qualifies as a security risk.
No, I would not call it like that :)

Please have a look at bug 12872. It is broader but covers this bug too.
The problem is that DOM indexes all under the Any keyword via the following lines:
            <xslo:template mode="index_all" match="text()">
                <z:index name="Any:w Any:p">
                    <xslo:value-of select="."/>
                </z:index>
            </xslo:template>
Look at the first obsolete patch too.
As discussed before on some older reports too, it would be nice to have an optional Anywhere (just like Any now), and a more restricted Any (only include the defined indexes as we were used too under GRS1). If you would not like Anywhere for various reasons (index size or hidden fields etc.), disable it some way.
Comment 8 Katrin Fischer 2015-11-14 15:35:39 UTC
Hm, my impression was that Any includes everything that is included in one of the other indexes - could there be another reason for the phenomenon?
Comment 9 Katrin Fischer 2015-11-17 18:31:59 UTC
Argh, ignore my last comment. I think this is a difference between GRS1 and DOM indexing?
Comment 10 Marcel de Rooy 2015-11-18 07:08:51 UTC
(In reply to Katrin Fischer from comment #9)
> Argh, ignore my last comment. I think this is a difference between GRS1 and
> DOM indexing?

Right: as we were used too under GRS1...
Comment 11 Katrin Fischer 2019-06-14 09:39:23 UTC
So since we switched to DOM indexing, the any (keyword) index will include all information from the bibliographic record and the item records. This can lead to confusing results in search:

- internal note is searchable
- A gift from someone that also appears as author will be found (the name was stored as bookseller in the item)

It causes confusion. Is there any way around it?
Comment 12 Marcy Dunning 2020-09-21 18:32:05 UTC
We have the same problem with basic searches on names pulling up donors as well as authors. While users can restrict the searches by limiting them to the author field, we want to make the OPAC as easy as possible to use. In some cases we have donors who have given hundreds of items, making a search on that name very confusing. Thanks for working on this.
Marcy
Comment 13 Katrin Fischer 2020-12-10 14:05:33 UTC
This pops up as an issue at our libraries quite regularly. Has someone found a solution to this yet?
Comment 14 Elaine Bradtke 2021-10-08 18:41:21 UTC
This problem applies to things like Locations as well.  
We have a lot of items set to display in the OPAC with the location 'storage'. This applies to everything from any room outside the reading room to the safe, archival storage room, and locked cases for rare books. The staff interface shows the specific location.
If you know what to search for, the specific locations will come up with a keyword search in the OPAC. 
Non public notes and the ability to really obscure certain things in the OPAC are in MARC for a reason. It can be a security issue, in the physical sense.  
Because we have so many locations, it's not practical to make up obscure codes for the staff interface.

We also have donor information in some non-public fields, which confuses the search results.
Comment 15 Fridolin Somers 2022-06-08 06:17:10 UTC
(In reply to Katrin Fischer from comment #11)
> So since we switched to DOM indexing, the any (keyword) index will include
> all information from the bibliographic record and the item records. This can
> lead to confusing results in search:
> 
> - internal note is searchable
> - A gift from someone that also appears as author will be found (the name
> was stored as bookseller in the item)
> 
> It causes confusion. Is there any way around it?

With Elasticsearch only MARC field defined with a search field are in default search field.
Maybe fields for OPAC and for staff can be configured separately.

Deeply I think MARC datas should always be public ;)
Comment 16 David Cook 2022-06-08 06:49:48 UTC
(In reply to Fridolin Somers from comment #15)
> With Elasticsearch only MARC field defined with a search field are in
> default search field.
> Maybe fields for OPAC and for staff can be configured separately.
> 
> Deeply I think MARC datas should always be public ;)

With Zebra, we could be using the "ViewPolicy" filter with the RecordProcessor, although that would mean those non-public things would also not be searchable on the staff interface either. 

Probably not realistic to have different indexes for OPAC and Staff Interface for Zebra at least.
Comment 17 Katrin Fischer 2022-06-08 08:28:01 UTC
> Deeply I think MARC datas should always be public ;)

I agree to a point. The problem is, that we have no way to store item specific internal information outside of MARC. I think this might be the missing feature that would resolve a lot of those issues.
Comment 18 David Cook 2022-06-08 23:16:37 UTC
(In reply to Katrin Fischer from comment #17)
> > Deeply I think MARC datas should always be public ;)
> 
> I agree to a point. The problem is, that we have no way to store item
> specific internal information outside of MARC. I think this might be the
> missing feature that would resolve a lot of those issues.

I was thinking the same thing last night. We have a tendency of storing everything in MARC, but maybe having internal "Koha notes" would be a good idea. 

Maybe we could have a way of automagically stuffing those into a MARC record if library staff needed to export to move the records somewhere...
Comment 19 Katrin Fischer 2022-06-09 07:08:13 UTC
Maybe it could be simpler... a list of item subfields NOT to add on export for indexing?
Comment 20 David Cook 2022-06-09 08:14:59 UTC
(In reply to Katrin Fischer from comment #19)
> Maybe it could be simpler... a list of item subfields NOT to add on export
> for indexing?

The caveat being that they won't be searchable for either the staff interface or the OPAC though.

--

As for mangling records for indexing:

In "sub marc_records_to_documents" in lib/Koha/SearchEngine/Elasticsearch.pm, there is some record mangling to handle indexing 880 fields. 

For Zebra, I created lib/Koha/Filter/MARC/Index880InZebra.pm, which is called in bin/migration_tools/rebuild_zebra.pl. 

It could be good to standardize indexing related mangling...
Comment 21 Katrin Fischer 2022-06-09 08:18:20 UTC
(In reply to David Cook from comment #20)
> (In reply to Katrin Fischer from comment #19)
> > Maybe it could be simpler... a list of item subfields NOT to add on export
> > for indexing?
> 
> The caveat being that they won't be searchable for either the staff
> interface or the OPAC though.
> 
> --

I think it might be ok, as we still have the SQL based item search for searching those fields internally.

 
> As for mangling records for indexing:
> 
> In "sub marc_records_to_documents" in
> lib/Koha/SearchEngine/Elasticsearch.pm, there is some record mangling to
> handle indexing 880 fields. 
> 
> For Zebra, I created lib/Koha/Filter/MARC/Index880InZebra.pm, which is
> called in bin/migration_tools/rebuild_zebra.pl. 
> 
> It could be good to standardize indexing related mangling...

Interesting, I wasn't ware of 880 indexing features with Elasticsarch - do you know more?
Comment 22 Elaine Bradtke 2022-06-09 17:44:59 UTC
I know this is specifically about item records, but there are non-public fields in the biblio records too. These are used by special libraries and archives who handle rare and otherwise unusual material.
Comment 23 David Cook 2022-06-09 23:28:38 UTC
(In reply to Katrin Fischer from comment #21)
> > In "sub marc_records_to_documents" in
> > lib/Koha/SearchEngine/Elasticsearch.pm, there is some record mangling to
> > handle indexing 880 fields. 
> > 
> > For Zebra, I created lib/Koha/Filter/MARC/Index880InZebra.pm, which is
> > called in bin/migration_tools/rebuild_zebra.pl. 
> > 
> > It could be good to standardize indexing related mangling...
> 
> Interesting, I wasn't ware of 880 indexing features with Elasticsarch - do
> you know more?

Yeah I've written more about it on Bug 15187. 

Basically, it also indexes the 880 as its associated field. 

E.g. "880 $6 245-01 $a 教牧書信" would be indexed as a 245 title as well. This is very handy for multilingual Koha libraries. I did the Zebra patches for a Chinese library of mine. (Learning basic Chinese for fun has been surprisingly helpful for working on Koha!)
Comment 24 Katrin Fischer 2022-06-10 06:03:13 UTC
> Yeah I've written more about it on Bug 15187. 
> 
> Basically, it also indexes the 880 as its associated field. 
> 
> E.g. "880 $6 245-01 $a 教牧書信" would be indexed as a 245 title as well. This
> is very handy for multilingual Koha libraries. I did the Zebra patches for a
> Chinese library of mine. (Learning basic Chinese for fun has been
> surprisingly helpful for working on Koha!)

Thx for pointing me to the bug, this is super interesting and a feature I always hoped for in Zebra :)