I know this bug is a little old now, but it is still pertinent. If you have an item type that cannot be viewed on the OPAC it still shows up in the facets. For example, we have an Interlibrary Loan item type that is not visible on the OPAC, but it appears in the facets when it meets the criteria of the search. Clicking on this item type will bring you to an error message. It's confusing to the patron and/or gives the impression that there are problems with the OPAC.

Created attachment 79233 [details]
Wrong count for facets

Facet should only take in count items that are not hidden.

Still valid !

Because the "OpacHiddenItems" is handled after results are returned from the search engine, it's really hard to do handle the logic well.

+1

We should have separate indexes for OPAC and staff interface.
Probably public_* for the OPAC. We could need some trick for when OpacHiddenItemsExceptions is matched (i.e. it should display the full thing).

Would this be just like 2 different itemtype indexes? or "full" index?
I think the first might be a good solution, but the opachiddenitems is currently a dymanic setting that works 'on the fly'. I think making new indexes will always require a full reindex so will limit the usefulness of the functionality a bit.

Is there an alternative solution where we could filter those from search dynamically?

(In reply to Katrin Fischer from comment #7)
> Would this be just like 2 different itemtype indexes? or "full" index?
> I think the first might be a good solution, but the opachiddenitems is
> currently a dymanic setting that works 'on the fly'. I think making new
> indexes will always require a full reindex so will limit the usefulness of
> the functionality a bit.
> 
> Is there an alternative solution where we could filter those from search
> dynamically?

I guess we could build smarter searches based on the context information (OpacHiddenItems, patron category, etc). It would be a not-so-simple development, but of course doable.

I wonder if the options of "exceptions" are not already a "killer" for the index idea - making things more complicated :(

(In reply to Katrin Fischer from comment #9)
> I wonder if the options of "exceptions" are not already a "killer" for the
> index idea - making things more complicated :(

Nah, you would use one query/index or the other depending on that context, which is a mix of settings and the logged-in user category and interface.

Still valid !

Using OpacSuppression is the only way to hide records without those searchengine side-effects.

(In reply to Tomás Cohen Arazi from comment #6)
> We should have separate indexes for OPAC and staff interface.
> Probably public_* for the OPAC. We could need some trick for when
> OpacHiddenItemsExceptions is matched (i.e. it should display the full thing).

This is something I've been thinking about when exporting data from Koha into other systems. OpacSuppression, OpacHiddenItems, OpacHiddenItemsExceptions... these all change behaviour without changing the actual data. 

With OpacSuppression it doesn't matter too much because it's just on/off, but the other 2 require potentially heavy calculations for updating an index...

Our workaround was to add CSS to remove the facet label (Where CustomType is the Item Type we're hiding.):
/*Hide the CustomType facet in search results*/
.facet-label a[title="CustomType"] {display: none !important; }

(In reply to Joseph Alway from comment #14)
> Our workaround was to add CSS to remove the facet label (Where CustomType is
> the Item Type we're hiding.):
> /*Hide the CustomType facet in search results*/
> .facet-label a[title="CustomType"] {display: none !important; }

To clarify this was added to the OPACUserCSS system preference.

Created attachment 164978 [details] [review]
Bug 14007: Hide hidden itemtypes in opac facets.

Test plan:

* Start koha-testing-docker with elasticsearch or opensearch (e.g., ku-es7)
* Make sure there is a built index for Elastic/OpenSearch:

  > koha-elasticsearch --reset --rebuild kohadev

* Search for "book" in opac and note the facet values for itemtype
* Change systempreference SearchEngine to ElasticSearch
* Search for "book" in opac and note the facet values for itemtype
* Go to the adminstration view of one of the itemtypes (for instance
  Visual Materials) and check "Hide in OPAC".
* Search for "book" in opac and make sure the hidden itemtype do
  not appear as facet.
* Search for "book" in staff interface and make sure the hidden
  itemtype still does appear as a facet.
* Change back the systemprefence to Zebra
* Search for "book" in opac and make sure the hidden itemtype do
  not appear as facet.
* Search for "book" in staff interface and make sure the hidden
  itemtype still does appear as a facet.

Created attachment 166030 [details] [review]
Bug 14007: Hide hidden itemtypes in opac facets.

Test plan:

* Start koha-testing-docker with elasticsearch or opensearch (e.g., ku-es7)
* Make sure there is a built index for Elastic/OpenSearch:

  > koha-elasticsearch --reset --rebuild kohadev

* Search for "book" in opac and note the facet values for itemtype
* Change systempreference SearchEngine to ElasticSearch
* Search for "book" in opac and note the facet values for itemtype
* Go to the adminstration view of one of the itemtypes (for instance
  Visual Materials) and check "Hide in OPAC".
* Search for "book" in opac and make sure the hidden itemtype do
  not appear as facet.
* Search for "book" in staff interface and make sure the hidden
  itemtype still does appear as a facet.
* Change back the systemprefence to Zebra
* Search for "book" in opac and make sure the hidden itemtype do
  not appear as facet.
* Search for "book" in staff interface and make sure the hidden
  itemtype still does appear as a facet.

Signed-off-by: David Nind <david@davidnind.com>

Testing notes (using KTD):

1. To start up Koha with Elasticsearch: ktd --es8 up (also --es7, os1, and os2) (tested with es7 and es8, os1 and os2 were having trouble starting up for me).

2. To reindex: koha-elasticsearch --rebuild -d -b -a kohadev

3. Note that in Elasticsearch some heading labels for facets are missing in the OPAC (but not in the staff interface or when using Zebra). I'm not sure whether this is related to the recent changes from bug 35138. I've created bug 36750 for this in case it isn't.

This looks reasonable to me but I'm not expert in the search code so I'm pining Nick to take a final QA eye ;)

Created attachment 166066 [details] [review]
Bug 14007: Reduce lookups

This works in the limited case of hiding itemtypes that are marked "Hide in OPAC" - for libraries using OpacHiddenItems to hide an itemtype this won't have any effect. I like how simple this is, however, I feel it is too narrow and following this path could end up with more post processing to handle all of the cases.

Code wise, I submitted a patch to reduce the lookups.

Moving to in discussion for now, I could be convinced this is better than not doing anything, but I think it's too late in this cycle for this one

(In reply to Nick Clemens (kidclamp) from comment #21)
> This works in the limited case of hiding itemtypes that are marked "Hide in
> OPAC" - for libraries using OpacHiddenItems to hide an itemtype this won't
> have any effect. I like how simple this is, however, I feel it is too narrow
> and following this path could end up with more post processing to handle all
> of the cases.
> 
> Code wise, I submitted a patch to reduce the lookups.
> 
> Moving to in discussion for now, I could be convinced this is better than
> not doing anything, but I think it's too late in this cycle for this one

Yeah I have similar thoughts. Patch doesn't fix the OpacHiddenItems issue but maybe better than nothing. OpacHiddenItems is almost always one of those "too hard basket" things unfortunately...

You know an alternative would actually be to take the OPACHiddenItems rules, check if there's itype, and then filter facet values accordingly?

That would help bug 37457

(In reply to David Cook from comment #23)
> You know an alternative would actually be to take the OPACHiddenItems rules,
> check if there's itype, and then filter facet values accordingly?
> 
> That would help bug 37457

I'm actually tempted to write a fix for this based on that idea. I think I could do it in a high performance way.

I'm going to take this one over. I've got a library who wants this fixed, and I've got an idea on how to do it, so planning to do this soon.

(In reply to David Cook from comment #25)
> I'm going to take this one over. I've got a library who wants this fixed,
> and I've got an idea on how to do it, so planning to do this soon.

Except that I totally misunderstood the problem...

Sure, if your OpacHiddenItem rule is an itype rule, you can filter it out of the facets.

But if your OpacHiddenItem rule is a withdrawn rule, it's not clear what facet value belongs to a hidden item. Yep. I get it now. Bit of an oversight there...

Yeah... this one might not be fixable.

(In reply to David Cook from comment #26)
> (In reply to David Cook from comment #25)
> > I'm going to take this one over. I've got a library who wants this fixed,
> > and I've got an idea on how to do it, so planning to do this soon.
> 
> Except that I totally misunderstood the problem...
> 
> Sure, if your OpacHiddenItem rule is an itype rule, you can filter it out of
> the facets.
> 
> But if your OpacHiddenItem rule is a withdrawn rule, it's not clear what
> facet value belongs to a hidden item. Yep. I get it now. Bit of an oversight
> there...

(In reply to David Cook from comment #27)
> Yeah... this one might not be fixable.

So... I think I make a good point there.

However, Liz said in the Description that the problem was that the Library hid item type "password" and they wanted to stop the item type "password" from showing in the facets.

That's the same problem reported to me by one of my libraries.

And that problem is fixable with my earlier idea of checking the facet values against the OpacHiddenItems.

It's not a perfect solution for values from items hidden by OpacHiddenItems appearing in the facets... but it *does* solve the problem of values hidden by OpacHiddenItems appearing in the facets.

So... I think I'll still go ahead with this fix (both locally and upstream).

I'm taking this one over. If someone wants to resurrect the patches from May on a separate bug, feel free, but I think the patch I'll provide here will solve the main problem that libraries are having.

Changing title from "Item types for items hidden by OpacHiddenItems turn up in facet results" to "Values listed in OpacHiddenItems should not appear in OPAC facets".

The patch I'm providing will also fix bug 37457  

--

Note: While the patch won't filter out all values that only belong to hidden items, it does filter out values that are explicitly listed in OpacHiddenItems, which is what the OP described, and what one of my client libraries is describing currently.

*** Bug 37457 has been marked as a duplicate of this bug. ***

Created attachment 173243 [details] [review]
Bug 14007: Filter search result values to remove OpacHiddenItems values

This change filters the values from the OpacHiddenItems syspref
out of their corresponding search results facets.

Note: This does not mean that all values from hidden items are filtered out of facets.
Rather, it just means that facet data that matches OpacHiddenItems is filtered out. This is
an imperfect fix, but it is a practical fix that has been requested by more than one library.

Test plan:
0. Apply the patch and koha-plack --restart kohadev
1. Go to http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=29
2. Add an item with an item type of "Maps"
3. Go to http://localhost:8081/cgi-bin/koha/admin/preferences.pl?op=search&searchfield=OpacHiddenItems
4. Fill in OpacHiddenItems with the following:
itype: ['MP']
5. Go to http://localhost:8080/cgi-bin/koha/opac-search.pl?idx=&q=test&weight_search=1
6. Note that "Maps" does not appear in the "Item types" facet
7. Go to http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=test
8. Note that "Maps" does appear in the "Item types" facet

9. ***Repeat this test plan using both Elasticsearch and Zebra indexing***

10. prove -v t/Koha/SearchEngine/Search.t

The important code here is in a centralized self-contained unit-tested function in Koha::SearchEngine::Zebra.

If we did want to add in some of the previous patches ideas, I think we could fit that into this code I've posted as well.

Anyway, let me know what you think. I'll probably be applying this locally later today to provide relief to my clients.

Created attachment 173309 [details] [review]
Bug 14007: Filter search result values to remove OpacHiddenItems values

This change filters the values from the OpacHiddenItems syspref
out of their corresponding search results facets.

Note: This does not mean that all values from hidden items are filtered out of facets.
Rather, it just means that facet data that matches OpacHiddenItems is filtered out. This is
an imperfect fix, but it is a practical fix that has been requested by more than one library.

Test plan:
0. Apply the patch and koha-plack --restart kohadev
1. Go to http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=29
2. Add an item with an item type of "Maps"
3. Go to http://localhost:8081/cgi-bin/koha/admin/preferences.pl?op=search&searchfield=OpacHiddenItems
4. Fill in OpacHiddenItems with the following:
itype: ['MP']
5. Go to http://localhost:8080/cgi-bin/koha/opac-search.pl?idx=&q=test&weight_search=1
6. Note that "Maps" does not appear in the "Item types" facet
7. Go to http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=test
8. Note that "Maps" does appear in the "Item types" facet

9. ***Repeat this test plan using both Elasticsearch and Zebra indexing***

10. prove -v t/Koha/SearchEngine/Search.t

Signed-off-by: David Nind <david@davidnind.com>

Testing notes (using KTD):

1. Test with both Elasticsearch 7 and Zebra. I couldn't start KTD with ES8, only ES7.

2. Top: start up KTD with Elastic search 7. Once tested with Elasticsearch, change SearchEngine to Zebra and reindex.

3. Note: The number of search results does not match. For example, I edited "Programming perl" (biblionumber = 262) to change the Koha item type to Maps, and edited the item so that it was Maps as well. Searching for 'perl' returns 10 actual results, which doesn't match with "Your search returned 11 results".

You're a legend, David (Nind).

One day we'll solve the OpacHiddenItems problem that causes the search result number mismatch problem... one day...

Created attachment 173365 [details] [review]
Bug 14007: Filter search result values to remove OpacHiddenItems values

This change filters the values from the OpacHiddenItems syspref
out of their corresponding search results facets.

Note: This does not mean that all values from hidden items are filtered out of facets.
Rather, it just means that facet data that matches OpacHiddenItems is filtered out. This is
an imperfect fix, but it is a practical fix that has been requested by more than one library.

Test plan:
0. Apply the patch and koha-plack --restart kohadev
1. Go to http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=29
2. Add an item with an item type of "Maps"
3. Go to http://localhost:8081/cgi-bin/koha/admin/preferences.pl?op=search&searchfield=OpacHiddenItems
4. Fill in OpacHiddenItems with the following:
itype: ['MP']
5. Go to http://localhost:8080/cgi-bin/koha/opac-search.pl?idx=&q=test&weight_search=1
6. Note that "Maps" does not appear in the "Item types" facet
7. Go to http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=test
8. Note that "Maps" does appear in the "Item types" facet

9. ***Repeat this test plan using both Elasticsearch and Zebra indexing***

10. prove -v t/Koha/SearchEngine/Search.t

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Pushed for 24.11!

Well done everyone, thank you!

Doesn't apply to 24.05.x cleanly, no backport.