> To top it off, if you do a search and there's only one search result, you get redirected directly to opac-detail, confusing Google even further

I would say Google should not be doing searches in the OPAC. An alternative to that might be sitemaps? Have you looked at the "koha-sitemap" script? 

See also: 

- man koha-sitemap
- https://perldoc.koha-community.org/misc/cronjobs/sitemap.html

Thank you, I wasn't actually aware of that script. It is some form of mitigation to have a sitemap to short anonical links (which it seems to default to), while simultaniously denying access to /cgi-bin/ dir with robots.txt rule.

It has its cons though, in that nothing besides bib pages and the main page would get indexed (perhaps it's okay though, it is what mostly needs to be indexed).

I will probably implement this solution, as it seems to be the best approach.

With that said, I think that semantically speaking the canonical links should still be implemented as I outlined in the bug, I believe it would still be beneficial both for libraries that have a sitemap and that don't (and it looks like most don't upon quick check-up of random libraries...).

(And Google was doing searches with default configuration probably by clicking authorities on bib pages. I think with canonical links one could have an option to prohibit just the search pages from robots.txt for example had they wanted to.)

I have this as a standard robots.txt: 

Sitemap: sitemapindex.xml

User-agent: *
Disallow: /cgi-bin/koha/opac-search.pl
Disallow: /cgi-bin/koha/opac-export.pl
Disallow: /cgi-bin/koha/opac-MARCdetail.pl
Disallow: /cgi-bin/koha/opac-ISBDdetail.pl
Disallow: /cgi-bin/koha/opac-reserve.pl

So there's a couple of issues with sitemap and robots.txt approach:

1. It's been a week and Google apparently can decide whether it even ever feels like even downloading the sitemap and pages on it, which in our case for a small site it decided not to do at all (apparently it's a common issue upon some research)

2. Thus blocking opac-search.pl with robots.txt limits the indexability of the page by Googlebot, because it's the only way effective to discover pages

The solution would be to:

1. Have canonical links for bibs as described above

2. Allow bots to access opac-search.pl, but have "noindex" <meta> attribute (but no "nofollow"!!). Contrary to robots.txt, this will allow the bot to download the page and extract links from it to index them, but will not index the search page itself. Ideal.

Created attachment 163987 [details] [review]
[PATCH 1/4] Bug 35812: add noindex attribute to search result pages in OPAC

Created attachment 163988 [details] [review]
[PATCH 2/4] Bug 35812: Add canonical URL for biblio details page

Created attachment 163989 [details] [review]
[PATCH 3/4] Bug 35812: specify canonical URL for the main page

Created attachment 163990 [details] [review]
[PATCH 4/4] Bug 35812: Fix search URL aliases in Plack Apache config

Testing notes:

1. Test whether affected pages now have the new <link> tags in HTML source
2. Check whether the canonical URL for bib links is correct and that the URL it points to works (http://LIBRARY.TLD/bib/12345)
3. For patch 4, if you open up searching on http://LIBRARY.TLD/search page, you should see nothing in `/var/log/koha/INSTANCE/plack.log`. After applying the patch, you should see `/opac/opac-search.pl` entries. This confirms the request now goes through Plack, similar to how `/bib/*` links should do already.

As for backports, patch 4 could be backported as technically it fixes a bug related to canonical links. The rest of the changes might be deemed not applicable for backport though.

I originally planned to also have canonical URL specified for /search itself too similar to /bib/, but I backed down from it because:
- its usefulness would be limited if search results have noindex attribute now
- it'd be actually somewhat messy to specify proper URL query params correctly, the current template parameters used for RSS link are not ideal, this would be lots of rework and rethink for no clear benefit, so...

This conflicts a bit with Bug 33317, which allows librarians to set the `meta name="robots"` tag via system preference.

Thank you, I commented on the other bug

Created attachment 164242 [details] [review]
Bug 35812: add noindex attribute to search result pages in OPAC

This will prevent search engines from putting the search results pages into their indexes, to prevent cluttering of the search results and give better visibility to the actual bibliographic records' pages.

Note that "noindex" is separate from "nofollow", meaning that the search engine is allowed to get and process the search results page and extract ("follow") further links from it, such as links to biblio records, which is exactly what we want.

This approach is better to the sitemaps, which search engines are free to choose whether to use or not, based on multiple factors such as the size of the page. In practice for small pages Google will just not download them at all, meaning that indexing by following links around is the only way the indexer will extract any links.

Note that for this to work properly, you must NOT block the affected pages in your robots.txt file (they're not blocked by default).

Also, this patch moves "cssinclude" block in opac-results.tt inside of <head>, because in all other templates it's inside, meaning its current positioning is a small mistake I fixed while at it.

Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>

Created attachment 164243 [details] [review]
Bug 35812: Add canonical URL for biblio details page

This adds a nice canonical URL for search engines to use. Will prevent duplicates with different URL query parameters from getting indexed separately, will also prevent search engine confusion when opac-search.pl redirects to opac-detail.pl when there's just one result.

Should result in prettier/simpler URLs in the search engines too, plus removal of constant warnings in Google Search Console and other tools.

Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>

Created attachment 164244 [details] [review]
Bug 35812: specify canonical URL for the main page

Currently main page can be accessed by:
- /
- /index.html
- /cgi-bin/koha/opac-main.pl

This specifies the canonical link for all of them to be "/", to aid search engines in assessing what's url to put in the index.

Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>

Created attachment 164245 [details] [review]
Bug 35812: Fix search URL aliases in Plack Apache config

Mimic missing stuff from apache-shared-opac.conf into apache-shared-opac-plack.conf, where /search and /opac-search.pl aliases were missing.

Note that while RewriteRule applies before reverse proxy directives and works, ScriptAlias applies purely to standard cgi perl scripts, meaning it didn't apply there.

Now it's mimicked analogously to how /index.html is configured.

This patch fixes an issue where someone might have just a reverse proxy to Plack configured without native Perl scripting, where this URL would then otherwise be failing.

Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>

Patches looked different on bugzilla when comparing to local git but they look ok now after I've resubmitted them with git bz.
Patches author is set as  Michał Kula <148193449+mkibp@users.noreply.github.com> - informing in case this is unintentional, ignore if not.

This looks good, explanations make sense, comment on the other bug is clear and constructive, not YASP, it's a SO from me.

Created attachment 166404 [details] [review]
Bug 35812: add noindex attribute to search result pages in OPAC

This will prevent search engines from putting the search results pages into their indexes, to prevent cluttering of the search results and give better visibility to the actual bibliographic records' pages.

Note that "noindex" is separate from "nofollow", meaning that the search engine is allowed to get and process the search results page and extract ("follow") further links from it, such as links to biblio records, which is exactly what we want.

This approach is better to the sitemaps, which search engines are free to choose whether to use or not, based on multiple factors such as the size of the page. In practice for small pages Google will just not download them at all, meaning that indexing by following links around is the only way the indexer will extract any links.

Note that for this to work properly, you must NOT block the affected pages in your robots.txt file (they're not blocked by default).

Also, this patch moves "cssinclude" block in opac-results.tt inside of <head>, because in all other templates it's inside, meaning its current positioning is a small mistake I fixed while at it.

Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Created attachment 166405 [details] [review]
Bug 35812: Add canonical URL for biblio details page

This adds a nice canonical URL for search engines to use. Will prevent duplicates with different URL query parameters from getting indexed separately, will also prevent search engine confusion when opac-search.pl redirects to opac-detail.pl when there's just one result.

Should result in prettier/simpler URLs in the search engines too, plus removal of constant warnings in Google Search Console and other tools.

Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Created attachment 166406 [details] [review]
Bug 35812: specify canonical URL for the main page

Currently main page can be accessed by:
- /
- /index.html
- /cgi-bin/koha/opac-main.pl

This specifies the canonical link for all of them to be "/", to aid search engines in assessing what's url to put in the index.

Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Created attachment 166407 [details] [review]
Bug 35812: Fix search URL aliases in Plack Apache config

Mimic missing stuff from apache-shared-opac.conf into apache-shared-opac-plack.conf, where /search and /opac-search.pl aliases were missing.

Note that while RewriteRule applies before reverse proxy directives and works, ScriptAlias applies purely to standard cgi perl scripts, meaning it didn't apply there.

Now it's mimicked analogously to how /index.html is configured.

This patch fixes an issue where someone might have just a reverse proxy to Plack configured without native Perl scripting, where this URL would then otherwise be failing.

Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Nice improvement here.. QA scripts happy and all working as I would expect.

Passing QA

Pushed for 24.05!

Well done everyone, thank you!

I tried to add some notes, please feel free to improve (but quickly).

Not backported to 23.11.x