Bug 35812 - Should specify canonical URLs to help search indexers
Summary: Should specify canonical URLs to help search indexers
Status: Pushed to main
Alias: None
Product: Koha
Classification: Unclassified
Component: OPAC (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Michał
QA Contact: Martin Renvoize
URL:
Keywords:
Depends on:
Blocks: 35302
  Show dependency treegraph
 
Reported: 2024-01-15 09:53 UTC by Michał
Modified: 2024-05-10 12:13 UTC (History)
9 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
24.05.00


Attachments
[PATCH 1/4] Bug 35812: add noindex attribute to search result pages in OPAC (4.13 KB, patch)
2024-03-27 12:28 UTC, Michał
Details | Diff | Splinter Review
[PATCH 2/4] Bug 35812: Add canonical URL for biblio details page (1.53 KB, patch)
2024-03-27 12:28 UTC, Michał
Details | Diff | Splinter Review
[PATCH 3/4] Bug 35812: specify canonical URL for the main page (1.28 KB, patch)
2024-03-27 12:28 UTC, Michał
Details | Diff | Splinter Review
[PATCH 4/4] Bug 35812: Fix search URL aliases in Plack Apache config (2.48 KB, patch)
2024-03-27 12:28 UTC, Michał
Details | Diff | Splinter Review
Bug 35812: add noindex attribute to search result pages in OPAC (4.19 KB, patch)
2024-04-02 08:47 UTC, Pedro Amorim
Details | Diff | Splinter Review
Bug 35812: Add canonical URL for biblio details page (1.59 KB, patch)
2024-04-02 08:47 UTC, Pedro Amorim
Details | Diff | Splinter Review
Bug 35812: specify canonical URL for the main page (1.35 KB, patch)
2024-04-02 08:47 UTC, Pedro Amorim
Details | Diff | Splinter Review
Bug 35812: Fix search URL aliases in Plack Apache config (2.54 KB, patch)
2024-04-02 08:47 UTC, Pedro Amorim
Details | Diff | Splinter Review
Bug 35812: add noindex attribute to search result pages in OPAC (4.26 KB, patch)
2024-05-08 12:49 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 35812: Add canonical URL for biblio details page (1.65 KB, patch)
2024-05-08 12:49 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 35812: specify canonical URL for the main page (1.41 KB, patch)
2024-05-08 12:49 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 35812: Fix search URL aliases in Plack Apache config (2.60 KB, patch)
2024-05-08 12:49 UTC, Martin Renvoize
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Michał 2024-01-15 09:53:07 UTC
So we've recently added our Koha site to Google Search Console and became bombarded by its complains about lack of defined canonical URLs for pages. This is mostly a problem for bib details page.

For main page, you can have three possible URLs that will all load main page:
- /
- /index.html
- /cgi-bin/koha/opac-main.pl

I think "/" should be specified as canonical URL, which could even "lazily" be achieved by following Apache directives (these will append the Link header to all three URLs above and nothing else) (they could be added to debian/templates/apache-shared-opac.conf):

Header always append "Link" "</>; rel=canonical" "expr=%{REQUEST_URI} == '/index.html'"
Header always append "Link" "</>; rel=canonical" "expr=%{REQUEST_URI} == '/cgi-bin/koha/opac-main.pl'"

The situation gets a bit more complicated with bibs, because trying to solve it with Apache directives becomes complex if even possible, instead of another solution (note that we can also have a HTML tag instead of HTTP header).
Koha links to pages with URLs in the format `/cgi-bin/koha/opac-detail.pl?biblionumber=1234`, but the Apache config has defined nice links in the format of `/bib/1234`. With `opac-detail.pl` page you get some junky parameters appeared after you reach the page from Koha search such as:
- /cgi-bin/koha/opac-detail.pl?biblionumber=1234
- /cgi-bin/koha/opac-detail.pl?biblionumber=1234&query_desc=se,phr:"Something"
- /cgi-bin/koha/opac-detail.pl?biblionumber=1234&query_desc=an:3200
- /cgi-bin/koha/opac-detail.pl?biblionumber=1234&query_desc=an:1846 and su-to:Something and itype:BK and su-to:Environment and su-to:Environment
- /cgi-bin/koha/opac-detail.pl?biblionumber=1234&fbclid=FSD8ufs98jf39jfes80jfds8jfsd
- /cgi-bin/koha/opac-detail.pl?bib=1234

To top it off, if you do a search and there's only one search result, you get redirected directly to opac-detail, confusing Google even further (I actually think it's worthy of a separate bug to add some kind of notice that you were redirected cause there was only one result, but that's a matter for another discussion...).

So the solution there would also be to specify the canonical link as "/bib/1234" with no params, but I think it might be best off done in code rather than Apache config there.

The end result of these would be that search results would have nice links, without weird params such as query_desc added to the indexed pages, which is awful (as the user coming from search has nothing to do with that query as when it was indexed).

This would solve also another potential problem, in that such silent redirect from search results could make a search engine think that the search URL is the canonical URL (as when it's not specified, it's up to the search engine to detect duplicate and decide, but it can do so wrongly and against its own rules sometimes, as we've observed).

And finally, the search URLs could also be made canonical from "/cgi-bin/koha/opac-search.pl" to "/search", however this time with keeping all URL query parameters.

All of this is somewhat related to Bug 18410, but not blocked by it by any means, as the URL aliases I've described above are actually in the Apache config already for a long time, and don't depend on overall Koha routing overhaul.

Btw: it's better to use full OPAC base URL instead of just "/" if we can, this will help with deduplicating indexed URLs in case your Koha is available under multiple aliased domains or is under process of changing domains (or is technically available under both http and https without a redirect, and search treats those as separate sites!). But doing so requires pure code solution instead of Apache directives.
Comment 1 Magnus Enger 2024-01-15 10:45:45 UTC
> To top it off, if you do a search and there's only one search result, you get redirected directly to opac-detail, confusing Google even further

I would say Google should not be doing searches in the OPAC. An alternative to that might be sitemaps? Have you looked at the "koha-sitemap" script? 

See also: 

- man koha-sitemap
- https://perldoc.koha-community.org/misc/cronjobs/sitemap.html
Comment 2 Michał 2024-01-15 11:04:21 UTC
Thank you, I wasn't actually aware of that script. It is some form of mitigation to have a sitemap to short anonical links (which it seems to default to), while simultaniously denying access to /cgi-bin/ dir with robots.txt rule.

It has its cons though, in that nothing besides bib pages and the main page would get indexed (perhaps it's okay though, it is what mostly needs to be indexed).

I will probably implement this solution, as it seems to be the best approach.

With that said, I think that semantically speaking the canonical links should still be implemented as I outlined in the bug, I believe it would still be beneficial both for libraries that have a sitemap and that don't (and it looks like most don't upon quick check-up of random libraries...).

(And Google was doing searches with default configuration probably by clicking authorities on bib pages. I think with canonical links one could have an option to prohibit just the search pages from robots.txt for example had they wanted to.)
Comment 3 Magnus Enger 2024-01-15 11:14:38 UTC
I have this as a standard robots.txt: 

Sitemap: sitemapindex.xml

User-agent: *
Disallow: /cgi-bin/koha/opac-search.pl
Disallow: /cgi-bin/koha/opac-export.pl
Disallow: /cgi-bin/koha/opac-MARCdetail.pl
Disallow: /cgi-bin/koha/opac-ISBDdetail.pl
Disallow: /cgi-bin/koha/opac-reserve.pl
Comment 4 Michał 2024-01-22 11:10:01 UTC
So there's a couple of issues with sitemap and robots.txt approach:

1. It's been a week and Google apparently can decide whether it even ever feels like even downloading the sitemap and pages on it, which in our case for a small site it decided not to do at all (apparently it's a common issue upon some research)

2. Thus blocking opac-search.pl with robots.txt limits the indexability of the page by Googlebot, because it's the only way effective to discover pages

The solution would be to:

1. Have canonical links for bibs as described above

2. Allow bots to access opac-search.pl, but have "noindex" <meta> attribute (but no "nofollow"!!). Contrary to robots.txt, this will allow the bot to download the page and extract links from it to index them, but will not index the search page itself. Ideal.
Comment 5 Michał 2024-03-27 12:28:12 UTC
Created attachment 163987 [details] [review]
[PATCH 1/4] Bug 35812: add noindex attribute to search result pages in OPAC
Comment 6 Michał 2024-03-27 12:28:20 UTC
Created attachment 163988 [details] [review]
[PATCH 2/4] Bug 35812: Add canonical URL for biblio details page
Comment 7 Michał 2024-03-27 12:28:28 UTC
Created attachment 163989 [details] [review]
[PATCH 3/4] Bug 35812: specify canonical URL for the main page
Comment 8 Michał 2024-03-27 12:28:30 UTC
Created attachment 163990 [details] [review]
[PATCH 4/4] Bug 35812: Fix search URL aliases in Plack Apache config
Comment 9 Michał 2024-03-27 12:36:17 UTC
Testing notes:

1. Test whether affected pages now have the new <link> tags in HTML source
2. Check whether the canonical URL for bib links is correct and that the URL it points to works (http://LIBRARY.TLD/bib/12345)
3. For patch 4, if you open up searching on http://LIBRARY.TLD/search page, you should see nothing in `/var/log/koha/INSTANCE/plack.log`. After applying the patch, you should see `/opac/opac-search.pl` entries. This confirms the request now goes through Plack, similar to how `/bib/*` links should do already.

As for backports, patch 4 could be backported as technically it fixes a bug related to canonical links. The rest of the changes might be deemed not applicable for backport though.

I originally planned to also have canonical URL specified for /search itself too similar to /bib/, but I backed down from it because:
- its usefulness would be limited if search results have noindex attribute now
- it'd be actually somewhat messy to specify proper URL query params correctly, the current template parameters used for RSS link are not ideal, this would be lots of rework and rethink for no clear benefit, so...
Comment 10 Lucas Gass 2024-03-27 17:28:05 UTC
This conflicts a bit with Bug 33317, which allows librarians to set the `meta name="robots"` tag via system preference.
Comment 11 Michał 2024-04-02 07:41:08 UTC
Thank you, I commented on the other bug
Comment 12 Pedro Amorim 2024-04-02 08:47:46 UTC
Created attachment 164242 [details] [review]
Bug 35812: add noindex attribute to search result pages in OPAC

This will prevent search engines from putting the search results pages into their indexes, to prevent cluttering of the search results and give better visibility to the actual bibliographic records' pages.

Note that "noindex" is separate from "nofollow", meaning that the search engine is allowed to get and process the search results page and extract ("follow") further links from it, such as links to biblio records, which is exactly what we want.

This approach is better to the sitemaps, which search engines are free to choose whether to use or not, based on multiple factors such as the size of the page. In practice for small pages Google will just not download them at all, meaning that indexing by following links around is the only way the indexer will extract any links.

Note that for this to work properly, you must NOT block the affected pages in your robots.txt file (they're not blocked by default).

Also, this patch moves "cssinclude" block in opac-results.tt inside of <head>, because in all other templates it's inside, meaning its current positioning is a small mistake I fixed while at it.

Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Comment 13 Pedro Amorim 2024-04-02 08:47:49 UTC
Created attachment 164243 [details] [review]
Bug 35812: Add canonical URL for biblio details page

This adds a nice canonical URL for search engines to use. Will prevent duplicates with different URL query parameters from getting indexed separately, will also prevent search engine confusion when opac-search.pl redirects to opac-detail.pl when there's just one result.

Should result in prettier/simpler URLs in the search engines too, plus removal of constant warnings in Google Search Console and other tools.

Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Comment 14 Pedro Amorim 2024-04-02 08:47:52 UTC
Created attachment 164244 [details] [review]
Bug 35812: specify canonical URL for the main page

Currently main page can be accessed by:
- /
- /index.html
- /cgi-bin/koha/opac-main.pl

This specifies the canonical link for all of them to be "/", to aid search engines in assessing what's url to put in the index.

Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Comment 15 Pedro Amorim 2024-04-02 08:47:54 UTC
Created attachment 164245 [details] [review]
Bug 35812: Fix search URL aliases in Plack Apache config

Mimic missing stuff from apache-shared-opac.conf into apache-shared-opac-plack.conf, where /search and /opac-search.pl aliases were missing.

Note that while RewriteRule applies before reverse proxy directives and works, ScriptAlias applies purely to standard cgi perl scripts, meaning it didn't apply there.

Now it's mimicked analogously to how /index.html is configured.

This patch fixes an issue where someone might have just a reverse proxy to Plack configured without native Perl scripting, where this URL would then otherwise be failing.

Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Comment 16 Pedro Amorim 2024-04-02 08:51:56 UTC
Patches looked different on bugzilla when comparing to local git but they look ok now after I've resubmitted them with git bz.
Patches author is set as  Michał Kula <148193449+mkibp@users.noreply.github.com> - informing in case this is unintentional, ignore if not.

This looks good, explanations make sense, comment on the other bug is clear and constructive, not YASP, it's a SO from me.
Comment 17 Martin Renvoize 2024-05-08 12:49:18 UTC
Created attachment 166404 [details] [review]
Bug 35812: add noindex attribute to search result pages in OPAC

This will prevent search engines from putting the search results pages into their indexes, to prevent cluttering of the search results and give better visibility to the actual bibliographic records' pages.

Note that "noindex" is separate from "nofollow", meaning that the search engine is allowed to get and process the search results page and extract ("follow") further links from it, such as links to biblio records, which is exactly what we want.

This approach is better to the sitemaps, which search engines are free to choose whether to use or not, based on multiple factors such as the size of the page. In practice for small pages Google will just not download them at all, meaning that indexing by following links around is the only way the indexer will extract any links.

Note that for this to work properly, you must NOT block the affected pages in your robots.txt file (they're not blocked by default).

Also, this patch moves "cssinclude" block in opac-results.tt inside of <head>, because in all other templates it's inside, meaning its current positioning is a small mistake I fixed while at it.

Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 18 Martin Renvoize 2024-05-08 12:49:21 UTC
Created attachment 166405 [details] [review]
Bug 35812: Add canonical URL for biblio details page

This adds a nice canonical URL for search engines to use. Will prevent duplicates with different URL query parameters from getting indexed separately, will also prevent search engine confusion when opac-search.pl redirects to opac-detail.pl when there's just one result.

Should result in prettier/simpler URLs in the search engines too, plus removal of constant warnings in Google Search Console and other tools.

Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 19 Martin Renvoize 2024-05-08 12:49:24 UTC
Created attachment 166406 [details] [review]
Bug 35812: specify canonical URL for the main page

Currently main page can be accessed by:
- /
- /index.html
- /cgi-bin/koha/opac-main.pl

This specifies the canonical link for all of them to be "/", to aid search engines in assessing what's url to put in the index.

Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 20 Martin Renvoize 2024-05-08 12:49:27 UTC
Created attachment 166407 [details] [review]
Bug 35812: Fix search URL aliases in Plack Apache config

Mimic missing stuff from apache-shared-opac.conf into apache-shared-opac-plack.conf, where /search and /opac-search.pl aliases were missing.

Note that while RewriteRule applies before reverse proxy directives and works, ScriptAlias applies purely to standard cgi perl scripts, meaning it didn't apply there.

Now it's mimicked analogously to how /index.html is configured.

This patch fixes an issue where someone might have just a reverse proxy to Plack configured without native Perl scripting, where this URL would then otherwise be failing.

Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 21 Martin Renvoize 2024-05-08 12:50:00 UTC
Nice improvement here.. QA scripts happy and all working as I would expect.

Passing QA
Comment 22 Katrin Fischer 2024-05-10 12:13:02 UTC
Pushed for 24.05!

Well done everyone, thank you!