Bug 33317 - Add system preference to set meta robots for the OPAC
Summary: Add system preference to set meta robots for the OPAC
Status: Signed Off
Alias: None
Product: Koha
Classification: Unclassified
Component: OPAC (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Hammat wele
QA Contact: Lucas Gass
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-03-22 21:56 UTC by Hammat wele
Modified: 2024-05-24 14:25 UTC (History)
8 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
This enhancement adds a new system preference, OpacMetaRobots which allows libraries to tell search engine robots how to crawl and index OPAC pages.
Version(s) released in:


Attachments
Bug 33317: Add system preference to set meta robots for the OPAC (4.06 KB, patch)
2023-03-22 22:19 UTC, Hammat wele
Details | Diff | Splinter Review
Bug 33317: Add system preference to set meta robots for the OPAC (4.13 KB, patch)
2023-03-23 09:21 UTC, ByWater Sandboxes
Details | Diff | Splinter Review
Bug 33317: Add system preference to set meta robots for the OPAC (5.04 KB, patch)
2023-04-28 22:26 UTC, Hammat wele
Details | Diff | Splinter Review
Bug 33317: Using checkoxes in OpacMetaRobots preference system to prevent incorrect data entry (3.24 KB, patch)
2023-11-10 19:48 UTC, Hammat wele
Details | Diff | Splinter Review
Bug 33317: (follow-up) Perltidy (1.35 KB, patch)
2023-12-07 15:09 UTC, Hammat wele
Details | Diff | Splinter Review
Bug 33317: (follow-up) Perltidy (1.32 KB, patch)
2023-12-20 15:28 UTC, Hammat wele
Details | Diff | Splinter Review
Bug 33317: (follow-up) Perltidy (1.36 KB, patch)
2023-12-20 15:38 UTC, Hammat wele
Details | Diff | Splinter Review
Bug 33317: (follow-up) Perltidy (1.45 KB, patch)
2024-01-04 13:28 UTC, Hammat wele
Details | Diff | Splinter Review
Bug 33317: Add system preference to set meta robots for the OPAC (5.06 KB, patch)
2024-01-04 15:20 UTC, Owen Leonard
Details | Diff | Splinter Review
Bug 33317: Using checkoxes in OpacMetaRobots preference system to prevent incorrect data entry (3.29 KB, patch)
2024-01-04 15:20 UTC, Owen Leonard
Details | Diff | Splinter Review
Bug 33317: (follow-up) Perltidy (1.54 KB, patch)
2024-01-04 15:21 UTC, Owen Leonard
Details | Diff | Splinter Review
Bug 33317: Add system preference to set meta robots for the OPAC (5.11 KB, patch)
2024-01-04 16:16 UTC, Lucas Gass
Details | Diff | Splinter Review
Bug 33317: Using checkoxes in OpacMetaRobots preference system to prevent incorrect data entry (3.34 KB, patch)
2024-01-04 16:16 UTC, Lucas Gass
Details | Diff | Splinter Review
Bug 33317: (follow-up) Perltidy (1.59 KB, patch)
2024-01-04 16:16 UTC, Lucas Gass
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Hammat wele 2023-03-22 21:56:51 UTC
Websites must have a robots meta tag to improve search engine crawling, so we could tells search engines to not index OPAC pages for example.

It needs to be added to koha-tmpl/opac-tmpl/bootstrap/en/includes/doc-head-close.inc

The content should also be available as a system preference.
Comment 1 Hammat wele 2023-03-22 22:19:25 UTC
Created attachment 148575 [details] [review]
Bug 33317: Add system preference to set meta robots for the OPAC

Websites must have a robots meta tag to improve search engine crawling, so we could tell search engines to not index OPAC pages for example.
To test:
1) Look at opac-main source and see that meta name=robots is missing.
2) Apply patch.
3) Go to system preferences > OPAC and add some directives to OpacMetaRobots (ex: noindex,nofollow).
4) Look at opac-main source and confirm that meta name=robots now has content equal to the text set in the OpacMetaRobots system preference.
5) This should Prevent search engines from indexing Opac pages (if OpacMetaRobots is set to noindex)
Comment 2 ByWater Sandboxes 2023-03-23 09:21:21 UTC
Created attachment 148593 [details] [review]
Bug 33317: Add system preference to set meta robots for the OPAC

Websites must have a robots meta tag to improve search engine crawling, so we could tell search engines to not index OPAC pages for example.
To test:
1) Look at opac-main source and see that meta name=robots is missing.
2) Apply patch.
3) Go to system preferences > OPAC and add some directives to OpacMetaRobots (ex: noindex,nofollow).
4) Look at opac-main source and confirm that meta name=robots now has content equal to the text set in the OpacMetaRobots system preference.
5) This should Prevent search engines from indexing Opac pages (if OpacMetaRobots is set to noindex)

Signed-off-by: Sally <sally.healey@cheshiresharedservices.gov.uk>
Comment 3 Sally 2023-03-23 09:34:16 UTC
I think this is 99% of the way there and I'm not sure if I'm being picky with this, so I've signed off so QA can take a look.

Adding a value works perfectly.
Changing a value works perfectly.

But if you've added a value and you later decide to reset to the original defaults, you can't do so by deleting the values and saving the empty input field.

To replicate:

- Enter a value into OpacMetaRobots
- Save 
- Open the OPAC and view the source; note the value matches
- Change the value in OpacMetaRobots
- Save
- Refresh the OPAC and view the source; note the value matches
- Remove the value from OpacMetaRobots so the input field is empty
- Refresh the OPAC and view the source; this time, note that the last saved value still remains

Given that 'index' is the default value if no robot meta tag is listed, maybe just a note to say 'to reset to default values, use: index' would work?
Comment 4 Sally 2023-03-23 09:43:40 UTC
(In reply to Sally from comment #3)
> I think this is 99% of the way there and I'm not sure if I'm being picky
> with this, so I've signed off so QA can take a look.
> 
> Adding a value works perfectly.
> Changing a value works perfectly.
> 
> But if you've added a value and you later decide to reset to the original
> defaults, you can't do so by deleting the values and saving the empty input
> field.
> 
> To replicate:
> 
> - Enter a value into OpacMetaRobots
> - Save 
> - Open the OPAC and view the source; note the value matches
> - Change the value in OpacMetaRobots
> - Save
> - Refresh the OPAC and view the source; note the value matches
> - Remove the value from OpacMetaRobots so the input field is empty
> - Save
> - Refresh the OPAC and view the source; this time, note that the last saved
> value still remains
> 
> Given that 'index' is the default value if no robot meta tag is listed,
> maybe just a note to say 'to reset to default values, use: index' would work?

This error didn't make sense to me, so I wondered if I was having sandbox woes again. 

I repeated the tests on a different sandbox and it all worked perfectly.

So please ignore the above comment - it's working as described.  Sorry for the noise.
Comment 5 Lucas Gass 2023-04-27 18:45:50 UTC
Hammat,

This is missing a DB update. Have a look at Bug 30403 for an example.
Comment 6 Hammat wele 2023-04-28 22:26:52 UTC
Created attachment 150420 [details] [review]
Bug 33317: Add system preference to set meta robots for the OPAC

Websites must have a robots meta tag to improve search engine crawling, so we could tell search engines to not index OPAC pages for example.
To test:
1) Look at opac-main source and see that meta name=robots is missing.
2) Apply patch.
3) Run ./installer/data/mysql/updatedatabase.pl
4) Go to system preferences > OPAC and add some directives to OpacMetaRobots (ex: noindex,nofollow).
5) Look at opac-main source and confirm that meta name=robots now has content equal to the text set in the OpacMetaRobots system preference.
6) This should Prevent search engines from indexing Opac pages (if OpacMetaRobots is set to noindex)
Comment 7 Lucas Gass 2023-05-01 17:32:05 UTC
I have a few follow-up questions:

1. Should a default value of noindex, nofollow be used here?  

2. Is freetext the right thing to use here? Using conflicting or inconsistent values can confuse search engine crawlers and cause them to make incorrect assumptions about how to treat your pages in their index.
Comment 8 Lucas Gass 2023-11-06 22:43:09 UTC
Failing QA here until my questions are addressed.
Comment 9 David Cook 2023-11-06 22:57:39 UTC
(In reply to Lucas Gass from comment #7)
> I have a few follow-up questions:
> 
> 1. Should a default value of noindex, nofollow be used here?  
> 
> 2. Is freetext the right thing to use here? Using conflicting or
> inconsistent values can confuse search engine crawlers and cause them to
> make incorrect assumptions about how to treat your pages in their index.

Good questions...

With Google is looks like there are quite a few valid rules: https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag#directives

But then free-text is error prone as Lucas says...
Comment 10 Hammat wele 2023-11-10 19:48:49 UTC
Created attachment 158820 [details] [review]
Bug 33317: Using checkoxes in OpacMetaRobots preference system to prevent incorrect data entry
Comment 11 Lucas Gass 2023-12-04 16:49:42 UTC
Looks good Hammat. Only issue now is to perltidy 'installer/data/mysql/atomicupdate/bug_33317.pl'.
Comment 12 Hammat wele 2023-12-07 15:09:11 UTC
Created attachment 159653 [details] [review]
Bug 33317: (follow-up) Perltidy
Comment 13 Owen Leonard 2023-12-07 16:36:40 UTC
The qa tool still has complaints about the atomicupdate. Please take another look.
Comment 14 Liz Rea 2023-12-11 23:12:46 UTC
This is a very good feature - put the power in the user's hands instead of the sysadmin. Strong approval.

Agree with Lucas that freetext is probably better avoided, since it can be error prone.

Liz
Comment 15 Hammat wele 2023-12-20 15:28:34 UTC Comment hidden (obsolete)
Comment 16 Hammat wele 2023-12-20 15:38:45 UTC
Created attachment 160126 [details] [review]
Bug 33317: (follow-up) Perltidy
Comment 17 Lucas Gass 2023-12-20 16:47:07 UTC
Hammat, 

The qa tool still complains. Please fix:

 FAIL	installer/data/mysql/atomicupdate/bug_33317.pl
   FAIL	  file permissions
		File must have the exec flag
   WARN	  tidiness
		The file is less tidy than before (bad/messy lines before: 0, now: 1)


-perltidy all of bug_33317.pl
-Also make sure bug_33317.pl has the proper permissions set ( File must have the exec flag )

sudo chmod 755 bug_33317.pl should do it.
Comment 18 Hammat wele 2024-01-04 13:28:46 UTC
Created attachment 160534 [details] [review]
Bug 33317: (follow-up) Perltidy
Comment 19 Owen Leonard 2024-01-04 15:20:56 UTC
Created attachment 160539 [details] [review]
Bug 33317: Add system preference to set meta robots for the OPAC

Websites must have a robots meta tag to improve search engine crawling,
so we could tell search engines to not index OPAC pages for example.

To test:

1) Look at opac-main source and see that meta name=robots is missing.
2) Apply patch.
3) Run ./installer/data/mysql/updatedatabase.pl
4) Go to system preferences > OPAC and add some directives to
   OpacMetaRobots (ex: noindex,nofollow).
5) Look at opac-main source and confirm that meta name=robots now has
   content equal to the text set in the OpacMetaRobots system
   preference.
6) This should Prevent search engines from indexing Opac pages (if
   OpacMetaRobots is set to noindex)

Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Comment 20 Owen Leonard 2024-01-04 15:20:58 UTC
Created attachment 160540 [details] [review]
Bug 33317: Using checkoxes in OpacMetaRobots preference system to prevent incorrect data entry

Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Comment 21 Owen Leonard 2024-01-04 15:21:01 UTC
Created attachment 160541 [details] [review]
Bug 33317: (follow-up) Perltidy

Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Comment 22 Lucas Gass 2024-01-04 16:15:52 UTC
Looks good now, passing QA.
Comment 23 Lucas Gass 2024-01-04 16:16:34 UTC
Created attachment 160544 [details] [review]
Bug 33317: Add system preference to set meta robots for the OPAC

Websites must have a robots meta tag to improve search engine crawling,
so we could tell search engines to not index OPAC pages for example.

To test:

1) Look at opac-main source and see that meta name=robots is missing.
2) Apply patch.
3) Run ./installer/data/mysql/updatedatabase.pl
4) Go to system preferences > OPAC and add some directives to
   OpacMetaRobots (ex: noindex,nofollow).
5) Look at opac-main source and confirm that meta name=robots now has
   content equal to the text set in the OpacMetaRobots system
   preference.
6) This should Prevent search engines from indexing Opac pages (if
   OpacMetaRobots is set to noindex)

Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Comment 24 Lucas Gass 2024-01-04 16:16:37 UTC
Created attachment 160545 [details] [review]
Bug 33317: Using checkoxes in OpacMetaRobots preference system to prevent incorrect data entry

Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Comment 25 Lucas Gass 2024-01-04 16:16:39 UTC
Created attachment 160546 [details] [review]
Bug 33317: (follow-up) Perltidy

Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Comment 26 Katrin Fischer 2024-01-31 21:37:50 UTC
I feel like the system preference description is a bit sparse. Could we add an example or a link to documentation to make it clearer what you are supposed to add?

+        -
+            - "Improve search engine crawling."
+            - pref: OpacMetaRobots
+              type: textarea

Also, the pref description shoudl be formed as a sentencee and connect with the input field.
Comment 27 Michał 2024-04-02 07:26:22 UTC
Wait, so this simply adds the same robots tag for the entirety of OPAC? I've been redirected here from Bug 35812 due to conflicts. And I think this bug right here is questionable and I'm not sure if it should be merged, at least as-is.

> Websites must have a robots meta tag

That is not true, this tag is very much optional and meant for granular page-level steering of crawling bots. The way this preference is implemented, the rules are going to apply to ALL opac pages, which I'm not sure if there's any reasonable use-case. The author mentions example "noindex,nofollow" to prevent ALL opac pages from being indexed. I think that if some library wants that, they'd be better off using the more widely used and known robots.txt file, which is more likely to be supported by various crawlers, and will prevent them from downloading the pages in the first place (instead of downloading the pages it wanted to and then discarding them upon discovering the meta tag for particular page).

I think the better direction is to diversify manually which pages should be crawlable by default and which shouldn't like in Bug 35812, ie. search results and so on (dynamic pages) shouldn't be indexed to decreate the amount of junk (but they should be crawled to extract links from it), while main page/info subpages/user-created lists/biblio records should probably be indexed by default and it's probably what most libraries would want by default.

With that said, there currently there's no "obvious"/"easy" way of specifying custom robots.txt file, apart from doing something like `Alias /robots.txt /var/www/html/robots.txt` in Apache config for opac (it works well enough btw).

So, in the spirit of what OP originally wanted, I believe it could be better to consider instead adding a user preference for textarea of robots.txt file contents, in place of site-wide robots meta tag contents. This would allow libraries to set more granular rules, but someone who wants to block everything could still just do:

User-agent: *
Disallow: /

Btw this is already documented in README.robots file in Koha's main git directory (last edited 13 years ago, the last paragraph there is probably outdated).

So I believe my idea above could solve the conflict between our two patches, robots.txt usage is more widely documented on the Internet I believe, and that'd override any rules that Koha devs could specify on per-template basis manually in robots tag on pages like I did in Bug 35812...
Comment 28 Michał 2024-04-02 09:18:46 UTC
By the way, having programmatic robots.txt generation could allow having sitemaps linked there with "Sitemap:" directive, provided the generated files are there, and that would remove one obstacle from having the sitemap generation enabled by default in the Debian installation.
Comment 29 Blou 2024-05-24 14:25:20 UTC
(In reply to Michał from comment #27)
> which I'm not sure if there's any reasonable use-case. The author mentions
> example "noindex,nofollow" to prevent ALL opac pages from being indexed. 
> I think that if some library wants that, they'd be better off using the more 
> widely used and known robots.txt file

Your solution doesn't work.  Robots.txt doesn't prevent sites from being indexed, Google itself instructs https://developers.google.com/search/docs/crawling-indexing/block-indexing

Our use case is very valid and a basic option in Wordpress: websites wants ways to not be found (while developping).  Here we have test and production systems, and of course our customers do not want their users to find the test system.  Two koha, similar data, user search google for their library and click on the test link.