|
Description
Hammat wele
2023-03-22 21:56:51 UTC
Created attachment 148575 [details] [review] Bug 33317: Add system preference to set meta robots for the OPAC Websites must have a robots meta tag to improve search engine crawling, so we could tell search engines to not index OPAC pages for example. To test: 1) Look at opac-main source and see that meta name=robots is missing. 2) Apply patch. 3) Go to system preferences > OPAC and add some directives to OpacMetaRobots (ex: noindex,nofollow). 4) Look at opac-main source and confirm that meta name=robots now has content equal to the text set in the OpacMetaRobots system preference. 5) This should Prevent search engines from indexing Opac pages (if OpacMetaRobots is set to noindex) Created attachment 148593 [details] [review] Bug 33317: Add system preference to set meta robots for the OPAC Websites must have a robots meta tag to improve search engine crawling, so we could tell search engines to not index OPAC pages for example. To test: 1) Look at opac-main source and see that meta name=robots is missing. 2) Apply patch. 3) Go to system preferences > OPAC and add some directives to OpacMetaRobots (ex: noindex,nofollow). 4) Look at opac-main source and confirm that meta name=robots now has content equal to the text set in the OpacMetaRobots system preference. 5) This should Prevent search engines from indexing Opac pages (if OpacMetaRobots is set to noindex) Signed-off-by: Sally <sally.healey@cheshiresharedservices.gov.uk> I think this is 99% of the way there and I'm not sure if I'm being picky with this, so I've signed off so QA can take a look. Adding a value works perfectly. Changing a value works perfectly. But if you've added a value and you later decide to reset to the original defaults, you can't do so by deleting the values and saving the empty input field. To replicate: - Enter a value into OpacMetaRobots - Save - Open the OPAC and view the source; note the value matches - Change the value in OpacMetaRobots - Save - Refresh the OPAC and view the source; note the value matches - Remove the value from OpacMetaRobots so the input field is empty - Refresh the OPAC and view the source; this time, note that the last saved value still remains Given that 'index' is the default value if no robot meta tag is listed, maybe just a note to say 'to reset to default values, use: index' would work? (In reply to Sally from comment #3) > I think this is 99% of the way there and I'm not sure if I'm being picky > with this, so I've signed off so QA can take a look. > > Adding a value works perfectly. > Changing a value works perfectly. > > But if you've added a value and you later decide to reset to the original > defaults, you can't do so by deleting the values and saving the empty input > field. > > To replicate: > > - Enter a value into OpacMetaRobots > - Save > - Open the OPAC and view the source; note the value matches > - Change the value in OpacMetaRobots > - Save > - Refresh the OPAC and view the source; note the value matches > - Remove the value from OpacMetaRobots so the input field is empty > - Save > - Refresh the OPAC and view the source; this time, note that the last saved > value still remains > > Given that 'index' is the default value if no robot meta tag is listed, > maybe just a note to say 'to reset to default values, use: index' would work? This error didn't make sense to me, so I wondered if I was having sandbox woes again. I repeated the tests on a different sandbox and it all worked perfectly. So please ignore the above comment - it's working as described. Sorry for the noise. Hammat, This is missing a DB update. Have a look at Bug 30403 for an example. Created attachment 150420 [details] [review] Bug 33317: Add system preference to set meta robots for the OPAC Websites must have a robots meta tag to improve search engine crawling, so we could tell search engines to not index OPAC pages for example. To test: 1) Look at opac-main source and see that meta name=robots is missing. 2) Apply patch. 3) Run ./installer/data/mysql/updatedatabase.pl 4) Go to system preferences > OPAC and add some directives to OpacMetaRobots (ex: noindex,nofollow). 5) Look at opac-main source and confirm that meta name=robots now has content equal to the text set in the OpacMetaRobots system preference. 6) This should Prevent search engines from indexing Opac pages (if OpacMetaRobots is set to noindex) I have a few follow-up questions: 1. Should a default value of noindex, nofollow be used here? 2. Is freetext the right thing to use here? Using conflicting or inconsistent values can confuse search engine crawlers and cause them to make incorrect assumptions about how to treat your pages in their index. Failing QA here until my questions are addressed. (In reply to Lucas Gass from comment #7) > I have a few follow-up questions: > > 1. Should a default value of noindex, nofollow be used here? > > 2. Is freetext the right thing to use here? Using conflicting or > inconsistent values can confuse search engine crawlers and cause them to > make incorrect assumptions about how to treat your pages in their index. Good questions... With Google is looks like there are quite a few valid rules: https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag#directives But then free-text is error prone as Lucas says... Created attachment 158820 [details] [review] Bug 33317: Using checkoxes in OpacMetaRobots preference system to prevent incorrect data entry Looks good Hammat. Only issue now is to perltidy 'installer/data/mysql/atomicupdate/bug_33317.pl'. Created attachment 159653 [details] [review] Bug 33317: (follow-up) Perltidy The qa tool still has complaints about the atomicupdate. Please take another look. This is a very good feature - put the power in the user's hands instead of the sysadmin. Strong approval. Agree with Lucas that freetext is probably better avoided, since it can be error prone. Liz Created attachment 160125 [details] [review] Bug 33317: (follow-up) Perltidy Created attachment 160126 [details] [review] Bug 33317: (follow-up) Perltidy Hammat, The qa tool still complains. Please fix: FAIL installer/data/mysql/atomicupdate/bug_33317.pl FAIL file permissions File must have the exec flag WARN tidiness The file is less tidy than before (bad/messy lines before: 0, now: 1) -perltidy all of bug_33317.pl -Also make sure bug_33317.pl has the proper permissions set ( File must have the exec flag ) sudo chmod 755 bug_33317.pl should do it. Created attachment 160534 [details] [review] Bug 33317: (follow-up) Perltidy Created attachment 160539 [details] [review] Bug 33317: Add system preference to set meta robots for the OPAC Websites must have a robots meta tag to improve search engine crawling, so we could tell search engines to not index OPAC pages for example. To test: 1) Look at opac-main source and see that meta name=robots is missing. 2) Apply patch. 3) Run ./installer/data/mysql/updatedatabase.pl 4) Go to system preferences > OPAC and add some directives to OpacMetaRobots (ex: noindex,nofollow). 5) Look at opac-main source and confirm that meta name=robots now has content equal to the text set in the OpacMetaRobots system preference. 6) This should Prevent search engines from indexing Opac pages (if OpacMetaRobots is set to noindex) Signed-off-by: Owen Leonard <oleonard@myacpl.org> Created attachment 160540 [details] [review] Bug 33317: Using checkoxes in OpacMetaRobots preference system to prevent incorrect data entry Signed-off-by: Owen Leonard <oleonard@myacpl.org> Created attachment 160541 [details] [review] Bug 33317: (follow-up) Perltidy Signed-off-by: Owen Leonard <oleonard@myacpl.org> Looks good now, passing QA. Created attachment 160544 [details] [review] Bug 33317: Add system preference to set meta robots for the OPAC Websites must have a robots meta tag to improve search engine crawling, so we could tell search engines to not index OPAC pages for example. To test: 1) Look at opac-main source and see that meta name=robots is missing. 2) Apply patch. 3) Run ./installer/data/mysql/updatedatabase.pl 4) Go to system preferences > OPAC and add some directives to OpacMetaRobots (ex: noindex,nofollow). 5) Look at opac-main source and confirm that meta name=robots now has content equal to the text set in the OpacMetaRobots system preference. 6) This should Prevent search engines from indexing Opac pages (if OpacMetaRobots is set to noindex) Signed-off-by: Owen Leonard <oleonard@myacpl.org> Signed-off-by: Lucas Gass <lucas@bywatersolutions.com> Created attachment 160545 [details] [review] Bug 33317: Using checkoxes in OpacMetaRobots preference system to prevent incorrect data entry Signed-off-by: Owen Leonard <oleonard@myacpl.org> Signed-off-by: Lucas Gass <lucas@bywatersolutions.com> Created attachment 160546 [details] [review] Bug 33317: (follow-up) Perltidy Signed-off-by: Owen Leonard <oleonard@myacpl.org> Signed-off-by: Lucas Gass <lucas@bywatersolutions.com> I feel like the system preference description is a bit sparse. Could we add an example or a link to documentation to make it clearer what you are supposed to add? + - + - "Improve search engine crawling." + - pref: OpacMetaRobots + type: textarea Also, the pref description shoudl be formed as a sentencee and connect with the input field. Wait, so this simply adds the same robots tag for the entirety of OPAC? I've been redirected here from Bug 35812 due to conflicts. And I think this bug right here is questionable and I'm not sure if it should be merged, at least as-is. > Websites must have a robots meta tag That is not true, this tag is very much optional and meant for granular page-level steering of crawling bots. The way this preference is implemented, the rules are going to apply to ALL opac pages, which I'm not sure if there's any reasonable use-case. The author mentions example "noindex,nofollow" to prevent ALL opac pages from being indexed. I think that if some library wants that, they'd be better off using the more widely used and known robots.txt file, which is more likely to be supported by various crawlers, and will prevent them from downloading the pages in the first place (instead of downloading the pages it wanted to and then discarding them upon discovering the meta tag for particular page). I think the better direction is to diversify manually which pages should be crawlable by default and which shouldn't like in Bug 35812, ie. search results and so on (dynamic pages) shouldn't be indexed to decreate the amount of junk (but they should be crawled to extract links from it), while main page/info subpages/user-created lists/biblio records should probably be indexed by default and it's probably what most libraries would want by default. With that said, there currently there's no "obvious"/"easy" way of specifying custom robots.txt file, apart from doing something like `Alias /robots.txt /var/www/html/robots.txt` in Apache config for opac (it works well enough btw). So, in the spirit of what OP originally wanted, I believe it could be better to consider instead adding a user preference for textarea of robots.txt file contents, in place of site-wide robots meta tag contents. This would allow libraries to set more granular rules, but someone who wants to block everything could still just do: User-agent: * Disallow: / Btw this is already documented in README.robots file in Koha's main git directory (last edited 13 years ago, the last paragraph there is probably outdated). So I believe my idea above could solve the conflict between our two patches, robots.txt usage is more widely documented on the Internet I believe, and that'd override any rules that Koha devs could specify on per-template basis manually in robots tag on pages like I did in Bug 35812... By the way, having programmatic robots.txt generation could allow having sitemaps linked there with "Sitemap:" directive, provided the generated files are there, and that would remove one obstacle from having the sitemap generation enabled by default in the Debian installation. (In reply to Michał from comment #27) > which I'm not sure if there's any reasonable use-case. The author mentions > example "noindex,nofollow" to prevent ALL opac pages from being indexed. > I think that if some library wants that, they'd be better off using the more > widely used and known robots.txt file Your solution doesn't work. Robots.txt doesn't prevent sites from being indexed, Google itself instructs https://developers.google.com/search/docs/crawling-indexing/block-indexing Our use case is very valid and a basic option in Wordpress: websites wants ways to not be found (while developping). Here we have test and production systems, and of course our customers do not want their users to find the test system. Two koha, similar data, user search google for their library and click on the test link. Hmm, I think for your use-case it might be the best to set the robots configuration as a header, by adding something like this to Apache config:
> Header set X-Robots-Tag "noindex, nofollow"
This way there's no need to modify all Koha templates, and I think it's better off there, as custom ways of hiding a test system, I think, would technically belong to a web server configuration on a layer above Koha, independent of its specific configuration (for example if it's a test system, someone might re-setup Koha and forget to set that setting up again etc. too).
Alternatively, `OPACPublic` preference might be an alternative, although it won't really *fully* hide it, it would do the job for the most part at least too.
Created attachment 168071 [details] [review] Bug 33317: (follow-up) Improve system preference description (In reply to Katrin Fischer from comment #26) > I feel like the system preference description is a bit sparse. Could we add > an example or a link to documentation to make it clearer what you are > supposed to add? > > + - > + - "Improve search engine crawling." > + - pref: OpacMetaRobots > + type: textarea > > Also, the pref description shoudl be formed as a sentencee and connect with > the input field. I added a follow-up to improve the sys pref description. Let me know if you agree with it. Created attachment 168072 [details] [review] Bug 33317: Add system preference to set meta robots for the OPAC Websites must have a robots meta tag to improve search engine crawling, so we could tell search engines to not index OPAC pages for example. To test: 1) Look at opac-main source and see that meta name=robots is missing. 2) Apply patch. 3) Run ./installer/data/mysql/updatedatabase.pl 4) Go to system preferences > OPAC and add some directives to OpacMetaRobots (ex: noindex,nofollow). 5) Look at opac-main source and confirm that meta name=robots now has content equal to the text set in the OpacMetaRobots system preference. 6) This should Prevent search engines from indexing Opac pages (if OpacMetaRobots is set to noindex) Signed-off-by: Owen Leonard <oleonard@myacpl.org> Signed-off-by: Lucas Gass <lucas@bywatersolutions.com> Created attachment 168073 [details] [review] Bug 33317: Using checkoxes in OpacMetaRobots preference system to prevent incorrect data entry Signed-off-by: Owen Leonard <oleonard@myacpl.org> Signed-off-by: Lucas Gass <lucas@bywatersolutions.com> Created attachment 168074 [details] [review] Bug 33317: (follow-up) Perltidy Signed-off-by: Owen Leonard <oleonard@myacpl.org> Signed-off-by: Lucas Gass <lucas@bywatersolutions.com> Created attachment 168075 [details] [review] Bug 33317: (follow-up) Improve system preference description Signed-off-by: Lucas Gass <lucas@bywatersolutions.com> (In reply to Michał from comment #30) > Hmm, I think for your use-case it might be the best to set the robots > configuration as a header, by adding something like this to Apache config: > > > Header set X-Robots-Tag "noindex, nofollow" The idea is to put more power in the hands of librarians and less burden on system admins. Pushed for 24.11! Well done everyone, thank you! Enhancement will not be included in the 24.05.x series |