We have experienced problems with Google indexing: it seems (controlled in Google search console) that Google does not "like" sitemaps with a name like sitemapNNNN.xml (does not read them). Changing the name to sitemap_NNNN.xml miraculously resolves the issue: individual pieces of sitemap are read as they are declared in sitemapindex.xml.
Created attachment 167230 [details] [review] Bug 36977: Google does not read sitemaps with the name sitemapNNNN.xml We have experienced problems with Google indexing: it seems (controlled in Google search console) that Google does not "like" sitemaps with a name like sitemapNNNN.xml (does not read them). Changing the name to sitemap_NNNN.xml miraculously resolves the issue: individual pieces of sitemap are read as they are declared in sitemapindex.xml. Test plan: ========== 1. Have your site configured to work with Google search console (cf. https://support.google.com/webmasters/answer/9008080). 2. Generate sitemap with: misc/cronjobs/sitemap.pl --dir /var/lib/koha/<instance>/sitemap (create the directory if necessary) 3. Check with you browser that the sitemap is generated: http://<OPAC-url>/sitemapindex.xml Check that the individual pieces are readeable: http://<OPAC-url>/sitemap0001.xml 4. Go to the Google seach console > Sitemaps > Add a new sitemap Enter sitemapindex.xml 5. Most probably Google won't read your sitemap chunks with the warning: "Couldn't fetch". 6. Apply the patch. Repeat p. 2, 3, 4, with the difference of the chunk name: http://<OPAC-url>/sitemap_0001.xml 7. You should see Google reading your entire sitemap with the info: "Success".
I'm not able to test this, and it's good that the change worked for you! However, I don't see any reason why the file name change affects whether Google tools can fetch the sitemap files (everything else being correct in the files): 1. Sitemap protocol information (https://www.sitemaps.org/protocol.html): the example on the page (you need to scroll down), doesn't have an underscore. 2. Google information (https://developers.google.com/search/docs/crawling-indexing/sitemaps/large-sitemaps): Google's own information doesn't have an underscore. A quick search shows there can often be multiple reasons why Google tools can't fetch the site map files, including, but not limited to: - something in the site robots.txt file - validation issues (https://www.sitemaps.org/protocol.html#validating) It would be good to hear from others about whether they have issues (or not) with Google tools accessing their sitemap files.
David, thanks for commenting. This is the theory we all know. But the reality seems not to conform to the theory. Just try to provide to Google a sitemap for a catalogue with 200+ K records and see if it will read it. And then try with the patch. So you will be able to valuate this. We discussed this issue during recent Hackfest with a colleague from KIT -- we had both, independently, the same experience and we both found the same work around. Hence the patch.
We at KIT (catalogue with over 1 million records) have independently noticed exactly the same behaviour from Google. Google did not want to index (or even download) our submitted sitemaps, only when we changed the file name (while the content was still the same) the sitemaps were indexed. We can therefore confirm both the problem and the patch. We also tried to contact Google but did not get any answer.
(In reply to Michael Skarupianski from comment #4) > We at KIT (catalogue with over 1 million records) have independently noticed > exactly the same behaviour from Google. Google did not want to index (or > even download) our submitted sitemaps, only when we changed the file name > (while the content was still the same) the sitemaps were indexed. We can > therefore confirm both the problem and the patch. > We also tried to contact Google but did not get any answer. Michael, thank you for your comment. So, could you--or somebody from KIT--sign off the patch, please?
Created attachment 167345 [details] QA Screenshot from the Google Search console (both files are identical)
(In reply to Janusz Kaczmarek from comment #3) > David, thanks for commenting. This is the theory we all know. But the > reality seems not to conform to the theory. Just try to provide to Google a > sitemap for a catalogue with 200+ K records and see if it will read it. And > then try with the patch. So you will be able to valuate this. > > We discussed this issue during recent Hackfest with a colleague from KIT -- > we had both, independently, the same experience and we both found the same > work around. Hence the patch. Thanks Janusz. It's good to have the confirmation from others that it is an issue (despite the theory!), and that renaming the site map files fixes the issue. David
Created attachment 167431 [details] [review] Bug 36977: Google does not read sitemaps with the name sitemapNNNN.xml We have experienced problems with Google indexing: it seems (controlled in Google search console) that Google does not "like" sitemaps with a name like sitemapNNNN.xml (does not read them). Changing the name to sitemap_NNNN.xml miraculously resolves the issue: individual pieces of sitemap are read as they are declared in sitemapindex.xml. Test plan: ========== 1. Have your site configured to work with Google search console (cf. https://support.google.com/webmasters/answer/9008080). 2. Generate sitemap with: misc/cronjobs/sitemap.pl --dir /var/lib/koha/<instance>/sitemap (create the directory if necessary) 3. Check with you browser that the sitemap is generated: http://<OPAC-url>/sitemapindex.xml Check that the individual pieces are readeable: http://<OPAC-url>/sitemap0001.xml 4. Go to the Google seach console > Sitemaps > Add a new sitemap Enter sitemapindex.xml 5. Most probably Google won't read your sitemap chunks with the warning: "Couldn't fetch". 6. Apply the patch. Repeat p. 2, 3, 4, with the difference of the chunk name: http://<OPAC-url>/sitemap_0001.xml 7. You should see Google reading your entire sitemap with the info: "Success". Signed-off-by: Michael Skarupianski <michael.skarupianski@gmail.com>
Created attachment 169815 [details] [review] Bug 36977: Google does not read sitemaps with the name sitemapNNNN.xml We have experienced problems with Google indexing: it seems (controlled in Google search console) that Google does not "like" sitemaps with a name like sitemapNNNN.xml (does not read them). Changing the name to sitemap_NNNN.xml miraculously resolves the issue: individual pieces of sitemap are read as they are declared in sitemapindex.xml. Test plan: ========== 1. Have your site configured to work with Google search console (cf. https://support.google.com/webmasters/answer/9008080). 2. Generate sitemap with: misc/cronjobs/sitemap.pl --dir /var/lib/koha/<instance>/sitemap (create the directory if necessary) 3. Check with you browser that the sitemap is generated: http://<OPAC-url>/sitemapindex.xml Check that the individual pieces are readeable: http://<OPAC-url>/sitemap0001.xml 4. Go to the Google seach console > Sitemaps > Add a new sitemap Enter sitemapindex.xml 5. Most probably Google won't read your sitemap chunks with the warning: "Couldn't fetch". 6. Apply the patch. Repeat p. 2, 3, 4, with the difference of the chunk name: http://<OPAC-url>/sitemap_0001.xml 7. You should see Google reading your entire sitemap with the info: "Success". Signed-off-by: Michael Skarupianski <michael.skarupianski@gmail.com> Signed-off-by: Michał Kula <148193449+mkibp@users.noreply.github.com>
I confirm, upon testing, that the patch indeed fixes the aforementioned issue, which we had also experienced. After applying it, we see that Google Search Console no longer shows 0 pages indexed in the sitemaps section, and now it shows up `/sitemapindex.xml => /sitemap_0001.xml` whereas previously it'd only show `/sitemapindex.xml`.
Created attachment 173612 [details] [review] Bug 36977: Google does not read sitemaps with the name sitemapNNNN.xml We have experienced problems with Google indexing: it seems (controlled in Google search console) that Google does not "like" sitemaps with a name like sitemapNNNN.xml (does not read them). Changing the name to sitemap_NNNN.xml miraculously resolves the issue: individual pieces of sitemap are read as they are declared in sitemapindex.xml. Test plan: ========== 1. Have your site configured to work with Google search console (cf. https://support.google.com/webmasters/answer/9008080). 2. Generate sitemap with: misc/cronjobs/sitemap.pl --dir /var/lib/koha/<instance>/sitemap (create the directory if necessary) 3. Check with you browser that the sitemap is generated: http://<OPAC-url>/sitemapindex.xml Check that the individual pieces are readeable: http://<OPAC-url>/sitemap0001.xml 4. Go to the Google seach console > Sitemaps > Add a new sitemap Enter sitemapindex.xml 5. Most probably Google won't read your sitemap chunks with the warning: "Couldn't fetch". 6. Apply the patch. Repeat p. 2, 3, 4, with the difference of the chunk name: http://<OPAC-url>/sitemap_0001.xml 7. You should see Google reading your entire sitemap with the info: "Success". Signed-off-by: Michael Skarupianski <michael.skarupianski@gmail.com> Signed-off-by: Michał Kula <148193449+mkibp@users.noreply.github.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Created attachment 173613 [details] [review] Bug 36977: (QA follow-up) Adapt tests Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
I cannot confirm this solves an issue or not. But the change is harmless on its own. QA scripts are happy. I adapted the tests to this new filename schema. Note: always check for existing tests on features. This time your change had broken the sitemapper tests. And if there weren't tests, we would require you to add them to avoid regressions.
Pushed for 24.11! Well done everyone, thank you!