From 83f480e39e04882231a8a4a3f72feadd5a6adab0 Mon Sep 17 00:00:00 2001 From: JAMES Mason Date: Mon, 30 May 2011 20:54:24 +1200 Subject: [PATCH] [SIGNED-OFF] add another example to README.robots Signed-off-by: Magnus Enger --- README.robots | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-) diff --git a/README.robots b/README.robots index a08371d..6865d0a 100644 --- a/README.robots +++ b/README.robots @@ -5,13 +5,23 @@ look for the file /robots.txt. If this file is found and has lines that apply to them they will do as instructed. A very basic robots.txt follow as an example: +------------------------------------------- # go away User-agent: * Disallow: / +------------------------------------------- This tells every search engine that cares (User-agent: *) to not index the site (Disallow everything past /). +Another slightly more intelligent robots.txt file example allows for some bot indexing (good for your site in google, etc), but also stops your Koha from getting thrashing by ignoring URLs that cause heavy search load + +------------------------------------------- +# do some indexing, but dont index search URLs +User-agent: * +Disallow: /cgi-bin/koha/opac-search.pl +------------------------------------------- + If you have installed Koha to /usr/local/koha3 then this file would be placed in the directory /usr/local/koha3/opac/htdocs/. This should prevent search engines from browsing every biblio record, and every view of each record, on -- 1.7.4.1