# search engine robot file. # Use this file to prevent areas # of your website from being indexed # by web robots. # Required for all crawlers to crawl your site ; User-agent: * # followed by Disallow: /whatever name User-agent: * Disallow: /ISP/ Disallow: /images/ Disallow: /cgi-bin/ Disallow: /private/ Disallow: /includes/ Disallow: /components/ Disallow: /modules/ Disallow: /Representing