User-agent: * Disallow: /docs/webdav/autopub/* Disallow: /webdav/* Disallow: /api/* Disallow: /html/* Disallow: /*/javadocs/* Disallow: /javadocs/* Disallow: /rensai/* Disallow: /docs/1.* Disallow: /docs/2.* Disallow: /docs/3.* Disallow: /docs/4.* Disallow: /servlets/* Disallow: /download/nightly-builds Disallow: /download/nightly-builds* Disallow: /accelerators/* Disallow: /shop/* Disallow: /doc/dotmarketing/* Disallow: /docs/com/dotmarketing/* Disallow: /browse/* Disallow: /devwiki/* Disallow: /pagefreezer-crawler/*