|
Hi,
My logs have been showing that Im getting hit hard by a bot python-urllib/1.15 and the ip traces back to Chalmers Innovation in Sweden.
It doesn't seem to me like they are a SE but rather a company that helps Startup companies (of which the one I work for is not a startup)
Does anyone know why I would want to keep allowing them to crawl my site? It doesn't seem to me to be worth it for us.
This is the url string I get.
2004-02-29 17:46:07 62.119.45.186 - w3svc4 myserver myserverip GET /robots.txt -200 0 421 90 140 80 HTTP/1.0 Python-urllib/1/15 - -
Any input/info on these guys would be appreciated...thanks
|
