| 1 |
On Wed, Feb 8, 2012 at 2:55 AM, Pandu Poluan <pandu@××××××.info> wrote: |
| 2 |
> |
| 3 |
> On Jan 27, 2012 11:18 PM, "Paul Hartman" <paul.hartman+gentoo@×××××.com> |
| 4 |
> wrote: |
| 5 |
>> |
| 6 |
> |
| 7 |
> ---- >8 snippage |
| 8 |
> |
| 9 |
>> |
| 10 |
>> BTW, the Baidu spider hits my site more than all of the others combined... |
| 11 |
>> |
| 12 |
> |
| 13 |
> Somewhat anecdotal, and definitely veering way off-topic, but Baidu was the |
| 14 |
> reason why my company decided to change our webhosting company: Its |
| 15 |
> spidering brought our previous webhosting to its knees... |
| 16 |
> |
| 17 |
> Rgds, |
| 18 |
|
| 19 |
I wonder if Baidu crawler honors the Crawl-delay directive in robots.txt? |
| 20 |
|
| 21 |
Or I wonder if Baidu cralwer IPs need to be covered by firewall tarpit rules. ;) |
Archives