You need to knock this shit off.

[dogweather]

This is likely breaking the law of my jurisdiction and yours. Can I send you an invoice for the rate-limiting fees I incurred because of you? What's your business address?

[tb0hdan]

Hey @dogweather, there's no need to be this rude.

Idun is a simple, open-source crawler that collects Internet domains. While I'm putting in my best effort to have it work
as efficiently as possible, incidents do happen.

There are several ways you can prevent things like this from happening in the future:

1. Ask your ISP to add HTTP/429 code
2. Implement rate limiting on web server
3. Add proper filtering on CloudFlare
4. Add Idun to robots.txt (yes, that works)
5. Last but not least - ban offending IP

Now, in order for me to fully understand your complaint, please attach here access log from your webserver. I'll work on a fix.

Thanks and take care.

[dogweather]

Why the copy and paste reply? If you look at my screenshots you'll see that I already performed all these steps.

I'm not a user of your software. I'm a third party who's injured by it.

I have to assume you're negligently abusing websites around the world.

[tb0hdan]

Your assumptions are wrong. I haven't pasted a single character.

Again, regarding your complaint - I'm just trying to get to the bottom of it. Like I did here: tb0hdan/domains#15

Please try and be constructive. Add access.log from your web server.

[dogweather]

It looks like this has been a problem for a while: https://webmasters.stackexchange.com/questions/132737/bot-calling-my-php-script-too-fast

I'm not running a technology that produces a legacy-style access.log. Those screenshots are a small portion of my Cloudflare WAF logs. You can see the multiple requests per second, the rate limiting, the timestamps, and the host and url that you were hitting.

[tb0hdan]

The link you've posted describes an issue that has been long resolved. I will, however, conduct further investigation and try to reproduce it. Meanwhile, I have added your website to no-crawl list.

[dogweather]

I have added your website to no-crawl list.

Thank you.