Internal Server Error

Seems like the bots are back at work…

Got this today

If you keep refreshing the error it loads the page but pretty annoying. Maybe making everything a members only site would stop attracting the ai bots but also drive less web traffic

Seemed just like the last time - maybe 3-4 years ago (could be 6-7...)  I didn't think much of it and just keep refreshing until it works - usually first refresh.

At least this shows who's willing to put a little work into browsing and contributing, anyway.   

I've been running a message board in one form or another for ~20 years (amazing i know).

I use more modern software, but also have been struggling with the amount of bot/scraper activity this past month or so.

Heavily driven by AI crawler traffic as far as I can tell.

I've been putting it off, but I'm going to invest the time soon and move my MB behind Clouldflare free tier.

You can also play whack-a-mole ip-blocking stuff but it's kindof an endless churn and constant work otherwise.

Heavily driven by AI crawler traffic as far as I can tell.

Interesting that AI would do more of this than, say, Google was already doing. In any case, it seems to me that the AI bots ought to be paying for the content that they are scraping, as this is a much different use case than Google directing traffic to a website.

Anthropic is supposedly going to pay out $1.5B for scraping info from a bunch of pirated copyrighted books that it found online, and 3 of my books were included in the settlement, so I'm supposed to get a payout--not a lot, but better than nothing. It doesn't take much to assert a copyright, and although I assume that has not been done in the past here at NCKA, perhaps it could be going forward...

I get this when my internet's not a-plus, so I was assuming the internal server error issues I had were flukes caused by spotty wifi. Sometimes I need to resubmit 3-4 times to get to the page, but I eventually get there

Heavily driven by AI crawler traffic as far as I can tell.

Interesting that AI would do more of this than, say, Google was already doing. In any case, it seems to me that the AI bots ought to be paying for the content that they are scraping, as this is a much different use case than Google directing traffic to a website.

Anthropic is supposedly going to pay out $1.5B for scraping info from a bunch of pirated copyrighted books that it found online, and 3 of my books were included in the settlement, so I'm supposed to get a payout--not a lot, but better than nothing. It doesn't take much to assert a copyright, and although I assume that has not been done in the past here at NCKA, perhaps it could be going forward...

Google is actually quite respectful of websites relatively speaking... they don't crawl at very fast rates, and if you have a robots.txt file that designates parts or all of the website offlimits to crawling, google will respect that.

Microsoft used to be more aggro about crawling, and I blocked them long ago.

Baidu and other chinese crawlers could be more aggressive as well, though the big names would still mostly respect robots.txt from what I could see.

Now there are just shitloads of random bots/crawlers I don't even recognize but it's pretty clear a lot of them are AI crawlers and they don't give two shits about robots.txt or crushing your server under load lol

It's pretty bad and only getting worse.

It feels like it's become mandatory to have to put servers behind some sort of service like Cloudflare or AWS WAF or equivalents now.



<edit> Just saw polepole in an old post mention Cloudflare and I just checked out of curiosity and they are already using it, so maybe it's just some setting to more aggressively throttle bots, not sure. I've not done this yet myself on my own mb so will be learning more about it myself soon.

So
Not much traction on killing this problem,  is it harmful to the end (us) users?

...is it harmful to the end (us) users?

Yes

On my own messageboard (since my last post) I successfully migrated behind Cloudflare and did several layers of lockdowns which took me from several hundreds of "guests" (basically bots) to around 5-20 at any given time. It's been a huge help to performance.

If any of the admins here ever want to hit me up for a rundown of my configs just email me.

maethlin - gmail.com

Happy to assist.

On my own messageboard (since my last post) I successfully migrated behind Cloudflare and did several layers of lockdowns which took me from several hundreds of "guests" (basically bots) to around 5-20 at any given time. It's been a huge help to performance.

If any of the admins here ever want to hit me up for a rundown of my configs just email me.

maethlin - gmail.com

Happy to assist.

I hope something like this is considered.
This issue seems to be getting worse for me 😔

Sad its come to this. I'd say around 30-50 pct of the time when I click to execute here I get that error.

The fact that ALL our information is being mined by AI has resulted in my personal choice to no longer post fishing reports of any kind. I posted here and on other forums freely so motivated fisherman could do the work and learn.  I have zero such interest in my "data" being mined to provide mindless jerkoffs from getting fishing information I never consented to release.

The fact that ALL our information is being mined by AI has resulted in my personal choice to no longer post fishing reports of any kind. I posted here and on other forums freely so motivated fisherman could do the work and learn.  I have zero such interest in my "data" being mined to provide mindless jerkoffs from getting fishing information I never consented to release.

I have same feeling but wouldn’t the “members only” section restrict crawler access?
Not sure how the site is configured.