this post was submitted on 05 Jul 2023
95 points (98.0% liked)

sh.itjust.works Main Community

7701 readers
1 users here now

Home of the sh.itjust.works instance.

Matrix

founded 1 year ago
MODERATORS
TheDude@sh.itjust.works
manifex@sh.itjust.works
biela@sh.itjust.works
imaqtpie@sh.itjust.works
95
Has anyone else been getting a lot more errors on sh.itjust.works recently? Could the server be overloaded? (sh.itjust.works)
submitted 1 year ago* (last edited 1 year ago) by seal_of_approval@sh.itjust.works to c/main@sh.itjust.works
20 comments fedilink hide all child comments
 

I've been getting a lot of these messages lately:

SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data

So I decided to look at the network requests in the browser console. After all, what JSON data was it trying and failing to parse?

Well, turns out that it was pretty much always:

Timeout occurred while waiting for a slot to become available

I don't do a lot of backend work, but I suspect that the lack of slots might be referring to the server running out of resources to process all of the ongoing user activity concurrently. Could we be running into scaling issues?

you are viewing a single comment's thread
view the rest of the comments
[–] biela@sh.itjust.works 96 points 1 year ago (4 children)

A lot of bots were spamming the instance which was causing an IO bottleneck, and thus timeout issues. We're looking into solutions to limit their impact, such as rate-limiting users. We're trying to do it in such a way that it doesn't impact regular users which is why it's taking a bit longer.

[–] jballs@sh.itjust.works 15 points 1 year ago

Awesome, thanks for the update. You guys rock.

[–] burgersc12@sh.itjust.works 10 points 1 year ago (2 children)

Does this instance have a bot problem?

[–] imaqtpie@sh.itjust.works 14 points 1 year ago* (last edited 1 year ago)

Not in the sense of the other servers that were overtaken by botswarms. But there have been a handful of bots recently.

[–] Barbarian@sh.itjust.works 10 points 1 year ago

It's hard to get a bot onto this instance, but not impossible. Once they're on here, they can spam out nonsense as fast as the server can take it (hence the rate-limiting solution).

[–] InEnduringGrowStrong@sh.itjust.works 7 points 1 year ago

Yea, rate-limiting per user instead of per IP might be a bit more complicated.
Appreciate the work y'all put into this.
You guys seem to handle fine, but if you need to toss a few ideas at a network engineer, ping me.