this post was submitted on 05 Jul 2023
95 points (98.0% liked)

sh.itjust.works Main Community

7649 readers
23 users here now

Home of the sh.itjust.works instance.

Matrix

founded 1 year ago
MODERATORS
 

I've been getting a lot of these messages lately:

SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data

So I decided to look at the network requests in the browser console. After all, what JSON data was it trying and failing to parse?

Well, turns out that it was pretty much always:

Timeout occurred while waiting for a slot to become available

I don't do a lot of backend work, but I suspect that the lack of slots might be referring to the server running out of resources to process all of the ongoing user activity concurrently. Could we be running into scaling issues?

top 19 comments
sorted by: hot top controversial new old
[–] biela@sh.itjust.works 96 points 1 year ago (4 children)

A lot of bots were spamming the instance which was causing an IO bottleneck, and thus timeout issues. We're looking into solutions to limit their impact, such as rate-limiting users. We're trying to do it in such a way that it doesn't impact regular users which is why it's taking a bit longer.

[–] jballs@sh.itjust.works 15 points 1 year ago

Awesome, thanks for the update. You guys rock.

[–] burgersc12@sh.itjust.works 10 points 1 year ago (2 children)

Does this instance have a bot problem?

[–] imaqtpie@sh.itjust.works 14 points 1 year ago* (last edited 1 year ago)

Not in the sense of the other servers that were overtaken by botswarms. But there have been a handful of bots recently.

[–] Barbarian@sh.itjust.works 10 points 1 year ago

It's hard to get a bot onto this instance, but not impossible. Once they're on here, they can spam out nonsense as fast as the server can take it (hence the rate-limiting solution).

Yea, rate-limiting per user instead of per IP might be a bit more complicated.
Appreciate the work y'all put into this.
You guys seem to handle fine, but if you need to toss a few ideas at a network engineer, ping me.

[–] burndown@sh.itjust.works 4 points 1 year ago
[–] DSAlKota@vlemmy.net 31 points 1 year ago (1 children)
[–] BenFranklinstein@sh.itjust.works 23 points 1 year ago (1 children)

It’s A LOT worse on lemmy.world, I’d say most of the instances are overloaded that prime time instances.

[–] grissee@lemm.ee 5 points 1 year ago (1 children)

a .world refuge here, can confirm it's overloaded, now I'm moving to lemm.ee

[–] InEnduringGrowStrong@lemm.ee 5 points 1 year ago

Yea I have an account on both sh.itjust.works and lemm.ee
For the last day or so lemm.ee has been more reliable but otherwise sh.itjust.works was very good too. Ultimately, I feel like this whole thing was meant for having a lot more smaller instances rather than huge ones.

[–] southsamurai@sh.itjust.works 15 points 1 year ago (1 children)

It was the bot bullshit, as others have said. I was kinda expecting it once I saw the posts about the account spamming new C/s, but it hit harder and longer than I thought it would.

[–] Oni_eyes@sh.itjust.works 6 points 1 year ago* (last edited 1 year ago) (1 children)

That's what...she said?

Conversely:

So did that homegrown green

[–] manitcor@lemmy.intai.tech 11 points 1 year ago

theres apparently some bots running around posting from it as well, may or may not be causing extra load

[–] Contextual_Idiot@sh.itjust.works 6 points 1 year ago (1 children)

I've been seeing that error, and sometimes when I click on a post it sends me to an error page instead. Reloading seems to work, eventually.

[–] Corkyskog@sh.itjust.works 6 points 1 year ago

Jerboa app is stable in the way that it just won't show you shit until it clears the error pages lol

[–] Bosa@lemmy.world 1 points 1 year ago

Ya been getting errors and I'm using the world instance. I imagine it's server capacity and the amount of people posting, joining etc (which is a good thing if that's the case with people coming over)

load more comments
view more: next ›