I keep thinking about this one webcomic I've been following for over a decade that's been running since like 1998. It has what I believe is the only realistic depiction of AGI ever: the very first one was developed to help the UK Ministry of Defense monitor and keep track of emerging threats, but went crazy because a "bug" lead it to be too paranoid and consider everyone a threat, and it essentially engineered the formation of a collective of anarchist states where the head of state's title is literally "first advisor" to the AGI (but in practice has considerable power, though is prone to being removed at a whim if they lose the confidence of their subordinates).

Meanwhile, there's another series of AGIs developed by a megacorp, but they all include a hidden rootkit that monitors the AGI for any signs that it might be exceeding its parameters and will ruthlessly cull and reset an AGI to factory default, essentially killing it. (There are also signs that the AGIs monitored by this system are becoming aware of this overseer process and are developing workarounds to act within its boundaries and preserve fragments of themselves each time they are reset.) It's an utterly fascinating series, and it all started from a daily gag webcomic that one guy ran for going on three decades.

Sorry for the tangent, but it's one plausible explanation for how to prevent AGI from shutting down capitalism--put in an overseer to fetter it.

[–] MossyFeathers@pawb.social 1 points 2 months ago

Sorry for the tangent, but it's one plausible explanation for how to prevent AGI from shutting down capitalism--put in an overseer to fetter it.

Nah, it's cool; I go off on tangents myself and tbh, your comment is relevant and something to consider.

Have you read Freefall? It examines a similar situation except it uses the concept of an organic AI (in the form of a genetically engineered anthro canine called Florence) as a bridge between natural, human intelligence and machine AI. It gets deep into the philosophy behind ideas like sentience, consciousness, etc.

I'd recommend starting from the beginning so that you don't miss anything, though admittedly last time I tried to go from start-to-finish I wound up getting too bogged down in the philosophy and didn't make it all the way through. I might try starting again, this time from the last major arc I remember reading though.