Programmer Humor

I mean alledgedly ChatGPT passed the “bar-exam” in 2023. Which I find ridiculous considering my experiences with ChatGPT and the accuracy and usefulness I get out of it which isn’t that great at all

Exactly. If it passed the bar exam it's because the correct solutions to the bar exam were in the training data.

The other side can immediately tell that somebody has made an imitation without understanding the concept.

No, they can't. Just like people today think ChatGPT is intelligent despite it just being a fancy autocomplete. When it gets something obviously wrong they say those are "hallucinations", but they don't say they're "hallucinations" when it happens to get things right, even though the process that produced those answers is identical. It's just generating tokens that have a high likelihood of being the next word.

People are also fooled by parrots all the time. That doesn't mean a parrot understands what it's saying, it just means that people are prone to believe something is intelligent even if there's nothing there.

ChatGPT refuses to tell illegal things, NSFW things, also medical advice and a bunch of other things

Sure, in theory. In practice people keep getting a way around those blocks. The reason it's so easy to bypass them is that ChatGPT has no understanding of anything. That means it can't be taught concepts, it has to be taught specific rules, and people can always find a loophole to exploit. Yes, after spending hundreds of millions of dollars on contractors in low-wage countries they think they're getting better at blocking those off, but people keep finding new ways of exploiting a vulnerability.