Technology

‘A Casual Theft of a Great American Artist’s Work’

Except.... maybe not?

Dudesy started an "AI podcast" as in a podcast "generated by AI" back when GPT-3 was just coming out. Their first episode included an extensive discussion of kayfabe. In other words, an elaborate hoax, using more traditional voice-masking tools, to record a human-written (perhaps AI-assisted?), human-voice, speaking the lines and having Carlin's voice replace the original voice speaking.

Long article, but worth the read. Certainly seems like kayfabe to me.

https://arstechnica.com/ai/2024/01/did-an-ai-write-that-hour-long-george-carlin-special-im-not-convinced/

It’s also worth remembering the context around AI at the time Dudesy premiered in March 2022. The “state of the art” public AI at the time was the text-davinci-002 version of GPT-3, an impressive-for-its-day model that nonetheless still utterly failed at many simple tasks. It wouldn’t be until months later that a model update gave GPT-3 now-basic capabilities like generating rhyming poetry.

When Dudesy launched, we were still about eight months away from the public launch of ChatGPT revolutionizing the public understanding of large language models. We were also still three months away from Google’s Blake Lemoine making headlines for his belief that Google’s private LaMDA AI model was “sentient.”

The strongest evidence that the Dudesy AI is just a bit, though, comes later in that first episode. It starts with a lengthy discussion of kayfabe, a popular professional wrestling term that Sasso extends to include any form of entertainment that is “essentially holding up the conceit that it is real… if you're watching a movie, the characters don't just turn to you and say, ‘Hi, my name is Tom Cruise’… he's an actor.”

Kultgen links the kayfabe concept to one of his favorite reality shows, saying, “For The Bachelor, most of that audience believes it's real. Almost none of the WWE audience believes it's actually real.”

That’s when Sasso all but gives up the game, as far as Dudesy is concerned. “Of course nobody believes [the WWE] is real,” he says. “It's not about it being real. It's sort of a... you know, they say it's like a burlesque for guys. And that's what Dudesy is, a burlesque for guys.”

When I first approached Willison with the question of whether a current AI could write the Dudesy-Carlin special, he said he’d “expect GPT-4 to be able to imitate [Carlin’s] style pretty effectively… due to the amount of training data out there.” Indeed, if you ask ChatGPT-4 for some Carlin-esque material, you’ll get a few decent short-form observations, though none of the vulgarity and little of the insight that characterizes a true Carlin bit.

After watching a bit more of the special, though, Willison said he grew skeptical that GPT-4 or any current AI model was up to the task of creating the kinds of jokes on offer here. “I've poked around with GPT-4 for jokes a bunch, and my experience is that it's useless at classic setup/punchline stuff,” he said.

Willison pointed specifically to a Dudesy-Carlin bit about the potential for an AI-generated Bill Cosby (“With AI Bill Cosby, you get all of the Cosby jokes with none of the Cosby rapes”). Willison said he’s “never managed to get GPT-4 (still the best available model) to produce jokes with that kind of structure… when I try to get jokes out of it, I get something with a passable punchline about one out of ten times.”

While Willison said that Dudesy’s Carlin-esque voice imitation was well within the capabilities of current technology, the idea that an AI wrote the special was implausible. “Either they have genuinely trained a custom model that can generate jokes better than any model produced by any other AI researcher in the world... or they're still doing the same bit they started back in 2022,” he said.