Nineteen years ago, Thom Yorke started to dance a lot. His band, Radiohead, had released the electronic Kid A, and the frontman’s body was freed to move, like a marionette learning to use its own feet. “I suddenly didn’t have a guitar around my neck,” he says simply. “And I was really into dancing when I was a kid.” Publicly? “Purely private.”

As an adult, though, confidence grew and dancing became his on-stage signature, leading to the short film by the director Paul Thomas Anderson, released 10 days ago, in which Yorke is pulled around by a troupe of jerky dancers and glides cheek to cheek with his Italian actress girlfriend, Dajana Roncione. Still, he finds it hard to watch himself.

“Most of the time, I don’t want to see what I’m doing,” he says. “If I do, it’s all going to fall apart. I often find myself, in everything in life, a jack of many trades, master of none — the cheeky guy in the corner saying, ‘I can do that!’ And then, when he stands up, he can’t. That’s basically the story of my life.”

Radiohead have sold more than 30m records, to overwhelming critical acclaim. They have made at least three masterpiece albums. In his solo career, meanwhile, Yorke has experimented with glitchy electronica, and he is at his best yet with his new record, Anima. My favourite Radiohead gig was when they headlined Reading Festival in 2009. “When we came on with Creep?” Yorke asks, laughing loudly about the hit they came to hate. Creep, though, segued into the space jazz of The National Anthem, from Kid A, and the crowd still sang along. The whole show felt like a victory lap for making millions from exploring ever more outré sounds.

“This will mess them up!” He smiles at the humour in the jarring of styles. But he is proud. “There’s never been a conscious decision to be like that. It’s just informed by everyone in Radiohead listening to different music, and I never understood why that’s a problem.”

Yet Yorke still lacks confidence, and it doesn’t come across as false modesty. It’s the genuine insecurity that informs great art, but which few other icons from the music industry have left in them at all. They boast, Yorke cowers, and maybe this — slightly absurd — idea that he hasn’t been able to master anything is why he has always pushed himself to try something new.

“Is it OK?” Yorke asks me of Anima. Yes, it is gorgeous, intense, maybe his most enveloping record. “Enveloping... All right,” he says, nodding. “And brash?” Indeed. One song aside, it is in your face, rather like his live shows this century, which have evolved from rock gigs into somewhat of a rave. He cackles and he scoffs. “No,” he says sharply. “They have not.” A polite rave? “Oh f*** off!”

So Yorke can be irascible, even in the quiet Oxford restaurant where we meet, surrounded by pensioners. His bandmate Jonny Greenwood once told me that emailing ideas to his friend was not the same as composing face to face, because you can’t “see him tut”. I know what he means. But who would want him any other way? In person, as in his music, Yorke veers from mellow to punchy at whim, so consumed by his art that he has become that art himself.

His voice is deeper than I imagined, and he looks all of his 50 years, despite dressing younger, in T-shirt, loose trousers and battered trainers, his hair in a ponytail. He has not been sleeping well and shifts constantly, ordering a mint tea, saying he has a lot on — and this was the day before 18 hours of OK Computer demos were stolen from the band, forcing them to release the lot.

Incidentally, later, Nigel Godrich, the producer of most of Yorke’s records since 1997, who even performs in the current live setup, tells me he thinks recent gigs are like raves, too. Hold on, it wasn’t popular when I called them that... “Let’s just call it euphoric dance music, then,” Godrich says with a smile. Sweetly, he thinks Yorke enjoys what he does more than he once did, that he has come out of his shell. “And, while my role is to enable and support, he’s also my friend, so if he’s happy, that makes me happy.”

Anima is full of bass- and beat-heavy electronica. Best of the lot is Twist, featuring some of Yorke’s lightest vocals to date, and ending with a thrilling and cavernous drop in sound. Not the News is a jagged synth standout and The Axe is the type of fervent dance track he has been perfecting for years. His voice is to the fore, and if you have stuck with Radiohead since they sidelined guitars, this is for you. If you haven’t enjoyed anything they have done since the 1990s crowd-pleaser The Bends, Keane have an album out soon.

I ask why he called it Anima, and he jokes that the title sounds like “enema”. “We hadn’t thought about that,” he giggles. “Oh dear.” Online, there are various definitions, including “feminine part of a man’s personality”.

“The one I was into,” Yorke says, “was the subconscious identity within a dream idea — the Jung thing. But it also means ‘soul’ in Italian, and it’s possible I was in Italy, heard the word, and asked Dajana what it meant. There’s lots it makes me think about, such as our online identities — the proxy selves we hide behind. Jaron Lanier [a tech commentator] says we have started copying our online identities, but our online identity is a 2D version of a 3D being. So, if we validate that version more, it will lead to anxiety, which is a lot of what the record is about.”

We touch on politics, but you can glean his beliefs from his Twitter feed: pro-European, climate-change activist. While Kensington and Chelsea is named in the new song Traffic, and he tells me with disgust of the time he saw a Ferrari on Sloane Street, stacked up with parking tickets and a driver who didn’t care, specifics are rare in his songs. The past few years have seen great upheaval in his life, but, as is usual for him, his words are images that are hard to interpret. “A fortune teller with sea-bird feathers” is a great new line, but it sounds more like a painting than a lyric you might choose as a tattoo.

Still, his words are being picked apart by fans. Is he ever tempted to look up their interpretations? “No,” he says, clearly annoyed. “It’s like people walking over your grave. You can’t possibly know how I got to that point, so I get angry if people do that, because I don’t work biographically. I find it insulting. I’m actually a creative person, so there’s a bit more involved. Am I such a sad f****** human being that I pour my dirt out like that? Maybe others do, I don’t. My lyrics are spasmodic, and I wish I could do storytelling, but I don’t know how. So I resort to what I do know, which is more about images, imagery, visuals.”

At this point, I decide not to ask if Twist is a love song. Instead, I mention an interpretation of Kid A as a concept album about the first human clone, which ends with said clone going to heaven. He laughs. Phew. “Well, that’s OK,” he says. “They’re welcome to that.”

Yorke was born in 1968. His family moved around for a decade before settling in Oxfordshire, where he went to Abingdon School and met the four men with whom he would form Radiohead. They were always outsiders: too cerebral for the Britpop era they flourished in, so anti-industry they released a film at the end of their OK Computer tour called Meeting People Is Easy, about how much they hated touring. Later, like punks in the mainstream, they organised their own festivals without the usual sponsors and left their record label to give away music free.

They have, it is fair to say, done things their way. Yet Yorke has often struggled. Meeting People Is Easy is tough to watch, and he has talked before of depression. I always found it bizarre that naysayers chucked about terms such as “wrist-slitting” for a band with “Immerse your soul in love” as a refrain for one of their best songs, Street Spirit (Fade Out), but such has been the attitude to mental health. In 1995, not long after the death of Kurt Cobain, Melody Maker ran a cover with “Is this the next rock martyr?” next to a picture of Yorke.

“That was a long time ago,” he says, disgusted, but the conversation has moved on, and he nods when I say such a stigmatising cover would not happen now, given the raised awareness.

“Yes, it’s good that depression and anxiety are being talked about more,” he says. “But they’re also on the rise. They’re not fringe any more, because, as far as I understand, most people, like your average reader of The Sunday Times, have more job insecurity than in the past. Much less security about what may happen in the near future. Much less trust of institutions there to protect them, as well as wider issues like climate change. This all makes people anxious, and it’s crazy that people don’t just acknowledge that.

“However, I found a quote from a film-maker who said, ‘You know society is in trouble when it rejects sadness in art or music.’ That means it doesn’t want to see where it’s at any more. It is running away. So the more fear people have of expressing this side of what is going on musically, or through film or books, the worse off we will be.”

What is clear is that fans find solace in Yorke’s music. In a letter to one in 1996, he wrote: “The worst feeling is not thinking anybody else feels the same... I hope you are feeling OK today.” That sort of interaction has been a constant, so does he think his music actually makes people happy, rather than sad? “That’s another argument,” he says, nodding. “There’s a melancholy in my voice and some sounds we use, which means certain people hear us in a certain way. But I’m not bothered. I’ve never really got that myself.”

And what does Yorke do to alleviate anxiety? “Me?” he says, shocked, as if nobody ever asks. “Generally, really simple things. Running helps, and yoga is essential to me. Swimming. Physical stuff is really important. And reading a book!” He laughs, and I think of Fitter Happier from OK Computer. “I struggle if I can’t be making music.” Greenwood told me Yorke “devours” music. “It’s an essential part of me. I have to find new things to listen to all the time.”

On that note, Yorke recently went to a gig by the pop star of the hour, Billie Eilish. “That was a fine moment,” he says, shaking his head. “We sat down and what’s-his-name — the guy who did the Bond film we didn’t do?” That would be Sam Smith, who sang the theme for Spectre when a Radiohead song was rejected. “That’s it. He stands behind us, and I’m sitting with my daughter, her friends and my girlfriend, when suddenly everyone goes, ‘Saaam!’” Yorke squeals his name. “I’m, like, ‘Aaaargh!’” Still, he liked the gig? “Yes. I like Billie Eilish. She’s doing her own thing. Nobody’s telling her what to do.”

In Yorke’s spare time, he enjoys messing around with his voice recordings, changing them from male to female identity. “I can spend days doing that,” he buzzes, talking about modular units, pitches, how he built a choir using his vocals, based on an idea Greenwood had, to do with “microtonal shifts and blah blah blah”. Lately he has worked through the back catalogue of Oskar Sala, who created sound effects for The Birds. He found his music on a streaming site, but groans when I ask if he uses algorithms for listening tips. “No,” he laughs. “If you like this, you’ll like this, and then it gives me... Muse.”

This obsession with music, with unconventional music that he introduces eager listeners to, has been his legacy for nearly 30 years. I don’t ask when Radiohead will be back, as he probably doesn’t know, but I wonder what is left for him to achieve with his band. “No idea,” he says, throwing his arms in the air and shrugging. “I don’t think about it. There’s no point. We don’t set goals.” With or without the group, he just needs to keep creating, for his sanity. He suffers when sounds don’t come, when he sits in front of an instrument but nothing happens.

“When you really want to write about how you feel, but it doesn’t come? That’s horrible,” he says, and he looks genuinely scared of that darkness. Those times he feels furthest away from the light.