Chef Stuart O'Keeffe Recalls Queer Eye Casting Before He Was Replaced by Antoni Porowski: 'A Missed Opportunity'

Stuart O'Keeffe was nearly a member of Queer Eye's Fab Five.

On Wednesday's episode of his Hurrdat Media podcast Don't Let It Stu, the chef and cookbook author revealed for the first time that he was initially cast on Netflix's reboot of the Bravo series before he was unexpectedly replaced by Antoni Porowski.

"I actually got chosen," said O'Keeffe, 42, of the series produced by Scout Productions. "Like, I was one of the Five."

"The worst part was I found out on Instagram I didn't get it," he added, describing a video that showed Porowski learning he'd been cast. "I was like, 'Fine, I'll just cry for the next year.' "

The Irish-born culinary star also shared a photo of himself posing with Queer Eye stars Jonathan Van Ness, Karamo Brown, Bobby Berk and Tan France.

The picture was snapped after producers told them they had made the show, O'Keeffe claimed.

"They came in. 'You're the Fab Five," O'Keeffe recalled, explaining that there were two groups of guys, the other that contained Porowski. "I was like, 'Holy s---.' We all went outside the room — like, screaming, jumping up and down. I was like, 'Hey, let's get a photograph. This is such a great moment for us all.'"

But the moment didn't last long for O'Keeffe.

Though he clicked with Brown, 43, and Berk, 42 — and was told filming was beginning in a month — he started to suspect something was afoot when he learned that the other guys all had their contracts. "[I was] like, 'Hey, what's going on?' " he said. "And they're like, 'Oh, we're working on it. We're working on it.' I'm like, 'Okay, that's fine. Like, whatever.' "

Then, during a scheduled group dinner, France didn't show up. "The other four do, and I'm like, 'What? Why is he not coming to dinner?'" O'Keeffe remembered. "So I'm like, 'Something's up. Somebody knows something. He was kind of being a bit of a d---. And I was just like, 'Oh, this is weird.' So four of us went for dinner in Silver Lake, talking about how excited we were."

After that, O'Keeffe claimed he got a call that he had to go in for another audition and meet with the heads of Netflix.

"Two days later, I get pulled out," said 'O'Keeffe. "And Antoni gets pulled in."

A representative for Netflix did not respond to PEOPLE's request for comment.

As for why he thinks it didn't work out, O'Keeffe said he could only speculate, but alleged that he'd been told that while he had worked with the producers before and was told they'd wanted him on the show, Netflix felt otherwise.

"My agent called me and he was like, 'Look, I know this sounds totally kind of s------ but they said you have done too much stuff,' " O'Keeffe shared. "Like, I was on the Today show, I was on Rachel Ray, 'cause I had a cookbook out and I had done multiple shows before, whereas the other four hadn't, so they kind of wanted everyone on the same level."

"That's kind of a lame excuse by fine, I guess I'll take it," he said.

Another theory floated by O'Keeffe was that Ted Allen, one of the original Queer Eye stars, insisted that producers cast Porowski, who was working as an assistant for him at the time.

"We'll never know," said O'Keeffe.

Ultimately, O'Keeffe — who has appeared on Food Network's Chopped and Private Chefs of Beverly Hills — noted that he's fine with the outcome. "It was such a missed opportunity for me, but you know what, everything works out," he said.

And he insisted he has no ill will towards Porowski, even if he baulked at the fact that Porowski didn't have a culinary education. "I didn't cross paths with him," O'Keeffe said. "He was there obviously. Nice guy, all good."

O'Keeffe's news comes amid reports of tensions on the set of Queer Eye among the Fab Five, that allegedly led to the departure of Berk.

(Berk has since been replaced by HGTV's Jeremiah Brent, who will make his debut in the upcoming season 9.)

Earlier this month, Rolling Stone published a report in which seven production sources alleged that Van Ness, 36, was difficult to work with. Van Ness has not commented on the allegations, and did not respond to PEOPLE's request for comment

Additionally, the piece included claims that France “campaigned to replace Berk” with Brent — something he was able to do with “alleged support” from castmate and friend Porowski. But France, 40, has denied that.

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from celebrity news to compelling human interest stories.

O'Keeffe, for his part, said he hadn't spoken with Berk about any of this, and only speculated on his experiences with Van Ness ("He knew he was a star," said O'Keeffe) and France, who he described as a loner.

Don't Let It Stu is available wherever podcasts can be found.