Wes Bergmann Shares the 1 Way He'd Return to ‘The Challenge’ After Announcing His Retirement (Exclusive)

Warning: This post contains spoilers for Thursday night's episode of The Challenge: USA.

Wes Bergmann has taken his final bow on The Challenge.

After being thrown into elimination on Thursday's episode of The Challenge: USA — in part thanks to a vote from friend and fellow Challenge vet Josh Martinez — Bergmann, 39, lost to Survivor’s Chris Underwood. Before heading home, Bergmann, who’s expecting his first child with wife Amanda Hornick, told his castmates and host TJ Lavin that he plans to retire from the game.

“I’m not coming back. And it’s been a privilege,” Bergmann said from the Arena floor. “To everyone making this, playing with y'all, like, I’m going to miss it all. So goodbye.”

In a confessional interview, Bergmann said The Challenge “is just a really large part of my life” and that he feels proud of the legacy he left behind.

The Real World: Austin alum detailed his decision a bit more on X (formerly known as Twitter) last month.

“It’s mounting entrepreneurial responsibilities paired with becoming a father (in the next month) - there’s no time,” he wrote Aug. 27. “Coupled with doing nothing to anyone and still people trashing my name behind my back in interviews - I’m above this.”

Bergmann explained to PEOPLE that he felt disrespected by his teammates on the green team at the start of the season.

“It's just the total picture of how green team — and some of them in particular — treated me, which was, ‘We're going to put you in a corner. We're going to lie to you. You're then going to come back very maturely and ask, “Why? And are we going to be able to move past this?” They're going to apologize. Then they're going to lie to me again and again and again,’” the three-time Challenge champ said. “Then I expect them in their interviews to be like, ’Man, I feel bad for what's going on, but we have to do it.’

"Instead, it's more like, ‘I don't believe anything that Wes is saying.’ They're buying into the propaganda that Wes is this master manipulator when really, I'm the guy that likes to work with new people and forge new alliances.”

Bergmann dissects the USA season 2 gameplay, how it felt for Martinez, 29, to vote him into elimination and how he’d approach a possible return to The Challenge.

PEOPLE: You got off to a not-so-great start in Thursday’s episode when you were the only one to miss the baton in the men’s heat. What happened?

WES BERGMANN: I did not win the high jump competition, which definitely is embarrassing, but I couldn't tell at the time what happened. After watching it, I paused it, because I had been wondering, why was I so bad? It went right through my hands, and it was a combination of slip/I jumped a little too far. I overthought it, and I went a little too far and it went right through my hands. But what some of the other guys were able to accomplish, [Johnny] Bananas and Fessy [Shafaat] in particular, I would've been nowhere near that. But it still stinks to lose right away on that first one — very embarrassing.

But you seemed confident going into the elimination.

I've been in a bunch of these things and I know that I can go in and stay focused. I know that there's going to be very little things that they can throw at me that I won't stand a really good shot at. Statistically, I'm one of the worst people that you could ever want to go up against, just because of my well-roundedness. No matter what you throw at me, you're going to have to side eye me, and I'm aware of that and I take that confidence into the game.

So why do you think anyone voted for you at this point in the game?

One thing that I've got to hand a little bit to Josh and definitely to Chris, there are predecessors of mine that I kind of hate because they feel the same way as Chris but they don't acknowledge it. Chris not only calls his shot and puts himself very much so in harm's way — there would've been less scary people to go in against — but he says, "Listen, I'm going to just say what everyone is thinking but no one is saying, which is that he is the best and we've got to get rid of him." That was brilliant. There have been people in the past that have gone at me, but they have to do so anonymously and secretly. He called his shot, he went at me, and I'm giving him his flowers. He beat me fair and square in a fair game, in a physical game. I brought it and he just brought it better. He took out somebody that was easily the biggest threat to win the final and that was smart.

And now, even though I am retired, if I ever un-retired — which I'm not — my strategy would be to come in completely out of shape. That's my only chance these days of getting to a final, because they see what I can do and they know good and well what I'm talking about. I need to come in 50 lbs. overweight. I'm going to start working on getting overweight right now, and that will be the only likelihood that I will ever make it to the end again.

C.T. Tamburello came back with a “dad bod” and performed very well.

It helped him so much. I'm not comparing myself to CT — CT is a better Challenger than me. He's a freak of nature, and he's a large man that can carry his body weight really well. And he's smart and he gets the game. He is on another level. But do you know when he used to be super ripped and young? He was enemy No 1! All of us would get together in hushed corners of the houses and we'd discuss anything and everything that it would take to get rid of him. Then he got a little bit chubby and we stopped having those meetings, and he still beat everybody. Even though he is on another level than me, some of us are going to need to do exactly that.

Bananas does a really good job of it. He walks upstairs and he's slow, and he grabs his back and he gets out of bed. Anytime the cameras are not around, he just makes fun of himself. It's brilliant because it makes people less worried about him. Meanwhile, I'm just jacked and running miles in front of their faces. And then obviously, I'm the smartest. I wish there was a way for my brain to get temporarily injured, so that way they could stop being worried about my brain and my cardiovascular fitness.

I don’t think your fans would wish that on you.

That's why I said temporary. If there was an ankle injury, but for the brain, that would be perfect for me if I ever did it again, like some sort of temporary amnesia or something. That's my only shot at this point.

Did you go into this season thinking it would be your last?

I came in knowing that this was going to be my last. Scheduling-wise, I can't make it happen anymore with my company. We're talking multiple months [away]. It's been almost impossible and it's been irresponsible for me to keep going, and that's without adding the variable of the child. The kid is definitively the straw that breaks the camel's back, but it's not the largest piece of the pie. Ultimately, I could hire help for the times that I'm away. People have traveling jobs as fathers. It's just that the CEOs of don't leave for two months to check into Czech prison, which is kind of what we're doing. I'm on the other side of the world without a phone, without internet, without email. So it's mostly the business, a little bit my kid, and then a little bit — let's be honest — what could I possibly accomplish that's left? I've won a bunch. I've been the face of this franchise for a long time. I've got legendary status. I do enjoy it, so that would be the main reason and only reason to come back.

Would you have liked to go further on your last season?

Of course. The ride out on the sunset thing would be fantastic, but when I saw the breakdown of things, I knew that that was not the case. We [the Challenge vets] were basically set up. We were brought in as actors is what it boils down to, because they didn't actually think that we stood a shot. The fact that we had been able to survive as long as we have is actually even remarkable.

How did it feel when you learned that Josh was one of the people who voted you in?

It stunk and it didn't make sense. He's right. I would beat him at the end with my arms tied behind my back, but so will all the other guys that are there, so I don't understand what going after me was going to accomplish. A lot of Josh and I's friendship is edited out, because it's boring. We play chess and we hang out. We do jive, or we did. And that's evident by, why is the man crying while voting for me? My message to Josh, or really any Challenger: If you are crying over voting for someone, you either should have been more upfront with the person, or you're a bad person. If Josh is ever able to play again, which would surprise me, if anyone is ever better than him, they have to be thinking now, "That's how Josh plays," which is he will cut anyone that he thinks is better than him, and that's a long list.

What are you going to miss most about doing The Challenge?

There's a lot of things that I'm going to miss about The Challenge. To be in a game that's multiple months long, that is happening 24/7, it's like, while you're brushing your teeth, you're kind of working. While you're walking down the hall, while you're eating, while you're in a bus, while you're doing the crazy, fun things that you'll remember for the rest of your life, while you're doing all those things, it's a constant game. And that's a box I will never be able to check off by just joining my intermural rec soccer league down the street. The Challenge is the only place to get that fix. The answer to that question is a book, so we'll start with that.

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

The Challenge: USA airs Thursdays at 10 p.m. ET on CBS.