Caeleb Dressel Says He Wants to 'Prioritize Myself' After 2021 Olympics: 'I Drove Myself Crazy'

Caeleb Dressel started swimming at age a young age — but as a form of childcare rather than out of his own personal interest.

"It was right by my dad's work," the Olympic swimmer, 25, tells PEOPLE in this week's 2021 Sexiest Man Alive issue of the local pool. "It was easy for my mom to throw us, literally, in a pool, burn off some energy. Swimming's been a part of my life very much so since around age 4, and I never really dropped it."

Though Dressel grew up in the pool, he didn't always want other kids to know about his favorite sport.

"I was super insecure in middle school; I didn't tell anyone I swam because it wasn't the cool sport to be doing," Dressel admits. "I didn't want anyone finding out that I wore a Speedo! I was super self-conscious about that. People would ask me like what sport I did — I mean, I was pretty tall — and I would lie to them."

Another thing the Florida native wanted to keep from his peers: the fact that he wore glasses. "I didn't have the greatest vision and contacts were a lot more expensive back then, so I got put in glasses, and I just wouldn't wear them," he recalls. "I had the glasses, I wasn't in a cool sport, I was super skinny, lanky, so, of course, that's the perfect concoction for being a pretty big weirdo about yourself."

The seven-time Olympic gold medalist came around in high school. "By that time I got to high school, I definitely started to respect the sport more — really started to obsess over it from a technical standpoint, a mental standpoint," Dressel says.

RELATED VIDEO: Kumail Nanjiani on Surviving Awkward High School Years and His Self-Care Secret: 'Not Looking in the Mirror that Much'

That obsession earned him a spot on the U.S. men's swim team, with whom Dressel made his Olympic debut in 2016. Despite Dressel's success with the sport, "I don't think I'm 'the guy' in the sport of swimming, but people want to label me as 'the guy,' " he says. "I don't see myself as that person because I know there are people out there with talent. I still have a lot left in this sport."

Dressel recognizes he needs to prioritize his mental health if he wants to continue to perform at an Olympic level.

"That's the biggest goal after coming back from Tokyo: being able to breathe," the University of Florida alum says. "I think I overdid it, to be honest. I got great results, but I think it was to a fault. Because I drove myself crazy and I don't think that's fair to myself. I don't want to end up leaving the sport early or hating it because I didn't give myself time to respect the water and I feel like the water has always respected me. I would like to prioritize myself a little bit more instead of swimming."

Dressel admits he had "some breaking points" while training for the Tokyo Summer Olympics during quarantine, and credits his coach with helping him move forward and achieve success at the Games.

"The pressure a lot of these athletes, myself included, are under, you have to keep so much bottled up" Dressel continues. "You can't let this emotion come forth because you'll crack, so you just keep shoving it down, ignoring it and trying to maintain the course that you're on and anything that's a distraction, you just throw it away and don't let it come out until the Games are over. It's a lot of pressure on that moment. Then all this stuff just comes forth."

With the Toyko Games in the books, Dressel plans to focus on other areas of his life. "I want to be a better son, a better brother, a better husband, better dog owner," Dressel, who married high school sweetheart Meghan in February, says. "Every aspect of my life I want to be better at in some way."

Dressel's favorite ways to indulge: Reese's, Five Guys and Gilmore Girls. "Megan started watching Gilmore Girls and it's so good," he says. "So I've been tearing up some Gilmore Girls."

For more Caeleb Dressel, pick up the latest issue of PEOPLE, on newsstands now, or subscribe here.

He also claims to have "a sweet tooth — by my standard."

"I really like Reese's, that's like usually my dessert with dinner," Dressel says. "If it's just one of those days where I get out of practice, I'm like, 'Hey, I need to go to Five Guys. I need to get two burgers and a large fry and a milkshake.'"

Dressel knows that even with two Olympics appearances on his resume, he doesn't have it all figured out just yet.

Never miss a story — sign up for PEOPLE's free weekly newsletter to get the biggest news of the week delivered to your inbox every Friday.

"I'm 25 at the moment, maybe I'll have it figured out when I'm 30. Probably not. Maybe when I'm 40. Maybe when I'm like 60, that would be cool," he says. "I would hope by then I would have it figured out and I just be a super happy guy who's still swimming. I hope I'm still wearing Speedos when I'm 60. I know I got a long way to go, but I've enjoyed it thus far."