Luke Evans Has ‘Terrible Anxiety’ About Not Looking ‘Good Enough’ Shirtless: ‘I’m Not Massively Confident’

Although Luke Evans has become known for his impressive and toned physique, he’s not always confident in his appearance.

The actor, 45, appeared on the November 27 episode of the How to Fail with Elizabeth Day podcast and got candid about aging and having to show off his body throughout his career.

“I still sometimes go to the gym and just feel very anxious,” he explained. “I look at myself in the mirror and just go, ‘You don’t look good enough,’ or ‘You’re letting it go.’ I look at my face…the terrible part of my industry is that you’re just reminded constantly of the decades of film you’ve done when you had not a crease on the face and not one gray stubble, whereas all my stubble is now gray.”

“I’ve had to learn to be kinder to myself, but I have terrible anxiety about feeling good enough physically,” he continued. “Part of that isn’t completely bad [because] it gives you a little something to fight for, but it can be overwhelming. I’ve been on a beach just recently, and I didn’t want to take my T-shirt off. I don’t want to be in that place. I know I shouldn’t feel like that, but you know we are sensitive creatures, we’re very delicate. As hard as I may look, I am quite delicate.”

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories. 

Evans said that staying fit and maintaining a certain fitness level is part of his job and admitted that if he wasn’t an actor, he wouldn’t prioritize his appearance as much as he does now. He's known for movies in which he has to be in top shape, like Beauty and the Beast, Fast & Furious and The Hobbit trilogy.

“I’m sure if I didn’t have to go and take my top off on a movie set every now and again, I’d probably let it all go,” he said. “I wouldn’t care so much, but it is part of my job. I get cast in certain roles that require a certain amount of physical strength and aesthetic. It’s been good because it’s probably kept me on the straight and narrow.”

Evans added that he often sees other men walking down the street or sitting at a bar confidently and wishes he could feel that way as effortlessly as they do.

“When I do, it’s a whole lot of work when I have to present that,” he said on the show. “I’m not massively confident about lots of things about me, but I’ve learned to deal with it, not ignore it and understand why and process it, but it doesn’t go away.”

The PEOPLE Puzzler crossword is here! How quickly can you solve it? Play now!

The star previously gave insight to his intense fitness plan whenever he takes on a role that requires him to have a muscular physique. "When I'm training hard the diet is miserable," he revealed during a 2014 interview with Men's Health U.K.

"I love my food and I like wine and it's really tough," Evans shared. "What I miss most is alcohol. When everyone is having a drink and you can't it stops you going out as much. Booze is a nightmare for empty calories and you can't train the day after. Socializing and enjoying a night out — not doing that is the hard bit."

However, when the opportunity is there, he eases up on intensity.

"It's good for your body to have a break," he told the outlet. "Even when you're training you have to have a cheat day every week. The body reacts better to training if you give it intervals of not training or you relax the diet."

Evans added, "I don't need to be super-ripped all year round. That's a pretty miserable way to live your life."