Comedian Jim Gaffigan Says He Keeps His ‘Fat Clothes,’ Jokes That They’re ‘Haunting’ Him After 50-Lb. Weight Loss

Jim Gaffigan is trying to adjust to his recent weight loss.

In an interview with Men’s Health, published Nov. 19, the comedian, 58, discussed his weight loss journey and new stand-up special, The Skinny. He explained why he mentioned in the Hulu special that since dropping 50 lbs., he’s still inclined to keep his “fat clothes.”

“Maybe I’m a pack rat, but I’m like, ‘I can’t get rid of that.’ And then there is something about the haunting,” he told the outlet. “They’re like, ‘You’re going to wear me again.’ ”

In his special, Gaffigan quipped, “I’m trying to enjoy being thin because I know I’m going to be fat again. This is not sustainable.” The star said he’s gone down two pant sizes so far, and pointed out that many of his jokes were often centered around his weight.

“People used to always say, ‘If you lost weight, you’d lose your whole act,’ ” he said. “It’s so strange, because in some ways, we’re not the funny big guy, are we? That’s the role that I’m supposed to play. But also in some ways, that may be the avenue of our warmth.”

Gaffigan slimmed down using Mounjaro, the brand name for tirzepatide, which works by reducing appetite and improving how the body breaks down sugar and fat. The medication — taken by injection in the thigh, stomach or arm — is FDA-approved to treat Type 2 diabetes and helps people with weight loss.

“Most of our lives we’ve been told, ‘Just control your appetite. Just stop being a f------ pig.’ The reality is, we can’t. Maybe our brains are a little wonky or whatever. I think that in some ways, the appetite suppressants are this justification for people who have compulsive eating, that they’re not weak—that some of it can be fixed,” he said.

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories. 

In addition to taking Mounjaro, Gaffigan also tried a low-carb diet and intermittent fasting and found a new hobby in growing his own vegetables.

“Growing vegetables makes you more interested in consuming those vegetables, which makes you healthier, which makes you kind of look at what you’re consuming, which makes fast food look like the crap that it is,” Gaffigan told the outlet.

Earlier this month, Gaffigan spoke to PEOPLE about his weight loss journey and said he's not trying to lose more weight and has yet to decide how long he will continue taking Mounjaro. He stays active by walking around New York City, where he lives with his wife, Jeannie, and their five children. 

The Skinny premieres Nov. 22 on Hulu.