Jimmie Allen Says He's 'Cool' with Wife Not Wanting More Kids, But 'Whatever Happens, Happens'

Jimmie Allen is supporting wife Alexis' plans to remain a family of five.

On Tuesday's episode of PEOPLE's podcast PEOPLE Every Day, the country artist, 36, clarified comments that he made on the ACM Awards red carpet about "retiring" from having more kids.

"It was actually her idea," Jimmie says of staying put with three kids. "Cause when it comes to babies, you know guys, we got the fun part ... women gotta carry the baby. Give birth to the baby. And 'cause she's told [people] before she doesn't mind giving birth, it's the pregnancy part that's normally gonna be hard on her."

"But I'm cool if she's done. My jerseys are retired, but then we did talk about maybe in a couple years having another one, but we'll kind of just see. Whatever happens, happens," adds Jimmie.

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

Jimmie, who co-hosted the ACM Awards alongside Gabby Barrett and Dolly Parton, is dad to daughters Zara James, 4 months, and Naomi Bettie, 2, with wife Alexis Allen plus son Aadyn, 7, from a previous relationship.

Speaking to Entertainment Tonight's Cassie DiLaura on the red carpet Monday, the singer said he and his wife "feel outnumbered" by their kids.

"Going from two to three is a big jump. It's cool, we love 'em and we're just definitely thankful for our kids," he said.

Asked about the possibility of having another baby, Jimmie replied without hesitation, "Oh, I am done."

The "Best Shot" artist and his wife first announced that they were expecting their second child together last June.

In August, the couple spoke with PEOPLE for its family issue, where they shared their excitement for their baby girl on the way. "That's like nine more years of Disney princesses!" Jimmie said. "I'm obsessed with Disney World. I've always wanted to go inside a Bibbidi Bobbidi Boutique."