Drew Barrymore and Hilary Duff Talk About Having a 'Healthy Relationship with Rejection Because of Our Job'

Drew Barrymore and Hilary Duff know a thing or two about rejection from their days as child stars.

The two stars bonded over losing out on several roles they auditioned for and the lessons they learned from it when they sat down to chat on Monday’s episode of The Drew Barrymore Show.

“There's been so many times when I felt like a loser or I didn't get the job or I've been sitting stagnant,” the How I Met Your Father star, 36, recalled. “But I had to fill my life with things that were real and that was enough at that moment.”

Barrymore, 48, said of her own shared experience: “I had such a bizarre relationship with rejection so early in my life that I appreciated what it gave to me, which was thinking there was enough room for everyone and that being competitive was unnecessary because there really is enough room for everybody.”

“Still a hard one sometimes,” Duff said of Barrymore’s point about competitiveness with a laugh.

“And people have a competitive spirit. That's a healthy thing. I get that,” the talk show host continued. “But it is weird to be a kid that you find out it's not going to be you and you're like, ‘OK, I have to figure out how I function through that.’”

Duff added, “When people ask you, ‘What job did you not get?’ I'm like, ‘There's been so many, I actually can't remember. I can't tell you. It wasn't worth holding on to it. So I dumped it back there somewhere.’”

Echoing her sentiment, Barrymore shared, “I won't ever admit it because I don't want the person who ended up doing it to feel weird.”

“You’re like, ‘That was mine’ and you’re still stewing on it” the Lizzie McGuire alum jokingly interjected.

“Yes, exactly! I’m like I won’t tell you!” Barrymore exclaimed as they both laughed. “But I’ll admit like things I auditioned for and then I guess they might find out that way.”

“I think the rejection thing is kind of coming full circle to me right now because I don't think I've quite processed how much rejection you have to deal with as a child actor, or just even in this business in general,” Duff said.

The actress then reflected on how her son Luca, 11, is learning the same lesson she did at that age.

“My son, he's doing club sports, and you have to continue to basically audition. It's not called 'audition' in sports, but that's my language,” Duff explained. “I'm like, ‘Are you ready for rehearsal? OK, it’s practice, got it. ‘You got your costume on? Oh, it's your uniform. Yeah, yeah, yeah.’”

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

“He's so annoyed,” added Duff, who shares Luca with ex Mike Comrie. She is also mom to daughters Mae, 2, and Banks, 5, with husband Matthew Koma.

She recalled an exchange with her son: “But he has to continue to make the team every season and he's always like, ‘What if I don't get it? What if I don’t get it?’ and I'm like, ‘You don't know how many auditions I didn't get. You don't know how many times I got told no. Like, we gotta just drop it. If it doesn't work out, we move on.’”

“I think I do have a healthy relationship with rejection because of our job,” noted Barrymore. “Or we're just beaten down,” her guest hilariously countered.

Barrymore then dramatically poked fun at the downside of their chosen careers, “You’re overstuffed, you’re ready to quit, you're beaten down, you feel liberated.”

“Turning pain into strength,” Barrymore celebrated as she high-fived Duff.