Oliver Hudson Felt 'Angry' as a Kid When Fans Would Ask Mom Goldie Hawn for Autographs: 'I Hated It'

Oliver Hudson is recalling how his mom Goldie Hawn's fame affected him as a child.

On his Sibling Revelry podcast, which he co-hosts with sister Kate Hudson, Oliver, 48, recounted feeling angry growing up when fans would approach Hawn, 78, while they were out together in public.

"I reflect on these feelings that I used to have when I was a young son and sitting with mom and people coming up and wanting her attention, and it was detracting from her energy towards me, and I hated it," he said, as the pair spoke with guest Cindy Crawford about growing up in a famous family.

"It would make me angry when people would come up to the table and want autographs and interrupt what we were doing, and it was this invasion for me as a young boy," he added.

"You know what I mean? Who needed his mother, no doubt about that as well. So it was definitely a negative experience on that side for me, whereas Kate was like, 'Bring it on.' " 

Oliver's younger sister Kate, 45, recounted that experiencing a parent's fame as a child can make someone "more of an observant observer when you grow up that way."

"It's like you really see what's happening around you when sometimes the parent that is in the center of the attention is trying to ignore it," she said, noting that Hawn more often dealt with fans approaching her in public when they were kids, as opposed to modern-day paparazzi.

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories. 

"The weird thing is that when we were kids, it wasn't necessarily paparazzi as much as it was fans," she said. "What happened with me when I became famous ... it was sort of this new world of internet paparazzi, like, all of those sites. So, like, I would go to a restaurant, and there would be, like, 20 papos outside, and then I'd get cars chasing me. I didn't grow up like that."

"Back in the day, it was just the airport. You're, like, walking out of the gate," Oliver added. "All of a sudden, like, 80 paparazzi come out of nowhere ... that was the '80s."

Oliver and Kate frequently discuss their famous family on their podcast; Oliver previously made news in the spring when he shared that he surprisingly found himself unpacking many emotions related to his childhood and Hawn during a week-long course at the non-profit Hoffman Institute.

He clarified in a follow-up episode that he felt his words were "completely taken out of context" and that he was simply speaking from his experiences as a "5, 6-year-old person." 

"I don't know who I would be without my mother," he said on the podcast's April 1 episode. "I can't even fathom it."