Freddie Prinze Jr. Says He and Sarah Michelle Gellar Are 'Deliberate' in Avoiding Spotlight: 'We Don't Like That Life'

Freddie Prinze Jr. is opening up about life with wife Sarah Michelle Gellar, and why they've kept their romance out of the public eye through their more than two decades together.

The actor reunited with his Freddie costar Brian Austin Green for an appearance on Green's Oldish podcast published on Monday, Nov. 4. During his chat with Green and co-hosts Sharna Burgess and Randy Spelling, Prinze, 48, said he and Gellar, 47, have remained out of the spotlight "by choice."

"Something that I've admired about your relationship is you both have seemed to stay out of the public," said Spelling, to which Prinze replied, "That's by choice, though. You can control it to a certain degree."

After Spelling, 46, said it "seems like it's very hard for a lot of people" to have that kind of control, he asked the She's All That star, "How intentional do you have to be to do that?"

"I don't know how deliberate [others] are in trying to avoid that BS," Prinze said. "I know how easy it's been for us because I don't go anywhere. I don't go to parties. People don't see Sarah and I out and about at the popular places. We have our restaurants that we like, and they're not super trendy in places where paparazzi hang out. So we're good at avoiding it."

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human-interest stories.

Prinze noted that he's "not trying to s--- on anyone," but said he feels like avoiding the attention is "not that hard ... if you really" want to.

"There's times where [paparazzi] wait at the bottom of our street, but there's people that are way more famous than me on my street," he said. "So I'll just make sure they leave first. They'll go follow Schwarzenegger and leave me alone. You know what I mean?"

For the I Know What You Did Last Summer star, "there's no secret to it, and it it wasn't hard," he said. "We just don't like that life. You know?"

He also pointed out how his wife "grew up in New York" and "experienced everything before she was 21 years old, like nightclubs and all that stuff."

"And I can count on one hand how many clubs I've been [to] my whole life. One was with you," Prinze told Green, 51. "They had live music though, it was cool."

The PEOPLE Puzzler crossword is here! How quickly can you solve it? Play now!

Prinze and Gellar, who first met on the set of their 1997 slasher I Know What You Did Last Summer, tied the knot in September 2002, and went on to welcome two children together: daughter Charlotte, 15, and son Rocky, 12.

And their relationship dynamic has clearly worked in favor of its longevity, as the actors recently celebrated 22 years of marriage.

Alongside a smiley and cuddly photo of the couple — in which Prinze wrapped his arms around Gellar’s waist — the Buffy the Vampire Slayer alum penned a simple caption.

"22 @realfreddieprinze,” Gellar wrote, adding a few wedding-themed emojis: “🥂💍👰🏼.”