Peter Gallagher Looks Back on 'Powerful' First Meeting with Wife Paula Harwood and Why It Took '7 Years' to Ask Her Out

Peter Gallagher had to work up the courage to ask out his now-wife Paula Harwood.

On The New York Times’ Modern Love podcast, the actor, 68, reflected on his 41-year marriage with Harwood. He first encountered her in the stairwell when they were undergraduates at Tufts University, and he said that despite his faulty memory, he “remembers almost every moment” of the meeting.

“No words were exchanged,” Gallagher reminisced. “And we just sort of, you know, glided by each other.”

While Harwood’s hair was done up and she wore “this tight turtleneck thing and a disco belt and tight corduroy bell bottoms and platform shoes,” The O.C. star remarked that he looked like he “had just left 8th-grade gym class.”

“It was just powerful,” Gallagher said of their first run-in. “And I spent the next seven years trying to figure out how I could go out with her.”

He admitted that after seeing her, he looked her up in the freshman yearbook and considered her “unattainable.” Still, he made an effort to show up in the cafeteria to chat her up during lunch and “make her laugh.”

“Frankly, that’s still a huge component of our relationship,” the Law and Order: SVU actor added.

Although they eventually got together, their relationship wasn’t without its bumps. During his appearance on the podcast, Gallagher also spoke about his mother urging him to return to Harwood and apologize after a brief breakup.

He shared how his mother said, “She is a wonderful young woman and any problem you're going to have with her you're going to be having with anyone because they're your problems. So you be damn sure you figure out what's going on with you before you start changing partners like socks.”

The PEOPLE Puzzler crossword is here! How quickly can you solve it? Play now!

In honor of their milestone 40th anniversary last year, the Grey’s Anatomy actor shared then-and-now photos of the pair on Instagram.

“Here we are at the church, on the same day with the same sunshine and blue skies that we were married in exactly 40 years ago. I'm so glad we did,” captioned Gallagher, who shares daughter Kathryn, 30, and son James, 27, with Harwood.

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

Harwood also posted a tribute to their enduring marriage on Instagram, writing alongside the same photos, “40 years ago today, May 7, 1983 , these two babies got married in the church we are standing in front of right now. We are two very lucky, grateful people."