Hilary Duff Says She Hangs Out with Ex Joel Madden and Their Spouses 'All the Time': 'It's Lovely'

Hilary Duff is keeping it friendly with at least one ex.

The How I Met Your Father star, 35, revealed on Watch What Happens Live that she and ex-boyfriend Joel Madden have long been good friends, after "the whole world's mind was blown for a hot second" at her and husband Matthew Koma's dinner with Madden and his wife Nicole Richie.

"By the way, I have a good relationship with their whole family. I think people were [blown away]. I don't know. We're neighbors. And we actually hang out all the time," she explained.

"I was just in her driveway, honking for her to come drink wine with us the other day," Duff added of Richie, 41, whom Madden, 43, married in 2010. "She wouldn't, she was asleep by 8."

She said now that they're neighbors, "We get them to hike with all the time," adding: "And you know what? It's lovely."

Duff previously posed for a photo last February during "date night" with Koma, 35; as well as Madden and Richie, and Pete Wentz and partner Meagan Camper. They were also joined by Pulse Music Group founder Josh Abraham and wife Gina.

"My husband is a giant troll, it's one of my favorite things about him," Duff noted. "Right when they moved in, for some reason Matt Photoshopped a picture of Joel and [brother Benji Madden] in a photo with me on Valentine's Day with like, 'Happy Valentine's Day, Honey.' "

"So, this was before we started hanging out, and I think Joel and Nicole were like, 'Oh my God, they don't like us and we just moved in here next to them.' But no, they just know that Matt is a big troll and likes to have lots of fun," said Duff.

The Disney Channel alum and the Good Charlotte frontman began dating in 2004. After working together on her 2005 compilation album Most Wanted, they called it quits in November 2006.

RELATED VIDEO: Hilary Duff on Life as Mom of 3 After Welcoming Baby Mae with Matthew Koma: "I Love This Mayhem"

Duff tied the knot with Koma in December 2019, and they share daughters Banks Violet, 4½, and 10-month-old Mae James. She also co-parents son Luca Cruz, 10½, with retired NHL ex-husband Mike Comrie, to whom she was married from 2010 to 2016.

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

Madden and Richie got married in December 2010, and they share daughter Harlow, 15, and 13½-year-old son Sparrow.