The Crown Cast Has a Behind-the-Scenes Chat About Making Season 4

The cast members of The Crown are giving their characters the royal send-off.

In a behind-the-scenes look at the Netflix series provided to PEOPLE, Olivia Colman, Tobias Menzies, Helena Bonham Carter, Erin Doherty, Josh O'Connor and Charles Dance look back at the making of The Crown's fourth season — including welcoming "newbies" such as Emma Corrin and Gillian Anderson into the cast.

"For us, it was really exciting to see Emma come in — and Gillian," Bonham Carter, 54, who plays Princess Margaret, says. "It was like, 'We're the six formers. Hee-hee.' "

While Anderson says she "couldn't tell anyone" that she was cast in the series as Margaret Thatcher, the actress nonetheless had a blast portraying the British prime minister and the "complexity" of her relationship with Queen Elizabeth (Colman, 46).

"We're two characters ,and the friction and the rhythm is so enjoyable to play," Colman says of their dynamic. "It was just really fun to have those 'horrible moments' with brilliant Gillian."

For the cast, one of their most memorable moments was when they all gathered to film a scene in which the royal family plays a game with Thatcher. According to Doherty, who plays Princess Anne, it "was so much fun" to let their hair down for a moment — even if there was so much tension between their characters.

"Just the tension that lies even in the idea on the page is so fantastic," Anderson, 52, says. "That was loads of fun."

Can't get enough of PEOPLE's Royals coverage? Sign up for our free Royals newsletter to get the latest updates on Kate Middleton, Meghan Markle and more!

However, season 4 is a bittersweet one for some cast members as they bid farewell to The Crown and hand off their roles to another successor.

"Having lived with her for two years, I felt tremendous sadness saying goodbye to her because she's been great company," Bonham Carter says of her role as The Queen's sister. "As an actor, you do feel like you're a guesthouse and people move in for a bit and then they move out. Sometimes Margaret has popped out when I'm not on camera."

"You spend so much time in these people's shoes that actually you do tend to take on their traits," Doherty, 28, admits. "It's just been a pleasure. It's been a bloody pleasure."

As for O'Connor, 30, he says he's "happy to pass on the baton" of Prince Charles to another actor.

"One of the things that attracted me to the role in the first place was that I knew it was going to end," he says. "I was always going to have two series and then some other lucky fellow was going to get the chance to take him on."

Season 4 of The Crown premieres on Netflix on Sunday.