Why Armie Hammer Won't Be Spending Thanksgiving with Kids and Estranged Wife Elizabeth Chambers

Armie Hammer won't be spending Thanksgiving with his kids.

The actor's estranged wife Elizabeth Chambers is currently staying in the Cayman Islands with their kids — daughter Harper Grace, who turns 6 on Dec. 1, and son Ford Douglas Armand, 3½ — where she and Hammer quarantined at the start of the pandemic. Hammer, 34, returned to Los Angeles after the two announced their separation in July.

Last month, Hammer filed for joint custody of his children and requested that Chambers, 38, return to the U.S. to arrange a custody schedule.

A source tells PEOPLE that the Call Me By Your Name actor "was supposed to fly to the Cayman Islands, quarantine and then spend Thanksgiving with the kids and his family, flying back to L.A. all together in mid December." Hammer, however, "has since decided to back out."

A separate source says the family was originally expected to have returned to their home in L.A. by now.

That source insists Hammer "never willingly abandoned his plan to go to the Cayman Islands, he was unable to travel there after repeated attempts." "When air travel reopened, he applied for a visa that has not been approved, " the source says.

Reps for Hammer and Chambers did not respond to PEOPLE's requests for comment.

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

On Wednesday, Chambers shared photos of herself teaching a baking class to young students ahead of Thanksgiving, as well as a photo of Harper jumping into a pool.

"From school to pool in 8 minutes," she wrote. "Grand Cayman obsessed with you and your commute."

In legal documents obtained by The Blast last month, Hammer's lawyers said the former couple "elected to stay with their children temporarily in the Cayman Islands where the pandemic appeared to be under greater control." When the actor returned to L.A. in July, though, "Elizabeth and the children remain in the Cayman Islands and Armie has not seen the children in several months now."

A source tells PEOPLE: "After filing the injunction, Armie backed off so the kids could still attend school in person in the Grand Caymans and not be in front of screens. Everyone agreed that's what's best, especially with cases getting worse in the U.S."

Chambers filed for divorce in July, seeking primary physical custody and joint legal custody of their two children. A source previously told PEOPLE, "These are crazy times with COVID-19, shutting schools down in L.A. and Elizabeth just wants the kids to have a normal experience."

Harper and Ford have enrolled at the same school Hammer attended when he was younger. (The actor's family moved to the Cayman Islands when he was 7 and they lived there for five years.)

RELATED VIDEO: Armie Hammer Files for Joint Custody & Requests Estranged Wife Elizabeth Chambers Return to the U.S.

RELATED GALLERY: All the Celebrity Couples Who've Called It Quits in 2020

In September, the actor told British GQ his separation from Chambers, whom he married in 2010, was a "seriously seismic event."

"I don't think you'll find anyone in the world who would ever say what I’m going through is an easy thing to go through," he told the outlet. "It’s not about whether it was your idea or not, or whether you think it’s the right idea or you don’t."

Hammer said when he and Chambers discussed their separation, they kept their focus on what was best for their kids.

"Elizabeth and I are two adults making this decision and the priority is how do we do it in a way that affects our children the least? Or, at least, how do we help get them through this in a way that mitigates as much consternation or fear or damage on their part," he said at the time.

That same month the father of two was seen having lunch with writer Jessica Ciencin Henriquez months after they both split from their respective partners.

The two were seen sitting close while chatting during the meal, with Hammer photographed leaning in to talk to Ciencin Henriquez. Hammer was later seen out walking arm-in-arm with Rumer Willis.