Gossip Girl's Kelly Rutherford Custody Battle with Daniel Giersch

Kelly Rutherford‘s two children they live an ocean apart from her, with a father who she says has told her they won’t set foot on American soil again.

It’s a battle the former Gossip Girl actress and her ex-husband, German businessman Daniel Giersch, have been fighting since before their daughter was even born – and one that Rutherford insists she never sought to wage in the first place.

“I’ve always wanted my kids to have a relationship with him, and I’ve really only encouraged that all along. When I went into court, I never asked for money, and I never asked for full custody,” Rutherford tells PEOPLE. “I just wanted what was right for them and their well-being moving forward. And what happened over the next six years was the most surreal, bizarre, unimaginable thing.”

This week, a federal court denied the mother’s request for her son Hermes, 8, and daughter Helena, 5, to move back home to New York. The kids have been living with Giersch, 40, in France ever since 2012, when a California family court judge ruled that because his work visa had been revoked, barring him from returning to the United States, it was in the children’s best interest for them to reside with him in Monaco, with Rutherford, 46, flying out to visit them.

Rutherford says she offered to bring the children to her ex-husband whenever he wanted or meet in a country they could both enter, but the judge was not swayed. Giersch was ordered to reapply for a visa, but according to the State Department, he has yet to seek a work or visitor’s one.

“There has never been a case like this in the history of this nation,” attorney Wendy Murphy, who is representing the children on behalf of Rutherford pro bono, tells PEOPLE. “There has never been a case of a family court judge issuing an order commanding American citizen children to live in a foreign country. And our concern is that if we don’t prevail, this will set a very dangerous precedent.”

“If this could happen to Kelly’s children, what would stop any judge from ordering any American born child to move to Iraq or Somalia?” she adds. “If a judge has discretion to do this, then the next step will be a different judge in another courtroom commanding American children to get on a plane and move to Afghanistan. There’s no prohibition on that under this court’s analysis, which I think is frightening.”

The travel costs and legal fees forced Rutherford to file for bankruptcy. She says Giersch does not help finance her trips – a friend paid for her plane ticket during her most recent visit, a 10-day trip last week – and she stays at other people’s houses or in an apartment while in France, meeting her children at the airport.

“He won’t let me in his home,” she says of Giersch. “He told me he’s not even letting them come back to the country anymore. … I think that this was probably his plan all along.”

Giersch is not commenting on the case or Rutherford’s claims.

“Daniel Giersch continues to protect the children from any negativity, and therefore will continue to not engage in any of these unfortunate and false media fabrications which only serve one person but clearly not the children,” his lawyer said in a statement to PEOPLE.

A Bitter Breakup

Rutherford and Giersch wed in 2007, welcoming Hermes the next year.

So, what went wrong?

“Obviously I was in love with him and things were good in the beginning, and then, just weird things started happening. People are not always who they say they are, and that’s okay, but that doesn’t mean you want to continue,” she explains. “The way we were living our lives were not aligned, and I was a bit scared, in all honesty, and I wanted to separate and be friends because of certain things that were going on.

“I was scared to leave for a while, too … so it took a lot for me to even leave. Even though it was a relatively short marriage, there’s a lot that happened within that short period of time.”

By 2009, the union was over: She filed for divorce and legal separation from the Giersch in the same day. She was three months pregnant at the time.

The split quickly turned sour: She moved to bar him from leaving the U.S. with their son and asked for a restraining order after claiming he threatened her. He tried to keep her from bringing Hermes back home to N.Y.C. from Los Angeles when Gossip Girl filming resumed. She hired a private detective. He claimed he wasn’t notified of Helena’s birth in June 2009 and that she refused to put his name on the little girl’s birth certificate.

And then, there’s the bombshell accusation that she’s the reason his visa was revoked in the first place.

“Kelly stated on the record that Daniel was dealing drugs and weapons in South America, which under the Victory Act is considered terrorism,” a source told PEOPLE in 2012, adding the claims are all untrue.

But the star staunchly denies having anything to do with it.

“I’ve never spoken to anyone about him in that way. I would certainly not even know who to call. I think that a lot of this has been set up this way … to make things look a certain way,” she says. “I’m saddened by that, because it’s just not true. I signed an affidavit saying it’s not true. I’ve been asked to write letters to help him get back [to the United States], and I’ve done that. I’ve done everything I can to help him.

“Look, if I had had the power to get him sent out of the country, I certainly would have had the power to keep my children in this country.”

Parenting from Afar

Helena and Hermes were last in America when they spent Christmas with their mom. She plans to fly out to France again in the next few weeks, and they talk regularly by video-chatting.

She’s in awe of how quickly they’ve grown up.

“I don’t know how it happened so fast, they’re like teenagers,” she says, laughing. “I’m like, ‘Wait a minute, you guys.’ It’s incredible how articulate they are and kind and loving. They’re just amazing people, and I’m really proud to be their mom, and I’m proud of how they’ve handled all of this. I tell them that.”

Even after all the drama, Rutherford doesn’t regret not seeking full custody initially, saying her dream is for her and Giersch to both raise their children in a more “fair” arrangement.

And though their goodbyes are always tearful, and her daughter has cried over Skype, she does her best to protect Helena and Hermes from all the ugliness, encouraging them to enjoy their new lives abroad.

“I just want them to look back and be able to say that they had a happy childhood,” she says. “So my goal is to say, ‘Hey, enjoy this time with your dad, you know I love you,’ and even though there’s things being said about me, I do my best to say, ‘Look, listen to your own heart. No matter what anyone says, just know that I love you and that I’m doing the best I can, and that I’ll never stop fighting for you.’ ”

What’s Next?

People ask Rutherford why she doesn’t just pack up her life and move to France to be closer to her children.

“How am I going to fight for my kids’ constitutional rights as U.S. citizens if I’m now also an exile in a foreign country, away from my family, my friends, my ability to work and make a living?” she asks. “I was put in a very strange predicament, to be bankrupted and have your kids sent to a foreign country. It doesn’t make sense, how it was set up, and it was not at the advantage of a mother having the ability to see her kids, if that was the goal, even 50/50.”

On top of that, a court in Monaco recently granted Giersch and the kids permanent residency, meaning there’s nothing preventing the millionaire from refusing to send them to the Rutherford for more visits, according to ABC legal analyst Dan Abrams.

As for the latest verdict, Rutherford was crushed and “disappointed” but says after such a long fight, “I didn’t have huge hopes, either.”

Murphy says the three-judge panel at the Second Circuit Court of Appeals was “clearly intrigued, concerned” and “perplexed, even” by Rutherford’s plight but ultimately decided it was a family court matter, not a federal one.

Murphy believes that by ordering the children to live in France, the California judge exceeded her authority – and that the current arrangement infringes on the children’s constitutional rights. She pointed to the 1967 Supreme Court case Afroyim v. Rusk, which guarantees that no American can de deprived of citizenship unless he or she willingly renounces it.

The ruling set the standard that even criminals can’t be deported to another country if they were born in here.

“My argument to the court was, if a serial killer or a terrorist, because that person happens to be a citizen, is protected and cannot be forced into exile,” Murphy says, “how on earth can two utterly innocent and defenseless American children be shipped like luggage to live in a foreign country.”

Her next step is seeking a re-hearing with the three judges and filing another appeal to a larger panel. If that fails, she’ll take Rutherford’s case to the Supreme Court.

Meanwhile, Rutherford is fighting in the court of public opinion, working with her Children’s Justice Campaign to advocate for other parents in difficult custody situations. She’s raising funds for the organization on Crowdrise and Represent.com, and she’s spreading awareness by writing a book about her ordeal.

Why has the battle been so bitter? Rutherford struggles to answer the question. All she ever wanted is shared custody, she says. But she points out that Giersch is a successful entrepreneur who has gone toe-to-toe with companies as big as Google, which he fought in 2005 over the Gmail trademark in Germany.

“Everyone of course in the beginning is going to be maybe hurt or maybe angry or disappointed, but for so many years and so long?” she says of their divorce. “Maybe he just saw this as a lawsuit versus a mother and children.”