Val Kilmer Reveals His Breakup from Daryl Hannah Was 'By Far the Most Painful'

He’s dated Cindy Crawford, Angelina Jolie and Cher, but Val Kilmer says his breakup from Daryl Hannah was by far the most difficult.

Hannah, 59, and Kilmer dated in 2001 after working together on the film In God We Trust. She would later go on to marry Neil Young in 2018

In his new memoir, I’m Your Huckleberry, Kilmer, 60, fondly recalls the time the two spent together in New Mexico and says the split was devastating for him.

“Lord knows I’ve suffered heartache. But Daryl was by far the most painful of all. (Neil Young, I always loved you, but I’m afraid I hate you now.),” writes Kilmer.

For more on Val Kilmer, pick up this week’s issue of PEOPLE on newsstands Friday.

In his new memoir, he also opens up for the first time about his battle with throat cancer. Kilmer was diagnosed with the illness in 2015 and says Cher, whom he dated in the early 1980s, helped get him through some of his darkest days.

“Once Cher works her way inside your head and heart, she never leaves. For her true friends, her steadfast love and loyalty never die,” he writes.

Kilmer underwent a tracheotomy, chemotherapy and radiation and also turned to his Christian Science faith and prayed for healing. Today, he is thankful to be cancer free.

“I have been healed of cancer for over four years now, and there has never been any reoccurrence. I am so grateful,” he says.

In his new book, he credits Crawford for helping him find happiness after his divorce from ex-wife Joanne Whalley.

He also says Jolie was “perhaps the most soulful and serious of them all.” “When people ask what she’s like, I say she’s like other women and other superstars, just MORE,” he writes of his ex.

I’m Your Huckleberry is available for purchase on April 21.