Barbra Streisand Says She Almost Replaced Mandy Patinkin in Yentl After He Told Her He Wanted to Have 'an Affair'

Barbra Streisand is recounting a difficult situation on the set of the first movie she ever directed.

In 81-year-old Streisand's new memoir My Name Is Barbra, the EGOT winner writes of her directorial debut, 1983's Yentl. Streisand wrote, directed and starred in the movie, which follows an early 20th-century Ashkenazi Jewish woman named Yentl who pretends to be a man after her father's death in order to receive a religious education.

In the book, Streisand recalls casting actor Mandy Patinkin to star opposite her as a character named Avigdor — and that the actor was the only person who "disturbed my equilibrium" during production on the movie.

" 'I thought we were going to have a more personal relationship,' " she recalls Patinkin, now 70, saying during a confrontational meeting with the actor inside Streisand's dressing room not long after production began. Streisand writes she asked to speak with Patinkin because he was acting angry and agitated on set.

" 'What?' I had no idea what he was talking about," Streisand writes. "[Patinkin says] 'I thought we were going to have an affair.' "

Representatives for Patinkin did not immediately respond to PEOPLE's request for comment.

"I looked at him as if he were crazy . . . 1) I would never have an affair with an actor I was directing, 2) he was married, and 3) I wasn’t at all attracted to him," she recalls. "But I couldn’t tell him he was not exactly fascinating to me. I didn’t want to hurt his feelings, so I simply said, 'I don’t operate that way.' Tears rolled down his cheeks."

Patinkin is married to actor and writer Kathryn Grody, whom he wed in 1980, per The New York Times. Streisand, who was married to Elliott Gould from 1963 to 1971 and has been married to James Brolin since 1998, was dating producer and hairdresser Jon Peters at the time of Yentl's production. She told PEOPLE in 1983 that her initial separation from Peters came while she made the movie and that Peters "told me straight out I’d never make the picture.”

Streisand writes that she told Patinkin that she would replace him in the movie and reshoot his scenes if his behavior did not change. She also described coaching the actor, then just a few years removed from his first film acting credit in 1978's The Big Fix, through a number of scenes to varying degrees of success.

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

Streisand recalls in the chapter that she wanted Patinkin's character Avigdor to serve as a strong co-lead for her character Yentl and for the film's audience. "Mandy was probably irritated. He must have felt as if I were putting him in a straitjacket," she writes. "I realize now, in retrospect, that he wasn’t used to being a leading man. And I was trying to make him into one."

Elsewhere in the chapter, Streisand writes that Patinkin reached out to her years later to ask if she would write a note on the back cover of one of his albums. Streisand declined. "I said, 'Mandy, you put me through hell. . . Why did you give me such a hard time?' "

'Well, I was scared.' 'Really?' " she writes. "I guess people have different ways of reacting to fear. I was scared, too, but it made me quieter. I listened more intently. But it was nice to hear the truth, finally. And I said, 'I didn’t know you were scared, because you hid it so well. And then you were so rude to everybody.'"

Streisand also details how her difficulties working with Patinkin resulted in her decision to change a climactic scene in which Yentl and Avigdor originally spend the night together.

"By this point Mandy had been making my life miserable for months, and I just couldn’t bear the thought of making love with him. I’m not that good an actress," she writes. "So I changed it. I rewrote the scene. And now that I look back on it, I wonder if I allowed my frustration with Mandy to overrule my instincts. Maybe I should have let Yentl . . . and the audience . . . have that moment."

Yentl earned Streisand a best director Golden Globe and an Oscar for Best Original Song Score following its 1983 release.

My Name Is Barbra is out now.

Celebrate Barbra Streisand's incredible career with the new PEOPLE Special Edition Barbra Streisand, available on newsstands and Amazon.com.