'Grey's Anatomy' Book Details Patrick Dempsey's 'Arguing Match' That Led to Isaiah Washington's Firing

The cast and crew of Grey's Anatomy are offering more details on the events that led to Isaiah Washington's exit from the show.

Washington, who played Dr. Preston Burke on the medical drama, was fired from Grey's back in 2007 after using a homophobic slur on set. He denied at the time that the slur was directed at anybody in particular, though many felt the outburst was aimed at costar T.R. Knight's sexuality.

Lynette Rice's new book, How to Save a Life: The Inside Story of Grey's Anatomy, details how it was actually a fight between Washington, 57, and costar Patrick Dempsey that ended in the P-Valley actor making the offensive comment.

"It was my episode," writer Mark Wilding recalls in the book. "I think one of them had been late to set one day and the other one then decided to pay him back by being late himself. Then it sort of exploded. They got into an arguing match, and then before you know it they were physically fighting. I was standing there in video village. I'm, like, six feet four inches. I'm bigger than both of them. But I didn't really jump in right away because I'm like, I don't know if I want to get involved."

"And Isaiah, for whatever reason that day, just took that the wrong way and he went after Patrick," adds Harry Werksman, another writer from the show who was there that day. "I guess he felt disrespected that he and the crew had been waiting. He went after Patrick, pushed him up against the wall, and said, 'You can't talk to me the way you talk to that little f----- T.R.' "

Marty Carrillo says he was on set and heard the argument. "I heard, 'You son of a bitch!' And then the f-word," Carrillo tells Rice, who is also an editor-at-large at Entertainment Weekly.

"Everyone knew T.R. was gay," Carrillo says. "For some reason, Isaiah just lost it and pinned Patrick up against the wall. You could see Isaiah's pupils go wide, like, 'What did I do?' It was quiet. Everyone had to leave the set."

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

Both Washington and Dempsey, who exited Grey's in 2015, also recall the altercation in the tell-all book. "We learned that we [had] to watch what we say," Washington says. "We [had] to make sure we [were] more accountable."

"I think the explosion really healed the show," Dempsey, 55, notes. "No one passed the buck, and everyone owned up to the situation and moved on."

RELATED VIDEO: Sandra Oh Saw a Therapist to 'Stay Grounded' During 'Traumatic' Rise to Fame on Grey's Anatomy

However, the incident came up again at the Golden Globes press room in 2007. When Washington was asked about it, he said: "No, I did not call T.R. a f-----. Never happened, never happened." Washington later apologized for using the slur at the awards show press room.

At the time, fellow Grey's alum Katherine Heigl, who was close friends with Knight, criticized Washington's statement. "I'm going to be really honest right now, he needs to just not speak in public. Period," she told Access Hollywood. "I'm sorry, that did not need to be said, I'm not okay with it."

In How to Save a Life, Heigl, 42, reflects on her comments, sharing, "I didn't have a courageous moment. I had a couple of glasses of champagne, and I was furious and frustrated for my friend and sick of the whole mess of it."

The actress continues, "I was recently talking to T.R. about this and I said, 'I hope I didn't embarrass you and draw more attention to something that you just wanted to go away.' And he said I could never have embarrassed him and that he was so grateful because no one had ever stood up for him that way before. So that is a proud moment for me. I don't regret it. And I'm just grateful that my friend doesn't regret it."

Washington also says he thought the issue was behind him before learning of his eventual firing that year.

"I did everything that the producers and the network asked me to do," he says. "I came back under great stress, and thought I was doing the job I was hired to do. I thought that was going to speak for my future at Grey's, but apparently that wasn't the same vision that the network and studio had for me."