Luke Grimes Says Not Having Kevin Costner on Yellowstone Set 'Felt Different': 'He's a Big Presence' (Exclusive)

For Luke Grimes, the gravity of Kevin Costner's departure from his leading role in Yellowstone was not lost while filming the show's concluding episodes without him.

The actor and musician, 40, tells PEOPLE that Costner's decision to not return after nearly eight years working together instilled "a finality to it."

"Him not coming back felt like, 'Okay, we're going to have to land this plane now for real,'" Grimes shares. "I think the patriarch leaving was always going to be part of the story. That's always where it was headed is like, what do these kids do? What does this family do when their rock is gone?"

Although he acknowledges the "drama" that predated Yellowstone's return to filming — including scheduling conflicts between Costner's Horizon: An American Saga and Yellowstone that led to the Academy Award winner bowing out of his role on the show in June — Grimes adds that he felt as though "it was always going to go that way."

"Whatever the circumstances may be, [it] definitely felt like, 'Okay, we're coming in for a landing,'" he says of Costner's absence. "It definitely felt different."

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories. 

"He's a big presence, man, guy's an icon," the actor adds. "So not having him around to set felt a little different. But as always, I mean, we're a big family now. It was just as special and hopefully, the fans will be able to connect with the story just as much ... I think they will be able to."

On the topic of family, Grimes highlights that although "Kayce as a character feels a little bit like an outsider in his own family ... I think that the thing that kind of put him in between a rock and a hard place was that deep down he felt the loyalty, even though he knew maybe it wasn't the best thing for him."

He notes the dynamic was "kind of fun to play with" over the years.

For more on Yellowstone, pick up the latest issue of PEOPLE.

With Costner's departure, fans are wondering what will come next for America's favorite Montana ranchers. Grimes suggests "there's always rumblings."

"I think when anything is this successful, there's a pressure to keep it going because they know they have an audience and that's what they're looking for," he says. "But it all comes down to Taylor [Sheridan's] vision. He has to see it to be able to write it, and I don't think he's going to write something that he doesn't want to. Basically we're just going to see if Taylor has an idea that he gets excited about, and that's going to make all the difference here."

Yellowstone returns on Sunday, November 10 at 8 p.m. ET/PT on the Paramount Network. All five seasons are streaming on Peacock and Paramount+.