Little Big Town Recall Wowing Taylor Swift with Their Vocals While Recording 'Better Man' (Exclusive)

Little Big Town knew better than to turn down a song from Taylor Swift. 

Karen Fairchild, Jimi Westbrook, Kimberly Schlapman and Phillip Sweet — the award-winning quartet that makes up the country music band — released song “Better Man” in 2016 as a single off their album The Breaker, and the track found huge success, both among the fans and commercially. 

And, while Fairchild, Westbrook, Schlapman and Sweet are skilled songwriters in their own right, Grammy-winning "Better Man" was penned by none other than Taylor Swift herself. 

"We were all at home, one day off the road. We were out with Luke Bryan on tour. I got an email one night, and it was Taylor Swift, so I was like, "What is this?" I read it, and she was like, "Hey, I have this song. I've had it for a little while. I hear your harmonies on it. If you like it, great. If not, no big deal,’” Sweet tells PEOPLE while discussing the band's Greatest Hits album, dropping Aug. 9.

The next day, Sweet played the song for the rest of the group, who were all “amazed that [Taylor] would send us the song. Then it turned out to be a beautiful match for us,” he says.

Little Big Town and Swift go way back to some of the now pop-icon’s earliest Nashville days, and she’s become a fast friend over the years. 

“We had known her for a long time,” recalls Fairchild, who says that receiving a song from Swift “seems like a slam dunk,” but after being the “only people that had heard her sing that song at the time, we needed to just make it our own.”

That process required “reinterpreting” the tune just “a little bit,” which Fairchild adds “turned out to be quite a challenge because you just have her in your head, and she's such a brilliant writer and performer.”

A question loomed for the band — who had decided in conjunction with Swift not to initially reveal that she had written the tune, as everyone wanted the song to stand on its own merit.

"Could we pull it off and make it feel like it was ours?” Fairchild remembers wondering.

The answer was a resounding yes. 

“It took us a little bit, and then we finally got there. We have such fond memories of bringing her to the studio to let her hear when it was done, and she turned around after she listened to it, and her mouth was open," says Fairchild of Swift. "She was like, 'I love it. I love it.' I love hearing her sing it now. We interpret it different ways, but a great song is a great song.”

LBT joined Swift on her 1989 World Tour in 2015, and Swift has now even performed her own rendition of “Better Man” while on her massively successful Eras Tour.

And, while “Better Man” was an easy shoe-in for the Greatest Hits album, the band does wish they were able to include so many other songs that didn’t make the cut. 

“We wanted to do more, and what people think of as a hit isn't always necessarily what we would deem a hit. There were a few other songs we wanted to include because they were just meaningful, and we could have gone on and on with that. I still long to do a deep cut,” says Fairchild, who’s sentiment is quickly echoed by her bandmates. 

“Me too,” Westbrook says. “The greatest songs you may have never heard!”

“Greatest songs you've never heard, yes,” agrees Schlapman. “Those songs, they're special to us, but they're also special to our fans, to our really die-hard fans. They've meant something to their lives also.”

And, even still, some of the songs that didn’t make it on the album were hugely popular. 

“We left off some things that did have some commercial notoriety, like 'Daughters,'" Fairchild shares. “There just wasn't room on the record according to the label. But favorites? I love "The Breaker" that was on The Breaker record, and I love 'Don't Die Young, Don't Grow Old,' and 'Silver and Gold'.”

“And 'Tumble and Fall,'” suggest both Westbrook and Sweet in unison. 

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

“There's some special ones for us— 'Sugar Coat'— that we could have just kept going,” Fairchild says, “so maybe we'll end up doing a —”

“Part one to part two. Part three?” Westbrook suggests. 

“Part one, part two, yeah,” agrees Fairchild with a laugh. “We can do that.” 

For more on Little Big Town pick up the new issue of PEOPLE, on newsstands everywhere Friday.