Matt James Says Everything He Went Through with Rachael Kirkconnell 'Prepared Us for Our Future'

Matt James never actually thought he'd be the Bachelor.

"I thought it was a joke initially," the entrepreneur, who has been dating his pick Rachael Kirkconnell for over a year, tells PEOPLE. "But I've learned a lot. And at the end of day, everything that we've been through has prepared us for the future."

Indeed, James, 30, whose new memoir, First Impressions, is on stands now, has certainly had his own ups and downs following his stint on his Bachelor season last year.

After giving Kirkconnell, 25, his final rose, James left The Bachelor bubble to discover racially insensitive photos of Kirkconnell at a 2018 antebellum plantation-themed fraternity formal resurfaced online.

"It was crazy," James recalls. "It felt like the world was against us. Some people felt like the situation was blown out of proportion and others felt like it hadn't gotten enough attention. We were just trying to enjoy each other and grow our relationship without having to deal with people we didn't even know weighing in.

Kirkconnell issued an apology but the couple eventually announced they had split during The Bachelor's After the Final Rose special. One month later, James told PEOPLE he was "pursuing" a relationship with Kirkconnell again.

"The best thing for us was to remove ourselves from the show, from everything," he now tells PEOPLE.

"We still just try to operate in our own lane. People are moving in, others are engaged... that stuff doesn't really matter to us. What matters is having a solid foundation and being able to communicate and care for each other.

RELATED VIDEO: Bachelor's Rachael Kirkconnell Urges People Defending Her Past Racist Actions

"That's what we're going to continue to build. Just because we met on a TV show doesn't mean our relationship needs a TV timeline."

That's not to say things aren't moving forward as Kirkconnell is looking for places to live in James' Miami hometown.

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

"Rachael has all the qualities I want and it's nice to find that," says James. "She's got a good sense of humor and we don't take each other too seriously."

Of the future, "We're not worried about pleasing everyone," says James. "It's not about checking boxes. it's about finding someone you're compatible with and focusing on that first."