Kane Brown Opens Up About Being Called the N-Word as a Kid: 'I Thought I Was Full White'

Breakout star Kane Brown is more than proud to bring added diversity to the country music scene. But long before rising to fame with his YouTube covers and becoming a game changer in the music industry, he came face to face with prejudice as a kid.

Born to a white mom, Tabatha, and a part Cherokee, part black dad who was out of his life, Brown admits he was once confused about his racial identity.

"I'm biracial; I didn't know that until I was 7 or 8 years old," the country newcomer, 25, tells PEOPLE in this week's issue, opening up about his amazing journey to Nashville stardom. "I thought I was full white, which honestly, I can't even really say because I didn't see colors."

So it came as a shock when Brown's peers started making fun of him for his skin color while he attended middle school in Georgia.

"I found out that I was biracial and I still wasn't thinking anything of it, but then I started getting called the N-word," he says. "I didn't even know what it meant. I learned what it meant, and that's when it started affecting me. I got in fights over it when I was little."

Some thick skin helped Brown move past the name calling. "I just kind of got over it," says Brown, who has since channeled his troubles into his music. "They just made me stronger. I guess it was God. Hopefully I can help kids and they can end up being stronger in the long run, too."

RELATED VIDEO: Why Kane Brown's Wife Katelyn Chose a 'Sleek and Simple' Wedding Dress

A growth spurt also helped keep the bullies away.

"Once I got to high school and hit my growth spurt, nobody really messed with me anymore," the "What Ifs" singer reveals. "I just kept to myself. I kept my circle small. I had my friends that I knew wouldn't do anything to hurt me and I would also stand up for them, and those are still my friends today."

Unfortunately, racism reared its head again when Brown began breaking into country music.

"When I first got into country, I started getting some of those comments like, 'He's an N-word.' Stuff like that," admits Brown, who married music management student and fellow singer Katelyn Jae in October. "I used to screenshot it and put it on Twitter, like, 'There still racism in the world.' But I didn't get into country music just to prove a point. I try to stay away from all negativity."

With his latest album Experiment debuting atop the Billboard 200 (Brown is one of only three country artists to achieve the feat this year!), plus three American Music Awards wins and a sold-out tour under his belt, Brown doesn't let haters bring him down.

"Now you can call me whatever you want," he says. "It just brushes off of me."

For more on Kane Brown and his new album, pick up the latest issue of PEOPLE, on newsstands now