Fat Joe Claims His Ex 'Abandoned' Son Joey After Learning His Down Syndrome Diagnosis: 'We Never Gave Up on Him'

Fat Joe is opening up about how he raised his son Joey, who has autism and Down syndrome.

As a guest on the Wednesday, Oct. 16 episode of the Club Shay Shay podcast hosted by Shannon Sharpe, the rapper, 54, claimed that his ex abandoned him and their son Joey, now 33, after their son was diagnosed with Down syndrome at birth.

"So, the doctor tells us, ‘I got bad news to tell you… he has Down syndrome and it’s gonna be a big challenge,' ” Fat Joe recalled to Sharpe, alleging, "His mother said, ‘Yo, I can’t do this, I’m going to have to give him up for adoption.’"

"My mother was like, ‘You crazy b----' And so, we raised him," he remembered, referring to his mother and father who helped parent Joey.

"I never seen his mother again is what I’m trying to tell you. She never visited him again," he claimed, adding, "She abandoned the kid.”

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories.

Fat Joe clarified that Joey's mom was always allowed to see her son if she wanted to. "It wasn’t like I kept the door closed where she couldn’t see her son," he explained. "It was always available for her to see her son."

Raising Joey was never a question for the musician, who noted, "[A] father's got to be a father."

"I don't know how to be a fake father," he continued. "I got to take care of my kids regardless and do everything I can for them. I don't know how to give up on my kids. I don't know how to not love on my kids. That's impossible."

In addition to Joey, the rapper also a dad to daughter Azariah, 17, whom he shares with wife Lorena Cartagena, as well as son Ryan, 27, from a previous relationship.

The PEOPLE Puzzler crossword is here! How quickly can you solve it? Play now!

Joey was also diagnosed with autism later in his life, but despite his "challenges," the proud dad said, "This guy is happy. I swear to God, he's always happy. He's never sad. He's like, if you're stuck in a place, he's stuck in happiness."

Fat Joe explains that Joey knows his place in his blended family of five, calling him the "Don."

"He does what he wants. He gets what he wants. He's the Don," the dad of three shared. "He sits at the end of the table, he knows 'Oh, those are my little brothers and sisters.'"

"He's our biggest blessing because we treat him the way we do because we never gave up on him," Fat Joe continued.