Brown was born with his twin brother, Wesley, in Liberty City, a low-income section of Miami, Florida. He was adopted by Mamie Brown, a 38-year-old single woman who worked as a cafeteria attendant and domestic assistant. Brown claims that he was declared "educable mentally retarded" in grade school, which damaged his self-esteem and confidence.

According to many of Brown's speeches, when he first decided to get his life into order, he went to a radio station he was repeatedly unsuccessful with his speeches. It was only after the on-air failures of the previous afternoon DJ that he was hired full-time. Upon his termination from the radio station, he ran for election in the Ohio House of Representatives and won. After leaving the Ohio State Legislature, he shifted his career to television and became a host on PBS.

Brown married Gladys Knight in 1995. They divorced in 1997.