Brown accepted an athletic scholarship to attend the University of Virginia, where he played for coach Al Groh's Virginia Cavaliers football team from 2001 to 2004. As a senior in 2004, he was recognized as a first-team All-Atlantic Coast Conference (ACC) selection and a consensus first-team All-American. In recognition of his outstanding college football career, the Virginia athletic department retired his jersey, No. 61, on September 24, 2011.
The Arizona Cardinals selected Brown in the fourth round (111th pick overall) of the 2005 NFL Draft, and he played for the Cardinals from 2005 to 2008. After four seasons with the Cardinals, Brown was cut on September 8, 2009. He had started fourteen of the thirty-four games he played for the Cardinals. He finished his professional playing career with the New York Sentinels of the UFL in 2009.