After red shirting his freshman year, Brown played in all 13 games in 2004, and caught his first collegiate pass against Western Michigan University that went for a touchdown. In 2005, he moved from tight end to offensive tackle, where he started every game and played in 673 offensive snaps during the regular season.
As a junior, Brown earned second-team All-Atlantic Coast Conference recognition and was voted state Offensive Lineman/End of the Year by the Richmond Touchdown Club.
During his senior season in 2007, Brown switched to left offensive tackle and earned second-team All-ACC honors and Offensive Lineman/End of the Year by the Richmond Touchdown Club for the second consecutive year.
Brown was drafted by the Houston Texans in the first round (26th overall) of the 2008 NFL Draft. He was only the second Virginia Tech offensive lineman ever drafted in the first round of an NFL Draft, after Eugene Chung in 1992. He started all 16 games as a rookie and became the first Texans rookie lineman to start every game since Chester Pitts in 2002. In 2009, Brown protected the blind side of Houston Texans quarterback Matt Schaub who led the NFL with 4,770 passing yards and helped the offense set franchise records with 6,129 total yards and 388 points.
During the 2010 season, he started 12 games, allowing only 6.5 sacks and had zero holding penalties.
On August 16, 2012, Brown and the Texans agreed on a six-year, $53.4 million extension, with $22.08 million guaranteed. Brown was in the final year of his rookie contract; the extension will kick in at the conclusion of the 2012 season. Brown was named onto the 2012 All-Pro Team as a first-team selection and was voted to play in the 2013 Pro Bowl at the end of the 2012 season. Bleacher Report named Brown the top left tackle in football and had the highest score of any tackle according to the site.