In 2004 Frazier was named the 23rd head football coach at Prairie View A&M University in Prairie View, Texas. His record at Prairie View was 43–30. Upon his resignation at Prairie View A&M, Frazier was the second winningest coach in Prairie View A&M football history.
On December 16, 2010, Frazier was introduced as the 21st head coach of the North Carolina Central University Eagles. Frazier succeeded Mose Rison and inherited an Eagles team that posted a 3–8 record during the 2010 campaign. His contract was for five years, with a base salary of $225,000. On August 22, 2013, following a domestic dispute, NCCU's administration announced that Frazier was relieved of his duties as the head coach of the Eagles.