Frazier led his team to three Class 4A state championships during his last three years in high school and earned MVP honors all three years at Shiloh Christian School. He was named USA Today's National Offensive Player of the Year. During his senior season, Frazier passed for 2,975 yards and 42 touchdowns and rushed for 1,164 yards and 22 touchdowns. He was listed as the nation's No. 2 quarterback by ESPN.com/Scouts Inc. and the No. 47 player overall on the ESPNU 150. Frazier was named an All-American, the nation's No. 4 quarterback, and the No. 1 recruit from Arkansas by SuperPrep and the No. 26 overall player nationally in the SuperPrep Elite 50. He in the Under Armour All-American game and was elected to play for USA Football's 2011 U.S. Under-19 National Team in the Team USA vs. the World game on February 2.
In 2011, as a true freshman, Frazier appeared in ten games, connected on 5-of-12 passes for 34 yards and rushed 76 times for 327 yards and three touchdowns. On December 31, 2011, Frazier ran for 55 yards and two touchdowns serving in a wildcat role during a 43–24 victory over the University of Virginia. In 2013, Frazier switched positions from quarterback to safety. The move came after Nick Marshall was named the Tigers' starting quarterback as part of coach Gus Malzahn's new offense. After the 2013 season, Frazier transferred to Ouachita Baptist University in Arkadelphia, Arkansas. In 2014, Frazier would lead the Ouachita Baptist Tigers to a 10–1 record, a Great American Conference championship, and a berth in the NCAA Division II football playoffs.