Gilmer was born in Birmingham, Alabama, where he attended and played high school football at Woodlawn High School. Gilmer often utilized the technique of leaping high into the air to pass the ball because, as a child, he often played pickup games with teammates who were much older and thus taller than he was; Gilmer was then one of the first players to popularize the "jump pass" when he continued using the technique at the collegiate level.
After high school, Gilmer played college football at the University of Alabama, where he was the left halfback from 1944 to 1947. As a freshman, he was 8 for 8 in passing attempts during a loss against Duke University in the Sugar Bowl. Gilmer's best year was his sophomore season, when he led the nation in touchdown passes–thirteen–and he ran for nine touchdowns. He had 79 rushing attempts with an average gain of 7.0 yards and a passing percentage of .648 on 88 attempts. His total offense, 1,457 yards, was second in the nation. Gilmer also spent time as a punter and kickoff returner and, in his junior year, he returned 37 punts; his average, 14.5 yards, led the nation. During that season, Gilmer led Alabama to the 1946 Rose Bowl, where they beat the University of Southern California 34–14. In his career Gilmer passed for 26 touchdowns and ran for 24. He passed for 2,894 yards and rushed for 1,673. His punting average was 36.4 yards. He averaged 28.7 yards on kickoff returns, 13.5 on punt returns.