While at the University of Arkansas, he was initiated into the Xi Chapter of the Kappa Sigma Fraternity. Scott's nickname while at the University of Arkansas was "Smackover", after his hometown. Scott was a three-time All-SWC player at RB and DB, and an All-American in 1948. He helped Arkansas win the 1946 Southwest Conference championship, as well as leading the Razorbacks to their first-ever bowl victory, beating William & Mary 21-19 in the 1948 Dixie Bowl on January 1, 1948. Scott was also a track star at the UA, competing in the 100-yard dash and 110-meter hurdles.
Scott was the eighth player overall chosen in the 1948 NFL Draft. Scott played five seasons in the National Football League for the Philadelphia Eagles and the Detroit Lions, appearing on two NFL championship teams. Scott had a career game on October 6, 1951 against the 49ers when he caught two touchdown passes (3 catches for 85 yards), and had a 40 yard run in the Eagles 21-14 upset win. The Eagles finished 4-8-0. The 49ers, at 7-4-1, finished 1/2 game behind the Conference Champion Rams. Scott was forced to retire in 1953 after suffering a knee injury.
In 2014 Scott was inducted into the Southwest Conference Hall of Fame. The University of Arkansas retired Scott's football jersey number 12, one of only two numbers the school has ever retired (the other is number 77, retired in honor of Brandon Burlsworth).
Scott died on January 30, 2018, at the age of 93.