Prior to joining the NFL, Graham attended and played college football at West Virginia University. At West Virginia, Graham excelled in football and basketball from 1921 to 1925 and earned first-team All-American honors in basketball in 1924. He also earned four letters in football as an end and served as the team's captain in 1924.
He was a member of West Virginia's only undefeated football team, which posted a 10–0–1 record in 1922. Graham also helped West Virginia collect its first bowl game victory, a 21–13 decision over Gonzaga in the 1922 San Diego East-West Christmas Classic. During his Mountaineer career, WVU posted a 30–6–3 record. On May 25, 2009, Graham was inducted into the WVU Athletics Hall of Fame.