Potter played his university football with the University of Ottawa Gee Gees, of whom he was made captain in 1972 and for whom he was named 1971 most valuable player. He began a 12-year career with the Eskimos in 1974. During this time, he was paired with Danny Kepley and Tom Towns for eight years, forming a linebacking unit that would lead Edmonton to 7 Grey Cups and 6 Grey Cup victories. Potter won Defensive Player of the Game and all star in 1980. He would finish his career in 1984 with the Toronto Argonauts, playing 5 games.