Born in Gloster, Mississippi, he graduated from Crosby High School in Crosby, Mississippi, then attended the University of North Carolina before transferring to the University of Mississippi. He also played baseball and basketball at Ole Miss, where his brothers Barney and Buster also became star athletes. Ray Poole was selected by the Giants in the 13th round of the 1944 NFL Draft while he was serving three years in the Marines during World War II. He returned to play at Ole Miss before joining the NFL. He was named to the all-NFL team in 1950.
He later played for the Montreal Alouettes of the Canadian Football League in 1953–54, and then became an assistant coach at Ole Miss under coach Johnny Vaught. He coached at Ole Miss from 1955 to 1974, then served as head coach at Northwest Mississippi Community College for the 1979 and 1980 seasons.