Casey was born in Natick, Massachusetts on May 16, 1894 to James Francis and Ellen (Ahern) Casey. He attended Phillips Exeter Academy where he graduated in 1915.
College coaching career
Casey started his football coaching career in 1920 at Mount Union College in Alliance, Ohio. In 1922, he moved on to Tufts College in Medford, Massachusetts where he coached until 1925. That year, he moved on to Harvard University. He was the Harvard freshmen coach from 1926 to 1928, the backfield coach from 1929 to 1930 and head coach from 1931 to 1934.