Kelly played college football for the Kentucky Wildcats of the University of Kentucky. In his first year on the freshman team, Kelly rode the bench most of the year. "I knew I could play and that I could run like hell" said Kelly. He finally got a chance against Centre in the season's final game, and scored three touchdowns. The 1929 team lost just a single game, to Wallace Wade's Alabama, a game that Kelly missed. In 1931, Kelly rushed for 1,074 yards averaging 6.3 yards per carry. He was second-team on the AP composite All-Southern team in 1930 and 1931. By Kelly's UK career's end he was dubbed "the fastest man in the South," running a 100-yard dash in 9.8 seconds.
At age 23, Kelly became a player/coach and later a player/coach/owner with the Dodgers football club, which he bought with a partner, Dan Topping. Kelly ran back the team's punts. He also led the league in receptions in 1933.
Kelly was a part of New York's cafe society and was frequently in attendance at the Stork Club, "21", and El Morocco.
Kelly was married in 1941, in New York City, to the "Millionaire Debutante" Brenda Frazier, after whom the long-running comic strip Brenda Starr, Reporter was named. The couple bought a new Packard Darrin convertible from the New York Auto Show, and travelled around New York City with people such as Jock Whitney and Tom Kerrigan. They were married for fifteen years, and had one daughter, Brenda Victoria. In 1956, he married Catherine Hannon. They had a son, John Kelly, who was in the Winter Olympics in 1980.
After the war, Kelly pursued a career as an investment banker, Florida real estate investor and became a champion amateur golfer. He played golf with the Duke of Windsor and Richard Nixon. He was also a big game hunter. He died of a stroke at age 76 and is buried in his home town of Simstown, Kentucky.