Kelley was an All-American and the captain of the Yale football team. Following his career at Yale, he signed a one-game contract with the Boston Shamrocks of the American Football League in 1937 but never played. Following his career in football, Kelley was a history teacher and alumni director at the Peddie School in Hightstown, New Jersey, his alma mater. He taught algebra at Cheshire Academy and spent 12 years in the glove manufacturing industry.
The Ian Graham Athletic Center at the Peddie School holds a replica of the Heisman Trophy donated to the school by Kelley. To benefit his nieces and nephews, Kelley sold his Heisman Trophy at an auction in December 1999 for $328,110 to the owner of The Stadium Museum, Restaurant & Bar in Garrison, New York, where it now resides. His health was visibly failing by then after having suffered a minor stroke and having open-heart surgery, and on June 27, 2000, Kelley died of a self-inflicted gunshot wound at his home in Hightstown, ruled a suicide by the police. He was 85 when he died. He was survived by his fourth wife and 18 nieces and nephews.