Ballard was a one-year starter at Wilmington College then transferred after his freshman year to Mount Union College. While at Mount Union, Ballard broke numerous school, conference and NCAA records en route to leading the Purple Raiders to their first ever Division III National Championship in 1993. A two time First Team All-American Team member in 1992 and 1993, and an Honorable Mention All-American in 1991, Jim shattered 17 Division III records and threw for over 12,000 yards and over 150 touchdowns. The two-time recipient of the Mike Gregory Award, which is given to the Ohio Athletic Conference's top offensive back, he was a three time All-Conference selection and suffered only one OAC loss during his college career. Ballard won the inaugural Melberger Award and the Gagliardi Trophy as Division III's Player of the Year. He held every Mount Union passing record by the end of his career and still holds the NCAA record for highest number of touchdowns thrown in a playoff game (8). In addition, Jim was selected as the 1993 Joe Fogg Award Winner, given to the top collegiate player in all divisions in the Midwest by the Cleveland Touchdown Club. In 2008, Jim was inducted into the College Football Hall of Fame. Ballard was an All-American at Mount Union College.