Swilling played Defensive Back for the Yellow Jackets from 1988 through 1991. He was an All-American and member of the 1990 National Championship team. Ken had a nickname of "Captain America" while at Tech. "If you were to draw a picture of a football player in a uniform, you'd draw Ken Swilling," said Coach Bobby Ross in a Sports Illustrated interview; "He's the guy you want to get off the bus first when you go visiting." Swilling is said to have called an undefeated season during before the 1990 schedule began after supposedly having a dream that it would happen.