Silver was the first player to hold the starting quarterback job at The University of Notre Dame for three consecutive years, from 1903-1905. He was also one of the first Jewish athletes to compete at the traditionally Catholic university, and was described as having exceptionally quick feet and being "slippery".
Led by Silver, the Irish achieved their first-ever undefeated season in 1903, outscoring their opponents by a combined total of 292-0 for a final record of 8-0-1. The combined record over the next two seasons would be a more modest 10-7. However, the 1905 season would include the most lopsided victory ever by a Notre Dame team: a 142-0 win over the American College of Medicine and Surgery, in which Silver's touchdowns included a 40-yard run and an 80-yard kickoff return. The margin likely would have been much larger had the game not been called after just thirty-three minutes of play.