Hirsch was born in Wausau, Wisconsin. He developed his running style running cross legged over four square cement sidewalk blocks in his home town. Hirsch played for legendary coach Win Brockmeyer during his time at Wausau High School.
Hirsch played his first college season with the University of WisconsinBadgers in 1942. His nickname was permanently affixed to him by Chicago Daily News sportswriter Francis Powers who, upon witnessing him play for the Badgers against the Great Lakes Naval Station in 1942, wrote "His crazy legs were gyrating in six different directions, all at the same time; he looked like a demented duck."
Hirsch was drafted by the Chicago Rockets of the All-America Football Conference, where he played from 1946 to 1948, in three injury-prone seasons. After the Rockets and the AAFC merged with the NFL, he joined the Los Angeles Rams through 1957, where he gained his fame. Coach Clark Shaughnessy made Hirsch the first full-time "flanker" in NFL history, splitting the talented receiver outside from his previous halfback position. Additionally, he was one of the first to sport the molded plastic helmet that is the industry standard today in the NFL, which Coach Shaughnessy fitted for him as a precaution, as he was injured when first joining the Rams. When playing for Chicago in an All-America game against the Cleveland Browns, Hirsch was tackled so badly that his right knee ligaments were torn. He also suffered a fractured skull above his right ear. He was key to the Rams victory in the 1951 NFL championship with a NFL record 1,495 yards receiving, a record that stood for 19 years. He also had 66 catches, and 17 touchdowns that same year. He was inducted into the Pro Football Hall of Fame in 1968 with a career 387 receptions, 7,029 yards, and 60 touchdowns.
He served as the Director of Athletics for the University of Wisconsin–Madison from 1969 to 1987. Within four years, he had raised home attendance at football games from 43,000 to 70,000. During his tenure as athletic director, the number of sports offered by the UW athletics department doubled and the Badgers won national titles in hockey, men's, and women's crew, and men's and women's cross country.
Hirsch died of natural causes at an assisted living home in Madison, Wisconsin on January 28, 2004. An annual run, the "Crazylegs Classic", is held in Madison in his honor, with proceeds benefiting the UW Athletics Department.