The 2004 season began with doubts and criticism for the Irish. With Julius Jones graduating as fourth-leading rusher in Notre Dame history, the Irish hoped to replace him with a talented recruiting class. However, Willingham struggled in his second full year of recruiting and the new class was ranked 30th in the nation.
Despite signing highly sought after recruit Darius Walker, the 17 man class only included three four-star recruits.
The season began poorly for the Irish with a loss at BYU. Despite Brady Quinn improving at the quarterback position, completing over 50 percent of his passes for 265 yards, the Irish only managed to gain 11 yards rushing. They next faced a highly ranked Michigan team at home and Willingham stated that an improved running game would be important if the Irish were to be able to beat the Wolverines. Darius Walker answered Willingham in his first collegiate game, gaining 115 yards and scoring two late touchdowns to lead the Irish in the upset. With the win the Irish were rejuvenated, and rallied to move to 3–1 on the season with wins over Michigan State and Washington. Some in the media began comparing Willingham to some of Notre Dame's legendary coaches and said the team would win seven or eight games in the season, and be back in national championship contention by 2005.
With renewed expectations, the Irish hoped to continue their streak and beat 15th ranked Purdue, who hadn't won at Notre Dame in 30 years. The Boilermakers' quarterback, Kyle Orton, torched the Irish defense handing them a 25 point loss to end the rally. The Irish got back on track and beat Stanford, making Notre Dame the second school to reach 800 wins, and Navy for the 41st straight time, to move into the rankings for the first time since their 2004 loss to Michigan.
The Irish didn't stay ranked for long, as Boston College once again beat the Irish on a late score. The Irish had three games left, and needed one win to become bowl eligible, but looked as if that win wouldn't come in their next game as they faced the 9th ranked Tennessee Volunteers in Knoxville. The Irish defense, however, stepped up, and, after knocking out quarterback Eric Ainge on a sack, returned an interception for a touchdown to upset the Volunteers and become bowl eligible. Once again ranked, the Irish returned home for their final home game against Pittsburgh. Losing on a late score, the team allowed five passing touchdowns by an opponent for the first time ever at home. Visiting USC for the final regular season game, the Irish again lost to the Trojans by 31. The Irish accepted a bowl bid to play in the Insight Bowl, however, in a highly criticized move, two days later fired Willingham. Defensive coordinator, Kent Baer, led the Irish, hoping to "win one for Ty," however, the Oregon State Beavers, led by four touchdown passes from Derek Anderson, beat the Irish in their seventh consecutive bowl loss. The Irish ended 2004 with a 6–6 record and in need of a coach.
The second bowl on December 28, the Insight Bowl held at Bank One Ballpark in Phoenix, Arizona, the home of the Arizona Diamondbacks, was the second of the 2004–05 bowl season to pit two BCS member teams. The Notre Dame Fighting Irish, the only independent BCS member, took on the Oregon State Beavers from the Pac-10. The Beavers never trailed in the game, and easily defeated the Irish 38–21. Beavers quarterback Derek Anderson threw for 359 yards and four touchdown passes, with no interceptions.
In firing Willingham, the Notre Dame athletic department cited a relatively poor record of 21–15, a weak recruiting class, and three losses, each by 31 points, to rival USC. However, the Irish also hoped to entice Urban Meyer, the head coach of Utah, to lead Notre Dame. Meyer had just led the Utes to an undefeated season and he had a clause in his contract that stated he could leave Utah without a penalty to coach for the Irish. When Meyer instead took the head coaching position at Florida, the Irish were ridiculed in the media, saying that the Notre Dame coaching position is no longer as prestigious as it was in the past. After over a week without a coach, the Irish hired New England Patriots' offensive coordinator Charlie Weis as head coach. Weis was an alum of Notre Dame, and became the first alum to coach the team since 1963. At least one sports writer stated that Weis was a choice that made sense for the program. Notwithstanding, three years after the fact, Willingham's firing remains highly controversial with many believing he was treated unfairly.