He attended the University of Maryland, where he played college football as a quarterback and punter for the Maryland Terrapins. Prior to the 1995 season, Milanovich and four other Maryland players received suspensions for betting on college football and basketball games. Milanovich received an eight-game suspension (later reduced to four) during his senior year for having bet between $25 and $50 on a total of six games. The bets had no impact on the outcome of the games. Despite throwing for 24 TD as a Sophomore and 20 TD as a Junior, Scott struggled mightily as a Senior throwing just 2 TD against 7 interceptions.
Milanovich was named the MVP for the Blue squad in the Blue-Gray Classic, and completed 9 of 20 pass attempts for 175 yards and two touchdowns. As of 2008, Milanovich still held the Maryland record for highest career pass completion rate.
After going undrafted in the 1996 NFL Draft, Milanovich signed as a free agent with the Tampa Bay Buccaneers. During his rookie campaign, he was designated as the team's third quarterback for 15 games, seeing action in one contest. In that game he completed two of three passes for nine yards. In 1997, he was declared inactive before all 16 regular season games and both playoff contests.
Milanovich was expected to be the starting quarterback for the XFL's Los Angeles Xtreme but lost the job to Tommy Maddox. Milanovich saw limited action as the Xtreme's second-string quarterback, behind Maddox.
Milanovich began his coaching career as the quarterbacks coach for the Rhein Fire of NFL Europe in the spring of 2003. Later that year he joined the Calgary Stampeders of the Canadian Football League where he also coached the quarterbacks. Between the 2004 and 2005 seasons, Milanovich was the quarterbacks coach and offensive coordinator for the now defunct Mansfield University Mountaineers football team. In 2004, he returned to the Rhein Fire where he served another season as quarterbacks coach. He was elevated to the position of offensive coordinator in 2005 and spent the 2006 season in the same capacity for the Cologne Centurions.
Milanovich returned to the CFL in February 2007 when he was named quarterbacks coach of the Montreal Alouettes. A year later he was promoted to offensive coordinator. In 2009, he added the title of assistant head coach.
On December 1, 2011, Milanovich was named the 42nd head coach of the Toronto Argonauts. His Argonauts won the Grey Cup in his first season at the helm. Following a 9–9 regular season in 2012, Milanovich led the Toronto Argonauts to a 35–22 Grey Cup victory in his debut season as a head coach, and was named the 2012 CFL Coach of the Year.
Milanovich's second season as Argonauts head coach was a successful one. The team battled plenty of injuries yet still managed to finish first place in the Eastern Division with an 11–7 record, their first division championship since the 2007 season. The Argonauts would eventually lose in the Eastern Final playoff game to Hamilton.
On January 26, 2017, Milanovich resigned as the head coach for the Argonauts as he was named as the quarterbacks coach for the National Football League's Jacksonville Jaguars. In November 2018, with the Jaguars offense struggling, offensive coordinator Nathaniel Hackett was fired and Milanovich assumed play-calling duties. Milanovich then relinquished offensive coordinator duties to the recently hired John DeFilippo.
Following his time in Tampa Bay he was charged with Driving Under the Influence (DUI) for an incident on April 11, 1998. He was pulled over by police in Pinellas Park, Florida and found to have a blood alcohol level of 0.135. He pleaded no contest and received a $550 fine, probation, 50 hours of community service and six-month driver's license suspension. He was able to pay the Salvation Army in order to avoid the community service requirement.