The Eagles remained competitive in 1961, winning 10 of 14 games, but fell to 3–10–1 in 1962 and 2–10–2 in 1963. Fired from the Eagles, Skorich took a job as a defensive assistant under Cleveland Browns coach Blanton Collier in 1964. The Browns promoted him to offensive coordinator four years later and head coach upon Collier's retirement after the 1970 season.
In 1970, the Browns had gone 7–7 in only their second non-winning season since beginning play in 1946. Under Skorich, the Browns went 9–5 in 1971, winning the AFCCentral Division before losing to the Baltimore Colts in the divisional playoffs. The following year, the Browns earned a wild card spot with a 10–4 record. In the playoffs, they came as close as anyone else that season did to beating the Miami Dolphins in that team's perfect season, losing 20–14 on a late Jim Kiicktouchdown.
After leaving Cleveland, Skorich served as supervisor of officials for the National Football League. He is credited with developing mechanics for umpires, the most demanding position on an officiating crew since the umpire is positioned behind the defensive line and is often caught in the middle of heavy traffic during play. The mechanics for umpires was changed by the NFL for the 2010 season, moving the umpire behind the quarterback, parallel to the referee, except for the last two minutes of each half.
He died in 2004, after complications from heart surgery. In his memory his family started the Nicholas L. Skorich scholarship fund, which, holds a yearly golf outing.