Clayborn made the Pro Bowl three times with the Patriots. From 1977-1982, he was paired with Hall of Fame player Mike Haynes. For his first three seasons, he also played kick returner. As a rookie, he returned 28 kickoffs for 869 yards and a league-leading 3 touchdowns, giving him an NFL-best and Patriots franchise record 31.0 yards per return average. He was a key player on three playoff teams, including 1978, 1982, and most notably 1985, recording a career-high six regular season interceptions (one for a touchdown) as the Patriots won three road playoff games en route to an improbable appearance in Super Bowl XX. In the Patriots 31-14 Conference Championship win over the Miami Dolphins, Clayborn helped frustrated Miami quarterback Dan Marino into having a terrible game during which he completed only 20 of 48 passes for 248 yards. Clayborn made one of two Patriots interceptions during the upset. In the subsequent 46-10 loss against the Bears in Super Bowl XX, the Patriots were undone by turnovers and smothered by Chicago's crushing 46 defense. Though Bears quarterbacks completed only 12 passes for 258 yards with no touchdowns, New England's secondary was beaten several times on catches for big gains. Clayborn recovered one of two Chicago fumbles while watching his offense turn the ball over six times.
Clayborn was involved in an infamous incident early in the 1979 season involving Boston Globe reporter Will McDonough. Clayborn, who had twice been involved in fights with teammates the week leading up to a 56-3 rout of the New York Jets, began snapping at reporters in the locker room after the game and threatening them. When McDonough tried to intervene and settle the situation, Clayborn poked him in the eye; McDonough responded by punching Clayborn twice. The National Football League handed down a $2,000 fine to Clayborn for "conduct involving members of the news media" over the incident. That followed another incident a week prior, in which Clayborn threatened Associated Press writer Bruce Lowitt for trying to ask Clayborn a question after the Patriots' season-opening loss to the Pittsburgh Steelers.
Clayborn finished his career with 36 interceptions, which he returned for 555 yards and a touchdown. He also returned 57 kickoffs for 1,538 yards and 3 touchdowns, and recovered 4 fumbles. At the time of his retirement, his 36 interceptions were a Patriots record. He is currently the NFL Uniform Program Representative for the Houston Texans.
He was elected to the New England Patriots Hall of Fame in 2017. On July 29, 2017, he signed a 1-day contract (at his request) prior to being inducted to officially retire as a New England Patriot.