And yet there was Brown on Thursday, answering questions on his 16th Longhorn squad.
"I looked at the history of Texas coaches," Brown quipped when asked if he saw himself coaching at Texas for 16 years. "No, I really didn't. I've been pleased with our years and what we've done, excited about a lot more for that reason. I think I am a Texan now. When we moved here, like Coach [Darrell] Royal, we were out-of-staters. Coach Royal said he came from Oklahoma and became a Texan. [I] grew up in North Carolina, came from Tennessee, but we moved here now and we are Texans."
Now, Brown is a Texan trying to get the Longhorn program back on track, pushing to meet the lofty expectations that Brown himself created. Texas won at least 10 games in every season from 1999, Brown's second year, through 2009, but hasn't hit that mark since. And the Longhorns bottomed out with a 5-7 mark in 2010, just one season after reaching the BCS National Title game.
"We were at a tough spot at 5-7," Brown said. "When you're there, you have to have some hard self-evaluations of you and your program, your players. I did a lot of that myself. I trusted friends to be very critical, as well."
How critical could they be, really? From 2005-2009, Texas competed in two national championship games, winning one, and were a couple bad breaks away from maybe playing in another one (2008) or two (2006). But in 2010, it became apparent that something wasn't quite right on the 40 Acres. The talent had fallen off, and was especially obvious when not disguised by historically great quarterback play. And some felt that there was a level of entitlement in the locker room ... players were still calling themselves a "great team" in interviews even as they entered the season finale with a 5-6 record.
The verdict: Change out the bulk of a coaching staff that had been intact when the Longhorns fell to the Crimson Tide. When defensive coordinator and coach-in-waiting Will Muschamp jumped for the Florida head coaching job and Duane Akina left for Arizona, Brown faced a daunting task: replace six of the nine assistant coaches on staff. Akina quickly returned, but the process was not unlike an episode of 'This Old House': gut everything, cut out the dead wood and build around the house's (the team's) strengths.
Not including the brief tenure of Jerry Gray and the switch back to Akina when Gray pounced on an NFL job, Brown hired five new assistants, including both coordinators in Bryan Harsin and Manny Diaz. And while the previous staff landed a boffo recruiting class in 2010 and the makings of another great one in 2011, the new staff was charged with hanging onto those 2011 recruits while steadily changing the culture.
Part one worked … the only recruit to jump ship was highly regarded offensive tackle Christian Westerman. And despite winning just seven regular season games — that talented 2010 class was just in its second year on campus — the Longhorn staff brought in a recruiting class ranked as the No. 1 class in the country by Scout.com.
Part two was a work in progress. Texas brought in Bennie Wylie to help upgrade the strength and conditioning of the young players. But Rome wasn't built in a day, and neither are college football bodies. Texas was brutally young in 2011, and last season, when the 2010 class were in their third year, the 2011 class was in its second and the 2012 crop its first.
Members of all three classes began to flash. David Ash, a 2011 signee, finished in the nation's top 25 in passing efficiency. Malcolm Brown and Joe Bergeron (2011) and Johnathan Gray (2012) gave the Longhorns a deep and talented group of running backs, while Mike Davis (2010) and Jaxon Shipley (2011) beefed up the receiving corps. Together with an offensive line made up of four different classes: Mason Walters (2009), Trey Hopkins and Dominic Espinosa (2010), Josh Cochran (2011) and Donald Hawkins (2012), the offense made a big leap forward.
In 2012, Texas averaged 434.6 yards and 35.7 points per game, up significantly from 2011's 392.5 and 28.1, and way up from the 2010 disaster season's marks of 382.5 and 23.8.
But even with those offensive improvements, the Longhorns didn't quite meet expectations. A defense that was a top-10 unit in S&P+ in 2011 struggled after losing four seniors up the middle, and later losing defensive anchors Jackson Jeffcoat and Jordan Hicks.
"We put things in place that take some time," Brown said. "We were ahead defensively two years ago. We were ahead offensively last year of where we were in that 5-7 season. We've made progress each year."
The result of that progress was a 9-4 season and a come-from-behind win in the Alamo Bowl. But even though the Longhorns nearly doubled their wins from 2010, that season still ranked under the standards that Brown set for the team.
"I think sometimes for the players, people have been a little hard on them," Brown said. "[A] 9-4 [record] is not a happy time around here, but I'd rather us be treated like 9-4 than 6-6. We have so many fans that are pumped, excited, can't wait till the season. They understand that we're better and we have a chance. I think they can't wait till the season."
You could certainly count Brown in that group. The Longhorns return almost every key piece on offense from a year ago, and lost just three players from the defensive two-deep. Linebacker, which struggled with inexperience last season, returns seven players who started at least one game a year ago, and Texas gets back a healthy Hicks and Jeffcoat.
Expectations, at least nationally, are somewhat varied. All have the Longhorns as a top-20 team, but, then again, Texas finished 18th nationally last year. Texas was picked to finish fourth in the Big 12 by the league's media, but the gap between No. 1 and No. 4 was tiny. Phil Steele picked the Longhorns fourth as well … but his rank was a national one.
Brown is certainly familiar with high expectations, and he said that in most of his big seasons as Texas coach, he knew before the season started that the 'Horns were in for a big year.
"The one we couldn't tell was '08," Brown said. "A lot of people thought we would win six or seven games. We had two freshmen starting with Earl Thomas and Blake Gideon. That team really developed and turned out to be a great football team. [They were] three seconds away from playing for a conference championship, maybe a national championship.
"The rest of them, you saw leadership, maturity, depth," Brown said. "All of that is here now."
It hasn't been an easy rebuild for Brown, as many fans expected him to turn the faucet and immediately get back to the expected excellence at the expectations monster that he helped to build. So, are the Longhorns back?
"Now we are back to a point where I feel like we can compete each week and have a chance to win," Brown said. "People ask, 'Do you have a chance to win all the games?' We do. That doesn't mean we're going to.
"As I've said before, we have to shut up and play, shut up and coach," Brown said. "I think every week we line up we have a good chance to win."