Since 2011, he has worked as Clemson's offensive line coach to turn his inexperienced blockers into a unit that hopes to be the foundation for this fall's Tigers.
And while the Tigers offense has risen from 88th in yards gained three seasons ago to ninth last season, that success is not what he is most often asked about - even these days.
What most people remember him for, Caldwell says, is his one-of-a-kind media gathering that stole the show at the 2010 Southeastern Conference media days just days after he unexpectedly had been named the head coach at Vanderbilt.
Caldwell, who'd gotten the job a week earlier, entered the room as a "Who's this?" curiosity and left as breakout star after more than 40 minutes of stories about growing up in rural South Carolina and working on turkey farms.
"I'm not a comedian," Caldwell said this week at Clemson's football gathering. "I just be myself and people laugh."
Caldwell gave them plenty to chuckle about that summer day three years ago.
He joked that some at the Wynfrey Hotel in Hoover, Ala., tipped him because they thought he was the doorman instead of the Commodores' coach. He shared stories about his life being raised in a farm-rich area about an hour south of Charlotte, N.C., and his early jobs like pouring concrete, pipefitting and framing. The biggest laughs came when he detailed his time at a turkey farm.
"I was on the inseminating crew. That's a fact," Caldwell told the SEC media in July 2010. "I worked my way to the top. Best job I ever had."
Caldwell figured that after a few minutes he'd get shuffled off in favor of a Steve Spurrier or Nick Saban. Instead, he kept the room riveted for most of an hour and got a standing ovation from those in attendance.
"It's kind of sad," he said. "Because I was just talking to people, being myself."
Caldwell's stay on top at Vanderbilt wasn't quite as crowd pleasing. He resigned four months later after the Commodores went 0-8 in the SEC.
"They talked about me being the head coach" at Vanderbilt, Caldwell said. "You can't consider me that. I just sat in the seat four months holding the fort down."
Caldwell landed back in the game in his home state three months later as an assistant for Clemson coach Dabo Swinney. Before his brief stay at the top, Caldwell was one of the game's most talented and respected offensive line coaches with stints at North Carolina State, North Carolina and Vanderbilt from 1986-2009 before joining the Tigers.
"I felt like a fish out of water, not coaching a position," Caldwell said of his time as head coach.
Caldwell's helped establish depth and consistency with a group that struggled at times with opening holes and holding off pass rushers.
The line is anchored by all-Atlantic Coast Conference first team left tackle Brandon Thomas. He'll be joined by experienced veterans Gifford Timothy at right tackle, and guards Tyler Shatley and David Beasley. Ryan Norton will get first crack to take over for Dalton Freeman, the Tigers longtime standout at center.
Caldwell's happy to be back in his element, digging in with the hard-nosed players he loves. "As I always tell them, it's hard to run an off-tackle play without a tackle," he said with a smile.
Tigers coach Dabo Swinney appreciates Caldwell's skill and plain-spoken attitude. "What you see with Robbie is what you get," Swinney said. "He's no different in meetings or on the field."
Caldwell was grateful for his short time in charge. He's glad to see the Vanderbilt program rising under his successor, James Franklin, and administrators committing resources to improve the program.
"It's always the same: Whoever comes in next is going to get more than the last one," Caldwell said. "The thing I'm proud of is they're using my plan. When I left from there I presented them a five-year plan to try to keep the job and I put together things they needed to do. They told me they were going to make a change but said, ‘Hey, can we keep the plan?' I said sure. I helped recruit those guys."
Now, Caldwell wants to help Clemson chase ACC and national championships - and maybe have people forget about his stealing the SEC spotlight.
"That's kind of sad if that's what you're remembered for," he said.
Caldwell Recalls SEC Stage
CUTigers.com Top Stories
RECRUITING RADARIn our weekly Recruiting Radar, CUTigers Recruiting Analyst Gary McDaniel updates you on the latest news and intel on the Clemson recruiting front!
CUTigers.comMonday at 5:53 PM
Visiting Recruit ListDeath Valley will play host to a star-studded list of recruits on Saturday, as #3 Clemson takes on #3 Louisville.
CUTigers.comYesterday at 5:24 PM
TUESDAY DABO PC AND INTERVIEWSWatch Dabo Swinney, Brent Venables and several Clemson players discuss their upcoming ACC showdown with #3 Louisville in Death Valley!
CUTigers.comYesterday at 9:07 AM
MONDAY WEST ZONE INTERVIEWSWatch Deshaun Watson, Jeff Scott, Ray-Ray McCloud, Ben Boulware and Clelin Ferrell discuss the Georgia Tech win and their upcoming showdown with #3 Louisville!
CUTigers.comMonday at 2:57 PM