But when South Carolina coach Steve Spurrier called in late January and offered Johnson a position, the South Carolina native just couldn't ignore his chance to go home.
"There were just some things for me personally that it was no question it was the right thing for me to do for my family," Johnson said. "It's just unfortunate the timeframe had to be what it was."
Johnson's three-week stint as Arkansas' defensive coordinator provides an interesting storyline this week as the Razorbacks (4-5, 1-4 Southeastern Conference) and Gamecocks (6-3, 3-3) prepare for Saturday's noon game in Williams-Bryce Stadium.
The 56-year-old, who accepted a position on Bobby Petrino's staff last winter and was gone a month later, is guiding a South Carolina defense that has been one of the nation's best.
The Gamecocks lead the SEC and are third in the nation in total defense (330.4 yards a game). South Carolina is third in the SEC in scoring defense (15 points) and is coming off a dominating, 27-6 win against Tennessee.
But Johnson enters Saturday's game with plenty of respect for Arkansas' coaching staff. And gratitude for Petrino, who Johnson believed understood his situation because of his decision to leave the Atlanta Falcons before completing his first year.
"It's one of those things where neither (Petrino) nor I created the situation. It wasn't the right way to do it," Johnson said. "But I do think he handled it as well as anyone could have. He was gracious."
Johnson retold the story Wednesday. He originally spoke with Spurrier about the South Carolina job in December, but Spurrier hired Brian VanGorder.
Soon after, Petrino hired Johnson and the coach was eager to settle in with the Razorbacks. He met some players, started laying out plans and was in the process of moving his family into a new home.
"We were tremendously excited about getting to go over there," Johnson said. "I knew (Petrino) was going to get it done and I know he will get it done."
But the coaching carousel started spinning again when VanGorder accepted a defensive coordinator position with the Atlanta Falcons. Spurrier called Johnson and said he wanted to hire him.
Johnson said he had to jump at the chance and let Petrino know his difficult decision.
"It's a place he feels like he has an opportunity to retire," Petrino said Monday. "It's worked out good for he and his family."
And for South Carolina where the Gamecocks are winning games thanks largely to the defense.
Johnson's work has been noticed. Two weeks ago, the Gamecocks announced a one-year extension to Johnson's contract. Spurrier called Johnson the best defensive coordinator in the country and said the defense has been "sensational" this season.
"We're fortunate to have Ellis here and probably the reason he's here is he's a South Carolina native and his wife is from Columbia," Spurrier said. "And he was really looking forward to getting back home, and we're all happy it worked out as it did."
Players appreciate the way Johnson has led the way, too. Linebaker Eric Norwood said his "old school" approach has added bite to a group that was embarrassed by Arkansas last November.
"I guess we're just playing the way we should've played last year," Norwood said. "He's just getting it out of us. Our attitude has changed a lot."
And Johnson, who is working at his fourth SEC school, has been thankful for the opportunity even if it was a little awkward getting there.
"We've been very happy with the move," Johnson said. "Wish it didn't happen the way it did, but really happy that this opportunity came."
Johnson Happy At Home
Hawgs Daily Top Stories
A decade of first commitmentsThese players were the first to commit to Arkansas in their respective classes.
Hawgs DailyTuesday at 3:53 PM
Arkansas adds first commitment for 2019Arkansas lands first commitment for 2019
Hawgs DailyTuesday at 10:27 AM
Froholdt, Ragnow named to watch listA pair of Hogs made the Outland Trophy watch list.
Losers and Gainers: Arkansas' weight changesHere is a breakdown of the Razorbacks who have gained and lost the most weight since last year.