He followed that up a year later by becoming the ACC's all-time reception leader, while breaking just about every receiving record at Clemson. But Kelly's career at Clemson is done and now Swinney is offering up that same challenge to junior Xavier Dye.
"It's time for him to be the man," Clemson's head coach said after spring practice earlier this week.
So far, it appears Dye is responding to that challenge.
After catching six passes for 75 yards as Kelly's backup last season, he has added a few pounds to his 6-foot-5 frame. He has worked harder in the weight room and though it has been just three days, he has been the Tigers' most consistent receiver in spring drills.
"I have been looking forward to this," Dye said. "Being behind Aaron and him being a fifth-year senior and all, it has been tough trying to take his position.
"You have a lot to learn. I'm just glad my opportunity is here. I'm just going to try and seize the moment and make every play that I can."
Dye has shown glimpse here and there of his big play potential the last two years. Last season, he hauled in a 22-yard pass against S.C. State and finished the day with three catches for 37 yards. He also had a 22-yard reception during his freshman season as well.
Xavier Dye is the heir-apparent to Aaron Kelly, but he still has much to prove this spring and next fall. (Kevin Bray/CUTigers.com)
"He's actually further along than Aaron was at this stage of his career," Swinney said. "This will be Xavier's third year, Aaron's third year was going into his sophomore year because he redshirted. From a fundamental standpoint, a physical standpoint and a knowledge standpoint, Xavier is further along.
"But he has to be productive like Aaron."
Kelly was very productive, especially the last two seasons, when combined he caught 155 passes for 1,803 yards and 15 touchdowns.
"Aaron was a very productive player and made a lot of plays," Swinney said. "We need (Dye) to step up and be the guy like Aaron did when (Chansi) Stuckey left."
Of course, Dye was the man when he was in high school.
During his junior season at Greenwood High School, he caught 37 passes for 614 yards – a 16.6 yards per catch average – and four touchdowns. After transferring to Byrnes High School his senior year and missing much of the season due to High School League rules and appeals, he eventually caught 32 Willy Korn passes for 524 yards and six touchdowns in just four games.
"Every play and every ball that comes your way, I don't care if it is a one-handed catch, you have to make it," Dye said. "This game is about production. That's the only way you are going to play is to perform.
"That's the only way (Swinney) is going to allow you to play for him."
Dye believes he can fill the void left by the graduating Kelly and he looks forward to the opportunity to prove it. He also looks forward to the opportunity to prove he is a different player than Kelly.
"I'm a little bit bigger and more physical at the point of attack," he said. "I like playing physical. I'm just trying to be consistent on a daily basis. And I'm trying to be a great player, not just a good player.
"I'm trying to make all the plays I can where I'm consistent and the coaches can count on me in the ball game whenever the game is on the line."
Playing for Swinney the first two years, Dye also understands what his head coach is looking for in him and the entire team, and pleasing his head coach is one of the more driving factors he has.
"To play for him you have to be a productive player," Dye said. "That is the main thing. You have to go out and look the best you can. He is watching and he knows what this guy is doing and he can tell if this guy really wants it."
And nobody wants it more than Dye.
"I want to win," the junior said. "We have been on this being average scale for a long enough time. It's time to step up and go out and do it. That's the main thing I'm trying to do."