But the third year sophomore is changing some minds.
His impressive performance in Tuesday's closed scrimmage inside Spartan Stadium turned some heads and made some people believe that he has enough talent to hold off freshman Connor Dixon and high school commitment Keith Nichol and have a career of his own once Stanton heads to the NFL.
Hoyer went 13-of-21 for 145 yards and two touchdowns, including a 54-yard bomb to wide receiver Deon Curry and a 20-yard flare pass to running back A.J. Jimmerson for a score. That's about the same output he managed in live competition in 2005, going 15-of-23 for 167 and two TDs.
So maybe Hoyer has a future as something other than a clipboard holder after all.
He says his main concern is letting people see that he is a capable backup to Stanton.
"These people are worried about something happening to Drew. I want to make sure they're not worried after that first time I'm in there," Hoyer told the Lansing State Journal.
Hoyer has established himself the old fashioned way; he's earned it.
He's had to fend off challenges from two highly regarded prospects.
First, it was Stephen Reaves, the son of a former NFL quarterback (John Reaves) and the guy Smith believed would emerge as his quarterback, even ahead of Stanton after the graduation of Jeff Smoker.
Reaves, though, after some limited early success, endured off-the-field problems, dropped on the depth chart and eventually decided to leave the program and transfer to Southern Mississippi where he's listed as second string.
Then came Domenic Natale, a big time prospect from the Hun School in New Jersey.
Despite throwing for 2,689 yards and 34 TDs in high school, Natale couldn't grasp Smith's spread offense and was mired deep on the depth chart. He also left the program and transferred to Rutgers.
That leaves Hoyer solidly entrenched as the backup to Stanton and in position to take over the reigns once the former Farmington Hills Harrison standout inevitably moves on to the NFL.
Hoyer knows their will be challenges.
Highly touted freshman Connor Dixon will be a viable alternative after red-shirting this season. Top commitment Keith Nichol of Lowell, MI., is one of the nation's Elite-11 high school quarterbacks and runs the spread offense in high school to perfection.
But you sense that Hoyer is going to get a shot.
That's all the Cleveland St. Ignatius product asks - a fair shot in the competition. He's already beaten out two top prospects, what's two more?