Then there's the fact that the Colts have had a tremendous home record in recent years. Setting aside those late-season games where they played their backups, they've only lost four games at home over the past five seasons.
Mainly, though, the biggest reason for worry is the unpleasant reality that Indy has simply dominated the Chiefs over the last decade. Since the Peyton Manning era began in 1998, the Colts are 6-1 against Kansas City, including two wins in the playoffs.
Adding all those factors together, it almost seemed crazy to look at the schedule and think the Chiefs might have a chance at pulling off the upset.
As we get closer to Sunday's game, though, more and more people around K.C. believe they actually have a shot at doing it.
Count me among them. I fully realize that I'm letting my optimism overrule my logic here, because even after the first two weeks of the season I wouldn't have given the Chiefs much of a chance to beat the Colts. And I know it's foolish to allow one game to completely change my perspective, but I'm letting the win over the 49ers do it anyway.
It's not because the Chiefs won that game by such a large margin. By itself, stomping a winless team like San Francisco doesn't mean the Chiefs are ready to match up with the likes of Indianapolis.
In week three, Chiefs Quarterback Matt Cassel came alive in the second half.
It was the way they won the game that has me feeling good about their chances in Indy. Watching them against the 49ers, it was impossible not to notice how the Chiefs' coaches were running circles around their counterparts on the other sideline.
The Chiefs' offense, particularly in the second half, seemed to know exactly how to attack the 49ers' defense. The Chiefs' own defense played the entire game like they were inside San Francisco's huddle. Throw in Todd Haley's aggressiveness – an onsides kick, running off the punt team to bring back the offense – and the 49ers spent all four quarters of that game on their heels.
Granted, apart from Romeo Crennel getting the defense to play at a high level, nothing about the season's first two games really hinted at any coaching magic on the Chiefs' part. From a coaching standpoint, perhaps the game with San Francisco was the equivalent of high school seniors picking on a group of timid freshmen.
But is much going to change on Sunday? Judging from the reactions of Colts fans over the past week, it doesn't sound like they have much confidence in head coach Jim Caldwell's ability to outmaneuver anyone in a mental chess match.
Caldwell rarely has to worry about that sort of thing with Manning leading the Colts' attack, of course. But when a game is tight – like last season's Super Bowl, or last week against Jacksonville – he always seems to be taking flak for getting outcoached along the way.
That's why I think that if the Chiefs' coaching staff can carry over their performance against San Francisco, it opens up all new possibilities for what could happen in Indianapolis.
With an extra week to prepare, what will Haley and Charlie Weis have in store for a Colts' defense that has struggled mightily this season? In fairness, three of Indy's four games have been played on the road, and the Colts play better at home when they're able to fly around on the turf and get off the ball faster with the crowd noise in their favor.
Colts Head Coach Jim Caldwell has found the footing a little slippery in 2010.
But simply being on the road doesn't explain away their bad performances. The 174 rushing yards they gave up to Jacksonville last week doesn't look half bad compared to the 257 they surrendered to Houston in the season opener. Then there's the Colts' secondary, which was picked apart by Denver quarterback Kyle Orton two weeks ago, even though the Broncos had no running game for Indy to defend.
Colts' defensive coordinator Larry Coyer, who spent most of the previous decade in Denver, is also taking plenty of blame from Colts' fans. In his second year leading their defense, many expected the unit to start making strides. Instead, they're struggling just to keep their heads above water. A scheming matchup against Weis, who hasn't come close to showing all his cards yet, seems like a clear advantage for the Chiefs.
But it's the other side of the ball where coaching will mean the most. The Colts have an offensive coordinator, but there's no mistaking that Manning is the one behind the wheel. And no defensive coordinator in league history has experienced as much success against Manning as Romeo Crennel has.
You've read all the statistics over the past two weeks, no doubt. While Crennel ran the Patriots' defense, they were a perfect 6-0 against Indy. While he was head coach in Cleveland, the Browns played the Colts twice and, despite losing, Crennel's defense held Indy under 14 points in both games.
And now with Ryan Lilja back with the Chiefs, Crennel has the advantage of talking with someone who spent the last six years as a major cog in the Colts' offense.
Does all this mean the Chiefs, who once went an entire playoff game without forcing the Colts to punt, are going to put on a defensive clinic on Sunday? Probably not. Between Eric Berry, Kendrick Lewis, and Javier Arenas, the Chiefs have three rookies seeing plenty of action in the secondary, which will provide Manning with opportunities no matter how great Crennel's game plan is.
But when the Chiefs need it most, can Crennel frustrate Manning just enough to lessen his effectiveness? I'm willing to bet that he can.
Of course, we have to acknowledge the fact that it seems a little dangerous to get our hopes up prior to a game against the Colts. Dating back to the mid 1990's, it seems like no team in the NFL has been present for as many of the Chiefs' heart-crushing losses as Indy has, particularly in the postseason.
Chiefs Tight End Tony Moeaki might have a career day against the Colts soft defensive secondary.
But given the team's surprising start, it's hard not to let a little optimism creep in. Especially when you consider that we've only covered the coaching issue here. We haven't even mentioned Jamaal Charles and Thomas Jones. We haven't mentioned emerging players like Dexter McCluster and Tony Moeaki, who made impacts against the 49ers beyond any coaching strategies.
We also haven't brought up the uncharacteristically poor play from the Colts' offensive line this season, or the injuries in their secondary that may force ex-Chief DaJuan Morgan into a starting role, or any other of the little items that seem to be popping up on the Chiefs' side of the ledger.
So could they really start the season 4-0? It's a complete 180 from just a few months ago, but when you take everything into account, it almost seems crazy to think they don't have a chance to win.