Wily Peralta (11-15) gave up five hits and walked five in five-plus innings for the Brewers. The Cardinals put the go-ahead run at the plate with two outs in the ninth, but closer Jim Henderson got slugger Matt Adams to strike out to end the game and earn his 26th save.
St. Louis learned earlier Sunday it was assured of at least an NL wild card after Washington lost its doubleheader opener against Miami.
The Cardinals chased Peralta after loading the bases in the sixth with nobody out. Reliever Donovan Hand allowed two runs to score on consecutive sacrifice flies to cut the Brewers' lead to 4-3, but he hot out of the jam by getting Matt Carpenter to strike out on a curve with the tying run at third.
Cardinals catcher Yadier Molina misplayed a throw to the plate for another error in a two-run seventh that gave Milwaukee back its three-run cushion. St. Louis also committed two errors in the Brewers' two-run second.
Hand struck out three in his two innings of work to settle down the Brewers. The Cardinals threatened again in the ninth with two outs. Carlos Beltran had an RBI single, and the Cardinals sent the potential go-ahead run to the plate but closer Jim Henderson got slugger Matt Adams to strike out to end the game and earn his 26th save.
St. Louis remains two games up in the NL Central, but now has two teams to fend off in second after the Reds defeated Pittsburgh 11-3 to pull into a tie with the Pirates.
Playing their final home game of 2013, the Brewers closed Miller Park on a high note in a season marked by injuries and the doping suspension of slugger Ryan Braun.
St. Louis began the day with an NL-low 71 errors.
Kelly also hesitated on what could have been an inning-ending double play in the first after getting Carlos Gomez to bounce sharply back to the mound with runners on first and third. But Kelly glanced just a split second long enough at Jonathan Lucroy leading off third. The Cardinals got the runner at second but couldn't get the relay to first and Lucroy scored easily for a 2-0 lead.
Leadoff hitter Aoki contributed to rallies in both the first and second. He's tortured Cardinals pitching all season, hitting .406 against them entering the day.
Aoki led off the first with a single and came home on an RBI single by Aramis Ramirez with one out. Aoki then reached on an RBI double in the second before scoring after Kelly threw away a sacrifice bunt attempt by Scooter Gennett.
Aoki also walked twice to reach base in each of his four plate appearances.
Kelly went seven innings, allowing seven hits and six runs - three of them unearned.
NOTES: LF Matt Holliday was scratched from the starting lineup with back spasms. ... St. Louis has a major league-leading 34 wins from rookie pitchers this season, their most since also getting 34 in 1952. ... Manager Ron Roenicke thinks SS Jean Segura will be available for pinch-hitting duties for the Brewers' three-game set starting Monday in Atlanta. Segura, the NL leader in steals with 44, has been sidelined since Wednesday with a strained right hamstring ... LF Khris Davis was out of the lineup Sunday due to a nagging wrist injury, but is also likely to return to action in Atlanta.
Link to box score: Milwaukee 6, St. Louis 4
Link to game discussion at The Cardinal Nation message board