The Spartans got within three multiple times in the final few minutes, but the Petteway came up with big shot after big shot as he hit three contested 3-pointers late.
Petteway finished with 23 points and forward Walter Pitchford had 18 points, hitting four 3-pointers.
Gary Harris led the Spartans with 18 points, but hit just 1 of 7 from 3.
As a team, the Spartans shot just 34 percent from the field.
MSU had the lead three times in the first half, but never led by more than two and trailed for the final 25:48 after Pitchford hit a 3-pointer at the 5:48 mark of the first half.
A Ray Gallegos layup put Nebraska ahead by 11 at 15:58 of the second half.
Michigan State pulled within two when Adreian Payne hit a layup in the post with 4:01 to go, but the Spartans couldn't get any closer.