Reserve Jared Jensen had 18 points and Travis Hansen added 15 for Brigham Young (17-10, 7-7) which lost its 10th straight road game. The Cougars were 0-7 in conference road games this season.
BYU led 35-26 at halftime, and Mark Bigelow's 3-pointer gave the Cougars a 52-38 lead midway through the second half.
Nelson's follow-up basket closed the gap to 57-55 with 34 seconds left. Greene hit two free throws and a layup in the last 23 seconds to tie the game.
Nelson, who was 12-for-16 from the line, hit two free throws with 1:52 left in overtime to put Colorado State ahead 69-65.
Hansen's 3-pointer got Brigham Young within 77-75 with nine seconds left. But the Rams converted their last eight free throws, the final two by Birley with 7.8 seconds left.
The Rams secured the 7th seed for the Mountain West Conference Tournament, and will face second-seeded Utah on Thursday at the Thomas and Mack Center in Las Vegas.