Smart sealed it with two free throws, and Cowboys fans stormed the court after OSU won its seventh straight game. Oklahoma State's last two home wins were by two points apiece.
Romero Osby led the Sooners (16-8, 7-5) with 18 points and 15 rebounds. Sam Grooms had a career-best 18 points.
In a sharply played game in which both teams finished in single digits in turnovers, Oklahoma State squandered an early 11-point lead and then had to rally back from 11 points down in the second half.
Nash also had a right-handed slam and a three-point play off a layup in overtime to lead the way for the Cowboys, who came into the game in a three-way tie for first place in the Big 12 with No. 10 Kansas State and No. 14 Kansas. Oklahoma had been just a game back, and the anticipation of the game attracted the first Bedlam sellout in Stillwater in five years.
It was Oklahoma State's second sellout of the season, and the first one involved a few thousand tickets being given away after a donation from booster T. Boone Pickens. The last capacity crowd of 13,611 to come to Gallagher-Iba Arena without a free ticket promotion was in January 2011, when the school marked the 10th anniversary of a plane crash that killed 10 men associated with the basketball team.
The Cowboys wiped away an 11-point deficit to tie it at 56, then fell behind by six before Phil Forte came up with a steal at halfcourt and a runout layup to even the score at 69 with 2:18 remaining in regulation.
Osby and Smart each banked in a running jumper - and each missed a free throw - and Smart was able to answer another Osby jumper by hitting two free throws to tie it at 73 with 42 seconds left in regulation.
Osby missed a jumper on Oklahoma's last second-half possession, and Smart missed a 3-pointer at the buzzer to send it into overtime.
Pledger also ended up with 18 points as Oklahoma played its first game since losing freshman Buddy Hield, the team's leader in minutes played, to a broken foot that will sideline him for four to six weeks. Fellow freshman Je'lon Hornbeak moved back into the starting lineup to replace Hield and had 10 points.
Brown had 18 points for Oklahoma State, which defended its home court against the Sooners for the fourth straight time.
The Cowboys got out to an early 11-point lead as Oklahoma missed 10 of its first 13 shots. Forte put Oklahoma State up 19-8 when he faked a shot, stepped forward and hit a 3-pointer from the left wing with 11:48 left before halftime.
The Sooners responded quickly, putting together a 9-0 run to tie the game at 21 on Pledger's layup from the right block with 5:40 to go in the first half. Grooms then hit back-to-back jumpers to spark another string of nine straight Oklahoma points to finish the half, with Pledger's 3-pointer making it 35-27 at the break.
The Sooners' lead reached as much as 45-34 after a 3-pointer by Hornbeak in the opening minutes of the second half, just before a string of nine straight OSU points. Oklahoma State finally tied it at 56 after a pair of free throws by Nash with 9:44 remaining.