Guard Sam Grooms, netting a career-high off the bench, was one of three Sooners to put in 18 points, but it still wasn't enough.
And in freshman guard Buddy Hield's absence, freshman guard Je'lon Hornbeak also reached double figures with 10 points, but even that couldn't get them over the hump.
That's because 17th-ranked Oklahoma State's Marcus Smart and Le'Bryan Nash poured in 28 and 26, respectively, and guard Markel Brown scored 14 and guard Phil Forte added 10 to boost a massive output against the conference's best defensive team.
A big moment came when Sooner starting forward Amath M'Baye picked up his fifth foul on a questionable and-one call with 1:34 remaining in regulation.
Though Smart was unable to come up with the bonus free throw, it was a damaging dint in OU's chances.
Even so, the Sooners still had a potential chance to ice it late when Brown missed a jumper, but Smart came up with the rebound and was fouled with 42 ticks left and made a pair of free throws to tie it up.
Both teams missed chances to potentially win it inside the final 10 seconds.
OU commanded overtime, never trailing until Nash's plus-one play at the 1:37 mark.
The Sooners got back on top at 79-78, but the Cowboys finished off with the game's final six points.
Oklahoma State leapt out to an early 19-8 lead in the contest, but OU finished the first half with 27-8 run to take an eight-point lead at intermission.
OU (16-8, 7-5 Big 12) falls to sixth place in the Big 12 Conference standings after potentially having a chance to sneak into a first-place tie heading into the day.
It'll get back to action with a 6 p.m. Wednesday contest at Texas Tech.