Marcus Paige missed two contested 3-pointers in one late possession and the Tar Heels (4-2) fouled Swing, who gave the Blazers a 62-57 lead.
Brice Johnson drove for a quick basket but Williams effectively put the game away when he made the second of two free throw attempts.
The Tar Heels were coming off a win over No. 9 Louisville to end the defending champions' 21-game winning streak, but coach Roy Williams couldn't beat his protege this time.
UAB second-year coach Jerod Haase played for Williams at Kansas and spent 13 years on his staff for his alma mater and the Tar Heels.
It was UAB's first win over a ranked team since beating No. 20 Butler in 2009, and it came against Haase's mentor and one of college basketball's most storied programs.
J.P. Tokoto led North Carolina with a career-high 16 points on 7-of-11 shooting. Paige had 13 points but made just 6 of 16 shots and missed all six 3-point attempts after scoring a career-high 32 against Louisville.
Nobody else picked up the slack in perimeter shooting for the Tar Heels, who made 1 of 12 attempts from beyond the arc.
The Blazers, who lead the nation in total rebounds, dominated the boards 52-37 to overcome shooting troubles. They made just 19 of 62 shots (30.6 percent) but collected 21 offensive rebounds.
Frazier made 12 of 14 free throws and hit three 3-pointers in five attempts for UAB. Rod Rucker added 11 points while Fahro Alihodzic had 13 rebounds and eight points.
UAB also made 22 of 28 from the line.
UAB led 39-25 early in the second half before a long field goal drought helped North Carolina come back.
Alihodzic snapped a 7:24 stretch without a made shot when he banked one in for a 54-48 edge with 3:27 left.
North Carolina had chances late but couldn't convert.
Paige scored on a drive, was fouled and hit the free throw to cut UAB's lead to 58-57 with 41 seconds to play.
Five seconds later, Frazier answered with two foul shots and North Carolina called time out.
Paige got a second chance after missing a 3-pointer from the top of the key but came up well short launching another attempt over two defenders.
The Blazers put it away on the line.