South Carolina went on a 12-2 run in the middle of the first half to pull away, and the Rebels never recovered.
UNLV cut the lead to eight points with 3:03 remaining, but Wallace hit a free throw and Powell drew an offensive foul on the Rebels' next possession.
Tre Kelley's dish to Rocky Trice under the basket put the Gamecocks (17-13) ahead 75-63.
The Gamecocks will host Georgetown in the quarterfinals.
Every time it looked as if South Carolina was going to put the game away, the Rebels hit a timely 3-pointer. The Gamecocks were slowed by 56.3 percent shooting from the free-throw line, but they were able to grab some offensive rebounds off the many misses.
South Carolina shot 38.5 percent from the field in the first half and used tenacious defense to force eight steals and limit the Rebels to 33.3 percent shooting.
UNLV (17-14) shot just 2-for-11 from behind the arc in the opening period, but shot 39.1 percent for the game.
The key run was highlighted by Kelley's steal off an inbounds pass, which led to an easy basket for South Carolina. The Gamecocks led by as many as 16 points in the first half and were up 37-24 at halftime.
Kelley finished with a career-high 9 assists.
Romel Beck led UNLV with 21 points.
South Carolina scored the first four points of the second half, but two 3-pointers by Odartey Blankson and Beck trimmed the lead slightly. UNLV got within 10 points on Michael Umeh's 3-pointer with 10:08 to go.
But Tarence Kinsey countered with a 3 on the Gamecocks' next possession and Kelley made a layup to put South Carolina ahead 60-47 with 7:44 remaining.
Kelley's no-look pass to Trice gave the Gamecocks their biggest lead at 41-24 about two minutes into the second half.
In the first half, UNLV went 4:22 without a basket, and finished the opening period with 11 turnovers. Blankson, who is averaging 17.7 points, scored just five and finished with 10.
This is the second time in three years that UNLV and South Carolina have met in the second round of the NIT. In 2002, the Rebels also advanced after a first-round win over Arizona State, but then fell to the Gamecocks in Columbia 75-65.
South Carolina holds a 5-1 record over the Rebels and is unbeaten at home at 4-0.
UNLV had won eight of its last 10 games, losing only to Utah and New Mexico, which both made the NCAA tournament.
Gamecocks Advance To NIT Quarterfinals
Gamecock Anthem Top Stories
New Head Coaches Mid-Season ReportThere were a lot of college football head coaching changes before this season. Let's take a look at how a few have fared so far this season.
ScoutWednesday at 4:59 PM
Scout's Top 25 Recruiting ClassesScout has a new top school in the recruiting rankings. Take a look at capsules for the top 25 classes in the country, including a new number one.
Scout FootballMonday at 10:20 AM
Updated Scout 300 Trend Meter For 2017The updated Scout 300 Trend Meter for the 2017 class takes a look at which schools are trending for each uncommitted member in the Scout 300.
UGA-USC Game Moved to SundayATHENS - Saturday’s game at South Carolina will be played Sunday afternoon.