The rigors of a military school aren't easy, and at times Brown wanted to just leave and return to his Roswell, Georgia home.
"When I first got here I kept calling different people... asking them to come get me and take me home," he said. "I didn't like it at all. It was tough.
"Everybody kept telling me though that I can make it... I can get through it. Coach Keatts and the other players also made me start feeling at home and that helped me get through it."
Brown had an outstanding season for Hargrave. Handling the point guard spot for arguably the country's top prep team, he scored at will, distributed the ball to his teammates, and consistently made big plays for his team.
The 6-foot-4, 185-pounder feels he improved tremendously working under Hargrave head coach Kevin Keatts.
"I think my game has improved a lot working with coach Keatts," said Brown. "I'm bigger now after being at Hargrave for a year, and I think I've improved my one-on-one game. I've learned a lot of tricks... figured out how to handle double teams better, pressure defenses, different things like that.
"Playing point has helped me improve my ballhandling, and I'm making my decisions... reads faster. They gave me the ball here and told me to try and make plays. Now the pressure doesn't faze me."
Last weekend Brown left Hargrave to take his official visit to NC State.
"The visit was great," said Brown. "It was good spending time with Ryan [Harrow] and C.J. Leslie... they both were there visiting with me.
"The coaches treated us well, and we got to see the campus and hang out with the players. Another thing is they took us out to eat a lot. It seemed like we ate the whole time we were there... it was a great visit for me."
Brown stated that the best part of the trip for him was playing pickup with the current Wolfpack players in the Dail Center.
"It was fun playing with them because it was like we already had that chemistry," said Brown. "We competed so the games weren't boring at all... everybody was playing hard.
"I thought everybody played well, but Johnny [Thomas] really stood out to me. Him, Jordan, and Josh. Then you have Scott [Wood]. He's lights out. He was on my team and didn't miss."
NC State's three-man class is considered by most to be one of the top recruiting classes in the country and the Wolfpack returns several players who received a lot of playing time last year. Brown doesn't think chemistry will be an issue next year.
"They welcomed us in," he said of the current players. "Chemistry's going to be big for us, but it's not going to be a problem. Everyone's cool and there was no talk about taking spots... everybody seemed comfortable.
"Coach Lowe recruited us because he felt like we were the extra pieces to the puzzle he needed, but I know we already have a bunch of guys who can play here. Just like Kentucky needed pieces with John Wall, DeMarcus [Cousins], and Eric [Bledsoe], I think we can be those extra pieces."
Regarded as a five-star prospect by Scout.com, some consider Brown to be the top post-graduate in the nation. He is expect to come in and contribute right away for NC State.
Because he didn't qualify last season some have wondered if he will make it academically this year, but Brown, who finishes his school year at Hargrave this week, says he will be good to go.
"I'll be sending all of my paperwork into the [NCAA clearinghouse] after I leave here on Thursday," said Brown. "That's my last day and I'll have all my final grades in. With what my grades are looking like and my test scores, they are saying I won't have any problems.
"I even took the test the last time they gave it just in case I needed to. I'm good to go though."
Brown heads back to Georgia tomorrow and will spend a few weeks at home before moving to Raleigh in late June. He's scheduled to attend the second session of summer school, and is excited to make his next move.
"We love all the excitement... the hype that's going around," said Brown. "The fans at NC State are great. They believe in us, and we feel like we can win like they want us to next year."