Cleveland (4-9) has dropped four in a row, the last two by a combined five points, and seven of eight.
"This was a bad loss. A tough one to take," Haden said. "This one hurts for real."
After missing Cleveland's previous game with a concussion, Jason Campbell passed for 391 yards and three touchdowns, including an 80-yarder to Josh Gordon with 1:25 left in third that put the Browns up 19-3.
Campbell's 4-yard TD pass to tight end Jordan Cameron made it 26-14 with 2:39 remaining, but that's when Brady took over.
"How many times has he done that in his career?" Browns coach Rob Chudzinski said. "We had our opportunity and we weren't able to close the game out how we wanted to.
"I'm sick about the outcome of the game."
The late comeback ruined another brilliant game by Gordon, who had seven catches for 151 yards. It was the third time in four weeks that he scored on pass plays of more than 70 yards.
Gordon set an NFL record with 774 yards receiving in four consecutive games. Calvin Johnson set the previous mark of 746 this season. Gordon also set a single-season team record for yards receiving with 1,400. Braylon Edwards established the previous mark of 1,289 in 2007.
But it wasn't enough - again.
The Browns had one last chance, but Billy Cundiff missed a 58-yard field goal attempt on the final play. Cundiff's calf had cramped up earlier in the game and he didn't kick off following Cameron's touchdown.
A pair of late penalties on the Browns fueled the Patriots' rally.
Jordan Poyer was called for unnecessary roughness on Edelman while making a hit near the back of the end zone, allowing the Patriots (10-3) to kick off from the 50-yard line, rather than their 35.
"I was there on that one," Haden said. "He hit him directly with his shoulder in his chest. It didn't look like a foul at all."
Those 15 yards became significant when Kyle Arrington recovered the onside kick at the Cleveland 40, falling on the ball after Cleveland's Fozzy Whitaker touched it.
Leon McFadden was then penalized on the game-winning drive for defensive pass interference against Josh Boyce in the end zone, putting the ball at the 1. McFadden and Chudzinski disagreed with the call.
"I felt like those two were both jostling for the ball," Chudzinski said. "It really doesn't matter what I think."
After Amendola's touchdown, the Browns moved to the Patriots 40 with 1 second left, but Cundiff's kick came up short.
Patriots tight end Rob Gronkowski departed with a right leg injury after he was hit by Cleveland safety T.J. Ward after a 21-yard completion in the third quarter. He missed the first six games of the season recovering from offseason surgeries on his left forearm and back. In his six full games after returning, he led NFL tight ends with 37 catches and 560 yards receiving.
"It's kind of being caught between a rock and a hard place," Ward said of new rules discouraging helmet-to-helmet hits. "It's a decision you have to make but you have to follow the rules at the same time.
"Gronk's a big dude. He's not small by any means. So, already he has that height. It just makes it difficult. I made a tackle. Unfortunately he got hurt."
The Browns led 6-0 at halftime on field goals of 43 and 37 yards by Cundiff.
NOTES: Brady completed 32 of 52 passes for 418 yards, two touchdowns and one interception. ... Campbell was 29 for 44 with no interceptions.