Arsenal would have provisionally supplanted leaders Chelsea on goals scored with victory and took a 20th-minute lead when Alexis Sanchez's free-kick deflected in off Williams.
But Seamus Coleman equalised with a 44th-minute header before Williams's first Everton goal ended his team's five-game winless run and sent Ronald Koeman's men up to seventh in the table.
"We showed two faces," Everton manager Koeman, who had defender Phil Jagielka sent off in stoppage time, told BT Sport.
"The first face, I hate that face because it is all about being nervous. You cannot lose the game like that.
"This was a good match though because we showed how we have to play -- with commitment and aggression. You see the reaction of the crowd when you go for it and that's what we did."
Arsenal manager Wenger said the game had been "very physical".
"We faced many physical challenges and that disturbed our game, but we were a bit unlucky to lose as well," he said.
"They were a bit scared at first, but they got their confidence. We did fight, I cannot fault that."
Meanwhile, champions Leicester City were brought down to earth after their 4-2 demolition of Manchester City as Marc Pugh's goal earned Bournemouth a 1-0 win at the Vitality Stadium.
Arsenal made only one change from their 3-1 win over Stoke City, fit-again right-back Hector Bellerin replacing injured centre-back Shkodran Mustafi, and took a 20th-minute lead.
The goal stemmed from a comedy of errors on the edge of Everton's box as Ross Barkley lost possession, Williams fouled his own team-mate Idrissa Gueye and Jagielka tripped Francis Coquelin.
- Thudding header -
Sanchez sent the free-kick curling towards Everton goalkeeper Maarten Stekelenburg and in seeking to clear the ball, Williams succeeded only in deflecting it into his own net.
It was Sanchez's 12th effort of the campaign, putting him level with Chelsea's Diego Costa as Premier League top scorer.
Everton did not look like equalising, with Romelu Lukaku and Aaron Lennon slicing shots off target, but with half-time looming they drew level.
Leighton Baines curled a right-foot cross towards the back post from the left and Coleman nipped in between Laurent Koscielny and Nacho Monreal to glance a deft header past Petr Cech.
Mesut Ozil spurned a glorious opportunity to restore Arsenal's lead early in the second half, wafting a shot over the bar from Sanchez's cut-back, while Barkley drilled wide at the other end.
The noise level rose as the game wore on and with four minutes to play Everton struck the winner, Williams flying in at the back post to send a thudding header past Cech from Barkley's corner.
Everton had Jagielka sent off after he picked up a second yellow card, but the hosts survived amid frantic late scenes that saw Arsenal substitute Alex Iwobi have a shot blocked on the line.
It was Arsenal's first league defeat since their 4-3 loss to Liverpool on the season's opening day and means they will be six points off the pace if Chelsea win at bottom club Sunderland on Wednesday.
Leicester remain without back-to-back league wins this season after defeat at Bournemouth -- their eighth of the campaign -- kept them in 14th place, just four points above the relegation zone.
Pugh marked his first start of the season with the game's only goal, slamming home the rebound after Leicester goalkeeper Ron-Robert Zieler saved from Benik Afobe in the 34th minute.
"The result is too severe against us because we created more chances than Bournemouth," said Leicester manager Claudio Ranieri.
"If we continue to fight at this level, sooner or later we will get a good result."
Eddie Howe's Bournemouth leap four places to eighth, nine points clear of the bottom three.
source: news.abs-cbn.com
