After Benteke netted his first goal in 20 games in the 17th minute, Mesut Ozil equalized at the start of the second half.
But Benteke set up Wilfried Zaha to restore Palace's lead in the 61st before James McArthur headed in the visitors' third.
Pierre-Emerick Aubameyang scored in the 77th but Arsenal couldn't find two more goals to extend its 10-match winning league run at home.
Arsenal remains in the fourth Champions League qualification spot but it is only ahead of Chelsea on goal difference with four games remaining. Manchester United is two points behind in sixth.
Palace rose to 12th place on a day when Shkodran Mustafi endured a day to forget in the Arsenal defense.
Mustafi let Benteke stroll into the box unmarked and head in the opener, wrongly thinking the striker was offside.
Benteke had a second effort chalked off for offside after Zaha's fine throughball as Palace continued to press.
Palace could have doubled its lead before the break, with Cheikhou Kouyate forcing a double save from Bernd Leno, then Benteke hit over from Zaha's fine tee-up.
Arsenal emerged transformed after the break, Ozil leveling almost immediately. Ozil ghosted onto Alexandre Lacazette's deft reverse ball and netted at the near post to level.
Mustafi's next mistake cost Arsenal dear, however, as Zaha raced onto Benteke's flick and slotted home.
Mustafi could only watch on as Zaha wriggled round him to stroll through on goal, having incorrectly expected goalkeeper Leno to race off the line.
Palace bolstered its lead when McArthur nodded home from a corner.
Aubameyang netted a neat second for Arsenal to tee up a tense finish, trading passes with Lacazette before prodding home. But there was to be no recovery.
More AP soccer: https://apnews.com/apf-Soccer and https://twitter.com/AP_Sports
Copyright 2019 The Associated Press. All rights reserved. This material may not be published, broadcast, rewritten or redistributed.