First things first, the reasons for the Neandertal(N) extinction are not known definitely. There is though no evidence of violence (*).
There is an overlap in Europe of N and anatomically modern humans (AMH) of several thousand years, even if N find sites are limited to the Mousterien industry alone. This is a conservative limit because N may have used more advanced technology like blade production as well in their last millennia, which would extend N presence to ~32,000BP. Sites like Gorham's Cave fit into that picture, but this long chronology of N is debated and counterdebated :-)). Anyway, long before the last glacial maximum (~23 to 20ky BP) they were gone.
Somewhat outdated is the view that N is a cold adaptation of the human lineage. The appearance of tools for clothing (bone needles) drastically rise with the Aurignacien, the first stone tool industry linked to AMH in Europe. That of course does not mean N didn't use clothes, but that they probably had not the same proficiency as Cro Magnon & consorts in making them. Correlation of find sites and climate confirm the picture that N avoided cold climates. N populations were smaller than AMH, though not really endangered by genetic bottlenecks (there is newer work concerning Denisovans, stating they never really split from N, but i can't find it).
By now it pretty clear that N and AMH interbred on many occasions.
So why did they become extinct ? Together with other ice age megafauna in a highly stressed environment, loosing their way of subsistence ? Competition of top predators ? Inbreading nevertheless ? Dwindling numbers and simple merge with AMH ? Bad luck ? Unclear ...
(*) I want to add a remark here to the Shanidar 3 guy mentioned below. First of all, the authors do not exclude an accident. Second, the work has been criticized by Trinkaus et al. for methodological errors, as is cited here. Similar arguments apply to St. Cesaire 1 (healed skull fracture) is likely an accident, the argument for an intentional blow is too weak. I do not categorically exclude violence in the Paleolithic, the accusation is unfair, I only say there is no (undisputed) evidence. The Shanidar burials, as well e.g. La Chapelle aux Saints 1 are seen as examples for compassion for those in need of care among N. Btw.: we haven't been at the meat of that yet, but i am not handing out fuel against my argumentation ;-)
It was stated that
"Our ancestors killed the males of other tribes and raped the few remaining females. That our ancestors killed Neandertals who looked rather different is not a far stretch from killing other anatomically modern humans who looked only slightly different."
Both claims are not backed by science, Cro Magnon and Neandertals were not that different (reconstructions Neandertal Museum, Mettmann, Germany) and normal people don't just attack others. Hunting strategies of late N and AMH in the early Aurignacien don't differ by much.
That men were killed and woman stolen happened in very isolated cases in the Neolithic and contemporary Mesolithic, and violence also becomes evident in the Epipaleolithic/Neolithic precursors ~10kyBP , in sedentary or semi-sedentary groups, but that's not the point here. Rather not did it happen in the cold steppe of oxygen isotope stage 3 in Paleolithic Europe for obvious reasons: when the loss of a hunter could endanger the whole group; and women - having one child at a time to keep up the pace of mobility - moved around with the group. At the time 40kyBP population was low, it must rather be assumed that any accidental meeting was a cause of celebration and interchange, as is reflected in the genome.