The most obvious reason why the South pole is colder than the North pole is altitude. The Arctic is mostly covered by ocean and low lying land whereas the Antarctic continent is really quite high, with an average height of about 2,400m. The environmental lapse rate in the troposphere is about 6.5°C/km and so this corresponds to about a 15°C temperature difference between the Arctic and Antarctic based on altitude alone. Obviously the actual temperature difference is greater than this so there must be something else at work. Most of Antarctica is only this high because it has a couple of kilometres of ice sitting on top of it and so this doesn't explain why there is a huge, permanent (at least for the last ~35 million years) ice sheet sitting there in the first place.
There are two main reasons why a large ice sheet can persist at the South pole. First is that the southern polar region is currently covered by a large continental land mass (Antarctica). Land has a much lower heat capacity than water and so in the Arctic, the ocean acts as a large reservoir of heat, mainly transported there from lower latitudes and this limits the sea ice thickness. Conversely, in Antarctica snow can accumulate on the land and, if it does not melt in the summer, it will be buried next winter, leading to ice accumulation.
The second reason is the existence of the cold Antarctic Circumpolar Current which flows from west to east and is the largest ocean current in the world, with a mean transport of 100-150Sv. This cold current, transporting huge volumes of water, serves to thermally insulate Antarctica from the warmer subtropical waters to the north, cooling the continent. The openings of the Drake Passage and the Tasman Gateway, the exact timings of which are debated but are generally thought to have both happened by about 30 million years ago, allowed the formation of the ACC and are roughly coincident with the onset of permanent glaciation in Antarctica.
An additional factor in the present day, as mentioned @winwaed, is that southern hemisphere winter solstice currently coincides with aphelion (roughly), which produces colder winters in the southern hemisphere, allowing extreme temperatures of -94.7°C to be reached. The flip side of this is that summer solstice coincides with perihelion, which produces higher summer temperatures.
