Why can I look directly at the sun during a sunset (with a lot less pain and unconfort) but not at noon?
At Noon, sunlight is coming from nearly directly above you, and goes through ~2 miles (4km) of reasonably dense atmosphere to reach you (Atmospheric pressure at 2 miles is about one half of sea level pressure). At sunset (and sunrise), since the sunlight is traveling mostly parallel to the surface, it travels through dense atmosphere for considerably longer. When traveling through dense atmosphere it's subjected to more scattering, which decreases the observed intensity of the light.
Or, considered a different way, when looking at a sunset/sunrise, you don't see the sun as a single well defined point. It's hazy, with the bits of sky near to the sun also giving off a fair amount of light. That additional light is some of the light that was scattered from an otherwise direct beam. If it were all re concentrated to a single point, it would be brighter/more painful to look at (though not as bright as the noon sunlight- for shaky statistical and light absorption reasons).
Using an Earth Curvature Calculator, sunlight at the horizon travels through ~120 miles of 'dense' atmosphere (ie has an altitude <2 miles). That will fall off rather quickly the higher above the horizon the sun is.
(dense atmosphere in this refers partly to just thicker air; but also to particulates/dust in the air, haze, mist, fog, etc)
An excellent explaination found here: https://www.youtube.com/watch?v=ZD1roP5wISU
Goes into detail about Rayleigh dispersion.