The reason is that the horizon is paler is that we are seeing all wavelengths of sunlight equally scattered into our eye from along the line-of-sight to (and beyond) the horizon, so the horizon appears to be the same color as sunlight, i.e. white.
This is because for far enough distances, the higher probability of short wavelength ("blue") light scattering in our direction from distant air is cancelled out by the same scattering processes causing the light to be attenuated as it travels through the air to your eye.
This explanation is essentially the same as given on page 269 of M.G.J. Minnaert's "Light and Color in the Outdoors".
If we are looking towards the horizon, what we see is sunlight scattered from the air along our line-of-sight. For simplicity, let's assume a constant density uniform atmosphere along that line of sight, which isn't a terrible approximation since the line-of-sight is close to the earth's surface until it passes well beyond the horizon. The amount of sunlight scattered from a small section $dx$ along that line-of-sight is proportional to $dx/\mu$, where $\mu$ is the scattering length in air. If the section is a distance $x$ away from us along that line-of-sight, the fraction of that scattered light that will reach our eye will be $e^{-x/\mu}$.
The total amount of light we will see scattered back to us from from along that line of sight will be proportional to
$$\int_0^D e^{-x/\mu} \frac{dx}{\mu}=1-e^{-D/\mu} \xrightarrow[D>>\mu]{}1$$
where $D$ is the distance to the farthest scattered light we can see.
(I first thought there should be a $1/x^2$ factor as the light source gets farther away, but this is cancelled out because for a fixed viewing solid angle, the observed source volume increases as $x^2$.)
As long as $D>\mu({\lambda})$ for all visible wavelengths, if the scattering probability for a given wavelength is larger, more of that wavelength is scattered from the element $dx$, but also more is scattered away along its path to you, and the two effects cancel out. The wavelength dependence of the scattering drops out of the integral and what is see is the colour of the initial sunlight.
Assuming the atmosphere ends at the earth's atmospheric scale height of $\sim 8.5$ km, the assumption of $D>\mu$ seems reasonable since on a perfectly clear day our line-of-sight in air extends over 300 kilometres beyond the horizon. This is longer than the Rayleigh scattering length for green light ($532$ nm) of about $100$ km.
The actual scattering lengths will usually even be shorter because of significant Mie scattering from aerosols such as water droplets and dust between us and the horizon.
Even if the distances are not enough to make the horizon pure white, it will still be much paler than the blue sky overhead.
(The reason the sky overhead looks blue is because there isn't enough distance (i.e. $D_{sky} \lesssim 8.5\,\textrm{km}<\mu(\lambda)$) for the scattering along the line-of-sight to cancel out the larger initial scattering of blue light.)