The Earth is always radiating heat to the space. But in the day the Sun delivers some heat. The net heat flux is then defined as the sum of those two factors. If the energy delivered by the Sun is bigger than the cooling rate, the Earth is net warming (positive net flux – we can imagine it like heat is travelling "to us"), as opposed to the opposite case (cooling; negative net flux – heat is travelling "away").
In the day, the Sun warms the ground until the evening. The Sun's heating rate is higher than the cooling rate, so the temperature is rising until it gets to a point where the heating rate is same as cooling rate. This happens in the evening, so the temperature is steady at that point. But the Sun goes even lower, so the net flux becomes negative. In the night there is only cooling of Earth, so the temperature is falling steadily until the Sun is high enough that it balances the cooling. This happens at a point of a minimum temperature.
Of course, this is valid for most of the days, but we can have some other effects that can change the time of the minimum temperature (clouds, fronts or advection, for example).
On graph:

So, the temperature is falling over night after the sunset, but rises again after the sunrise. Thus, the temperature is at its lowest point in the morning.
Appendix for all you loving calculations:
Note: Simplified to toy model, no atmosphere
The cooling rate of the Earth is approximatelly given by the Stefan-Boltzmann equation:
$$j_E=\sigma\cdot T^4=5.670 \cdot 10^{-8} \frac{W}{m^2 K^4} \cdot (288.15 K)^4 = 390 \frac{W}{m^2}$$
The maximum heating rate of Sun in the zenith is $j_{\text{S max}}=1361 \frac{W}{m^2}$. So, the heating rate of Sun at altitude $\alpha$ is:
$$j_S=j_{\text{S max}}\cdot \sin{\alpha}=1361 \frac{W}{m^2}\cdot \sin{\alpha}$$
When is the heating rate equal to zero?
$$0=j_S-j_E=1361 \frac{W}{m^2}\cdot \sin{\alpha} - 390 \frac{W}{m^2}$$
$$1361 \frac{W}{m^2}\cdot \sin{\alpha} = 390 \frac{W}{m^2}$$
$$\alpha = 17 °$$
So, with our calculations, the minimum temperature is at the time when the altitude is equal to 17°.