The temperature does indeed increase with depth, something that is a problem in deep mines or deep drilling, but a benefit for geotermal heating. The heat originates mostly from radioactive decay, but there is also a fraction of primordial heat from the formation of the Earth.
As a rule of thumb, geologists often assume 25°C/km temperature increasements within continental plates, but steeper increasements in oceanic plates or volcanic regions. The temperature curve is not constant, however. Due to the convection, the temperature increases slower with depth in the mantle and outer core. In that sense, the crust is an isolating cover. The temperature also varies due to subductions, the dynamics of the mantle and radioactivity within the crust.
The shallow soil temperature and the temperature in the air depend mostly on the insolation, the radiation heat from the sun. The average annual solar isolation is 1361 $W/m^2$ but only 20-500 $mW/m^2$ heat flux comes from Earth's internal heat production. However, this tiny fraction can sometime sbe important, e.g. to understand basal melting of ice sheets and glaciers.
Temperature gradient from crust to core. Recent research have improved the accuracy, but the principle in the figure remains (Annual Review of Earth and Planetary Sciences, 24(1), 15–40)
So why do you experience the temperature colder underground when you are digging?
The temperature at depth is the average temperature for day and night or even the year. The energy from the sun reaches only the uppermost soil layer that absorbs heat but the thermal conductivity is relatively low and it will not transport the heat down. The ground is reluctant to change its temperature regardless of variations on the surface.
How strongly you experience the temperature difference depends on where in the world you are and the local conditions. In some areas, the ground remains frozen, known as permafrost, even as the temperatures in the air can be pleasant during the summer and the surface layer melts. The depth of the suggests that it's been formed over a long time. It reaches down to the depth where the mean temperature over hundreds or thousands of years is 0°C. At larger depths, the heat from the Earth's interior takes over and melts the frost.
Temperature in the active layer and permafrost. The thickness of active layer is less than a couple of meters and the thickness of the permafrost depends on its age, the bedrock and the distance to the coast. (By German Wikipedia contributor, "HylgeriaK")
See the plot, linked by @JeopardyTempest in the comments to better understand the variations. Plotted on another scale and depth, annual variations would probably be seen as well.