To extend @Poutnik's invisible river analogy, the cold air can also 'pool' in small dips.
On clear nights, temperature fluctuations are quite a frequent occurrence due to radiative cooling. If the air cools sufficiently to reach the dew point, then cloud can act as a 'tracer' for the air pockets.
An example of this are fog patches that you potentially pass through in certain conditions. The air in the fog patches is a little colder (or higher absolute humidity, for example over a lake) than the surrounding fog-free areas. Even if the air wasn't cold enough to form cloud, the same temperature variation can occur invisibly.
Here's an example from Wikimedia user Simo Räsänen:

In the mountains, the invisible rivers of cold, descending air that @Poutnik mentions can sometimes be seen by tracer clouds flowing down hillsides. Colder air is denser than the surrounding air and has less friction so gravity pulls it down. The areas with cloud will be colder than the adjacent cloud-free zones. This effect can lead to strong down-slope katabatic winds. In Antarctica this effect is responsible for some of the strongest winds in the world.
Here's a good example from Wikimedia user Andrew J. Kurbiko:

To summarise, the key to the effect you noticed is almost certainly radiative cooling and the movement of colder denser air. Radiative cooling occurs due to any object above absolute zero giving off radiation. On a clear night this radiation passes through the Earth's atmosphere and is lost into space. This cools down the object (land) that the radiation came from. The land, in turn, cools down the air above it. On a cloudy night, the radiation is reflected by the clouds and reabsorbed by the land keeping the temperature constant.