I notice that in my town sometimes the pollution/haze/smog sticks around on some days but then on others the sky can be completely clear. I don't necessarily see any difference between windy and non windy days so it seems like it is something else at work.
My guess would be a combination of boundary layer, cloud cover, and emissions (though that isn't terribly specific).
Smog is a combination of nitrogen oxides (which cause smog's brown color), ozone, and particulate matter. These in turn form from emissions of hydrocarbons and nitrogen oxides from vehicles and industrial combustion. If those emissions are lower on some days, the smog's going to be better. But that's a boring answer if you don't have a way of tracking your town's emissions.
Ozone and nitrogen oxides break down and reform in sunlight, driving a number of chemical reactions; this is why smog is usually more of an issue in the daytime. Cloud cover can reduce these sunlight-driven reactions, which can help with smog. Rain washes out most of the particulates, so even light rain overnight can help with smog.
Since the emissions are coming from the ground, they're usually trapped within the boundary layer (https://en.wikipedia.org/wiki/Planetary_boundary_layer). Air within the boundary layer is fairly well-mixed. If the boundary layer is low, then the concentrations are high and the smog is bad. Boundary layers increase in size throughout the day as the ground heats up and convection mixes higher into the atmosphere, so a number of places (including my hometown of Los Angeles) have their worst smog in the morning. How long the smog sticks around depends on how quickly the boundary layer can grow. This depends on a ton of interesting factors -- there are entire textbooks on the subject (The Atmospheric Boundary Layer, An Introduction to Boundary Layer Meteorology) -- but start with the Wikipedia article and let us know if you have more questions about it!