The two biggest changes are fairly straight forward. Ice cover and Ocean area. During the peak of the last ice age, ice extended over most of Canada, maybe half of Russia and parts of the Northern United States and snow probably extended well south of that during winter. Ice has a high albedo.
As glaciers grow, ocean cover decreases. The yellow on this map indicates loss of ocean cover and increase in land. Oceans have low albedo, which means shrinking oceans also raises Earth's albedo. As a rule, during glaciation periods, Earth's albedo is higher and more light gets reflected directly back into space, which fits with the planet getting colder. This is one of the ice age feedback mechanisms.
Source of image.
Other factors may be less predictable, like deserts have comparatively high albedo and forests, lower albedo and the Sahara, for example, becomes partially forested over depending on variations in weather patterns driven primarily by Milankovich cycles (not necessarily in sync with ice age cycles), but I suspect, compared to the ice and ocean surface area changes, desert to forest changes have less effect.
Today's global warming is expected to (and already has) decreased ice and show cover, which lowers albedo and this trend is expected to continue. Over time, enough glacial ice might melt that oceans become measurably larger as well. This ice melt and ocean expansion lowers albedo further and is one of the positive feedbacks that effects and is expected to continue to effect man made climate change.