In addition to the answer above (and I realize this is an old question and perhaps better for world building), but what we know about climate, some subtle changes can have pretty big effects.
If you increase the axial tilt a few degrees that would make the variation in sunlight winter to summer greater and would make colder winters and warmer summers, but the effect of warmer summers is that glaciers would have a harder time growing. You could also increase the planet's greenhouse gas.
You could also create more ocean surface area and less land. Oceans are good temperature stabilizers and with strong ocean currents, you might get a year-round warm water flow towards the continent on the pole, leading to a warming of the land on the pole, much the same way Europe is warmer than similar latitudes in other places on earth, due to the warm mid Atlantic current. More ocean surface area and a strong oceanic conveyor might be the simplest explanation, creating a regular stream of warm air that blows onto your continent on the pole.
Or you could give the planet a slower rotation and longer days (colder nights), but during the summer, snow would melt during the longer day - also preventing glacial buildup. For your scenario, glaciers on Mountains is fine, but you don't want glaciers spreading across land like we have in Greenland and Antarctica. For a warmer pole continent, you need some combination of effects that would prevent glacial growth.
Glacial growth on land and a decline in oceanic circulation is thought to be the cause of the ice age period that began about 2.6 million years ago. Similarly, land over the pole is thought to be what lead to the Karoo ice age. You'd need a mechanism to make your land on the pole warmer (like some of the mechanisms above).
On river runoff:
River water usually (but not always) requires snow melt. If you have a lot of snow that melts over the year (like the Himalayas), that would create a lot of large rivers but I don't think those rivers would warm the land very effectively. Rivers respond to land temperature and snow/ice melt and weather patterns, more than they control them. Other methods like more greenhouse gas, a bigger ocean with a strong conveyor or some orbital changes are the best methods I can think of for your scenario.