I also mixed Longitude and Latitude until I heard someone saying '... at high latitudes.' in a conference talk.
In my mind, the picture of the Earth shows the Earth with the North and South poles at the top and bottom (or reversed, doesn't matter). High describes a vertical orientation. Therefore, the latitude changes in North-South direction.
Additionally, because the horizontal degree scale has a discontinuity at 360 degree (0 degree), it does not make much sense to distinguish high and low values in horizontal direction. Hence, high values are in vertical direction (= latitude).