The reference in the figure (where the plot was cropped from - the period 70M-50M years is missing) is helpful.
The resolution on that plot is quite low but it can be seen that, for the past 50M years, the temperature is not always decreasing. There appear to be four main cooling trends flanked by periods of constant or increasing temperatures.
According to the authors, the long-term cooling trend is a result of several processes that work together in a nonlinear way to modify the distribution of heat on the surface of the Earth. However, we require many more global records of temperature before we are able to quantify their respective contributions.
From the paper:
The long-term cooling over this period is thought to have resulted
from a combination of factors that altered the amount and distribution
of heat over Earth’s surface. Variations in atmospheric greenhouse
gases (principally CO2), Earth’s albedo, and linked changes in
ocean-atmosphere circulation patterns, which are also influenced by
the opening and closing of ocean gateways, could contribute to this
Lear, Caroline Helen, H. Elderfield, and P. A. Wilson. "Cenozoic deep-sea temperatures and global ice volumes from Mg/Ca in benthic foraminiferal calcite." science 287.5451 (2000): 269-272.
Actual figure from the original paper: