The simple answer to this question is that cold seawater is denser than warm seawater, so it sinks and fills up the abyssal ocean.
The water that fills up the abyssal ocean comes from the polar regions. Here's a 1990s plot of temperature in the Atlantic, retrieved from the WOCE Atlantic Ocean Atlas:

You can see that the coldest water is coming from the Antarctic region (dark blue stuff on the left hand side of the plot) and spreading north.
If you look at 1990s salinity instead (plotted below) you can see that the water from the Arctic isn't as cold, but is much saltier (yellow part of the plot spreading from the right hand side and sitting above the water coming north from Antarctica).

When thinking about the density of the ocean, it's important to remember that the ocean isn't filled with water, it is filled with seawater. Seawater contains approximately 35 g of salt per kilogram of seawater. The density of seawater therefore depends on both the temperature and the salinity. You could write this as
$$\rho = \alpha T + \beta S$$
which is correct enough for what we need here. (Small aside, the density also depends on pressure and the coefficients $\alpha$ and $\beta$ depend on pressure, salinity, and temperature - you can see this in the curvature of the lines of constant density in this figure. Truly understanding the density of seawater is a massive undertaking.)
The salt makes two other very important things happen:
- as seawater gets colder, it keeps getting denser (but $\alpha$ gets pretty small at low temperatures, so salinity really dominates the density)
- seawater freezes at ~-2°C
This is why we see water at -0.4°C spreading away from Antarctica in the bottom left corner of that figure. It was colder when it was near the surface, but as it sank it entrained ambient fluid, and warmed up.
This explains how we get cold water down into the abyss, but why doesn't the geothermal heat flux simply warm it up?
The reason that the geothermal heat flux doesn't heat up the abyssal ocean is that it is too small. The geothermal heat flux is up to ~0.5 W/m$^{2}$. By comparison, heat fluxes at the surface of the ocean range from -200 W/m$^{2}$ to 200 W/m$^{2}$ (where positive means into the ocean in both cases). The surface heat fluxes and ocean dynamics simply overwhelm the much much smaller geothermal heat flux. Lots of models include a heat flux through the ocean floor, but because it is so small the abyssal ocean remains cold. Here's an example of the heat budget from an ocean model that includes a flux of heat through the seafloor.
So, not much hard maths in there, but I'm not sure we need it.