The biggest controlling factors are:
- Velocity Model
- Composition (though velocity model and composition are related)
I believe you understand the first factor, so I will focus on the other two. The most common GLOBAL velocity model is PREM, which is depicted below(Image Source):
Velocity models basically assume the internal P and S velocity structure of Earth. If the crust is thinner than average in the area where the source is, the arrival times will be faster than if the crust is thicker than average, though depending on the distance this could be negligible. This is because seismic waves travel faster in than they do in the crust, so the least time tends to involve travel through the mantle.
The other controlling factor is composition, or in other words, heterogeneities within the earth. Depending on the composition the wave travels through, arrival times could be slower or faster. For example, imagine an interplate earthquake traveling through a subducting slab and into a seismometer: the arrival times are slower because the wave speed of lithosphere is slower than the wave speed of mantle. In fact, its these deviations from our velocity model predictions that help us find heterogeneity in the earth to begin with! This is essentially how we find our natural resources, though instead of using earthquakes as the source we use man made active seismic waves.