In "An Introduction to Seismology, Earthquakes and Earth Structure", authors Seth Stein and Michael Wysession made the statement: "Faulting will occur along the plane where the shear stress is the highest". However, no further clarification was given. What are the implications of this statement? Is the statement true? What are the processes that cause faulting to occur along the plane of maximum shear stress?
This statement is referring to depths within the crust where the lithostatic pressure is sufficiently high to suppress the formation of dilatant fractures. Rock is a brittle material, and if loaded to failure while under little or no confining pressure, it may fail by the growth of mode-I fractures oriented normal to the least compressive stress. At depth in the crust, the overburden weight of rock may cause sufficient pressure to suppress the growth of mode-I fractures, but if the stress is sufficient to cause mode-II fractures, these tend to grow in the plane of maximum shear stress.
This statement is known as the Wallace-Bott hypothesis. It is mostly valid for reactivated faults rather than fresh rocks(where cohesion comes into play). It says that the slip in a fault plane will occur along the direction of maximum shear stress.
So we have a stress vector acting on a fault plane(the direction given by the direction of force), it has two components-the normal component and the shear component. The normal component is responsible for opposing the slip as frictional force is proportional to the normal. And the shear force will be responsible for the slip to occur. This direction of shear stress is the direction of maximum shear stress and is dictated by the stress vector. (Note that it can have several components in the fault plane none of which will be maximum).
Just think on the lines of high school physics involving friction, and it'll be easier to understand.
Regarding Coulomb's criterion, although it gives an equation which relates shear strength, normal stress, cohesion in rocks and the angle of internal friction(You can google the equation), a research paper by Byerlee(1978) suggests that it is reasonable to take the shear stress/normal stress ratio as a fixed high value(around 0.85) for geologically realistic condition.