There are 4,660 known minerals, so why are most of them so rare?
Is it to do with the structure of minerals?
Mineral rarity is dependent on several factors that include these:
The mass-abundance of the nine most abundant elements in the Earth's crust is approximately: oxygen 46%, silicon 28%, aluminum 8.2%, iron 5.6%, calcium 4.2%, sodium 2.5%, magnesium 2.4%, potassium 2.0%, and titanium 0.61%. Other elements occur at less than 0.15%. (reference)
So minerals composed of elements other than (O, Si, Al, Fe, Ca, Na, Mg, K, and Ti) represent far less than 0.15% chemical composition of the earth.
Redistribution of elements by natural processes.
Consider the top 9 elements (O, Si, Al, Fe, Ca, Na, Mg, K, Ti), notice Chlorine is not in the list but by mass Cl should be in the top 9 for the surface of the earth. Natural processes will alter the distribution of elements.
Top elements in the oceans (71%) of earth's surface are: Oxygen, hydrogen, chlorine, sodium, magnesium, sulfur, calcium, potassium, bromine, carbon, and boron.
Atmospheric elements are: Oxygen, Carbon, Hydrogen,Nitrogen ,Calcium, Potassium
Stability (physical and chemical characteristics) of mineral species can affect how common a mineral may be.
Example: Water soluble mineral species - There are at least 90 mineral species that readily dissolve in water. (reference)
Some minerals are solution readily affected by acidic or alkali solutions.
Some minerals are inherently unstable are near surface conditions.
Mineral formation environments impact mineral distribution.
Minerals that commonly form in the mantle of the earth are naturally going to hard to find at the surface regardless of the chemical position.
Volcanic mineral assemblies require a volcanic activity which is typically limited to plate boundaries and isolated mantle hot spots.
Lastly thermodynamics (Gibbs free energy) can be used to measure the how probability of specific chemical reaction is occur.
Consider: Fe + S = FeS2 (pyrite) verses Fe + S = FeS2 (marcasite) Free energy state of an environment can be used to explain why pyrite or marcasite will form.
Some chemical reactions are very improbable due to elements involved with in a specific environment.
Cubic zirconia (CZ) is the cubic crystalline form of zirconium dioxide (ZrO2). It is not found in nature because there are no stable environments for formation where Silicon is absent. Zircons are formed instead. (zirconium silicate (ZrSiO4).
Most minerals are rare because of two reasons: They are not rare, but they inaccessible, or they require very unusual conditions to form.
There are some very common minerals in Earth or the universe that we hardly get to see. One of them is moissanite, or silicon carbide (SiC). Both silicon and carbon are very common elements in the universe and they commonly combine to form this mineral. On Earth, however, there is abundant oxygen and both silicon and carbon combine with it instead, so it's rare on Earth. The problem is that space is out there in space, so while it's common in the universe it's rare for us on the surface of our planet. Another example is ringwoodite, the high pressure analogue of olivine. Olivine is one of the most common minerals in the crust, and ringwoodite is one of the most common minerals in the mantle. However, we rarely get to see it up on the surface.
As for minerals that require unusual conditions (on Earth), that would be the majority of minerals. The four most important parameters for mineral formation are temperature, pressure (equivalent to depth), chemical environment (i.e. what elements are available) and oxygen fugacity (a measure of how oxidising an environment is). Because a lot of the rocks in the crust form in just a few combinations of these four parameters, few minerals comprise most of the minerals by volume. But then, you can deviate each one of those parameters in so many different combinations, each deviation causing the formation of a different mineral that would be extremely rare.
Here's an example: Tellurium is a rare element, but it forms rather common minerals together with silver and gold. Sylvanite ((Ag,Au)Te2) and petzite (Ag3AuTe2) are probably the two most common. Notice how there is no oxygen there: that's because the geological environments you find tellurium are not those you usually find (enough) oxygen. These minerals are called tellurides. But what if you do find oxygen? You end up having minerals with tellurium and oxygen, and those are names tellurates. These minerals are less common. What if we add some more elements that are even more rare? The Aga Mine in California is one such case where you have tellurium, oxygen, and the addition of the rare lead and copper. This causes for formation of lead-copper-tellurates, extremely rare minerals that occur in only that single mine. Look at the list of minerals found in that mine! Every mineral in bold and (TL) in its name is found only in that single mine. All of them are lead-copper-tellurates, with various proportions of the elements.