...which allow for exposure of various rocks, still mountains are mostly grey/brown.
The colourful minerals are apparent when you look up close. When you look from far away, the minerals are all mixed in your vision. My 2-year old daughter prepared a wonderful example of how this works:

Take a bunch of beautiful bright water colours, mix them together, and you end up having a greyish brown, dull colour.
Take another example. Sand, or sandstone. Boring yellow, white, or brown. But it's basically quartz. Finely divided quartz, unlike any of the beautiful quartz crystals you see for sale. Grain size is an important factor of the perception of colour.
That said, one of the skills geology undergraduates learn in their first year of university, is distinguishing colours in a geological context. What to the untrained eye may seem brown or grey, appears like a world of colour. I still remember that when I just started my geology B.Sc about 10 years ago, we were given a bunch of rocks that all looked grey. After one year, they did not look grey any more. They were red-grey, brown-grey, blue-grey, green-grey, etc.
Mountains can be coloured. Here are some pictures that I took myself a while ago:


On a closer look I'm certain you will see that there are many colours in there: green, yellow, orange, pink, red, brown, white, black. No need for geological education to see that.
Why isn't there a single mountain out of Olivine, Pyrite, Tourmaline
or, say, Autunite, or any other brightly coloured rock?
Most rocks are not composed of a single mineral. However, there are rocks composed only of olivine. They're call "dunites". Here's one:

See the rock behind? It's brown. That's because olivine contains ferrous iron. Once it's exposed to atmospheric oxygen, it becomes ferric iron. A better known name for ferric iron is "rust". This is generally the case for a lot of minerals. Colours commonly arise because of transition metals in a more reduced cationic state. Higher oxidation states are usually black, grey, red, and all kinds of earthy colours. The boring colours you see are commonly the result of "rusting" of rocks.
This is also the case for pyrite. It is an iron sulfide - shiny yellow. When exposed to the atmosphere, it rusts. In fact, patches of red soil in otherwise non-red soil are an indicator for a hidden ore deposit (gossan).