Suppose I could drill a hole to mantle at any arbitrary point in the crust, will magma start to rise from the hole and finally become a volcano?
As Bon says, the mantle is solid, albeit somewhat viscous / plastic under the prevailing pressure and temperature conditions, so your hypothetical borehole isn't going to puncture any liquid magma chamber. Could it liquify if suddenly depressurized in a borehole? I doubt it, although it might do so if there was a local concentration of volatiles to depress the melting temperature.
But the question is forever going to be hypothetical because there is no drill string capable of cutting into the lower crust, let alone the upper mantle. The heat would soften the drilling bit to the point where it was like putty. Also, the lateral rock pressure would be so high that the borehole walls would implode.
The mantle itself is solid, and melting only happens under special conditions like in subduction zones. This molten fraction is the magma, which can then rise closer to the surface and into the lithosphere, creating pockets of magma that can be penetrated by not too deep drilling.
A part of OP's question is whether, if we hit magma while drilling a hole, this magma would rise and erupt resulting in a volcano. The answer is "not necessarily". Some kinds of magma are relatively tame and don't have too much trapped gas in them.
There have been several encounters of magma while drilling. One of them (the first time) was on the Big Island of Hawaii. Here's what the result was:
Workers at Ormat, one of the world's largest geothermal producers, discovered the magma in October 2005 when they hit a chamber of the magma about a mile and a half down while drilling an injection well.The substance quickly rose about 20 feet up into the drill hole before becoming glasslike as it cooled. Ormat workers redrilled the area several times, with the same result.
The article in Nature cites the magma chamber depth of 2.5 km, so it's not really deep, and quite far from the mantle. But this already gives us some understanding of behavior of some kinds of magma.
Actually we can answer the question experimentally because the upper mantle is exposed in some places, such as ocean floors. Such exposures are not exceptionally volcanic by any means. Instead their main property is the mafic/ultramafic minerals that make up mantle rock weather quickly upon exposure to the atmosphere. A mantle outcropping will be more likely covered with rust (formed from the oxidation of the iron-bearing components of the minerals) than with lava.
Photos of mafic rock weathering are shown below from northwestern Russia ; weathering of exposed mantle rock, also mafic, would likely appear similar.
1. Ekaterina V. Vasyukova, Priscia Oliva, Jerome Viers, Francois Martinal, Bernard Dupré, Oleg S.Pokrovsky. "Chemical weathering of mafic rocks in boreal subarctic environment (northwest Russia) under influence of glacial moraine deposits", Chemical Geology 509 (30 March 2019), 115-133. https://doi.org/10.1016/j.chemgeo.2018.12.033