You are looking for a portable XRF.
It does (almost) exactly what you said. It sends electromagnetic radiation to the rock (X-rays), which excites electrons in the atoms and when they bounce back they return X-ray of different energies which are detected by the instrument. It looks like this, for example:
Using the different peaks you can figure out what the rock is made of.
This is not as good as lab-based XRF, but it's so easy to use it's now standard practice for exploration and mining geologists to use it.
Note that this will only give you surface information. It analyses what it can see. If you want to know what is inside the rock, you have to drill or break it.
If you want to know what is inside the rock without breaking it, you can use a CT scanner, but it only gives information on density. Inferring what the material is based on its density can be a challenging task and it is made easier if you have pre-existing information on what it could be. There are no portable CT scanners however (at least not hand-held). They can me mounted on trucks though.
There are other methods, such as infrared, Raman, LIBS, etc etc.
The Mars Curiosity Rover has quite a lot of those instruments on board. It is essentially a semi-automated geologist exploring another planet.