There are these rocks we have -- not too common, usually find them in loads of small rocks, very seldom at random. They are perfectly oblong, and sort of smooth, but not quite as smooth as a river stone, no, but a low grit sand-papery feel. They are almost hollow: within their centers are a color (usually red or orange,) that is, in my experience, typically dry-powdered. I have heard of, from one person, them occasionally still having the liquid within. They can be easily broken open by hand -- we have always called them "Paint Rocks --" the innards can, again, be used as a pigment in mineral paints. Unfortunately, I cannot say for certain where the literal ton of gravel I found these in came from, I can say this that they do seem to be a natural rock here in Louisiana -- I assume the ton of rocks from which I found these were from here in Louisiana, but I know not for certain. These pictured aren't actually that good of examples for perfect shape, but they are the same kind of rock. The "Pigment" within looks almost like clay.The "pigment" can be broken up mostly into a fine powder, but some chunks stay that can't be crushed up and are hard as, well, a rock! But, amazingly, when a little water is added, ALL suspends -- amazingI do say, the paint from this sample I've had is lovely, and in person looks like rather pure gold, but glitters not -- even in the picture it, though darker, looks golden.
I am not sure how the rock itself is called, but the pigment inside is most likely ochre. It is a mixture of several iron oxides. Yours seem to be dominated by Fe(OH)3 or similar mixtures of hydrated Fe3+ oxides called limonite. These are commonly known as "yellow ochre".
Because the individual grains of the ochre are so small, they stick together very well. This is why sometimes it's hard to break it down to powder. But as you've noticed, adding water just turns it into a paste. It is also possible it contains clay minerals. These have a tendency to stick together when dry. One way to know if you have clay in there is to make a paste and then wait until it's dry. Do you see mud cracks? If yes, it also has clay inside.
Ochre has been used as a pigment since prehistory, also in your area. It is the pigment used in a lot of rock art and cave paintings all over the world. There are three main reasons for that: it sticks, it's abundant, and it's really easy to prepare. Just mix it with water. It's also non-toxic*!
As for why it occurs in this form you're finding it, I can only speculate because I don't know much about the samples or where you find them from. One probably explanation I can think about is that these began their "life" as iron sulfides, maybe pyrite or marcasite. As these are exposed to oxygen in near-surface environments, the reduced iron and sulfur are oxidised. Iron bonds with water and oxygen to form the ochre, and sulfur is washed away as sulfate. It is possible that the crust around the ochre is some sort of sulfate, maybe calcium sulfate (gypsum?). The crust can also be calcite, or some sort of silica.
Some interesting reading: Ochre - The Oldest Known Natural Pigment in the World
*Regarding non-toxicity: ochre is my best guess. There are other naturally occurring compounds with similar properties that are toxic. Lead oxide(s) or cadmium oxide are two that come to mind. I wouldn't eat your powder, and be sure to thoroughly wash your hands when you're done. I think it's unlikely to be those, but until you're not certain - you're not certain.