It may be difficult to ascertain the hardness of a mineral relative to an other mineral in a Mohs streak test if one (or both) are not homogeneous. Already a small local grain of an impurity may put you off; you might remember the occasions of the sudden scratching sound of chalk on the green board. Which is why pen-shaped hardness picks are advantageous: to use a tip made a (synthetic) alloy with reproducible hardness (except for #2 and #3 made of plastic, or copper) to aim right on your region of interest:
Equally, it is a question of technique (if adhering to this protocol), too, as mentioned by, e.g.,
«One must also use enough force to create the scratch (if you don't use enough force even diamond will not be able to scratch quartz - this is an area where practice is important).»
(credit to Mineralogical Society of America, emphasis added)
«Don't be wimpy! This is a very common problem. Some people casually rub one specimen back and forth against another and then look for a mark. That is not how the test is done! It is done with a single, slow, determined motion, with firm pressure, with the goal of cutting a scratch.»
(credit to geology.com, compiling some potential pitfalls)
while both a) the relative orientation of the mineral to be tested .vs. a reference, and b) the origin of a mineral may yield different degrees of hardness.
(Hardness by Mohs (and streak) isn't the only scale. In materials science, there equally are hardness tests by indentation e.g., Rockwell or Brinell.)