I do have a large dataset containing nearly a decade of particulate matter measurements. At the lower precision limit of the measuring instrument, there are values given like < 8. For statistical analysis however, a defined value is needed.

What is the best possible way to deal with this?

  • Replacing it as missing values?
  • Replacing it by 8 or even 7 (< 8)?
  • Assuming 0?
  • Assuming the mean value between something like 0 and 8; e. g. 4?

Or is there a better solution, maybe considering the natural background value?

EDIT: I'd like to calculate the intra-day- and intra-year distribution, as well as searching for patterns by considering wind directiond and speed.

  • 1
    $\begingroup$ It depends on what you want to measure/calculate. So we need more info. $\endgroup$ – Jan Doggen Oct 3 '16 at 10:05
  • $\begingroup$ @JanDoggen: Added some more information, thanks! $\endgroup$ – Arne Oct 3 '16 at 10:39
  • 1
    $\begingroup$ Perhaps one standard deviation around the limit of determination would suffice, but it depends upon the range of the dataset you are using, and the accuracy / precision of whatever you are trying to analyse. If the limit of determination, say 7 or 8, has a high frequency relative to other values, then you may have a statistical problem from too uniform a lower limit, in which case, counter-inuitively, it may be beneficial to introduce synthetic randomness into your lowest value. I sometimes do this when I have to synthesize missing rainfall data. $\endgroup$ – Gordon Stanger Oct 3 '16 at 13:03
  • $\begingroup$ In addition to Gordon Stanger's comment: replacing <8 values by missing values might alter the distribution but replacing < 8 values by noise (or replacing it by other non-missing values) might yield artificial correlations. Therefore, it might be reasonable to do the noise and the missing-value approach and get a feeling on which one is actually working 'better'. If there are less than approximately 5% (not a fix threshold) I would suggest to use missing values. $\endgroup$ – daniel.heydebreck Oct 4 '16 at 7:45
  • $\begingroup$ Maybe consider changing the title of your question. Reading the title, I assumed that the question would be focused on programming-related. $\endgroup$ – daniel.heydebreck Oct 4 '16 at 7:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.