9
$\begingroup$

The context of this question is computational earth science specifically meteorology/climatology but I am sure it applies to all the scientific disciplines where numerical methods are used.

Recently I had asked the author of a peer reviewed paper for the software that was used in the publication of his paper. He not only immediately provided the software but also gave me test data to verify the results of his paper. The same was done by a climate change institute affiliated to NOAA. They gave me their software and asked me to download the relevant data from the NOAA site to validate their conclusions and perhaps reuse their software for solving my problems. In both these cases the only requirement was that the software had to be cited in any paper that I publish in the future.

These two instances are not the problem. In another couple of cases when I asked the authors of the peer reviewed paper for their software to validate their work they refused claiming that is "private". My questions are therefore:

  • Which one of these two occurrences is more common in meteorology/climatology?
  • As somebody who is originally from a CS background (where code reviews are extremely common) I'm curious what is the current standard of reproducibility in meteorology/climatology?
Improve this question
$\endgroup$
5
  • $\begingroup$ BTW, this would probably be closed & deleted as off-topic at academia.se $\endgroup$
    – 410 gone
    Sep 17, 2015 at 6:23
  • 1
    $\begingroup$ This is a very interesting question, but it feels like it's inviting a discussion — you're only going to get people's opinions — and there is no 'answer'. Journals have policies, enforced or not, and individual researchers, readers, and reviewers have their preferences. I don't know, but this feels like the wrong venue for the conversation. $\endgroup$
    – Matt Hall
    Sep 17, 2015 at 11:53
  • $\begingroup$ @kwinkunks I feel that points 2 and 4 ("which occurrence is more common" and "how does this work in Earth Sciences") should be on-topic: essentially it's asking "what are the norms for reproducibility in earth science", which at least in theory should be possible to answer objectively -- cf. academia.stackexchange.com/a/29192/32532 for a similar question on CS. On the other hand, that question was considered on-topic at academia.se despite being exclusively CS-focused, so maybe its earth science equivalent would also fit better there. $\endgroup$
    – Pont
    Sep 18, 2015 at 8:31
  • $\begingroup$ @Pont - while the current answers are fine( more like individual experiences/suggestions) the kind of answer that I want is what is presented in that question asked in academia.SE. In other words an objective assessment of what it is in ES. $\endgroup$
    – user1066
    Sep 18, 2015 at 11:11
  • $\begingroup$ @EnergyNumbers Still think this question would be closed on academia ? academia.stackexchange.com/questions/134338/… $\endgroup$
    – user1066
    Aug 7, 2019 at 10:13

3 Answers 3

Reset to default
5
$\begingroup$

Let's start with the trivially true: programming is not earth science; earth science is not programming. Programming is one tool of many in earth science. In programming, the end result is the program itself. In earth science, that's never the case: but one end result is algorithms developed that inform us about the real world, and implemented in a progam.

What if there is a bug in a peer-reviewed paper? Well, of course there is, just as there are bugs in pretty much every program that does anything useful in the real world. So what?

Validating source code is the domain of programming. Validating algorithms and observations is the domain of earth science. Re-running someone else's code with the same input data is barely any kind of validation at all, and rarely the interesting kind. Taking someone else's algorithms and implementing them yourself, that's the sort of validation I see across the physical, life, social sciences.

So I don't see someone else's source code? No big deal. Indeed, that's often a good thing: it means I can genuinely clean-room their algorithm.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ Good points. If someone wants to validate my trajectory analyses, I'd much rather see them implement their own trajectory code than just run mine again. Much more interesting to see how two implementations differ (and investigate why) than just verifying a program behaves consistently across platforms. $\endgroup$
    – casey
    Sep 17, 2015 at 12:49
  • $\begingroup$ Well, you can only get to the interesting part of investigating how two implementations differ if you have access to the code. $\endgroup$
    – Antonio
    Mar 3, 2016 at 21:18
3
$\begingroup$

In addition to the answer by farrenthorpe, the other way to try to valid someone's results is to develop procedures and computer programs yourself, from first principles.

This is a lot more tedious and time consuming and can be difficult, particularly for complex models and data. However, it is a method by which validation can be confirmed or refuted independently.

Improve this answer
$\endgroup$
3
$\begingroup$

Most Earth-Science models are comprised of several sub-models that are developed by multiple members of the international community, requiring an open-source framework to be successful long-term. However, sometimes there are model developments that are done by a consulting company which may have caveats as to how the code can be distributed. Or, you may have a grant that is funded by a private source that can dictate who has access to the final product. So, it really does depend on the source of the money and what the terms of the grant are. Some privately funded products do become well-recognized in the Earth Science community, but it is quite rare, and typically the fundamental equations are published. Sometimes its something as simple as confidential business data that is embedded in the code logic. You can get access to the executable and still do your own testing, but the code itself is private.

Usually, though, Earth Science computational models are developed with grant money that comes from publicly funded agencies like the National Science Foundation, NASA, or NOAA. In these cases, the end-product is by definition open-source and will be shared with the scientific community.

Improve this answer
$\endgroup$
4
  • $\begingroup$ thanks for the answer. So the publications whose results cannot be validated are not to be taken seriously ? $\endgroup$
    – user1066
    Sep 17, 2015 at 4:18
  • 3
    $\begingroup$ In my experience, you can still validate their results, you just can't read all their code. Sometimes its something as simple as confidential business data that is embedded in the code logic. You can get access to the executable and still do your own testing. $\endgroup$
    – f.thorpe ♦
    Sep 17, 2015 at 4:29
  • $\begingroup$ under what category is the Office of Naval Research Award ? Private or public ? $\endgroup$
    – user1066
    Sep 17, 2015 at 5:42
  • 1
    $\begingroup$ Most things concerning the military are usually private/secret/confidential. $\endgroup$
    – Fred
    Sep 17, 2015 at 5:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.