6
$\begingroup$

I have been brushing up on numerical methods for PDEs lately, and climate models provide such a rich set of models and ideas to practice with. However, I was a little confused about how academics can do research on large climate models--like the CESM, WRF, or other models produced by GDFL or NCAR--given the computational challenges of running these models? These are global circulation models, or even limited regional atmospheric models with physics happening at multiple scales in the simulation. I was not sure if most institutions have setups to run these models locally, or is there some way to send a new model to NCAR or similar institute to test the accuracy, etc.

To run large climate simulations requires supercomputers or large cluster resources, as well as delicate MPI and mapreduce operations. Since a lot of that model code is written in fortran, that just exacerbates the portability problem since there is less ability to "abstract" away some of this configuration in objects.

Writing and running an individual component model or "parameterization" model for one of these larger models seems manageable. That task just requires the scholar to write something doable over one or a few machines. But then how can that same person test his/her model inside of one of the larger climate models?

I have been doing some research on this, but have not found much. I found some descriptions of how these larger scale models are built from their development documentation. The code is available as open source, but something like the WRF has 1.5 million lines of code. So debugging it or customizing it to a different cluster seems pretty tough--though I don't have any person experience doing that with a climate model. I have also spoken with some climate folks at Caltech who are working on developing their own large scale climate models, but they could only explain the challenges that they faced with building a flexible meshing scheme, etc., The Caltech folks did not tell me about their experiences using the established large scale climate models.

Hence, I figured I would ask the SE community.

UPDATE: As per some feedback below, I just wanted to explain why this question is posted on ES instead of the Academics.SE. The thought was that Academics.SE is a much more general site across all of academia and many people there don't know about the nuances of numerical computing and the computational setup that goes along with it. Hence, I posted the question on ES where the audience is more familiar with running these types of simulations. I recognize that this question is "soft" however it seemed relevant to others conducting ES and Atmospheric research--especially those who want to do research on this topic and come from other disciplines.

$\endgroup$
5
  • $\begingroup$ This is not really a ES question. You can post this on academics. For as long as this question is here academics can apply for grants for running their simulations on supercomputers such as at Pittsburgh etc. These grants come via NSF. All of this assumes a US context. In Switzerland something similar is present with their national supercomputing center. $\endgroup$
    – gansub
    Sep 15 at 15:10
  • $\begingroup$ @gansub this is helpful. So there is a workflow for running sims on some established implementations of the models. Yeah, I thought about posting this on academics, but that is a much more general site. So I thought that the audience there might not have detailed knowledge of the computational and numerical challenges of large PDE simulations. I understand that this is a "soft" question, but it is also something that is not very clear from reading academic papers--since those might explain a numerical setup but not how the corresponding computational setup was created. $\endgroup$
    – krishnab
    Sep 15 at 15:18
  • $\begingroup$ But perhaps I can add a comment to explain why I did not post on Academics.SE, as your point is valid. $\endgroup$
    – krishnab
    Sep 15 at 15:19
  • $\begingroup$ I agree with krishnab that it seems a very reasonable Earth Science question to me, as I'd think the answer may in fair part be about the scale of the models rather than getting super computing resources itself. I know some of the highest resolution may use super computers, but I would think there's a chance quite reasonable climate models can be run at a pretty decent resolution. $\endgroup$ Sep 17 at 6:26
  • $\begingroup$ Relevant: theconversation.com/… $\endgroup$
    – Gimelist
    Sep 18 at 13:15
5
$\begingroup$

This question is a little vague, with multiple aspects, but I shall do my best.

Computing resources

Some universities have their own clusters or supercomputers. But many countries also have regional or national facilities. For example, ARCHER2 is the latest iteration of the UK's national supercomputer service. Small amounts of computing time are available for free to UK academics, but for large projects they must bid for it.

It's rare for simulations to require something of the scale of ARCHER or ARCHER2; depending on the model being run, often a departmental or institutional system of just a few nodes may be adequate, especially if one isn't in a hurry.

Building code for different systems

So long as the code has been well-written, this is not as hard as it sounds. FORTRAN is reasonably well standardised (though some codes will require the use of a particular compiler), and MPI - despite its varied implementations - is too. And most modern supercomputers are massively parallel machines whose actual compute nodes are standard x64 servers, using CPUs such as Intel Xeons. So you can do a lot of development on a desktop PC before you try to build it somewhere big and expensive. The biggest differences between different systems tend to lie in the interconnectors (the high-speed, low-latency networking between nodes), but usually the system admins will have optimised an MPI installation for that system and made it available to users, so all the details of inter-process and inter-node communications are abstracted.

Making changes or improvements

As you've identified, for most people the key is to improve one small aspect at a time, such as a particular parametrisation. Once somebody has the standard code working, then if it is well written in a maintainable manner (not a given in the scientific world) then it is relatively straightforward for them to modify one aspect and test it. The hardest thing often is obtaining measurements or other data against which to validate the new version.

$\endgroup$
10
  • $\begingroup$ Yes, this makes a lot of sense. As a non-specialist in climate models, your answer clarifies a lot. It seem like there is a lot of compartmentalization of these different modelling levels. So a person can innovate on a parameterization model or develop a new model on their computer. Once that works, they can use their local cluster for larger scale simulation (perhaps at finer resolution). If the model is useful, then someone can write the code for ARCHER2, etc. $\endgroup$
    – krishnab
    Sep 17 at 16:16
  • $\begingroup$ @krishnab sort of, but also it is often possible to run the SAME CODE on all those computers. Just faster or slower. Remember that these days, clusters and supercomputers are basically the same as your desktop computer, repeated many times. $\endgroup$ Sep 17 at 16:35
  • $\begingroup$ Good answer & I know this isn't a discussion site, ... but, as a supplementary question, is distributed computing used for serious climate modeling? $\endgroup$
    – Fred
    Sep 18 at 11:14
  • $\begingroup$ @fred by distributed computing, you mean computers in different locations working together? Not so far as I know, but I do not claim complete knowledge of the climate modelling space. In the types of models I'm familiar with - which, again, is not exhaustive - latency between nodes is important to performance. $\endgroup$ Sep 18 at 14:28
  • $\begingroup$ @Fred I think distributed computing usually refers to a cluster right, meaning a bunch of servers that are connected across a local network, and which breakup a large compute job into pieces. So it seems the answer is yes. Architecture for this can vary, but seems like someone was using the Amazon cloud for running climate models. I am not sure about distributing server over longer geographic distances--the latency would be a killer as Simon indicated. $\endgroup$
    – krishnab
    Sep 18 at 15:10
3
$\begingroup$

Another UK perspective here to supplement Semidiurnal Simon’s answer (which reflects my experience too).

The UK research community is dominated by a single family of models known collectively as the Unified Model. The code for these is owned by the UK's national weather and climate modelling center, the Met Office, and used gratis under license by academics. That license gives academics access to repositories of model source code and experiment setups shared across the MO and academic community. Simon mentioned ARCHER2, which is shared by the academics across all disciplines, but the Met Office also provides Monsoon specifically for atmospheric modelling collaborations with the academic community.

Because the MO is an operational forecast center, they put a lot of effort into making sure the code and support software run efficiently and reliably, so what academics have access to is pretty battle hardened. But the academic side also has NCAS-CMS, whose job it is to make sure the model works on machines like ARCHER2 for the whole community. All in all there’s a good level of national support for this model on these machines, and when I send a student on the training course for the model they can be running climate simulations on that hardware within 30 minutes of arriving.

something like the WRF has 1.5 million lines of code. So debugging it or customizing it to a different cluster seems pretty tough

Well, the UM has about 1.3 million lines of code and I’d estimate that I know about 15% of that code very very well (mainly a particular area of science) and the rest of it barely at all. When I encounter bugs they’re almost always because of the thing that I’ve just changed or something closely related in areas of the code I know well. When bugs lead into other parts of the model, it’s usually best to go and ask someone who knows about those areas rather than digging too hard yourself.

Writing and running an individual component model or "parameterization" model for one of these larger models seems manageable... But then how can that same person test his/her model inside of one of the larger climate models?

Yes, new parameterizations are often developed separately from the full climate model before being added to it. But the longer a parameterization is developed in isolation the more likely it is that it will be conceptually or technically incompatible with the full model. The trick is knowing early on that you want to couple it into the larger model later and to design your parameterization accordingly. In my experience, however modular we aspire to make these models, it can still be quite a pain to couple in parameterization code that’s had a well-established, independent life outside of the climate model. In general though, this is where the shared code repositories and an active community are really useful.

Since a lot of that model code is written in fortran, that just exacerbates the portability problem since there is less ability to "abstract" away some of this configuration in objects.

I remember years ago, as a student, a computer scientist friend of mine avoided doing an industry placement at a climate modelling center because he had such a low opinion of their software. "It's so basic and boring", he said, "They just use Fortran!" But those same things that are off-putting to a computing student are beneficial to the largely self-taught programmers (i.e., physical scientists) who are working with these models. Fortran is a fairly straightforward and safe language to learn and use, with relatively few concepts and gotchas. Compare that with OOP paradigms, which are hard to use well without significant training.

But those are more comments on the portability of the programmers than of the programs. As Simon mentions, the difficult bits of getting a climate model running on new hardware (big or small) tend to be handled by the support staff of that hardware rather than the academic researchers themselves.

$\endgroup$
2
  • $\begingroup$ yes this is very helpful. I can see how the ARCHER2, Monsoon, or other systems allow researchers to plug-in their models into the larger frameworks. So that makes a lot more sense. I understand the workflow better now. I can see how highly standardized the code has to be to work with the UM models/hardware. Sorry if it sounded like I was criticizing Fortran, as that was not my intent. I was really just talking about the portability of the code to other hardware or environments. $\endgroup$
    – krishnab
    Sep 19 at 18:05
  • $\begingroup$ just an additional question. So given that the UM models are so highly standardized, is it possible to experiment with different meshing strategies for a model. Like if I wanted to experiment with different discretizations or use spectral elements versus discontinuous galerkin methods? I just wanted to understand what the range of flexibility was when using these kinds of models. Or is the meshing strategy set by the system, and everyone has to use that defined method--to ensure the rest of the model pieces work? $\endgroup$
    – krishnab
    Sep 19 at 18:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.