How do academics/institutions run large climate simulations given the computing challenges?

Question

I have been brushing up on numerical methods for PDEs lately, and climate models provide such a rich set of models and ideas to practice with. However, I was a little confused about how academics can do research on large climate models--like the CESM, WRF, or other models produced by GDFL or NCAR--given the computational challenges of running these models? These are global circulation models, or even limited regional atmospheric models with physics happening at multiple scales in the simulation. I was not sure if most institutions have setups to run these models locally, or is there some way to send a new model to NCAR or similar institute to test the accuracy, etc.

To run large climate simulations requires supercomputers or large cluster resources, as well as delicate MPI and mapreduce operations. Since a lot of that model code is written in fortran, that just exacerbates the portability problem since there is less ability to "abstract" away some of this configuration in objects.

Writing and running an individual component model or "parameterization" model for one of these larger models seems manageable. That task just requires the scholar to write something doable over one or a few machines. But then how can that same person test his/her model inside of one of the larger climate models?

I have been doing some research on this, but have not found much. I found some descriptions of how these larger scale models are built from their development documentation. The code is available as open source, but something like the WRF has 1.5 million lines of code. So debugging it or customizing it to a different cluster seems pretty tough--though I don't have any person experience doing that with a climate model. I have also spoken with some climate folks at Caltech who are working on developing their own large scale climate models, but they could only explain the challenges that they faced with building a flexible meshing scheme, etc., The Caltech folks did not tell me about their experiences using the established large scale climate models.

Hence, I figured I would ask the SE community.

UPDATE: As per some feedback below, I just wanted to explain why this question is posted on ES instead of the Academics.SE. The thought was that Academics.SE is a much more general site across all of academia and many people there don't know about the nuances of numerical computing and the computational setup that goes along with it. Hence, I posted the question on ES where the audience is more familiar with running these types of simulations. I recognize that this question is "soft" however it seemed relevant to others conducting ES and Atmospheric research--especially those who want to do research on this topic and come from other disciplines.

Answer 1

This question is a little vague, with multiple aspects, but I shall do my best.

Computing resources

Some universities have their own clusters or supercomputers. But many countries also have regional or national facilities. For example, ARCHER2 is the latest iteration of the UK's national supercomputer service. Small amounts of computing time are available for free to UK academics, but for large projects they must bid for it.

It's rare for simulations to require something of the scale of ARCHER or ARCHER2; depending on the model being run, often a departmental or institutional system of just a few nodes may be adequate, especially if one isn't in a hurry.

Building code for different systems

So long as the code has been well-written, this is not as hard as it sounds. FORTRAN is reasonably well standardised (though some codes will require the use of a particular compiler), and MPI - despite its varied implementations - is too. And most modern supercomputers are massively parallel machines whose actual compute nodes are standard x64 servers, using CPUs such as Intel Xeons. So you can do a lot of development on a desktop PC before you try to build it somewhere big and expensive. The biggest differences between different systems tend to lie in the interconnectors (the high-speed, low-latency networking between nodes), but usually the system admins will have optimised an MPI installation for that system and made it available to users, so all the details of inter-process and inter-node communications are abstracted.

Making changes or improvements

As you've identified, for most people the key is to improve one small aspect at a time, such as a particular parametrisation. Once somebody has the standard code working, then if it is well written in a maintainable manner (not a given in the scientific world) then it is relatively straightforward for them to modify one aspect and test it. The hardest thing often is obtaining measurements or other data against which to validate the new version.

Answer 2

Another UK perspective here to supplement Semidiurnal Simon’s answer (which reflects my experience too).

The UK research community is dominated by a single family of models known collectively as the Unified Model. The code for these is owned by the UK's national weather and climate modelling center, the Met Office, and used gratis under license by academics. That license gives academics access to repositories of model source code and experiment setups shared across the MO and academic community. Simon mentioned ARCHER2, which is shared by the academics across all disciplines, but the Met Office also provides Monsoon specifically for atmospheric modelling collaborations with the academic community.

Because the MO is an operational forecast center, they put a lot of effort into making sure the code and support software run efficiently and reliably, so what academics have access to is pretty battle hardened. But the academic side also has NCAS-CMS, whose job it is to make sure the model works on machines like ARCHER2 for the whole community. All in all there’s a good level of national support for this model on these machines, and when I send a student on the training course for the model they can be running climate simulations on that hardware within 30 minutes of arriving.

something like the WRF has 1.5 million lines of code. So debugging it or customizing it to a different cluster seems pretty tough

Well, the UM has about 1.3 million lines of code and I’d estimate that I know about 15% of that code very very well (mainly a particular area of science) and the rest of it barely at all. When I encounter bugs they’re almost always because of the thing that I’ve just changed or something closely related in areas of the code I know well. When bugs lead into other parts of the model, it’s usually best to go and ask someone who knows about those areas rather than digging too hard yourself.

Writing and running an individual component model or "parameterization" model for one of these larger models seems manageable... But then how can that same person test his/her model inside of one of the larger climate models?

Yes, new parameterizations are often developed separately from the full climate model before being added to it. But the longer a parameterization is developed in isolation the more likely it is that it will be conceptually or technically incompatible with the full model. The trick is knowing early on that you want to couple it into the larger model later and to design your parameterization accordingly. In my experience, however modular we aspire to make these models, it can still be quite a pain to couple in parameterization code that’s had a well-established, independent life outside of the climate model. In general though, this is where the shared code repositories and an active community are really useful.

Since a lot of that model code is written in fortran, that just exacerbates the portability problem since there is less ability to "abstract" away some of this configuration in objects.

I remember years ago, as a student, a computer scientist friend of mine avoided doing an industry placement at a climate modelling center because he had such a low opinion of their software. "It's so basic and boring", he said, "They just use Fortran!" But those same things that are off-putting to a computing student are beneficial to the largely self-taught programmers (i.e., physical scientists) who are working with these models. Fortran is a fairly straightforward and safe language to learn and use, with relatively few concepts and gotchas. Compare that with OOP paradigms, which are hard to use well without significant training.

But those are more comments on the portability of the programmers than of the programs. As Simon mentions, the difficult bits of getting a climate model running on new hardware (big or small) tend to be handled by the support staff of that hardware rather than the academic researchers themselves.