I have been brushing up on numerical methods for PDEs lately, and climate models provide such a rich set of models and ideas to practice with. However, I was a little confused about how academics can do research on large climate models--like the CESM, WRF, or other models produced by GDFL or NCAR--given the computational challenges of running these models? These are global circulation models, or even limited regional atmospheric models with physics happening at multiple scales in the simulation. I was not sure if most institutions have setups to run these models locally, or is there some way to send a new model to NCAR or similar institute to test the accuracy, etc.
To run large climate simulations requires supercomputers or large cluster resources, as well as delicate MPI and mapreduce operations. Since a lot of that model code is written in fortran, that just exacerbates the portability problem since there is less ability to "abstract" away some of this configuration in objects.
Writing and running an individual component model or "parameterization" model for one of these larger models seems manageable. That task just requires the scholar to write something doable over one or a few machines. But then how can that same person test his/her model inside of one of the larger climate models?
I have been doing some research on this, but have not found much. I found some descriptions of how these larger scale models are built from their development documentation. The code is available as open source, but something like the WRF has 1.5 million lines of code. So debugging it or customizing it to a different cluster seems pretty tough--though I don't have any person experience doing that with a climate model. I have also spoken with some climate folks at Caltech who are working on developing their own large scale climate models, but they could only explain the challenges that they faced with building a flexible meshing scheme, etc., The Caltech folks did not tell me about their experiences using the established large scale climate models.
Hence, I figured I would ask the SE community.
UPDATE: As per some feedback below, I just wanted to explain why this question is posted on ES instead of the Academics.SE. The thought was that Academics.SE is a much more general site across all of academia and many people there don't know about the nuances of numerical computing and the computational setup that goes along with it. Hence, I posted the question on ES where the audience is more familiar with running these types of simulations. I recognize that this question is "soft" however it seemed relevant to others conducting ES and Atmospheric research--especially those who want to do research on this topic and come from other disciplines.