I still don’t have it all figured out.

And I did not express myself as being opposed to a rewrite.

I have always been responsible for obtaining funding for the projects on which I work. Why do you and Gavin think that it’s my (or our collective) responsibility to obtain funding for you. Plus as a practical matter exactly how would that work? If I help you write the proposals do I get a cut of the work action?

More later, maybe.

Re #9, alas, yes. I am as uncomfortable as any of you with nonmilitary publicly funded codes being closed and simply can’t fathom any reason for it. I do hope all of you are as strident advocates for open source elsewhere in the public sector as you are in climate research.

I suppose you won’t take my word for it that there are all sorts of pressures extraneous to the scientific community to keep science codes closed. The best solution is to have the grant agencies specify an open source license in their proposal calls. I’ll be Microsoft will be thrilled by that.

That aside, many of the demands you all are making, however reasonable (and some of them are very reasonable), are expensive, and climate work (other than remote sensing which eats up the bulk of the big numbers often quoted) has always been run on a shoestring, as Gavin rightly points out.

I wonder whether most of the readers of this blog will support the increase in funding required to get this thing done right from top to bottom. Dan has already expressed himself opposed to a rewrite, calling the idea “naive”, though I don’t really understand why, being naive and all myself.

I promise you that doing it right will not make global warming go away, though, so if that’s what you’re after you probably shouldn’t advocate for ponying up.

A million lines of code and no software engineering is not a pretty picture.

It is very likely, with high confidence, that the resulting code won’t be pretty either.

I hope you don’t mind me dropping this in here, it seemed an appropriate place to put it.

There is an article in this months Physics World that will probably make you chuckle, “Fortran at 50″, discussing how the HadSM3 climate model consists of “over one million lines of Fortran code”. It adds a quote at the end:

Despite the wealth of off-the-shelf software packages, there often comes a time in scientific research when they do not do quite what is needed. It is then usually much simpler for a researcher to write the necessary software than for a software expert to understand the scientific requirements. And, [...] Fortran is still one of the easiest languages for a scientist or engineer to learn.

If a researcher is experimenting, I can’t imagine Fortran being easier to use than something like Matlab or R. And if you’re writing a million lines of code then you really need to be getting software engineers involved if you want any hope of getting something with a fairly low level of bugginess. The difficulty in writing reliable software goes far beyond just learning the syntax of the language. Whilst I have no major problems with Fortran per se, it is the naivety of the problems of producing software and complex models within the scientific community that stands out in this article.

http://physicsworld.com/cws/article/print/31912

For item (3) I’m still confused, but maybe I have made some progress. I have come to the conclusion that the complete equation for conservation of KE has not been used. And the dissipation is ‘backed out’ of an approximate equation for KE. Given the velocity distribution solution from the momentum equations I don’t understand why this is necessary.

A complete equation for conservation of KE requires accounting on the RHS for (1) bulk flow of KE, (2) work done by pressure of surroundings, (3) rate of reversible conversion to internal energy, (4) rate of work done by viscous forces, (5) rate of irreversible conversion to internal energy (dissipation), and (6) rate of work done by gravity. See Aris, for example. I think this equation would usually be solved for the change in KE and not used as an equation for the viscous dissipation. The viscous dissipation must be consistent with the symmetric stress tensor used in the momentum equations for the fluid flow. If the momentum equations solutions are available why not simply calculate the dissipation? Equally important, when a phrase like ‘conservation of energy’ is used it should always be a reference to the fundamental form for which energy is in fact conserved. Approximate forms do not conserve energy.

If I have correctly followed your comments, and the coding, the viscous dissipation in the ModelE model is approximated by

Diss = (KE_1 + PE_1) – ( KE_0 + PE_0 ), where _1 is a final state and _0 is an initial state.

Only items (5) and (6) in the above list are retained, all others are neglected. Conservation of KE cannot be satisfied if only parts of the correct equation are retained. Additionally I’ll still say that the calculation in the code accounts for both the approximate explicit parts retained plus implicit contributions due to numerical solution methods.

That this is not a complete measure of the viscous dissipation is shown by the fact that it gives the wrong answer for all flows for which the change in KE and PE are zero. There can be viscous dissipation into internal energy when the change in KE and PE are zero. Some of the terms that have been dropped from the complete KE conservation equation make it possible.

I also have not been able to understand how KE+PE can be a negative-definite (for the ModelE sign convention), monotonically decreasing function of time for general, transient flows of compressible materials. Of course the coding does not tell me the sign used for the dissipation. But the coding does allow for the possibility that ediff can be either positive or negative.

I have seen the range of dissipation seems to be from about 2 W/m2 to about 15 W/m^2. Have experiments been conducted to validate the model?

Maybe I’ll get back to this if I can find some papers and reports that develop the approximations used in the codes from complete statements of the continuous equations. I find that explicit statements of assumptions and approximations and the steps in the processes used to arrive at the final forms of the continuous are very helpful

(5) I have never used the words ‘true’, ‘bug free’ ‘correct’ or ‘complete’ (actually I have used ‘correct’, but not in the sense ‘proof of correctness’). No one who has been part of a software V&V and SQA project for any non-toy model/code has ever used these words either. Generally it seems that those seeking an exemption for their particular software project are the first to mention these concepts. Sometimes they also throw in something about the squelching of creativity, trampling of personal devotion to excellence, and smashing of creative coding too. An accepted and proven procedure for conducting V&V and SQA for real-world, evolved models/codes is outlined in the report linked in this post.

So far as I can tell, there has never been an independent, formal, documented V&V and SQA audit of any AOLGCM code as these procedure are understood to mean in the software-development world.

(6) When a paper is reviewed for publication, I think reviewers carefully check the source of where the numbers presented in the results come from (I always did). Reviewers do not pass over the equations and all other necessary parts of the problem statement as something of secondary importance simply because it looks like something that has been previously published. Direct verification of the problem statement and its solution are integral parts of the peer-review process.

Numerical solution methods are notorious, and rightly so, for producing numbers that might in some rough approximate way satisfy the discrete equations and yet at the same time have no relation to the solutions of the continuous equations. The actual source of the numbers is of great importance, and it is this source that must be directly verified by persons independent of the author of the work under review. The same process applies to software. Comparison of computer-code output does not represent verification, validation, or SQA. And most certainly it does not provide information relative to ‘truth’, ‘bug freeness’, ‘correctness’, or ‘completeness’.

There are journals published by professional engineering societies that will not accept papers for publication given absence of direct demonstration of Verification of the numerical solution methods. The science journals have given a free pass to papers that are based on numerical calculations. If they had not, there would be requirements that the calculations be directly demonstrated to be actual solutions of the continuous equations. A paper based on a single calculation on a single grid (with the exact same code) would be rejected out of hand and returned to the authors.

> Documentation both external and internal to the code is either not available or not sufficiently detailed.

True. Creating good documentation takes time and effort, and unfortunately the scientists are not paid to do that. You won’t be impressed that the situation now is significantly better than it used to be, however, even perfect documentation does not impart understanding – that is always going to take work.

> If this is indeed a calculation of the physical dissipation why doesn’t the comment simply state that. Additional information pointing to references containing both the continuous and discrete equations for the calculation would be most helpful.

If the dissipation isn’t included (which it wasn’t for a long time) it would be a small error that was ‘fixed’ by this code fragment. I doubt you really need a reference to the continuous and discrete versions of the conservation of energy!

(2) Request for Constructive Efforts to Improve the code
> more documentation is absolutely needed.

Thanks, but that is well known. Bears repeating I suppose.

> the situation surrounding the AOLGCM models/codes is such that the very foundation of the scientific process cannot be applied. That is, independent verification and replication of reported results cannot be carried out.

Absolutely untrue. You can redo almost any reported experiment and test the sensitivity of the results. You can download CCSM3 and do the equivalent experiment and see whether the same phenomena occurs, you can test predictions against real world data.

(3) The ediff Coding

> You state, � …. ediff, it is simply the dissipation term in the KE equation …�.

Slight mis-statement. It is the term for the dissipation of KE in the temperature equation.

> the coding seems to say that it is in fact the difference between the sum of kinetic energy (KE) plus potential energy (PE) at two different states of the fluid, or stages of a calculation.

No contradiction. The different stages are before and after the momentum/advection equations are dealt with. Any loss in total energy, associated principally with KE dissipation, is turned to heat.

> viscous dissipation always acts to increase the thermal energy of a fluid. As shown in the coding, the quantity ediff can be both positive and negative.

The coding does not tell you the sign. ‘ediff’ in fact is always negative (which leads to an increase in temperture). The sign convention is arbitrary.

> Citations to papers and reports or textbooks that cite exactly what continuous equations are being used would have made this entire discussion unnecessary.

Energy is conserved. $d/dt (\int E dA) =0$ ; $ d/dt \sum E_{ij} = 0$.

(4) Execution of the Code by Anyone

It is difficult, but not impossible, and it isn’t just amateurs doing it. It works best when they keep us appraised of their ideas and we keep them up-to-date with fixes and suggestions. I disagree that good documentation is a substitute for using the model. You would have worked out the importance (or not) of ediff in much less time than these postings if you’d just run the model and put in some print statements and looked at the diagnostic printout.

> Such details include development of the modeling and associated continuous equations, the discrete approximations to the continuous equations, the numerical solution methods for the discrete equations, the details of the structure of the code, and even the details of each and every calculation in each and every routine in the code.

Fine. Please inform our funders of the need to double our budget.

(5) Not Operational Forecasting Products

This is a distinction not between projections and forecasting, but a disinction between research and operations. One is a science, the other is engineering (roughly, though obviously the distinction is not clear cut). It is conceivable that climate forecasting could be made into an operational unit (a National Climate Service), but as it stands all climate model devlopment groups are research based – we do not have a mandate to produce ‘products’. Compare that to the National Hurricane Center which does.

> The scientific journals associated with the climate-change community will accept papers for publication the basis of which are calculations by computer software that has not been peer-reviewed. So, AOLGCM-based papers get peer-reviewed and published and then cited in the IPCC reports. This is not correct because it bypasses the independent verification and replication processes of the scientific method.

This is simply not the case. No complex code can ever be proven ‘true’ (let alone demonstrated to be bug free). Thus publications reporting GCM results can only be suggestive. If a result is interesting it gets looked at by other model groups to see if it’s robust. Thus a background of replicated results gets built up and sense of what is and what isn’t ‘bankable’. This is much more useful than showing that any one code is ‘correct’ since it can never be shown to complete. With limited resources, this is the approach that the field has taken.

(6) Newer Versions of the Code

> We have been directed to investigate a part of the AOLGCM contributions to the climate-change community that apparently is no longer in use.

Model development is continuous. We released the ‘frozen’ version that was used in the IPCC AR4 runs. All of the results currently being published use physics as coded in that version. However, newer runs will use updated code, that is inevitable. Changes will be documented in the literature.

Science using GCMs does work differently from that in other fields. There is no agreed upon set of equations that completely cover the problem and the need for parameterisations of unresolved processes means that there never will be. Thus the evaluation of models will always be against real world data, not against pure theoretical solutions.