Models, Methods, Software

A quick look at some of the info provides evidence that an Independent organization to Review and Verify all results, each and every number, that are to be used to set public policy is critically important.

Climate policy remains the sole exception to this world-wide universal standard.

The present system has been fatally compromised by the very organizations that resist Independent Review and Verification.

Maybe we’ll soon see a govrenment-level Carbon Regulatory Agency approach designed and implemented in a progressive country.

Steve McIntyre has discovered and presented yet another problem with a long-term temperature reconstruction: Yamal: A “Divergence” Problem. While Steve has named his blog ClimateAudit, in my opinion his work is in fact Independent Verification.

The old, and rapidly becoming tedious, “Peer Review Publication is Required” nonsense has already appeared in the comments. Ironically, it seems that no one who raises this issue is aware that Steve is in fact discovering problems in peer-reviewed publications.

A very short summary of a major problem area relative to this olde rant follows.

Read the rest of this entry »

I received the following e-mail from Pat Roache regarding his new book about Verification and Validation. He has also made arrangements for reduced prices on two others of his books.

I am pleased to announce the publication of
“Fundamentals of Verification and Validation” by Patrick J. Roache.
Copyright 2009, ISBN 978-0-93478-12-7. 476 pages, subject index.

The book is the successor to my 1998 book “Verification and Validation in Computational Science and Engineering.” About 1/3 of the material is new, including a new Chapter 11 describing the Total Validation Uncertainty approach of ASME ANSI Standard V&V 20 (2009).

The attached file outlines the new features of the book. A complete Table of Contents, including designators for new and modified Sections, will be found on the website.

The price is the same as the 1998 book, U.S. $85.00 (but shipping charges have increased). The book is available directly from our fulfillment house, BookMasters, by email order to

orders@BookMasters.com

or from Amazon.com at the following.
http://www.amazon.com/Fundamentals-Verification-Validation-Patrick-Roache/dp/0913478121/ref=sr_1_7?ie=UTF8&s=books&qid=1253150836&sr=1-7

The orders to BookMasters tend to ship faster since they do not run out of stock. Also, wholesale orders can be placed by contacting BookMasters at orders@BookMasters.com.

I would appreciate it if you would forward this email to any of your colleagues who might be interested.

Thanks for your consideration.

Respectfully,
Patrick Roache

p.s.
If anyone is interested, the 1998 V&V book is on clearance sale for 1/2 price at US $42.50. Available from orders@BookMasters.com or from Amazon at
http://www.amazon.com/Verification-Validation-Computational-Science-Engineering/dp/0913478083/ref=sr_1_4?ie=UTF8&s=books&qid=1253718713&sr=8-4.

Likewise, the book “Fundamentals of Computational Fluid Dynamics” is on clearance sale for 1/2 price at US $37.50. Available from orders@BookMasters.com or from Amazon at
http://www.amazon.com/Fundamentals-Computational-Dynamics-Patrick-Roache/dp/0913478091/ref=sr_1_5?ie=UTF8&s=books&qid=1253718713&sr=8-5

[Updated September 26]
I think the name of the program originally known as the Advanced Strategic Computing Initiative ( ASCI ) is now known as Advanced Simulation & Computing ( ASC ).

[Updated August 26]

I have uploaded an excellent summary, developed by Los Alamos, of the ASC V&V Program at LANL. It is here.

A couple of quotes.

Confidence in simulation extrapolation comes via confidence in physics & numerics models, not calibration to experimental data.

Having “good agreement” between calculations and observations is not sufficient to establish scientifically credible predictive capability.

More and more it seems that the Climate Change Community remains the only holdout among all compute-intensive enterprises relative to application of rigorous, independent V&V to computer software.

Update
Read the rest of this entry »

I sent a slightly revised copy of the comment on the Proposed CO2 Ruling, given in this post, to Ms. Lisa jackson, Administrator of EPA. The letter to Ms. jackson is shown below. I received a reply from Rona Birnbaum, Chief, Climate Science & Impacts Branch, Climate Change Division. (The actual signing of the letter was a task delegated to someone whose name I can’t read.)

As in all previous cases in which I have attempted to convey the critical necessity of Independent Verification and Validation to persons outside the software development community, I failed again. It is obvious that Ms. jackson, Rona Birnbaum, or whoever read and responded to the letter, have no idea what I’m talking about.

Read the rest of this entry »

I have started working on a toy model and plan to include analytical sensitivity analysis as part of the methods.  These notes, and an associated extended discussion that I have up-loaded, serve as a short introduction to the subject.

The file is here:

Summary

These notes introduce a few of the ideas and concepts associated with sensitivity analysis for algebraic and ordinary differential equations. By sensitivity I mean what are the effects of changes in the numerical values of the parameters in a system of equations relative to a response function of interest. The response function can take any mathematical form, but I will focus on the values of the dependent variables of the equation system.

Read the rest of this entry »

Some time ago it was suggested to me something to the effect that most of the people in the Climate Change Community to whom I suggest V&V and SQA issues are critical and require significant attention don’t even know what I’m talking about. Here’s some information that lends support to that observation.

Read the rest of this entry »

The Intergovernmental Panel for Climate Change (IPCC) Fourth Assessment Report (AR4, IPCC, 2007) states:

“A major advance of this assessment of climate change projections compared with the TAR is the large number of simulations available from a broader range of models. Taken together with additional information from observations, these provide a quantitative basis for estimating likelihoods for many aspects of future climate change.” [ My bolding. ]

Do the numbers from these “large number of simulations available from a broader range of models” GCM calculations have any Meaning. My answer is No.

One crucial and necessary first step is that application of Verification procedures have shown that the numbers produced by the software accurately reflect both (1) the original intent of the continuous equations for the models, and (2) the numerical solution methods applied to the discrete approximations to the continuous equations. That is, Verification shows that the equations have been solved right. Do the numbers actually satisfy the Verified-to-be-correct-as-coded discrete equations and do the solutions of the discrete equations converge to solution of the continuous equations. Neither of these has been demonstrated for any GCM. I will be pleased to be shown to be wrong on this point.

All software can be Verified. Objective technical criteria and associated success metrics can be developed and applied in a manner that provides assurances about the correctness of the coding of the equations and their numerical solutions. Lack of Verification leaves open the potential that the numbers from the software are simply results of “bugs” in the coding.

The present-day software development community, in all kinds of applications and organizations, is keenly aware that lack of SQA policies and procedures, and successful applications of these to the software, leaves open a significant potential for problems to exist in the software. So far as I am aware, there are no precedents whatsoever for public policy decisions to be based on software for which no SQA procedures have been applied.

Another example of the lack of even the most basic concerns with verification of calculations has been discovered in GISS/NASA work.

The response by GISS/NASA employees is extremely disconcerting. That response seems to me to say that if the application procedure simply produces numbers, we’re ok with that. Verification of the numbers is left to others.

Read the rest of this entry »

The calculation requires Independent Verification. The calculation includes the coding, the data provided to the code at run time, built in options chosen by the user at run time, and the qualifications of the user.

An excellent example has been provided by the work over on Climate Audit. The results of the excellent work by Climate Audit is being discussed at Climate Science.

The example started with this, which led to this, and leading finally to this.

An excellent piece of Independent Verification accomplished without the benefit of source coding, and actually without any publicly available equations whatsoever. Exactly why the equations and codes are kept secret from the people who paid for them is another very significant issue. Keeping models, methods, procedures, and associated source code secret is not a part of the accepted scientific method.