Climate policy remains the sole exception to this world-wide universal standard.

The present system has been fatally compromised by the very organizations that resist Independent Review and Verification.

Maybe we’ll soon see a govrenment-level Carbon Regulatory Agency approach designed and implemented in a progressive country.

The degree of inconsistency that RC exhibits becomes more astounding every day.

How many times have we been told that replication does not require that all the original material be readily available. Indeed, we have been repeatedly lectured that true replication is not obtained if only the original material is used. Yet, Tom P. could not have carried out his rapid-response function if Steve had not made all the material available.

By RC’s usual standards, Tom P.’s results could have been considered true replication only if he had started from a clean piece of paper, gone out in the field and gathered up some tree rings, properly selected the data to be used for analyses, developed the analysis methods, produced the coding for the solution of the methods, and then carried out the analysis. ( I’ll let Tom P. skip the Verification and Validation procedures as Climate Science does not do these. )

More and more, it seems to me that RC’s views are shaped by what is convenient in contrast to what they say is the correct approach.

By the usual RC standards, RC cannot accept Tom P.’s analyses and conclusions.

Thinking about this some more led me to the following:

A list of bona fides required by RC prior to Tom P.

1. Certified Climatologist

2. Peer-Reviewed Publications

3. In only the Properly-Certified Journals

4. In only those Journals having a High-Impact Factor

5. Known and verified Educational Pedigree

6. Comments limited to areas consistent with 5.

7. Comments of 6 also limited to the extent of 5 at the time of graduation

8. Replication from a clean slate

9. No reliance on any aspects of previous work

10. No funding by Big Oil, Big Gas, Big Coal, or any organizations associated with Big Fossil in any way

11. No previous work experience in any of 10

12. No funding by “conservative” think tanks.

13. No funding by any think tanks to which 10 gives support

14. Previous work results are known so that source can be used to dismiss or accept all results

15. Proper identification so that 14 can be invoked

So far as I know, Tom P.’s status for any of these 15 items has not been established.

I can reject Tom P.’s results without even looking at the analyses.

Someone at RC will now label Tom P. for convenience in future discussions.

This is astounding to me. Tom P. meets none of the previously required criteria, and yet NASA’s Gavin Schmidt has accepted the results of Tom P.’s black-box analysis. Unquestionably without any provisos whatsoever.

In the RealClimate post linked above appears this statement:

“So along comes Steve McIntyre, self-styled slayer of hockey sticks, who declares without any evidence whatsoever that Briffa didn’t just reprocess the data from the Russians, but instead supposedly picked through it to give him the signal he wanted. These allegations have been made without any evidence whatsoever.”

These statements are false. And they are known to be a false by those you made the statements. The statements are lies and the persons who made the statements are liars. If Gavin Schmidt and Michael Mann cannot point to where Steve McIntyre has made these statements, Gavin and Michael are liars.

By this action Schmidt, Mann, and RealClimate have taken the next step and moved to a higher level of promoting dis-information. That is, while always refusing to correct mis-information stated by commenters, and even RealClimate in-line responses, Schmidt and Mann purposefully lie. The purpose being to jump-start the peanut gallery’s, aided by RealClimate, usual attacks on persons while ignoring the technical issues. The Science of Personal Destruction; invented at RealClimate.

A post and 176 Responses ( and counting ) and not a single error on the part of Steve McIntyre has been identified. Not a single error !! This is reflective of the way RealClimate has of diverting attention from the real issues. In effect, the entire post hinges on becoming discussions of nothing that the subject of the post has said or done. Yet another naked strawman post. Additionally, the comments, with supporting responses from RealClimate, rapidly deteriorate to nothing more than de-meaning, unfounded attacks on people; shooting the messenger and ignoring the message.

To paraphrase, RealClimate continues to take absolutely no responsibility for the ridiculous fantasies and exaggerations that their supporters broadcast, apparently being happy to bask in their acclaim rather than correct any of the misrepresentations they have engendered.

RealClimate, having been unable to point to any technical errors, have continued the Climate Science policy of the Science of Personal Destruction.

They must continue to feed the small-minded, foul-mouthed labelers such as tamino and dhogaza. It’s interesting to me that those who chose to label with the most extreme and debasing labels chose also to remain anonymous. The, mostly non-scientists, peanut gallery at RealClimate seems to revel in this race to the bottom, cheering on tamino and dhogaza at every opportunity.

NASA’s Gavin Schmidt and Penn State’s Michael Mann are liars.

Gavin Schmidt and Michael Mann lied in the subject post, therefore everything they have written is suspected of being lies.

Climate Science . . . The Science of Personal Destruction

The old, and rapidly becoming tedious, “Peer Review Publication is Required” nonsense has already appeared in the comments. Ironically, it seems that no one who raises this issue is aware that Steve is in fact discovering problems in peer-reviewed publications.

A very short summary of a major problem area relative to this olde rant follows.

A proper calculation suitable for archival publication requires that the following be applied prior to submittal of a paper.

Independent Verification of:

1. The input data.

2. Correct handling of the data prior to calculations; pre-processing.

3. The equations to be used in the analyses.

4. The coding of those equations.

5. The solution of the equations.

6. Correct handling of the calculated results; post-processing.

If this procedure is not carried out, the paper has not been peer-reviewed and is not fit for archival publication.

Steve McIntyre provides sufficient information to allow application of the procedure to his work; Independent Verification is attained. Peer-reviewed journals do not require that evidence be presented that the procedure has been applied to submitted papers.

On a related issue that frequently comes up in these Peer-Reviewed Publication rants. High-Impact Factors count for nothing if the papers are not of archival quality. Not to mention that the Impact Factor for a Journal says absolutely nothing about the citation history of any individual paper. And the citation history count of an individual paper says nothing about the nature of the citations; praise or condemnation are both available.

Please consider dropping a few coins to help with covering his costs and time.

Thanks.

I am pleased to announce the publication of
“Fundamentals of Verification and Validation” by Patrick J. Roache.
Copyright 2009, ISBN 978-0-93478-12-7. 476 pages, subject index.

The book is the successor to my 1998 book “Verification and Validation in Computational Science and Engineering.” About 1/3 of the material is new, including a new Chapter 11 describing the Total Validation Uncertainty approach of ASME ANSI Standard V&V 20 (2009).

The attached file outlines the new features of the book. A complete Table of Contents, including designators for new and modified Sections, will be found on the website.

The price is the same as the 1998 book, U.S. $85.00 (but shipping charges have increased). The book is available directly from our fulfillment house, BookMasters, by email order to

orders@BookMasters.com

or from Amazon.com at the following.
http://www.amazon.com/Fundamentals-Verification-Validation-Patrick-Roache/dp/0913478121/ref=sr_1_7?ie=UTF8&s=books&qid=1253150836&sr=1-7

The orders to BookMasters tend to ship faster since they do not run out of stock. Also, wholesale orders can be placed by contacting BookMasters at orders@BookMasters.com.

I would appreciate it if you would forward this email to any of your colleagues who might be interested.

Thanks for your consideration.

Respectfully,
Patrick Roache

p.s.
If anyone is interested, the 1998 V&V book is on clearance sale for 1/2 price at US $42.50. Available from orders@BookMasters.com or from Amazon at
http://www.amazon.com/Verification-Validation-Computational-Science-Engineering/dp/0913478083/ref=sr_1_4?ie=UTF8&s=books&qid=1253718713&sr=8-4.

Likewise, the book “Fundamentals of Computational Fluid Dynamics” is on clearance sale for 1/2 price at US $37.50. Available from orders@BookMasters.com or from Amazon at
http://www.amazon.com/Fundamentals-Computational-Dynamics-Patrick-Roache/dp/0913478091/ref=sr_1_5?ie=UTF8&s=books&qid=1253718713&sr=8-5

[Updated August 26]

I have uploaded an excellent summary, developed by Los Alamos, of the ASC V&V Program at LANL. It is here.

A couple of quotes.

Confidence in simulation extrapolation comes via confidence in physics & numerics models, not calibration to experimental data.

Having “good agreement” between calculations and observations is not sufficient to establish scientifically credible predictive capability.

More and more it seems that the Climate Change Community remains the only holdout among all compute-intensive enterprises relative to application of rigorous, independent V&V to computer software.

Update

It’s been a while since I looked into the general activities in IV&V in scientific and engineering computing. Today I did a quick search over at www.osti.gov and got some good hits.

But first, relative to the two quotations in the original post, I failed to mention my usual mantra; Verification must always precede Validation. In the absence of Verification, Validation is not possible.

Reviewing today’s hits indicates to me that IV&V has been established to be a priority of the highest order, as it necessarily should be, in the Advanced Simulation & Computing Program (ASC) work underway at LANL (Los Alamos), LLNL (Livermore), and SNL (Sandia).

This Sandia report SAND 2008-5517 (Unlimited Release Printed January 2009) provides an extensive specification for the overall general SQA requirements. So far as I know there is no document of a similar nature for any piece of software in all the Climate Change Community.

This report, Enhanced Verification Test Suite for Physics Simulation Codes ( LA-14379 Issued: September 2008; aka LLNL-TR-411291; aka SAND2008-7813 Unlimited Release Printed April 2009 ) is an excellent report on Verification.

The report deals strictly with Verification and reiterates that Verification is:

Verification deals with mathematical correctness of the numerical algorithms in a code, while Validation deals with physical correctness of a simulation in a regime of interest. This document is about Verification.

The report open with this quotation:

In an age of spreading pseudoscience and anti-rationalism, it behooves those of us who believe in the good of science and engineering to be above reproach whenever possible. Public confidence is further eroded with every error we make… As Robert Laughlin noted in this magazine, ‘there is a serious danger of this power [of simulations] being misused, either by accident or through deliberate deception.’ Our intellectual and moral traditions will be served well by conscientious attention to verification of codes, verification of calculations, and validation, including the attention given to building new codes or modifying existing codes with specific features that enable these activities. Patrick Roache [Roa04]

where [Roa04] is this paper: Roache, P., “Building PDE Codes to be Verifiable and Validatable,” Comput. Sci. Engrng. 6, pp. 30–38 (2004).

The Introduction in this report provides excellent descriptions of the fundamental and critical necessity for Verification and the how-tos for implementing the Verification process. It should be required reading for everyone involved in production of software the results of which have the potential to influence the establishment of policies that affect the health and safety of the public. Reading the Introduction is a good way to get a firm handle on the concepts of Verification.

I would like to put all of the Introduction here, there is much good info in it, but it’s kind of long. I especially like this:

What is Verification?
Verification is the process of demonstrating that numerical solutions of the discretized algorithms in simulation software are the correct solutions of the corresponding continuum equations. Consequently, verification represents an important aspect of the development, assessment, and application of simulation software for physics and engineering. An essential element of the verification process is the quantitative analysis of simulation code performance on well-defined problems. The outcome of such analyses provides hard evidence of mathematical consistency between the mathematical statements of the physics models and their discrete analogues as implemented with numerical algorithms in the simulation codes.

and this part:

Code Verification
Verification can be summarized as the analysis of whether the numerical solutions of the discrete algorithms provide accurate solutions of the corresponding continuum equations. Distinct numerical schemes based on the identical continuum equations can produce radically different quantitative (and qualitative) results; therefore, while one may obtain nominally correct solutions of the discretized schemes, those results might be inaccurate solutions of the underlying continuum equations. Consequently, verification analysis constitutes a critically important aspect of the development, assessment, and application of simulation software for physics and engineering. It is important to distinguish between verification for the purpose of proving code correctness, and for providing algorithmic assessment; both activities have high value, but differ in details and tenor. An essential element of any verification process is the quantitative analysis of the simulation code performance on well-defined problems. The outcome of such analyses provides defensible evidence of mathematical consistency between the mathematical statements of the physics models and their discrete analogues as implemented with numerical algorithms in the simulation codes.

As in all previous cases in which I have attempted to convey the critical necessity of Independent Verification and Validation to persons outside the software development community, I failed again. It is obvious that Ms. jackson, Rona Birnbaum, or whoever read and responded to the letter, have no idea what I’m talking about.

Here is my letter to Ms. Jackson:

April 27, 2009

Ms. Lisa Jackson, Administrator
Environmental Protection Agency
Ariel Rios Building
1200 Pennsylvania Avenue NW
Washington, DC 20460.

SUBJECT: Software Verification and Software Quality Assurance

Dear Ms. Jackson:
I have over 40 years experience in development of mathematical models of physical processes in the multi-phase thermal sciences, analytical and numerical solution methods for these models, and incorporation of these into large, complex computer codes designed for applications to complex engineering problems. I have developed software with the BASIC, FORTRAN, F77, and F90/F95 computer languages, among others, on machines using the Unix, Windows NT, and Mac OS operating systems. I have an extensive and continuous peer-reviewed publication and presentation record and have made major contributions to writing computer code manuals. My experience and expertise have direct relationships to development of almost all engineering and scientific models, methods, and software.

Recently my work has focused on documentation and Verification and Validation of software maintained under software quality assurance (SQA) plans. I will focus on related aspects of the Proposed Findings of Docket ID No. EPA-HQ-OAR-2008-0508.

The Intergovernmental Panel for Climate Change (IPCC) Fourth Assessment Report (AR4, IPCC, 2007) states:

“A major advance of this assessment of climate change projections compared with the TAR is the large number of simulations available from a broader range of models. Taken together with additional information from observations, these provide a quantitative basis for estimating likelihoods for many aspects of future climate change.” I have added the emphasis on the models.

Do the numbers from these “large number of simulations available from a broader range of models” GCM calculations have any meaning. My answer is No.

One crucial and necessary first step is that application of Verification procedures have shown that the numbers produced by the software accurately reflect both (1) the original intent of the continuous equations for the models, and (2) the numerical solution methods applied to the discrete approximations to the continuous equations. That is, Verification shows that the equations have been solved correctly. Verification procedures are designed to answer the question, Do the calculated numbers actually satisfy the coded discrete equations and do the solutions of the discrete equations converge to solution of the continuous equations. Neither of these extremely critical properties has been demonstrated for any GCM. None of the GCM codes, and very likely none of any of the enormous number of other computer codes, used in the IPCC processes have been Verified to be correct. Equally important, none of the Journals in which the papers reviewed by the IPCC process are published have editorial policies that require that the software on which papers are based to be Verified.

All software can be Verified. Objective technical criteria and associated success metrics can be developed and applied in a manner that provides assurances about the correctness of the coding of the equations and their numerical solutions. Lack of Verification leaves open the potential that the numbers from the software are simply results of “bugs” in the coding.

The present-day software development community, in all kinds of applications and organizations, is keenly aware that lack of SQA policies and procedures, and successful applications of these to the software, leaves open a significant potential for problems to exist in the software. The computer models and associated software developed for analyses of the Earth’s climate seem to be the sole exception to the understanding of the absolute necessity for application of Verification and SQA procedures to these very complex and extremely important issues. The more complex the physical situation and thus the computer codes, the more likely that bugs are present. Likewise, the more critical the decision that affects the health and safety of the public, the more important that software be Verified.

The situation in CO2 and Climate Science is in stark contrast to that in several engineering societies. These professional societies and their Journal editorial boards have put into place technical requirements on the Verification of the software before the paper can be considered for publication. If the requirements have not been met the paper will not be published; in some cases the paper will be rejected out-of-hand and not be sent out for review. Papers for which the basis is a single calculation on a single grid with no investigations of convergence and other stopping criteria are typically sent back to the authors.

Some of these professional organizations and associated Journals include: The American Society of Mechanical Engineers (ASME) Journal of Heat Transfer and Journal of Fluids Engineering; The American Institute of Aerospace and Astronautics (AIAA) Journal of Spacecraft and Rockets; and the International Journal of Numerical Methods for Fluid Flow. Other professional societies and journals are sure to follow the lead of these. References for the editorial polices for these journals are as follows.

The ASME Journal of Heat Transfer, Editorial Board, “Journal of Heat Transfer Editorial Policy Statement on Numerical Accuracy,” ASME Journal of Heat Transfer, Vol. 116, pp. 797-798, 1994.

The ASME Journal of Fluids Engineering, C. J. Freitas, “Editorial Policy Statement on the Control of Numerical Accuracy,” ASME J. Fluids Eng., 115, pp. 339–340, 1993. http://tinyurl.com/9fs98 and J. Fluids Eng. 130, 2008. http://timyurl.com/8phpar

The AIAA Journal of Spacecraft and Rockets, AIAA, Editorial Policy Statement on Numerical Accuracy and Experimental Uncertainty, AIAA Journal, Vol. 32, No. 1, p. 3, 1994.

The International Journal of Numerical Methods in Fluids, P. M. Gresho and C. Taylor, “Editorial,” International Journal of Numerical Methods in Fluids, Vol. 19, p. iii, 1994.

The results of my direct correspondence about these matters with several of the US agencies and personnel involved in GCM development and applications, and analyses of CO2 and Climate Science, with a strict focus on software Verification and SQA, have shown that Verification and SQA are not considered necessary aspects of any analyses of these important issues. My correspondence has been with appropriate personnel at NASA and GFDL, among others. Correspondence with editors of several of the high-impact Journals in Climate Science, Journals such as Nature, Science, and those published by the AGU, AMS, AAAS, for examples, indicate the same results. These Journals do not have Verification and SQA requirements in place for the papers submitted for publication. This is not an acceptable position for Journals the intent of which is to publish archival papers.

I think it is certain that the EPA has in place mandatory Verification and SQA requirements for all the software applied to all analyses of all other matters that come before the Agency. So far as I am aware, there are no precedents whatsoever for public policy decisions to be based on software for which no Verification and SQA procedures have been applied.

The EPA should not allow decisions on probably the most important issue to come before the Agency to set this type of precedent. No other Agency of the US Government, none whatsoever, has ever allowed and will never allow such a precedent to be set. It is simply unthinkable.

Thank you for your attention to these matters.

Sincerely,

Dan Hughes

Here is the EPA response:

UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460

OFFICE OF
AIR AND RADIATION

Mr. Dan Hughes
324 Plank Road
Porter Corners, New York 12859

Dear Mr. Hughes:
Thank you for your letter dated April 27, 2009 to Administrator Lisa Jackson concerning software verification and software quality assurance. My office has been asked to respond. I appreciate your interest in this matter.

Referencing the Environmental Protection Agency’s (EPA’s) Mandatory Reporting of Greenhouse Gases; Proposed Rule (Docket ID NO. EPA-HQ-OAR-2008-0508), your letter argues that Global Circulation Model (GCM) calculations are flawed. First, the GCMs EPA has relied on for technical analysis of climate change have their foundation in widely accepted physical principles. Second, they have an ability to accurately reproduce key features of past and current climate. Third, they have undergone an extensive peer-review process and been validated by numerous scientific bodies. For all these reasons, GCMs that have been thoroughly vetted by the scientific community provide valuable and insightful information about climate change.

While serving as a highly useful tool. GCM results constitute just one part of the large and growing body of scientific evidence about climate change that is guiding EPA’s evolving policy response to the problem. Additional strands of data EPA uses to guide policy include observed changes in temperature, precipitation and other key climatic features, as well as observed impacts of climate change on terrestrial and oceanic systems.

For more information about the strengths and limitations of climate models, please see “Climate Models: An Assessment of Strengths and Limitations.” This report was published in 2008 by the U.S. Climate Change Science Program and is accessible from http://www.climatescience.gov/Library/sap/sap3-l/final-report/default.htm. If you would like for your letter to be considered as an official comment on the proposed Mandatory Reporting rule or would like to submit additional comments, please follow the instructions for submitting written comments available at http://www.epa.gov/climatechange/emissions/ghgrulemaking.html. lf you would like to make an official comment EPA’s Proposed Endangerment Cause or Contribute Findings for Greenhouse Gases Under Section 202(a) of the Clean Air Act (Docket ID N0. EPA-HQ-OAR-2009-0171), please follow the instructions available at http://www.epa.gov/climatechange/endangerment.html.

Again, thank you for your letter. I appreciate the opportunity to be of service and trust the information provided is helpful.

Sincerely,

Rona Birnbaum
Chief, Climate Science & impacts Branch
Climate Change Division