Model Performance Benchmarking and Ranking

This section provides a concrete example of how a comparative performance as­sessment study can be conducted when a large number of models is involved. In the present case, a rather exhaustive literature survey provided a list of 35 clear – sky broadband irradiance models that can predict instantaneous or short-term (e. g., hourly) direct and diffuse irradiances from limited information on the optical prop­erties of the atmosphere. A computer program that can use the benchmark dataset

Fig. 20.6 Comparison between meteorological (METSTAT) site-based and (Perez) satellite-based direct normal irradiance (DNI) solar radiation estimates for the 1991-2005 US National Solar Radiation Data Base update (Wilcox et al. 2007). Grayscale contours (originally in color) indicate the average 1998-2005 DNI predictions from the Perez model, and similarly color-coded circles indicate those from the METSTAT model

mentioned in Sect. 5.1 to evaluate all these models is included on the CD-ROM (file ‘Models_performance_compar. f’). Only a subset of 15 models is described in what follows, owing to the demonstration purpose of this validation exercise and space limitations. Most of the models have been described and discussed in previous stud­ies (e. g., (Gueymard 1993, 2003b), so that only a summary is provided here, except where more details are necessary. The models are listed in alphabetical order below.

• Model 1—ASHRAE

This is the model used by engineers to calculate solar heat gains and cooling loads in buildings. It was first introduced in 1972, but new monthly coefficients have appeared recently (ASHRAE 2005), which are used here. Note that, beyond these empirical coefficients and solar zenith angle, this model does not depend on any atmospheric data.

• Model 2—Bird

This is the original Bird model (Bird and Hulstrom 1981a, b), with only one modification, required by changes in turbidity measurement practice. At the time this model was developed, the aerosol optical depth (AOD) was measured by sunphotometers with at most two channels, centered at 380 and 500 nm, hence the model’s requirement for the AOD at these two wavelengths. Since the early 1990s, networks of multiwavelength sunphotometers, with typically 5-7 aerosol channels, have expanded worldwide (see, e. g., http://aeronet. gsfc. nasa. gov). Therefore, it is now easier than ever to obtain the turbidity coefficients a and в by fitting the ex­perimental AOD at various wavelengths, xax, to Angstrom’s Law:

таХ = в(Х/Хо)-а (20.10)

where Ao = 1000nm. When a and в are known, the specific AOD at 380 and 500 nm required by Bird’s model can be replaced by в0.38-а and в0.5-а, respectively. This respects the model’s integrity while considerably expanding its applicability.

• Model 3—CLS

The Cloud Layer-Sunshine model (Suckling and Hay 1976, 1977) is based on original work by Houghton (1954) and Monteith (1962). The CLS model has been independently validated for average sky conditions during the IEA Task IX men­tioned in the previous section, but not for clear skies only—at least outside of Canada. The original expression for the aerosol transmittance, Ta = 0.95m, where m is the air mass, is used here.

• Model 4—CPCR2

This two-band model (Gueymard 1989) has already been tested extensively in various studies (Battles et al. 2000; Gueymard 1993, 2003a, 2003b; Ineichen 2006; Olmo et al. 2001). It normally requires separate values of a and в for each of the two wavebands (290-700 nm and 700-4000 nm) considered by the model, i. e., (a1, в1) and (a2, вг) with the constraint в1 = в20.7а1-а2. This information can be derived from the current sunphotometric data by appropriate application of Eq. (20.10). If not possible, the model can be accommodated with simply a1 = a2 = a and в1 = вг = в yielding only a modest degradation of performance.

• Model 5—ESRA2

The original version of this model (Rigollier et al. 2000) has been used to derive the latest edition of the European Solar Radiation Atlas (Scharmer and Greif 2000). The Linke turbidity factor, TL, was then the basis to evaluate the effect of aerosols. However, this factor cannot be measured directly, and therefore needs to be evalu­ated by inversion of an appropriate irradiance model, using experimental clear-sky direct irradiance as the input. This creates a problem in the context of validation studies since the measured direct irradiance cannot be used both to test the model’s predictions and to derive its inputs (see Rule #1 in Sect. 5.3). A new version of the model, which is tested here, rather calculates TL from air mass, precipitable water, and в (Remund et al. 2003).

• Models 6-8—Iqbal’s Parameterization Models A, B and C

These models are fully described in the original publication (Iqbal 1983), and have been tested previously, to some extent (Battles et al. 2000; Gueymard 1993). Scatterplots of Iqbal C’s model appear in Fig 20.3 for the benchmark dataset con­sidered here.

• Model 9—Kasten

This classic model (Kasten 1980, 1983; Kasten and Czeplak 1980) has been ex­panded to provide direct and global irradiance, both as a function of TL (Davies and McKay 1989). The latter version is used here. To overcome the difficulty in using TL while respecting the model’s intentional simplicity, a simple linear function of в has been used,

TL = 2.1331 + 19.0204 в (20.11)

where the numerical coefficients have been obtained by combining the empirical determinations of в = f(TL) proposed by different authors (Abdelrahman et al. 1988; Grenier et al. 1994; Hinzpeter 1950; Katz et al. 1982).

• Model 10—METSTAT

As mentioned in Criterion #6 of Sect. 2, this model has a deterministic algorithm that can be combined with statistical features so that correct frequency distributions of hourly irradiances can be obtained despite the use of daily or monthly-average turbidity and cloud input data (Maxwell 1998). The model is used here without these statistical corrections since short-term input data are available for validation, there­fore respecting Rule #4 in Sect. 5.3. A modification to the model, however, is nec­essary since it uses the broadband aerosol optical depth, та, to evaluate the aerosol transmittance. Like TL, та can only be obtained indirectly from an inverted model and irradiance measurements. To circumvent this problem, a convenient methodol­ogy (Molineaux et al. 1998), which uses the concept of equivalent wavelength for broadband turbidity, is used here to derive та from a and в through

Ta = в [0.695 + (0.0160 + 0.066в0.7~“)да]~“. (20.12)

• Model 11—MAC

The McMaster (MAC) model has evolved slightly between its original deriva­tion (Davies et al. 1975) and the latest performance assessment results (Davies and McKay 1989). The version described in the IEA Task IX report (Davies et al. 1988) is used here, with a Rayleigh transmittance formula corrected for its typographic error. In the absence of specific information on the most appropriate aerosol trans­mittance to be used here, the original formula Ta = 0.95m (Davies et al. 1975; Davies and Hay 1979), as for the BCLS model, is selected.

• Model 12—MRM5

This new, version 5, of the Meteorological Radiation Model, is described in Chap. 14. It contains important changes from the previous version 4 (Muneer 2004), which used incorrect numerical coefficients that considerably affected the model’s irradiance predictions (Gueymard 2003a, 2003b).

• Model 13—REST2

This two-band model (Gueymard 2008) is based on CPCR2, but incorporates completely revised parameterizations, which have been derived from the SMARTS spectral model (Gueymard 2001, 2005b). REST2 uses the same inputs as CPCR2, with the addition of the amount of nitrogen dioxide in a vertical atmospheric col­umn, which can be defaulted if unknown. See Fig. 20.3 for scatterplots involving this model and the benchmark dataset.

• Model 14—Santamouris

This model’s algorithm (Santamouris et al. 1999) is in essence similar to that of the Bird model, except that a fixed turbidity is considered.

• Model 15—’Yang

The direct irradiance predictions of this model (Yang et al. 2001; Yang and Koike 2005) have been shown to perform very well (Gueymard 2003a, b). Its diffuse irradiance predictions have not been evaluated so far.

All radiation models require at least one input variable, namely the solar zenith an­gle, Z. The simplest models (e. g., ASHRAE) do not need more inputs (besides some empirical coefficients). The most sophisticated models (e. g., REST2) may require as many as 10 more inputs, mostly atmospheric data. For clarity, all the inputs (besides Z) required by the 15 models considered here are compiled in Table 20.1.

Using the specially-developed Fortran program contained on the CD-ROM (file ‘Models_performance_compar. f’), all these models (and more) have been run using the 30-point benchmark dataset recently proposed (Gueymard 2008) and previously mentioned in Sects. 3 and 5.1. This dataset is contained in file ‘Models_perf_exp. dat. txt’. Of course, to the benefit of the reader, this Fortran program can also

Table 20.1 Inputs required by the 15 clear-sky models under scrutiny

#

Name

m

En0

Pg

p

T

Uo

Un

w

Tl

Ta a

в

Ma

1

ASHRAE

2

Bird

/

/

/

/

/

/

/

/

3

CLS

/

/

/

/

/

4

CPCR2

/

/

/

/

/

/

/

/

/

5

ESRA2

/

/

/

/

/

6

Iqbal A

/

/

/

/

/

/

/

/

/

7

Iqbal B

/

/

/

/

/

/

/

/

8

Iqbal C

/

/

/

/

/

/

/

/

/

9

Kasten

/

/

/

/

10

METSTAT

/

/

/

/

/

/

/

11

MAC

/

/

/

/

/

/

12

MRM5

/

/

/

/

/

/

/

13

REST2

/

/

/

/

/

/

/

/

/

/

14

Santamouris

/

/

/

/

/

/

15

Yang

/

/

/

/

/

/

/

Key: m, air mass; En0, distance-corrected extraterrestrial irradiance; pg, far-field ground albedo; p, site pressure;T, dry-bulb temperature; Uo; total ozone in the vertical column; Un, total nitrogen dioxide in the vertical column; w, precipitable water; TL, Linke turbidity coefficient; Ta, broadband aerosol optical depth; a, Angstrom wavelength exponent; в, Angstrom’s turbidity coefficient, a>„, aerosol single-scattering albedo.

Table 20.2 Performance statistics, and their associated ranking (in bold italics), for direct irradi – ance predicted by the 15 clear-sky models under scrutiny, relative to a benchmark dataset

#

MBE (%)

RMSE (%)

R2

t

d

AS

1

3.6

10

12.0

11

0.703

15

1 . 70

5

0.897

11

2.71

11

2

-0.6

2

2.9

3

0.983

5

1 . 21

2

0.995

3

4.46

4

3

13.0

14

17.0

14

0.730

11

6.47

11

0.823

14

2.28

14

4

-1.2

5

2.2

2

0.994

3

3.68

8

0.997

2

4.49

3

5

3.1

8

4.4

8

0.979

7

5.50

10

0.989

8

4.38

7

6

2.7

7

3.0

4

0.996

2

10.76

15

0.995

4

4.45

5

7

9.8

12

11.0

10

0.949

10

10.58

14

0.928

10

3.59

10

8

1.2

4

3.1

5

0.982

6

2.31

6

0.995

5

4.40

6

9

1.5

6

5.8

9

0.961

9

1.50

3

0.984

9

3.97

9

10

3.4

9

4.0

7

0.994

4

8.86

12

0.990

7

4.28

8

11

9.9

13

14.9

13

0.725

12

4.87

9

0.846

13

2.35

13

12

-5.8

11

12.7

12

0.718

13

2.82

7

0.897

12

2.66

12

13

0.1

1

1.0

1

0.998

1

0.49

1

0.999

1

4.66

1

14

-20.0

15

23.3

15

0.715

14

9.20

13

0.759

15

1.97

15

15

0.9

3

3.4

6

0.977

8

1.54

4

0.994

6

4.58

2

accommodate other, more voluminous datasets, if prepared with the same format. The present example is purposefully limited to a small dataset, so that the con­clusions reached here should not be considered of general or “universal” validity. However, owing to the benchmark status of this dataset, it can be said that a model should not be considered as universal if it does not perform well under the conditions of this dataset.

A statistical analysis has been conducted from the differences between predicted and measured direct, diffuse and global irradiances; the summary performance re­sults appear in Tables 20.2, 20.3 and 20.4, respectively. These tables include the

Table 20.3 Performance statistics, and their associated ranking (in bold italics), for diffuse irradi – ance predicted by the 15 clear-sky models under scrutiny, relative to a benchmark dataset

#

MBE (%)

RMSE (%)

R2

t

d

AS

1

-5.6

5

33.9

12

0.351

14

0.92

4

0.595

14

2.41

10

2

1.0

2

10.8

2

0.933

7

0.49

3

0.980

2

3.94

2

3

-8.8

6

35.3

13

0.289

15

1 . 41

5

0.583

15

2.13

13

4

14.2

9

16.8

6

0.969

3

8.60

12

0.963

5

3.41

6

5

-19.5

11

24.5

8

0.947

6

7.18

11

0.879

8

2.91

8

6

13.2

8

15.3

5

0.966

4

9.19

14

0.966

4

3.52

4

7

-28.5

15

33.3

10

0.826

9

9.10

13

0.847

9

2.29

11

8

-0.4

1

11.3

3

0.928

8

0.20

1

0.977

3

3.88

3

9

-21.0

12

33.4

11

0.594

10

4.43

8

0.808

10

2.20

12

10

-22.0

13

22.6

7

0.985

2

24.63

15

0.926

7

3.12

7

11

-14.9

10

36.0

14

0.365

12

2.49

6

0.604

13

2.06

14

12

-1.8

3

32.7

9

0.360

13

0.30

2

0.669

12

2.51

9

13

2.7

4

4.4

1

0.993

1

4.28

7

0.997

1

4.16

1

14

28.4

14

43.0

15

0.366

11

4.83

9

0.671

11

1.82

15

15

-10.1

7

15.2

4

0.961

5

4.83

10

0.955

6

3.45

5

Table 20.4 Performance statistics, and their associated ranking (in bold italics), for global irradi – ance predicted by the 15 clear-sky models under scrutiny, relative to a benchmark dataset

#

MBE (%)

RMSE (%)

R2

t

d

AS

1

1.7

9

3.4

8

0.996

13

3.14

6

0.999

1

4.97

1

2

-0.6

2

1.8

4

0.998

7

1 . 78

2

0.843

9

3.21

9

3

8.5

14

9.2

14

0.998

8

13 . 45

14

0.865

2

1 . 76

14

4

1.3

6

1.5

3

1.000

3

9.55

10

0.847

7

4.08

3

5

-0.9

4

3.9

9

0.994

15

1.37

1

0.841

11

3.90

6

6

4.2

11

4.6

11

1.000

2

13.08

13

0.859

3

2.98

12

7

1.3

7

2.3

6

1.000

6

3.84

7

0.850

5

3.54

8

8

0.6

3

1.5

2

1.000

5

2.42

4

0.848

6

3.93

4

9

-1.6

8

3.9

10

0.995

14

2.47

5

0.841

12

3.17

10

10

-1.7

10

2.0

5

1.000

4

9.90

11

0.833

13

3.92

5

11

4.7

13

5.3

13

0.998

9

10.63

12

0.855

4

3.01

11

12

-4.2

12

4.9

12

0.998

10

9.39

9

0.814

14

2.96

13

13

0.5

1

0.8

1

1.000

1

4.84

8

0.844

8

4.26

2

14

-9.1

15

9.7

15

0.998

11

14.62

15

0.782

15

1.50

15

15

-1.1

5

3.2

7

0.996

12

2.02

3

0.842

10

3.66

7

MBE, RMSE, R2, t-statistic, Willmott’s index of agreement d, and Muneer’s ac­curacy score AS. For each of these performance indices, the corresponding model ranking is indicated. As could be expected, the models that use detailed atmospheric information (particularly on aerosols and water vapor) perform better than those with little or no such inputs. The main disturbing fact, however, is that the ranking methods disagree widely, particularly for diffuse and global irradiance. This con­firms the need for more in-depth investigations on this issue.

6 Conclusions

The primary focus of this chapter has been to emphasize to the newcomer as well as the experienced solar radiation model developer, tester, or user, the nuances of model validation and performance evaluation. Section 2 addressed seven criteria describing typical solar radiation model approaches or types. Sections 3 and 4 de­scribed the principles of model validation and uncertainty analysis required for both measured data and uncertainties in model estimates. Sections 5.1 and 5.2 addressed some aspects of qualitative and quantitative model performance. Section 5.3 em­phasized seven constituent elements of model validation that must be addressed in any evaluation. These include validation and input data quality, independence, and consistency of temporal and spatial extent, and validation limits. Section 6 dis­cussed evolution and validation of model component parts, the importance of, and difficulties associated with, interpreting independent model validation, as well as demonstrated the practice (and difficulties) of comparing the performance of many models).

As the solar energy industry and scientific research into the detailed energy bal­ance of the Earth continues to grow and evolve, it is critical that computational models be validated and tested as stringently as possible to provide decision mak­ers, and the scientific community in general, with the most accurate, comprehensive, and well documented information possible.

[1] Inference. According to the rule-base from Table 7.1, six rules are set-up. At this step the fuzzy inputs are combined logically using the operator AND (Eqn. 7.14a) to produce the output values:

[2] Fuzzyfication. Crisp inputs are transformed into confidence levels of input lin­guistic variable attributes, being computed with the equations (7.18) and (7.20 a, b). For At = 14°C, the linguistic variable air temperature amplitude is charac­terized by three attributes with the corresponding confidence levels:

T4: mAt,4 = 0.2

T5: mAt,5 = 0.7

Te. mAt,6 = 0.8

Julian day j = 90 have both attributes SUMMER and WINTER:

S: mS = 0.4

W: mW = 0.6

Lv, z {1 + c[exp(d0s) -exp(dn/2)} + ecos20s} ■ {1 + a■ exp(b)}

It can be appreciated that the formulation is similar to the one proposed by Perez in his all-weather model. Nevertheless, some differences must be emphasized. On the one hand, a new term (exp(dn/2)) appears in the scattering indicatrix func­tion. It was introduced by Kittler (1994) as a necessary correction according to the concept of relative scattering indicatrix.

On the other hand, six groups of a and b coefficients and other six groups of c, d and e coefficients are considered instead of employing continuous parameters for establishing the type of sky and the value of the coefficients of Eq. (17.9). Sub­sequently, six different gradation and indicatrix functions exist whose combination may produce up to thirty six different types of sky. Notwithstanding, the standard only includes those fifteen considered to be of more interest, although some work in low latitudes seems to make clear the need of increasing the current type of skies under consideration (Wittkopf and Soon 2007).

In order to choose the type of sky to be used in each case, one of the following procedures may be employed:

[4] By comparison of the theoretical functions of gradation, g(0), and indicatrix, f (£), with their observed counterparts. The observed gradation function may be determined from luminance measurements in different points of the sky located on the plane of the solar meridian and on another one perpendicular to it. In contrast, for establishing the observed indicatrix function, it is necessary to know the luminance in different points of almucantars of different altitudes.

[5] By the analysis of the ratio between the zenith’s luminance and the diffuse illumi­nance (Dv) on a horizontal plane. In this respect, Kittler et al (1997) graphically