Obtaining hourly weather data, including direct and diffuse insolation

Question

I really need some reliable measured hourly weather data for a few sites across the world. It needs to include solar diffuse and direct values or their equivalents.

I've tried the usual sources: Met Office, Weather Underground and Weather Analytics but none of them provide what I need. They are either not hourly, don't contain solar radiation, or have difficult licence/ access restrictions. This is for some research to determine if there's any relation between position of sun, temperature, relative humidity, wind speed and direction. So I'd need a couple of years data for a couple of locations. Aggregate data isn't much use I'm afraid.

Answer 1

If reanalysis data (ERA, ECMWF, CSFRR) is too aggregate for you, try looking for the datasets used to calibrate photovoltaics models (PVGIS, PVWATTS and the like) - they include both direct and diffuse insolation.

Also, follow the citation trail for papers that describe the calibration of the PV models, and the reanalysis models - some of the datasets you are seeking will be somewhere in that trail.

Barry Carter has pointed you at the NOAA ISD-Lite dataset, and I'd like to extend that pointer to the whole Integrated Surface Data from NOAA, which is enormous: 20,000 global stations. Because it's an amalgamation of many different sources, they don't all share the same metadata, but this pdf lists the breadth of the metadata: the insolation variables are on pages 64-77.

Answer 2

EDIT: see also the community wiki answer at: https://opendata.stackexchange.com/questions/10154/sources-of-weather-data

ftp://ftp.ncdc.noaa.gov/pub/data/noaa/isd-lite/ includes hourly data for thousands of stations. The data includes temperature, dew point (from which you can compute relative humidity), wind speed and direction, and you can of course calculate solar position.

It doesn't, however, have all the information you seek.

Answer 3

It will take effort and programming knowledge to get data from Chilbolton Observatory, Hampshire, England, one of the few cloud research sites in the world. http://www.stfc.ac.uk/chilbolton/default.aspx

To get data you need to create an account at BADC and stick to the few terms and conditions. https://badc.nerc.ac.uk/data/

Data time resolution is logger resolution, sub 1 minute. You will have to decimate. Data is supplied as NETCDF. You get raw data, assume nothing. You will have to detect and handle any missing data.

I think you also need to understand the instruments. (in this case you will need to know and ignore some things which are widely stated and wrong)

Full up and down from direct, indirect and "nocturnal". Wind data, yes. Temperature and simple humidity, yes. Exotics, yes.

I think reality is going to strike, perhaps you need to consider what you are really trying to do. If you put in the effort I think you will reach few definite conclusions above what you can learn from a lot of reading. The gain is a working understanding of reality.