Getting the CRN data

Many questions have been raised over the quality and reliability of the data that is being used to determine the global temperature. Issues such as microsite biases, UHI and TOBS have been raised. A number of sites such as Climate Audit, Surface Stations.org and others, have made strides in determining problems with the temperature stations. In this decade an attempt to provide reliable sites has been implemented by NOAA
<blockquote>The U.S. Climate Reference Network (USCRN) consists of 114 stations developed, deployed, managed, and maintained by the National Oceanic and Atmospheric Administration (NOAA) in the continental United States for the express purpose of detecting the national signal of climate change. The vision of the USCRN program is to maintain a sustainable high-quality climate observation network that 50 years from now can with the highest degree of confidence answer the question: How has the climate of the nation changed over the past 50 years? These stations were designed with climate science in mind.

Given the availability of the CRN data some analysis can be undertaken to see if these sites show the same trends as the sites that are used to create the US portion of the temperature data for the global temps and to make some determination if corrections that are in use are actually doing what their implementors claim.For my initial effort I decided to download the daily data through their FTP site manually rather than attempting to write R code.  The files are as described in the readme file in the ftp directory:

CRN FTP DAILY STREAM
NCDC provides ftp access to data from the Climate Reference Network (CRN) via
anonymous ftp at the address:
ftp://ftp.ncdc.noaa.gov/pub/data/uscrn/products/daily01
That directory contains separate subdirectories for each year that the ftp data
are available.  Each annual subdirectory contains a collection of ASCII files
named according to the following convention:
CRNDAILYTT-YYYY-${name}.txt
where
TT = 2-character file format number (currently always 01)
YYYY = 4-digit year
${name} = Unique station name (state location vector) with spaces replaced by
underscores (e.g. AZ_Tucson_11_W)
Also contained in the daily01 directory are important announcements relating to
the Daily01 product.  Please be sure to review these announcements before using
any product data.
There is also a separate collection of near-real-time files produced daily in:
ftp://ftp.ncdc.noaa.gov/pub/data/uscrn/products/daily01/updates
This subdirectory contains a collection of annual subdirectories, which contain
ASCII files that are named according to the following convention:
CRNDAILYTT-YYYYMMDDHHmm.txt
where
TT = 2-character file format number (currently always 01)
YYYY = 4-digit local year
MM = 2-digit local month (01=Jan, …, 12=Dec)
DD = 2-digit local day of month
HH = 2-digit local standard time hour of day (always 23)
mm = 2-digit minute (always 59)
Each one of these files contains CRN data for the period of one day ending at
local standard time YYYYMMDDHHmm.  The 2-character sequence TT indicates the
file format number.  It is currently always 01.  If at some point in the
future for format for CRN ftp data changes, this number will be changed.
Each file contains fixed-column formatted lines, each of which represents one
hour’s observation data from one station.  The format of each line is as follows:
cols 1 — 5 [5 chars] WBANNO
The station WBAN number.
cols 7 — 12 [6 chars] COOPNO
The station COOP number.
cols 14 — 21 [8 chars] LST_DATE
The local calendar date of the observations.
cols 23 — 28 [6 chars] CRX_VN
The version number of the station datalogger program that was in
effect at the time of the observation.
cols 30 — 36 [7 chars] LONGITUDE
Station longitude.
cols 38 — 44 [7 chars] LATITUDE
Station latitude.
cols 46 — 52 [7 chars] T_DAILY_MAX
Maximum temperature, in degrees C, during the day.  Note: CRN
stations have multiple co-located temperature sensors that record
independent measurements.  This value is a single temperature
maximum that is calculated from the multiple independent
measurements. The independent measurements are the maximum for
each sensor of 5-minute average temperatures measured every 10
seconds during the day.
cols 54 — 60 [7 chars] T_DAILY_MIN
Minimum temperature, in degrees C, during the day.  Note: CRN
stations have multiple co-located temperature sensors that record
independent measurements.  This value is a single temperature
minimum that is calculated from the multiple independent
measurements. The independent measurements are the minimum for
each sensor of 5-minute average temperatures measured every 10
seconds during the day.
cols 62 — 68 [7 chars] T_DAILY_MEAN
Mean temperature, in degrees C, calculated using the typical
historical approach of (T_DAILY_MAX + T_DAILY_MIN) / 2.
cols 70 — 76 [7 chars] T_DAILY_AVG
Average temperature, in degrees C, during the 24 hours of the day
. Note: CRN stations have multiple co-located temperature sensors
that record independent measurements. This value is a single
temperature that is calculated by averaging 24 full-hour averages
derived from the multiple independent measurements of 5-minute
intervals during each hour.
cols 78 — 84 [7 chars] P_DAILY_CALC
Total amount of precipitation, in mm, recorded during the 24
hours of the day. Note: CRN stations have multiple independent
measures of precipitation; this P_DAILY_CALC value is a single
precipitation total that is calculated from the station hourly
totals derived from multiple independent measurements of 5-minute
intervals during each hour.
cols 86 — 93 [8 chars] SOLARAD_DAILY
Total solar energy received, MJ/meter^2, recorded during the day
. This value is calculated from the hourly average solar
radiation rates and converted to energy by integrating over time.
cols 95 — 101 [7 chars] SUR_TEMP_DAILY_MAX
Maximum 5-minute temperature, in degrees C, during the day.
cols 103 — 109 [7 chars] SUR_TEMP_DAILY_MIN
Minimum 5-minute temperature, in degrees C, during the day.
cols 111 — 117 [7 chars] SUR_TEMP_DAILY_AVG
Note: This value is a single temperature that is calculated by
averaging 24 full-hour averages.
cols 119 — 125 [7 chars] RH_DAILY_MAX
Maximum clock hour relative humidity, in %, for the day.
cols 127 — 133 [7 chars] RH_DAILY_MIN
Minimum clock hour relative humidity, in %, for the day.
cols 135 — 141 [7 chars] RH_DAILY_AVG
Average hourly relative humidity, in %, for the day.  This
statistic is not yet available, but we’re holding a place for it
for the future.
cols 143 — 149 [7 chars] SOIL_MOISTURE_5_DAILY
Average soil moisture, in % volumetric water content, during the
entire day at 5 cm below the surface. Note: CRN stations have
multiple co-located soil moisture sensors that record independent
measurements. This value is an average of 24 hourly soil
moisture values that each are calculated from the multiple
independent measurements. This statistic is not yet available,
but we’re holding a place for it for the future.
cols 151 — 157 [7 chars] SOIL_MOISTURE_10_DAILY
Average soil moisture, in % volumetric water content, during the
entire day at 10 cm below the surface. Note: CRN stations have
multiple co-located soil moisture sensors that record independent
measurements. This value is an average of 24 hourly soil
moisture values that each are calculated from the multiple
independent measurements. This statistic is not yet available,
but we’re holding a place for it for the future.
cols 159 — 165 [7 chars] SOIL_MOISTURE_20_DAILY
Average soil moisture, in % volumetric water content, during the
entire day at 20 cm below the surface. Note: CRN stations have
multiple co-located soil moisture sensors that record independent
measurements. This value is an average of 24 hourly soil
moisture values that each are calculated from the multiple
independent measurements. This statistic is not yet available,
but we’re holding a place for it for the future.
cols 167 — 173 [7 chars] SOIL_MOISTURE_50_DAILY
Average soil moisture, in % volumetric water content, during the
entire day at 50 cm below the surface. Note: CRN stations have
multiple co-located soil moisture sensors that record independent
measurements. This value is an average of 24 hourly soil
moisture values that each are calculated from the multiple
independent measurements. This statistic is not yet available,
but we’re holding a place for it for the future.
cols 175 — 181 [7 chars] SOIL_MOISTURE_100_DAILY
Average soil moisture, in % volumetric water content, during the
entire day at 100 cm below the surface. Note: CRN stations have
multiple co-located soil moisture sensors that record independent
measurements. This value is an average of 24 hourly soil
moisture values that each are calculated from the multiple
independent measurements. This statistic is not yet available,
but we’re holding a place for it for the future.
cols 183 — 189 [7 chars] SOIL_TEMP_5_DAILY
Average soil temperature, in degrees C, during the entire day at
5 cm below the surface. Note: CRN stations have multiple co-
located soil temperature sensors that record independent
measurements. This value is an average of 24 hourly soil
temperature values that each are calculated from the multiple
independent measurements. This statistic is not yet available,
but we’re holding a place for it for the future.
cols 191 — 197 [7 chars] SOIL_TEMP_10_DAILY
Average soil temperature, in degrees C, during the entire day at
10 cm below the surface. Note: CRN stations have multiple co-
located soil temperature sensors that record independent
measurements. This value is an average of 24 hourly soil
temperature values that each are calculated from the multiple
independent measurements. This statistic is not yet available,
but we’re holding a place for it for the future.
cols 199 — 205 [7 chars] SOIL_TEMP_20_DAILY
Average soil temperature, in degrees C, during the entire day at
20 cm below the surface. Note: CRN stations have multiple co-
located soil temperature sensors that record independent
measurements. This value is an average of 24 hourly soil
temperature values that each are calculated from the multiple
independent measurements. This statistic is not yet available,
but we’re holding a place for it for the future.
cols 207 — 213 [7 chars] SOIL_TEMP_50_DAILY
Average soil temperature, in degrees C, during the entire day at
50 cm below the surface. Note: CRN stations have multiple co-
located soil temperature sensors that record independent
measurements. This value is an average of 24 hourly soil
temperature values that each are calculated from the multiple
independent measurements. This statistic is not yet available,
but we’re holding a place for it for the future.
cols 215 — 221 [7 chars] SOIL_TEMP_100_DAILY
Average soil temperature, in degrees C, during the entire day at
100 cm below the surface. Note: CRN stations have multiple co-
located soil temperature sensors that record independent
measurements. This value is an average of 24 hourly soil
temperature values that each are calculated from the multiple
independent measurements. This statistic is not yet available,
but we’re holding a place for it for the future.
IMPORTANT NOTES:
1. All fields are separated from adjacent fields by at least one space.
2. Leading zeros are omitted.
3. Missing data are indicated by the value -9999 (or -9999.0).
4. There are no quality flags for these derived quantities. When the raw
data are flagged as erroneous, these derived values are not
calculated, and are instead reported as missing. So these fields may
be assumed to always be good (unflagged) data, except when they are
reported as missing.
5. The update files are named according to the calendar date of the
observations.  The daily records are generated several times during
the next calendar day so as to insure that late arriving data are
included.  Early in each month, the annual-station files will be
reconstructed, both to add the previous month to the current annual
file, and to backfill data that were received through manual recovery
at the station site as opposed to being received via satellite
transmission.

CRN FTP DAILY STREAM
NCDC provides ftp access to data from the Climate Reference Network (CRN) via anonymous ftp at the address:
ftp://ftp.ncdc.noaa.gov/pub/data/uscrn/products/daily01
That directory contains separate subdirectories for each year that the ftp data are available.  Each annual subdirectory contains a collection of ASCII files named according to the following convention:
CRNDAILYTT-YYYY-${name}.txt
where       TT = 2-character file format number (currently always 01)     YYYY = 4-digit year  ${name} = Unique station name (state location vector) with spaces replaced by             underscores (e.g. AZ_Tucson_11_W)
Also contained in the daily01 directory are important announcements relating tothe Daily01 product.  Please be sure to review these announcements before usingany product data.
There is also a separate collection of near-real-time files produced daily in:
ftp://ftp.ncdc.noaa.gov/pub/data/uscrn/products/daily01/updates
This subdirectory contains a collection of annual subdirectories, which containASCII files that are named according to the following convention:
CRNDAILYTT-YYYYMMDDHHmm.txt
where       TT = 2-character file format number (currently always 01)     YYYY = 4-digit local year       MM = 2-digit local month (01=Jan, …, 12=Dec)       DD = 2-digit local day of month       HH = 2-digit local standard time hour of day (always 23)       mm = 2-digit minute (always 59)
Each one of these files contains CRN data for the period of one day ending at local standard time YYYYMMDDHHmm.  The 2-character sequence TT indicates the file format number.  It is currently always 01.  If at some point in the future for format for CRN ftp data changes, this number will be changed.
Each file contains fixed-column formatted lines, each of which represents one hour’s observation data from one station.  The format of each line is as follows:
cols 1 — 5 [5 chars] WBANNO        The station WBAN number.

cols 7 — 12 [6 chars] COOPNO        The station COOP number.

cols 14 — 21 [8 chars] LST_DATE        The local calendar date of the observations.

cols 23 — 28 [6 chars] CRX_VN        The version number of the station datalogger program that was in        effect at the time of the observation.

cols 30 — 36 [7 chars] LONGITUDE        Station longitude.

cols 38 — 44 [7 chars] LATITUDE        Station latitude.

cols 46 — 52 [7 chars] T_DAILY_MAX        Maximum temperature, in degrees C, during the day.  Note: CRN        stations have multiple co-located temperature sensors that record        independent measurements.  This value is a single temperature        maximum that is calculated from the multiple independent        measurements. The independent measurements are the maximum for        each sensor of 5-minute average temperatures measured every 10        seconds during the day.

cols 54 — 60 [7 chars] T_DAILY_MIN        Minimum temperature, in degrees C, during the day.  Note: CRN        stations have multiple co-located temperature sensors that record        independent measurements.  This value is a single temperature        minimum that is calculated from the multiple independent        measurements. The independent measurements are the minimum for        each sensor of 5-minute average temperatures measured every 10        seconds during the day.

cols 62 — 68 [7 chars] T_DAILY_MEAN        Mean temperature, in degrees C, calculated using the typical        historical approach of (T_DAILY_MAX + T_DAILY_MIN) / 2.

cols 70 — 76 [7 chars] T_DAILY_AVG        Average temperature, in degrees C, during the 24 hours of the day        . Note: CRN stations have multiple co-located temperature sensors        that record independent measurements. This value is a single        temperature that is calculated by averaging 24 full-hour averages        derived from the multiple independent measurements of 5-minute        intervals during each hour.

cols 78 — 84 [7 chars] P_DAILY_CALC        Total amount of precipitation, in mm, recorded during the 24        hours of the day. Note: CRN stations have multiple independent        measures of precipitation; this P_DAILY_CALC value is a single        precipitation total that is calculated from the station hourly        totals derived from multiple independent measurements of 5-minute        intervals during each hour.

cols 86 — 93 [8 chars] SOLARAD_DAILY        Total solar energy received, MJ/meter^2, recorded during the day        . This value is calculated from the hourly average solar        radiation rates and converted to energy by integrating over time.

cols 95 — 101 [7 chars] SUR_TEMP_DAILY_MAX        Maximum 5-minute temperature, in degrees C, during the day.

cols 103 — 109 [7 chars] SUR_TEMP_DAILY_MIN        Minimum 5-minute temperature, in degrees C, during the day.

cols 111 — 117 [7 chars] SUR_TEMP_DAILY_AVG        Note: This value is a single temperature that is calculated by        averaging 24 full-hour averages.

cols 119 — 125 [7 chars] RH_DAILY_MAX        Maximum clock hour relative humidity, in %, for the day.

cols 127 — 133 [7 chars] RH_DAILY_MIN        Minimum clock hour relative humidity, in %, for the day.

cols 135 — 141 [7 chars] RH_DAILY_AVG        Average hourly relative humidity, in %, for the day.  This        statistic is not yet available, but we’re holding a place for it        for the future.

cols 143 — 149 [7 chars] SOIL_MOISTURE_5_DAILY        Average soil moisture, in % volumetric water content, during the        entire day at 5 cm below the surface. Note: CRN stations have        multiple co-located soil moisture sensors that record independent        measurements. This value is an average of 24 hourly soil        moisture values that each are calculated from the multiple        independent measurements. This statistic is not yet available,        but we’re holding a place for it for the future.

cols 151 — 157 [7 chars] SOIL_MOISTURE_10_DAILY        Average soil moisture, in % volumetric water content, during the        entire day at 10 cm below the surface. Note: CRN stations have        multiple co-located soil moisture sensors that record independent        measurements. This value is an average of 24 hourly soil        moisture values that each are calculated from the multiple        independent measurements. This statistic is not yet available,        but we’re holding a place for it for the future.

cols 159 — 165 [7 chars] SOIL_MOISTURE_20_DAILY        Average soil moisture, in % volumetric water content, during the        entire day at 20 cm below the surface. Note: CRN stations have        multiple co-located soil moisture sensors that record independent        measurements. This value is an average of 24 hourly soil        moisture values that each are calculated from the multiple        independent measurements. This statistic is not yet available,        but we’re holding a place for it for the future.

cols 167 — 173 [7 chars] SOIL_MOISTURE_50_DAILY        Average soil moisture, in % volumetric water content, during the        entire day at 50 cm below the surface. Note: CRN stations have        multiple co-located soil moisture sensors that record independent        measurements. This value is an average of 24 hourly soil        moisture values that each are calculated from the multiple        independent measurements. This statistic is not yet available,        but we’re holding a place for it for the future.

cols 175 — 181 [7 chars] SOIL_MOISTURE_100_DAILY        Average soil moisture, in % volumetric water content, during the        entire day at 100 cm below the surface. Note: CRN stations have        multiple co-located soil moisture sensors that record independent        measurements. This value is an average of 24 hourly soil        moisture values that each are calculated from the multiple        independent measurements. This statistic is not yet available,        but we’re holding a place for it for the future.

cols 183 — 189 [7 chars] SOIL_TEMP_5_DAILY        Average soil temperature, in degrees C, during the entire day at        5 cm below the surface. Note: CRN stations have multiple co-        located soil temperature sensors that record independent        measurements. This value is an average of 24 hourly soil        temperature values that each are calculated from the multiple        independent measurements. This statistic is not yet available,        but we’re holding a place for it for the future.

cols 191 — 197 [7 chars] SOIL_TEMP_10_DAILY        Average soil temperature, in degrees C, during the entire day at        10 cm below the surface. Note: CRN stations have multiple co-        located soil temperature sensors that record independent        measurements. This value is an average of 24 hourly soil        temperature values that each are calculated from the multiple        independent measurements. This statistic is not yet available,        but we’re holding a place for it for the future.

cols 199 — 205 [7 chars] SOIL_TEMP_20_DAILY        Average soil temperature, in degrees C, during the entire day at        20 cm below the surface. Note: CRN stations have multiple co-        located soil temperature sensors that record independent        measurements. This value is an average of 24 hourly soil        temperature values that each are calculated from the multiple        independent measurements. This statistic is not yet available,        but we’re holding a place for it for the future.

cols 207 — 213 [7 chars] SOIL_TEMP_50_DAILY        Average soil temperature, in degrees C, during the entire day at        50 cm below the surface. Note: CRN stations have multiple co-        located soil temperature sensors that record independent        measurements. This value is an average of 24 hourly soil        temperature values that each are calculated from the multiple        independent measurements. This statistic is not yet available,        but we’re holding a place for it for the future.

cols 215 — 221 [7 chars] SOIL_TEMP_100_DAILY        Average soil temperature, in degrees C, during the entire day at        100 cm below the surface. Note: CRN stations have multiple co-        located soil temperature sensors that record independent        measurements. This value is an average of 24 hourly soil        temperature values that each are calculated from the multiple        independent measurements. This statistic is not yet available,        but we’re holding a place for it for the future.
IMPORTANT NOTES:
1. All fields are separated from adjacent fields by at least one space.
2. Leading zeros are omitted.
3. Missing data are indicated by the value -9999 (or -9999.0).
4. There are no quality flags for these derived quantities. When the raw     data are flagged as erroneous, these derived values are not     calculated, and are instead reported as missing. So these fields may     be assumed to always be good (unflagged) data, except when they are     reported as missing.
5. The update files are named according to the calendar date of the     observations.  The daily records are generated several times during     the next calendar day so as to insure that late arriving data are     included.  Early in each month, the annual-station files will be     reconstructed, both to add the previous month to the current annual     file, and to backfill data that were received through manual recovery     at the station site as opposed to being received via satellite     transmission.

Advertisements

Leave a reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s