could you explain or help?
Thanks
Jörg

http://data.giss.nasa.gov/gistemp/sources/GISTEMP_sources.tar.gz

Because that is the link given on the relevant web page:

http://data.giss.nasa.gov/gistemp/sources/

When we unpack this tarball, it used to make a directory called GISTEMP_sources. Now it makes a directory called GISTEMP_sources_open. Our preflight.py/fetch.py code is now independent of the name of this directory.

We’ve never used the GISS_Obs_analysis link.

We fetch various files such as station metadata from the GISTEMP source
archive. The top-level directory in this archive has changed name from
GISTEMP_sources to GISTEMP_sources_open. Our preflight code fails as a
result. We should be flexible about this: for instance, by using a regular
expression to match either old or new directory names.

Is that URL correct?

ftp://data.giss.nasa.gov/pub/gistemp/GISS_Obs_analysis/

Exists.

ftp://data.giss.nasa.gov/pub/gistemp/GISS_Obs_analysis/GISTEMP_sources/

Exists.

GISTEMP_sources_open doesnt exist

MISSING: input/ushcnV2_cmb.tbl
MISSING: input/mcdw.tbl
MISSING: input/ushcn2.tbl
MISSING: input/sumofday.tbl
MISSING: input/v2.inv
MISSING: input/oisstv2_mod4.clim.gz
PROBLEM: Tried fetching missing files but it didn’t work.
====> STEPS 0 to 5 ====
Traceback (most recent call last):
File “tool/run.py”, line 225, in
sys.exit(main())
File “tool/run.py”, line 206, in main
data = step_fn[step](data)
File “tool/run.py”, line 56, in run_step0
data = giss_io.step0_input()
File “/Users/mosher/Downloads/ccc-gistemp-0.4.1/tool/giss_io.py”, line 674, in step0_input
input.ushcn_stations = read_USHCN_stations(‘input/ushcn2.tbl’, ‘input/ushcnV2_cmb.tbl’)
File “/Users/mosher/Downloads/ccc-gistemp-0.4.1/tool/giss_io.py”, line 609, in read_USHCN_stations
for line in open(ushcn_v1_station_path):
IOError: [Errno 2] No such file or directory: ‘input/ushcn2.tbl’

It struck me as notes one would write to oneself when deep into coding where you live and breath the code like it was your native tongue. But it was helpful to me to just get my feet back under me WRT gisstemp code.

Part of this is I’m putting together a bunch of metadata for any one to use, so I’m painfully aware that I better get all the ids correct etc
so some of the id correction stuff was important, also learning R and trying to get the other data ( antartica ) into a GHCN format just as a R learning tool.

I was hoping for some clarification as Im trying to understand exactly what files gisstemp uses, what get added, what gets modified, what gets, disgarded, the various errors in station ids.

Let’s say with Zeke’s program its easy to understand. He uses GHCN. v2mean.

Anyways, I’m not being critical of the file or the operation I’m just trying to understand it.

When I get some time, I’ll try to come up with clearer questions.

That looks like a copy of most of the file doc/step0-update-notes, which I wrote in early 2009-12, as we were working towards ccc-gistemp release 0.2.0 – our first all-Python release, which we made on 2010-01-11. There is a corresponding file doc/steps1-5-update-notes, which I wrote at the same time. These files can be fetched with SVN or browsed, with revision history, here.

I wrote these documents to help myself to understand, and to communicate to ccc-gistemp people (mainly David Jones at the time), changes that GISS had made to GISTEMP over the preceding year or so. I discarded these notes shortly after they were written, and haven’t revisited them since. So some of them may be terse, opaque, ambiguous, or downright wrong. But the summary at the top of the file, which I will have written last, is probably both accurate and sufficient.

At the time, in 2009-12, we were working towards our first all-Python version of ccc-gistemp, release 0.2.0 which was actually released on 2010-01-11. When we started ccc-gistemp, back in 2008, ccc-gistemp 0.1.0 was equivalent to GISTEMP in 2008-09 (the document says 2008-09-11, although I think the actual GISTEMP release may have been on 2008-09-10). We wanted ccc-gistemp 0.2.0 to match current GISTEMP sources. So I downloaded the GISTEMP sources released on 2009-12-03, ran diff over the two GISTEMP source trees, and spent a while pondering the results and writing these two documents (step0-update-notes and steps1-5-update-notes).

I emailed Reto Ruedy at GISS on 2009-12-17 to ask about the first version of these documents (I sent him a link to the SVN browser); his reply informed my next update (r102).

So this document isn’t really about versions of data (except inasmuch as the GISTEMP changes included a switch from USHCNv1 to USHCNv2, and some updates to SCAR data and metadata); it’s about versions of code.