Certainly no global data. As for arrays, in any application like this they will always be there under the skin.

As to it being “obscure”…I’ve actually found it easier for people to use the OO approach over a procedural one that utilizes arrays and global data. .

I’ve found this useful even when delivering software to other people for their use in particular types of data analysis. The particular application I am thinking of involved using classes in the MATLAB language, which as you know has very crude support for the OO paradigm.

In this particular case, it has been successfully reused by other people who had no previous experience in OO code, but who were very rapidly able to incorporate my software package into their existing software framework.

[Once you give them examples of how to create objects and manipulate them, it's not that different conceptually from creating arrays, loading them with data, then supplying them to a function... it's just a lot easier and a lot more bullet proof.]

Regarding “a truly object-oriented version of the code”: I don’t think that’s the best direction for ccc-gistemp. OO is a powerful style, but it can also be very obscure for newcomers. If we go down that road then before we know it we will have factory classes, I/O monads, and so forth, which would add hundreds or thousands of lines of code which would be impenetrable to any non-programmers. It also wouldn’t make switching modules of code any easier. Python has first-class functions and modules, which give just as much flexibility for a much lower cost in clarity. We do use objects in a few places.

In terms of the simplest experiments, this month I am changing the code to lift all the numerical parameters out to parameters/py.

What would be ideal would be to migrate to a truly objected-oriented version of the code, in which different homogenization codes could be written and “dropped” in as different subclasses as class functions..

And of course the ability to “drop in” Monte Carlo tests in place of real data. I don’t know if you guys have thought of that, but a means for doing that would be really slick.

[I imagine starting with the homogenized code, feeding this back in to an earlier stage of the analysis, but with Monte Carlo errors introduced, and using that to study the efficacy of the various corrections that have been applied.]

Firstly, I am glad you were able to easily use our code. I hope you will find it fairly easy to understand as well, and to develop the details of any experimental changes you want to try.

Your suggestions are exactly the sort of thing we are considering for future work. The next version (0.4.x) will have a common data set structure: i.e. the data objects output by (for instance) the “STEP1″ station-combination step will have exactly the same form as the data objects output by (for instance) the “STEP2″ peri-urban adjustment step (at present step1.py and step2.py follow the Fortran in producing, respectively, a plain-text Ts.txt file and a Fortran binary Ts.GHCN.CL.PA file). All the I/O is going to come out of the core code and move into the tool/ directory. Paul Ollis is working on that right now. Once that is done, it will be trivial for third parties to, for instance, switch steps on or off, or to modify the code for a step, or even to reorder steps.

After that version, or possible the one after, all the core algorithms will be a set of parameterized iterator functions – filters – each of which takes a data set and returns a modified data set. These functions won’t exactly correspond to the Fortran STEPs: for instance the part of STEP0 which combined USHCN data with GHCN data would be one such function; STEP1 currently has a total of 5 functions. STEP2 has one – the peri-urban adjustment – but preparative to that is the calculation of annual anomaly series, from which it is separable. And so on. My hope is to clearly expose the parameters to each of these functions (e.g. the peri-urban radius) and also to make each of these functions switchable (so, for instance, one could choose to omit the peri-urban adjustment entirely, or the St-Helena-adjustment part of STEP1). This will make the sort of experiment you describe a simple matter of changing a line in a configuration file and re-running the code.

Have you guys thought about how to go about making it smooth to turn on or off different parts of the package externally?

In particular (at the moment), I’m interested in comparing the output with and without the UHI correction. I imagine I’ll be able to do this with a few minutes of work, but it would be pretty awesome if there were a way to specify the output file (have the default be same as now), but to be able to change things like whether homogenization and UHI get performed, as well as be able to change the grid spacing and radius in step 3 and so forth.

The main point of these is to look at magnitude of the effect of the various manipulations. Knowing how important they are tells you something about where to spend your QC effort first. I imagine for example homogenization is a big effect, UHI not so much. Anyway, in my experience, this sort of manipulation of the algorithm has always been an important part of the software validation itself.

I could go in and add flags to the code to turn these on and off, change values and so forth, but that seems a bit like duplicate effort. I will probably do so anyway, but I’d love to see the evolution of this code include automating tweaks like this.

Thanks again for your effort.

Carrifck