netCDF Primer   | Software   | netCDF Online     | WOCE Conventions

Dear WOCE DAC/SAC Managers,

First, please pardon the extraordinary length of this email. In the past month, there have been several activities that are moving us towards developing WOCE Global Data V3. We summarize these developments and, more importantly, lay out the vision of what we will accomplish and deliver to the ocean research community by November 2002, the date of the final WOCE conference (WOCE and Beyond).

Finally, we are providing the details necessary for all WOCE data units (ie DACs, SACs) to begin developing data files and other components of the off-line and on-line V3 dataset. Achieving the vision will require some effort on the part of the DACs/SACs, the DIU, as well as the DPC. By sharing the vision and the development plan, all involved will have an opportunity to assess their roles and plan accordingly towards producing what will no doubt be one the richest, well-integrated, and comprehensive suite of global ocean data available.

We are quite confident our goal of an integrated data set can be achieved. However, we are aware of some hurdles that must be overcome and are committed to working with the DACs/SACs, the DIU, as well as the DPC to find solutions. Consequently, the last section of this message outlines an immediate request to DAC/SAC managers for some feedback.

At DPC-14, we discussed the goals of V3:

· V3 will be more complete

· V3 content will be more consistent across DACs

· V3 will be integrated

The DACs/SACs have worked very hard and all signs indicate they will achieve the first goal. The second and third goals are linked. Data files and structure with more consistency are easier to integrate when searching the data or synthesizing it for scientific study. The level of consistency maps directly into the level of integration achievable when using the dataset. Additionally, when such consistency extends not only across DACs/SACs, but also to other ocean and/or climate data sets as well as to modeling and forecast communities, one can then take advantage of tools developed in these communities. V3-compliant netCDF files have been recognized in the netCDF tools we have tested (e.g. ncview, ncbrowse, MatLAB). Additionally, this consistency encourages the use of the WOCE data by more scientists, and by doing so imparts momentum to follow-on efforts that will ultimately improve the WOCE data, widen its use, and increase its long-term impact.

The V3 integration was envisioned to have two components. The first is an off-line tool to search the static V3 CD's for data files of interest. The second was an on-line presence offering, potentially, a fuller feature set and opportunities for dynamic evolution into an improved data and integration tool.

In pursuing these goals, the DPC should be mindful of other complementary efforts such as the DODS system (<http://www.unidata.ucar.edu/packages/dods/>) for data distribution as well as the many netCDF tools (<http://www.unidata.ucar.edu/packages/netcdf/software.html> ) that may assist users gaining the maximum use of the data.

The V3 Working Group (V3WG), led by Reiner Schlitzer, has met to map out a plan to achieve this vision, while remaining mindful of the capabilities of the DACs/SACs and their other efforts to collect and process WOCE data. We are very pleased to share their plan with you.

The V3WG spent considerable time and effort identifying a set of actions required that would lead to an integrated V3 dataset. The plan is comprised of two components:

· Specifications for WOCE V3 netCDF data files

· Specifications for WOCE V3 inventory files

Specifications for WOCE V3 netCDF data files

As described in the articulation of the V3 vision, maximum consistency in file structure, content, and conventions is crucial. After careful consideration of user requirements (based on a survey), the existing WOCE netCDF format conventions, and widely adopted community netCDF conventions, a revised set of conventions for WOCE V3 netCDF files was developed (see sections 1-3 in attachment/links on above mentioned web page). They correspond to the widely accepted "COARDS" standard now prevalent in the model/forecasting community.

While the new specification is lengthy, the differences are largely not in content, but in how the content is organized/structured/named. The differences/improvements fall into three categories:

1) adding a new variable "time" with specified format

2) requiring certain variables and relevant attributes

3) providing more explicit specifications for structure, variables names, and a range of attributes

The new time variable will be the most difficult, yet the most critical to add. The DIU is collecting tools to easily translate between WOCE date/time and the new time variable.

The increased consistency achieved through adoption of the file specification (and suitable search tool) facilitates off-line searching/retrieval of data by date, latitude, longitude, depth, and variable - an option previously achievable only through piecewise extraction and tailored searching.

For those DACs wishing to extend their consistency and make their files EPIC compatible, (<http://www.pmel.noaa.gov/epic/document/convention.htm>), please feel free to add additional variable attributes, but do not alter any convention covered in the V3 specifications. Bernie Kilonsky is most familiar with the EPIC conventions and is willing to help those wishing to pursue this additional enhancement.

Specifications for WOCE V3 inventory files

A critical component of the search/integration tools are the inventory tables that provide the critical parameter space for each WOCE netCDF file. Each WOCE netCDF file will have an entry in these tables. These tables will serve as the foundation for searching the WOCE V3 netCDF files. They will be parsed to provide the user a list of files containing data satisfying their search criteria. A tool that DACs could utilize to parse conforming WOCE V3 netCDF files and produce the required table entries has been identified for testing. Additionally, the DIU will be providing DAC-specific sample entries for the table.

The DIU has developed an off-line search tool. The tool, as demonstrated at the recent V3WG meeting, had a very nice graphical user interface and successfully searched the V2 inventory files produced by the DIU. This search tool will soon be released for testing (see above links for more information). Additionally, the V3WG has identified a likely pathway to establishing an online DODS server for all the V3 WOCE data. The DIU tool along with the inventory tables will provide the fundamental underpinnings to query the server, thus it may be possible for users to access WOCE V3 data on-line. Additionally, the netCDF specifications along with the inventory tables provide all necessary information for users to not only access, via DODS, file information, but also to retrieve V3 WOCE data. Thus we anticipate this will make it easier to expand on the existing DIU tool and/or use other data query tools and interfaces to access the on-line V3 data, thus fulfilling our vision of a dynamic and widely adopted V3 dataset.

Actions requested

By DPC-15, we expect the DACs to demonstrate a V3-compliant CD as well as their inventory table entries (a sample if not all entries). This is an ambitious schedule and one that all DACs may not be able to immediately meet.

We are asking all DACs to immediately review the new netCDF specifications in order to respond to these questions:

a) Are there any technical and/or other objections to adopting the V3 specifications?

b) Can a V3 compliant CD-ROM be developed and demonstrated at DPC-15 (mid-March)? If not, what is the explicit nature of the difficulty (scheduling, cost, people-time, etc.) and when would a first "draft" version of such a CD-ROM be available for review?

Nathan and I will be making personal contact with each DAC manager in the coming weeks to a) insure we are aware of DAC-specific issues that should be addressed; and b) to enlist help as needed for DACs.

Finally, let me again thank all the DACs/SACs for the efforts required meeting these goals. The DPC is fortunate indeed to have such willing partners devoting their time and energy to the WOCE effort. The V3 dataset will be an unparalleled achievement and one that will be looked upon in hindsight as one of the most enduring legacies of the WOCE program.

Best Regards,   David Legler and Nathan Bindoff Co-Chair WOCE DPC