CDIP banner
CDIP banner recent historic documents
FAQs & Summaries
  History and Funding
  Program Goals
Wave Measurement
  Wave Generation
  Wave Dynamics
  Irregular Waves
  Spectral Analysis
  Gauging Waves
  Hurricane Events
  Tsunami Events
  Underwater Sensors
  Surface Buoys
Data Acquisition
  System Organization
Data Processing
  System Organization
  Quality Control
Data Management
  Stations and Sets
  Files and Storage
CDIP Products
  Data Formats
  Web Products
  COOS Integration
  Wave Eval Tool
  Custom Products
  NDBC Dial-A-Buoy
  Access Instructions
Related Links

                    META_PROC - DF FILE PROCESSING

Under CDIP's new processing structure, meta_proc is the program that
manages the processing of diskfarm files. It handles all data types - 
energy, temperature, wind, etc. - and produces all of the standard CDIP data 
products: spectral files, databank files, and parameter files. 

A. The processing archive

All of the information that meta_proc uses to manage the processing of df
files comes from the processing archives. A station's processing archive
(e.g. .stations/095/095.proc) is basically an instruction book, explaining 
exactly how that station's files should be processed. Since the processing 
procedure for a station can change over time, archive files are organized by
time frames. Thus once meta_proc determines the time of a file, it
can consult the processing archive and determine the correct processing
procedure. (See .docs/processing/processing_archive.txt for more details.)

B. Program flow - pre-, mid-, and post- processing

To process a file, meta_proc first opens and loads the processing archive, and
then it assembles a list of tasks to be completed. It categorizes these
tasks into three groups: pre-, mid-, and post-processing.

  I. Pre-processing: Currently there is only one type of pre-processing 
     task: gauge comparison. This pre-processing is used to ensure that 
     pressure sensors in close proximity (e.g. within an array) are in
     agreement. If one of the sensors does not agree with the others, it is 
     flagged bad and not processed further.

 II. Mid-processing: The bulk of meta_proc's work comes in mid-processing.
     All of the spectral, databank, and report products are made by the 
     mid-processing routines. There are, in fact, two types of mid-processing: 
     default and combined. 
     a. The default processing works with single-sensor data - a 
        temperature time series is run through the temperature routine, 
        a directional buoy file is run through datawell processing, etc.

     b. The combined processing uses multiple files to generate products - 
        a wind speed file and a wind direction file are combined to 
        produce a databank wind entry, four water column files are put through
        array processing to produce a directional spectral file, etc.

III. Post-processing: Currently there is only one type of post-processing:
     parameter file generation. During the mid-processing, all of the
     parameters calculated for each product are saved. Meta_proc's final task
     is to add these parameters to the appropriate parameter files.

Two special data structures are used to pass information between meta_proc
subroutines. If any pre-processing is performed, an 'energy_block' is used
to pass the edited time series, the transform coefficients, etc. into the
mid-processing. Similarly, the mid-processing routines write parameter data
into the 'pm_info_block', which is then passed into the post-processing.

  INPUT FILES                PROCESSING TASKS                  OUTPUT FILES
  -----------                ----------------                  ------------

                             'meta_proc rd08300200201150845'


  083.proc           ->      Load processing archive


  ix08300200201      ->      Read index, assemble df file list


  df08301200201...   ->      PRE-PROCESSING:
  df08302200201...   ->        gauge_comparisons
  ...                ->                          \
  df08306200201...   ->                           \
                                    V             /
                             MID-PROCESSING:             -->  sp08301, sp08302..
                               energy                    -->  de08301, de08305..
                               directional               -->  sp083p2, dd083p2..
  df08307200201...   ->        barometer                 -->  db08307200201
                               tidal             \       -->  + reports, .update
                                    V             /
                               run_parameter             -->  pm083p1200201


C. Subdirectories and subroutines

Descriptions of meta_proc's major subroutines are given below.

   gauge_comparisons - 	pre-processing agreement checks for pressure sensors.
                        Can be used for any number of sensors; a prerequisite 
			for array processing.

   energy - 		processes single-point pressure sensors and 
                        non-directional buoys.

   detox - 		processes directional buoy data.

   temperature - 	processes temperature data.

   wind - 		processes wind and wind buoy data.

   directional -	performs array processing on wave data.

   update_reports - 	routines for writing to the .mondat and .update reports.

   parameter -          routines for writing to the parameter files.
Official UCSD Web Page