CDIP banner
CDIP banner recent historic documents
sub-menu
Documentation
 
FAQs & Summaries
Glossary
Publications
 
Introduction
  History and Funding
  Program Goals
Wave Measurement
  Wave Generation
  Wave Dynamics
  Irregular Waves
  Spectral Analysis
  Gauging Waves
  Hurricane Events
  Tsunami Events
Instrumentation
  Underwater Sensors
  Surface Buoys
  Meteorological
Data Acquisition
  System Organization
  Hardware
  Software
Data Processing
  System Organization
  Software
  Quality Control
Data Management
  Stations and Sets
  Files and Storage
CDIP Products
  Data Formats
  Web Products
  COOS Integration
  QARTOD
  Wave Eval Tool
  Metadata
  Custom Products
  NDBC XML/NWS Format
  NDBC Dial-A-Buoy
  Access Instructions
 
Related Links

Data Acquisition

From its inception, CDIP has considered automated data acquisition to be a core responsibility. Data is never merely recorded locally, at the sensor; it always transfered back to a central archive. This is done at regular intervals, without any operator intervention. For a program monitoring such a vast spread of locations - in 2003, for instance, CDIP stations ranged from Guam to Georgia - there are two distinct challenges in this process. First, data from the sensors must be organized and stored locally. Second, these data must be tansferred back to a central location for processing and analysis. Over the course of its history, CDIP has employed a variety of hardware and software techonologies to meet these requirements in a reliable and cost-effective manner.

System Organization

As noted above, CDIP's data acquisition is a two step process. The first step is recording the stream of measurements from the station's sensors on a local computer called a 'shore station'. This is a constant, non-stop process; the shore station is continuously linked to the sensors and records their readings - generally one or more sample per second - 24 hours a day, 365 days a year.

The second step in the process is to transfer the data from the shore station back to CDIP's central data processing and archiving facility. This facility, located in La Jolla at SIO, is known as the Lab. To accomplish this, the Lab's computers contact the shore stations once or twice per hour, using a modem or network connection to collect all the accumulated data.




One key aim of the acquisition system is to keep CDIP's data as up-to-date as possible. Most stations are now contacted every 30 minutes, and the data they return are generally available on the website within minutes. (All the data on the web appear to be at least 30 minutes old because the files and products are labeled with start times, which means that a file running right up to the current time has a start time of 30 minutes ago.) The shore stations themselves also feature several display pages with real-time data straight from the sensors. In many places CDIP's shore stations are located in the offices of local users like harbor masters and lifeguards, who can refer to the most current measurements at any time.

Another key aim of the system is reliability, and to this end the shore stations maintain a local archive of all data acquired by their sensors. When the shore-to-Lab link experiences a failure - i.e. if the network or modem connection goes down - the local archive can then be used to recover any data which were not successfully transmitted back to the Lab. Note, however, that none of CDIP's sensors are equipped to archive data themselves. This means that if there is a failure on the sensor-to-shore link - as occasionally occurs because of radio interference - data are lost and cannot be recovered.

Hardware

Although the overall setup and operation of CDIP's data acquisition system has remained remarkably consistent since the mid-1970s, technological advances have resulted in numerous changes to the hardware used in the system, both at the shore stations and in the Lab.

Shore stations

In the 1970s and 1980s, shore stations were composed of custom-designed circuit boards that operated exclusively under hardware control. Holding a very limited amount of data, these boards would upload their contents to the Lab when contacted via modem. In the early 1990s, more control and flexibility was acheived with the introduction of the PC-based 'smartstation'. Unlike its hardwired predecessors, the smartstation was based on a generalized architecture, allowing it to bring in data from any type of sensor. Equally important, the smartstations opened up two-way communications between the Lab and the shore stations. Instead of simply dumping data back to the lab, the smart stations sent back diagnostic information and could receive and interpret a range of instructions. This allowed numerous sensor parameters - sample rate, record length, etc. - to be controlled remotely from the Lab. Another major advance introduced by the smartstation was the real-time display: at shore stations where interested users wanted to view the data, video output displayed sensor reading and derived parameters in real time.

At the end of the 1990s a gradual shift was begun to a new generation of shore stations. Based on Sun Microsystems' Ultra 5 and Blade 100 computers, these new stations greatly expand upon the flexibility and control of smartstations. These shore stations' numerous features include:
  • full remote shore station access via modem or network links;
  • a multi-screen, graphical display of all measured parameters;
  • continuous data archiving with a multi-year capacity;
  • a seconday, independent zip-based backup data archive; and
  • a smart UPS that ensures graceful shutdown and startup.
As of 2004, the vast majority of CDIP shore stations are Sun systems, although a few smartstations remain in use.

An Ultra 5 shore station


Shore station dimensions and photos

The Lab

The hardware in the Lab is responsible for contacting all of the shore stations, collecting available data, and archiving it permanently. When CDIP first started operations, this was all handled by a Nova minicomputer. The Nova contacted shore stations with a specially designed modem and communications boards, and then archived the data on nine-track reel-to-reel tapes. In the 1980s the switch was made to PCs. While the modem and communications remained largely the same, the x86 PCs in use archived data to floppy disks in addition to tapes, making the data much easier to handle. The use of a PC as the Lab's main data acqusition machine continued until the end of the 1990s, when the switch was made to a Sun Ultra 10 workstation. The Sun, still in use today, uses standard modems and network connections to contact the shore stations, and it archives all the data collected on a hard drive and DLT tapes.
Evolution of Lab hardware:
Click any thumbnail for full image

Nova 1200: 1975-1980

x86 PC: 1980-1999

Sun Ultra 10: 1999-present

Software

Complementary sets of software are used in the field and in the Lab. For communications and data transfer, kermit is used on both the shore stations and lab computers. Customized kermit scripts transfer data and system information using a number of network and modem protocols. These scripts provide extensive diagnostic information, allowing any communcations problems to be quickly and automatically addressed.

To read data from gauges and manage it locally, the shore stations use a combination of CDIP-developed FORTRAN and C code. The very same FORTRAN libraries are used in the Lab to manage and analyze the data once they come back from the field. For displaying the data and calculated parameters, the shore stations use MATLAB to generate a range of tables and plots. Similarly, in the Lab MATLAB is used for creating a great many of the graphical products found on CDIP's website.

For more details on the data acquisition software and setup, please refer to the documents listed below. Any questions regarding the content of these documents should be directed to the webmaster.
  • Shore station software: - A description of all the shell scripts and compiled code in use on CDIP Sun shore stations.
  • Data/processing flowchart: - A chart showing data flow on the field stations and in the lab, based on a buoy station.
  • Field QC - buoys: - An outline of the quality control checks used by the data acquisition code with Datawell buoys.

Back to top

CDIP's major funding contributors are the US Army Corps of Engineers and the California Department of Boating and Waterways.
Official UCSD Web Page