The openPDC is a system that can be used to process complex events and respond to dynamic changes in fast moving data. More specifically, the openPDC can process complex events that can be described as “time-stamped measured values”. These measured values are simply numeric quantities that have been acquired at a source device and are typically called points, signals, events, time-series values or measurements. Examples of measurements include temperature, voltage, vibration, location, luminosity and, of course, phasors. When a value gets measured, an exact timestamp is taken, typically using a GPS-clock for accuracy – the value, along with its timestamp, is then streamed to the openPDC where it can be “time-aligned” with other incoming measurements so that an action can then be taken on a complete slice of data that was all measured at the exact same moment in time.
Although the system was specifically developed to manage real-time streaming synchrophasors and generate time-sorted concentrated data streams using standard protocols, its completely modular based design makes the system a general purpose distributed stream processing engine (see event stream processing). This will have a broad range of uses outside of synchrophasors where real-time streaming measured data needs to be processed and archived, for example: consumer energy usage (smart-grid), seismic metering, high-speed location tracking, fast changing temperature monitoring, surveillance applications, network traffic processing, etc.
Below is an image that shows the openPDC modular system design used to manage streaming measurements. User defined input adapters are used to collect streaming data and assign incoming measurements an ID. Measurements are then sorted by time and passed along to custom action adapters that process the data. Finally, all measurements are sent to output adapters that are used to queue up data for archival. The high level data flow is: map, sort, queue where map is the step of assigning an ID to a measurement, sort is the step of sorting the measurements by time so that an action can be taken, and queue is the step of queuing up measurements so they can sent to different outputs. The system loads user defined “assemblies” to perform all of these steps – that is, users can develop their own input, action and output adapters to provide extensible processing of any streaming measured data. The project includes a variety of adapters to make the system operate as a “phasor data concentrator” as well as to archive time-series measurements into files that can be processed in bulk by Hadoop. The map-reduce code needed to allow Hadoop to mass process the time-series archive files is also included.
- openPDC_Data_Flow.jpg (80.42 KiB) Viewed 493 times