Comments on CEP as a discipline

[hgilde]

Here are some comments and questions on CEP as a discipline, based on recent experience writing the Knol.

I happen to find use in some of the software that bills itself as CEP when working with real-time data. Mostly, I find that this software adds some good functionality that works in cooperation to traditional techniques like custom coding, or connecting ILOG or some statistical software to a message stream.

When I talk with people involved in mathematical finance, signal processing or pattern detection and analytics, and if they have a need for real-time processing, I'll show them some of this "CEP" software. Usually they get the idea pretty quickly and determine that to use this software, they would need to evaluate the EPL and other functionality in the context of solving their particular problem.

But then the question comes: "so, what exactly is CEP?" Is it just a term for "all the stuff involved in real-time processing" or is there some concrete structure to CEP that can help me in a definable way? Most people, in my experience, who do a little research into CEP find that, even though there is a book (the PoE) on this topic, it is very rarely referenced. Again, most people see a value in the products themselves, but don't understand exactly what CEP purports to add to existing disciplines.

For example, a friend read the Wikipedia page and asked me if all there is to CEP is the practice of looking at signals (in the form of events) and inferring a context, state or situation? He asked this in a skeptical tone because there are plenty of better established disciplines that deal with this.

He then asked me if there is any component of CEP that is not handled better by another, better established discipline.

So I started thinking about whether CEP wants to add anything, or if it's just an umbrella under which people combine ideas from other disciplines. In other words, is there a discipline called CEP?

If not, then what is CEP?

Is it an architecture paradigm? And if so, how is it different from an EDA?

Is it a collection of terminology and design patterns for detecting patterns in events? If so, in what way is it more useful than (or even as useful as) the multitude of existing terminologies and design patterns?

Is it the integration of several disciplines in order to process events? And if so, how exactly does CEP integrate these disciplines, other than just listing a bunch of stuff and stating that you might want to use some of it?

Does it have something to do with pattern detection? If so, what exactly? Does this boil down to using causality? And if so, what about other disciplines that also have algebraic operators like causality?

So I thought that I'd start writing the Knol in order to see if I could discern some concrete component to CEP. And I have to say that I found some things, but putting it together is tough.

I was trying to stick to CEP literature rather than just supplying my own advice on building software and detecting patterns. And clearly I was able to make some progress, but I have to say that there is no clear path to describing exactly what CEP adds to the many existing disciplines in this field. There are surely some design patterns to describe, although most of them come from other disciplines. And the work on causality and the causality operator is applicable to many problems.

Of course, no one said that CEP has to be its own discipline. It would be fine, I think, to use CEP as a word to describe the building of real-time applications or real-time pattern detection, or whatever.

So at the moment, what I have found is some interesting work on causality and event-related terminology, and mostly references to those disciplines from which CEP gets its design patterns.

So does that summarize CEP? I don't mean to imply something negative here, because even as a term for combining many disciplines, CEP has value.

[DLuckham]

Hans,
Let me understand. First you wrote the knol. Then you got to thinking that maybe there wasn't anything in CEP?
Or was it the other way round, you got to thinking, then you wrote the knol?

Actually, it is fair to say that some of CEP can be found in other disciplines. Event processing has been going on in one form or another, for the past 50 years. Simulation, Networking, Active DBs, Middleware. I think I deal with that in part 1 of a short history,
http://complexevents.com/?p=321
So its reasonable to say "we were doing that before".

CEP emphasizes certain aspects of event processing such as event patterns, event abstraction and event hierarchies. And CEP is about the technology for this kind of processing.

However, only the simplest aspects seem to get discuss here most of the time. Which speaks to the educational function of the website. But technical articles are very welcome too! Send them!

We don't often deal with the technologies and the research problems:

what is a good event pattern language design? Is SSQL the best we can do? I certainly dont think so!
How can we implement pattern detection for complex patterns, say patterns with 20 or 30 event templates in them, or long-lived patterns?
What are the linear algorithms (say linear in the number of events in the pattern)?
Can we use causality in patterns to cut down the number of spurious matches?
How about distributed event pattern detection?
How do you know the rules engine you're using is doing pattern matching correctly? Did it miss a million dollar trade yesterday - you'll never know!

Nothing has been done yet on defining event hierarchies for, say, business processing problem domains (see the discussion earlier in this forum).
There are no libraries of event patterns yet.

CEP has only just begun. The foundations are unexplored. Its an open field of research issues. And of course, just to complicate matters, some of the most challenging work on applications development is being done in stealth mode so we dont get to hear about it. Competitive advantage etc. Must be something in it, I guess!

I want to see more Open Source stuff in CEP. More event pattern engines like Esper, but based on new event pattern languages.
The Orbitz open source effort should be strongly supported.
Here are the links:

Extremely Reusable Monitoring API (ERMA):
https://launchpad.net/erma
http://erma.wikidot.com/

Graphite:
https://launchpad.net/graphite
http://graphite.wikidot.com/

Press coverage:
http://www.infoq.com/news/2008/06/orbit ... ource-erma
http://news.cnet.com/8301-13505_3-9979041-16.html

[Tim Bass]

Nothing has been done yet on defining event hierarchies for, say, business processing problem domains (see the discussion earlier in this forum).
There are no libraries of event patterns yet. .... CEP has only just begun. The foundations are unexplored. Its an open field of research issues.

Hi David,

Excellent comments. I agree with you.

In my opinion, and I am confident that Marc Adler over at Citi agrees, we see too marketing, fuffy self-promotion, and (in prior years) "my engine is better than your engine" discussions; but few seem to be working at the level of patterns and abstractions for representing the knowledge, or key indicators, for a lack of a better term.

Folks mostly "dumb" CEP discussions down to performance benchmarking against some simple trading algorithm, as if this is the whole of CEP, algo trading. Unfortunately, the "benchmark" is latency vs. "who can detect this pattern / model of this critical complex event" with the most accuracy.

For the past few weeks I have been working on performance tuning a web application, discussed in The Attack of the Spiders from the Clouds

In following on to that post, I would like to see an event stream processing application that can receive Apache access log file events and detect, with nearly 100 percent confidence, which networks and IP addresses are robots, not human users, compare bot discoveries to a database and then update the database, robots.txt and iptables (blocking command) based on rules. In the case, the pattern of the behavior of a web robot (the model) needs to be defined as well as the ECA rules.

We are missing these accurate, working models (patterns) and, in my opinion, as you said, these models would be great for the future of CEP. WIthout accurate models, the engines are not so useful.

Not to many businesses want to pay big bucks for "high performance, but not-so-smart, engines", they want "engines with a lot of useful patterns/models based on their business". I see this in every continent I travel. Folks are interested in a higher level discussion - knowledge, models, key indicators.

Cheers.

Yours faithfully, Tim

PS: C-level executives get paid not to believe in "steath mode", "secret projects", "the easter bunny" , "the tooth fairy" or "santa claus" - they all just myths that serve one or more purposes. In general, CEOs and CIOs believe in facts, not myth, not secrets and not "behind the curtain" success stories.

[hgilde]

I didn't mean to imply that there isn't anything to CEP. If that's how my post reads, it was a mistake on my part.

I wanted to share my experience of how hard it is to summarize and explain CEP. It's easier to give people a gut feeling about CEP than to address a concrete description. I'm not an evangelist, so I don't rehearse such things. But coming from the perspective of a user, describing something of possible use to a colleague, I do find that CEP is tough to explain in concrete terms. Unless my experience is unique (and I know it's not), I think that this is a topic of interest to anyone involved in CEP.

So I am sharing common questions. I would say that directly addressing some of these questions would go a long way toward clarifying CEP in the mind of the average user.

For example, one might look up the term CEP and find a phrase like "CEP can be used for XYZ." But that begs the question "what exactly is the concrete component of CEP that can be used for XYZ?" More often than not, the answer is to pull in language and concepts from another field and describe how that can be used. But note that this is not a description of CEP, but rather a description of another field.

So what I have done in the Knol is to describe CEP as an umbrella term for many disciplines and technologies that deal with events. CEP adds some terminology and ideas, but is mostly composed of components drawn from outside. I wanted to see if this is how you and others see CEP.

Your post here has given me more ideas to clarify the Knol, thanks for that.

[Tim Bass]

Actually, it is fair to say that some of CEP can be found in other disciplines. Event processing has been going on in one form or another, for the past 50 years. Simulation, Networking, Active DBs, Middleware.

{ .... }

CEP has only just begun. The foundations are unexplored. Its an open field of research issues.

Kindly note my response, On CEP as a Discipline.

Yours faithfully, Tim