ESP, CEP, Partially ordered events, all over again

[DLuckham]

Discussion of the differences between Event Stream Processing (ESP) and Complex Event Processing (CEP) has cropped up again in the eGroups. Seems the correct place for this, since it may well lead to a partial ordering of threads if it continues, is here on the Forum!

The term "Event Streams Processing" was introduced (so I'm told) by a vendor who was getting resistance from potential customers to the term "Complex Event Processing". Software was "complex" enough without it actually doing "Complex Event Processing" - thank you very much! Other vendors reported no such push back from customers at all. Over time, the first vendor rethought the issue and decided maybe introducing the terminology ESP was a mistake. After all, that vendor claimed to be doing, and was in fact doing, CEP. One wanted to be selling everything, simple software doing complex things, simply! Dream on!

The two terminologies are both useful. And the issue that keeps on arising is "what's the difference"? And of course, marketing departments always want to be selling a product that "does both very well" - which tends to raise a level of confusion that doesn't seem to ever go away.

From the Event Processing Glossary:

Abstraction: An event is an abstraction of a set of events if it summarizes, represents, or denotes that set of events.

Complex event: an event that is an abstraction of other events called its members.

Examples:
• the 1929 stock market crash (an abstraction denoting many thousands of member events, including individual stock trades),
• the 2004 Indonesian Tsunami (an abstraction of many natural events) ,
• a CPU instruction (an abstraction of register transfer level (RTL) events),
• a completed stock purchase (an abstraction of the events in a transaction to purchase the stock).
• A successful on-line shopping cart checkout (an abstraction of shopping cart events on an on-line website).

Relationships between events: Events are related by time, causality, abstraction and other relationships. Time and causality impose partial orderings upon events.
Simple event: an event that is not an abstraction or composition of other events.
Complex-event processing (CEP): Computing that performs operations on complex events, including reading, creating, transforming or abstracting them.
Event stream: a linearly ordered sequence of events.
Notes: Usually, streams are ordered by time, e.g., arrival time. An event stream may be bounded by a certain time interval or other criteria (content, space, source), or be open ended and unbounded.
A stream may contain events of many different types.
Event Stream Processing (ESP): Computing on inputs that are event streams.

Is all ESP contained in CEP? Yes. Is all CEP contained in ESP? No. So ESP /= CEP? Yes. What's the difference? Ah ..........
Gentlemen (of both sexes) start your engines!

[DLuckham]

If I might be allowed a quick start on this one:
To my mind ESP is marketing terminology. The claimed advantage being speed (by limiting the operations and abstractions performed on the events, usually stock market feeds - i.e., streams). Although the marketeers didn't want to fess up to the limitations!
For example, in a claimed ESP engine, an event relationship like --> (followed by) would have the semantics of "comes before in the stream order of arrival".
Whereas in a CEP engine, "followed by" would be "," and the operator "-->" might have the semantics "is a cause of".
Result, a richer set of abstractions is possible in the latter case, but usually at reduced processing speed.

Final analysis: the inputs being event streams is irrelevant. Its the abstractions that can be computed with the events themselves that is the differentiator.

But if your event processor does take an event steam input and limits its computations to the stream relationships (order in the stream and timing) then it is doing ESP.

[hgilde]

David, I don't believe that anyone is using ESP in marketing at the moment as a positive aspect of their product. Even Mark Palmer wants the term to go away. So if anyone is still talking about ESP, it's in a negative sense, like "that other product only does ESP while ours does more broad CEP."

My main problem with arguments on this topic are that they try to define what is complex enough to qualify as complex. Everyone seems to say that unless something meets their own definition, it is not CEP. But no one can provide a definition for CEP that is less broad than "operations on complex events", meaning operations on composite events, and everything can do that.

So all of these arguments boil down to "if it doesn't do what I want, it's not CEP." What's the point of discussing that? The first step has to be a positive assertion on what a term means, or else the arguments couldn't get anywhere.

Some arguments go so far as to claim that "real" CEP products will be magical constructs that just do everything you might need. And the assertion seems to be that only true visionaries see this future of CEP. Again, not much to discuss there except the possibility of a naked emperor.

If someone could make positive assertions, that would be easier to discuss than simply claiming that undefined terms are not what many people think.

[opher]

I agree with Hans here. There was an attempt in the beginning to say that there are two markets - simple and quick vs. complex and slow. No vendor classifies itself today in either of them; the term ESP has disappered from the market, the term "stream processing" still exists in the academic community, and also getting to describe things like video stream processing, audio stream processing, news stream processing etc.. Since all vendors call themselves CEP vendors, there is no use classifying anybody as ESP, one can classify CEP products in various usefull categories - i.e. according to - non-functional properties, type of applications, programming model, environment (stand-alone or part of middleware), level of tooling etc.. - but I don't see any real motivation to look at "ESP engines", and as far as I am concerned, this term can be omitted from the glossary.

cheers,

Opher

[Tim Bass]

Hi David,

I don't think the issues are as simple as some folks are making them out to be (then again, what is!)

For example, dropping the term ESP from the vocabulary will not put an end to the underlying issues; most of which seems to have it's roots in folks with a lack of operational experience dealing with truly complex event processing problems, which I have elaborated on in my web log, drawing from a old Buddhist proverb or story,which I paraphrased here:

CEP and the Story of the Fish

Furthermore, I happen to agree with Claudi, who resurfaced this issue. Reductionism is not the right answer - and as Claudi eloquently elaborated, most, if not all, of the current products in the market are not really doing CEP, they are doing SEP (simple event processing) - very little about what is happening, from a software engine processing perspective, has to do with "complex event processing", I am sorry to say.

I hope that good technical thought will drive this discussion, not marketing hype and counter points by well intended folks with little actual operational experience working to solve complex detection-oriented problems in cyberspace.

Yours faithfully, TIm

[hgilde]

I have also posted on my blog about this conversation. The title of my post is "CEP is Magic".

http://hansgilde.wordpress.com/2007/12/17/cep-is-magic/

Hans

[opher]

I am not sure if I am the fish from Tim's story, or just a frog hiding somewhere in the background, but I have answered with a story of my own - entitled: " CEP and the captured traveller"...
http://epthinking.blogspot.com/2007/12/cep-and-story-of-captured-traveller.html

have fun.

Opher

[Tim Bass]

Hi Opher!

No, I was not thinking about you when I wrote the story of the fish.

In fact, I must not have written the story very well, because to understood the story of the fish, is to know that we are all like the fish, in certain aspects of life, and there is nothing negative to be gleaned from the story.

I have elaborated here, in this post:

End Users Should Define the CEP Market.

Yours faithfully, Tim

[opher]

Hi Tim. So let's agree to disagree here, I think that I have already said everything I have to say, and hate to repeat myself. One point -- since you have based the CEP definition on David Luckham's intentions about what is CEP -- well, I don't have the ability to read David's mind, so I'll let him explain his intentions. Reading his book, papers, and the glossary definitions he co-leads, I got the impression that "complex event processing" in all of these is interpreted as "processing of complex events" - but, as you know, English is not my mother tongue, so I may fail to understand subtle aspects of the language

cheers,

Opher

[Tim Bass]

Hi Opher,

I am not sure we disagree, but we seem to have different perspectives on what classes of problems businesses need to solve and the relationship to the term, CEP.

I read David's books and his papers, all in detail. He was quite clear, based on my read of the material, that CEP was a new technology for detecting patterns or situations, both opportunities and threats, in a complex, perhaps unbounded, distributed networks.

His work has mentioned fraud detection, information security, and a host of other domains where their is a great deal of prior-art in this field; hence, there must be some new "value added" to the state-of-the-art, because detection-oriented problems, and their solutions, have existed for many years.

David's papers, and prior work, also talked about the simulation and debugging of distributed systems. He used phrases like "event cloud" to describe this large, complex, distributed network. He discussed partial order and the complexity of determining cauasality.

David's background as a distingushed professor was in AI and a significant portion of his work was funded by DARPA, who fund research related to complex problems and systems, especially detection-oriented problems. His team's work on Rapide was an architectural abstraction to help folks simulate and debug large-scale, complex systems, as I recall.

Hence, without the "noise" of the marketing folks, I think the term complex, as it describes event processing, is fairly straight forward. The notion of fraud, for example, is a complex abstract object. Detecting fraud, in the mist of much more "noise", is a problem with a very tiny signal-to-noise ratio, generally speaking. The environment is constantly changing and very little is static.

This is the environment. This is complexity. The use of the term "complex" in information systems is consistent with the problems that need to be solved.

In closing, I had no idea that "complex event processing" is not "intelligent" enough to encompass advanced techniques. Your views are quite new to me! If that is the case, the market is going to be quite small, as I do not think companies will be willing to spend much money on "unintelligent CEP" solutions.

There is little value in "unintelligent complex event processing", IMHO.


Yours faithfully, Tim