again with the POSETs

[hgilde]

Ok, I'll be the one to reintroduce the POSET thing here again.

First of all, David, IMO the term POSET is currently being badly abused with respect to CEP. I'm not sure, but maybe you would be surprised at the conclusions that are being drawn using this term POSET. It has become a little like a Cargo Cult, where the notion of processing a POSET takes on a mystical quality.

AFAIK, your references to POSETs in The Power of Events can be summarized by this blog post.

Now contrast this with a recent comment on this other blog post, stating that "a linearly order set is still a linearly ordered set even when the elements of the set are observed out of sequence." This statement implies that it matters if you are processing a sortable set. David, this is crazy talk. If I can take a sortable set, in any order, and find a causality relation between the elements without using the ordering, then it makes no difference whether that set was sortable to begin with. And if I find a causality relation using the implicit ordering of a set of events that is more accurate than another causality relation that does not use the ordering... there is nothing that makes my causality relation inferior, just because I chose to use a known ordering of the events.

And we have not even begun to discuss the fact that there are plenty of people who are interested in an obtaining accurate prediction or some operationally actionable information, and do not care one whit about constructing a causality relation and thus a POSET.

[DLuckham]

I agree Hans, there seems to be rather a lot of confusion going on at the moment.
Confusions are rather like wild fires. They spring up arbitrarily, all over the place, fanned by the winds of rumor and misunderstanding. Put one out and another springs up!

The notion that the activity in a distributed system is a set of events ordered by various relationships, time, causality, ... and that some of these relationships are partial orderings, seems simple enough. So, POSET (partially ordered set of events) was coined.

Some confusions seem to arise from a perception about how events are observed (feeds, order of arrival), and others from a need for simplicity. Yes we do have linear streams of events. Or even just single individual events. But the general form of a system's activity IS a poset under the common event relationships. When we get into less well behaved relationships, mathematically speaking, the structure on the event domain may not be a partial order. These kinds of event spaces are still to be researched.

I dont have time at the moment to deal with each of the confusions.
Suffice it to say that the whole question of the correctness of pattern detection in the current generation of event processing engines needs to be studied. Is this bad? Well, its no different from the software field in general. Do MSFT products behave as expected?

More later .......

[hgilde]

Hi David, I agree that it's perfectly reasonable for one to say that they don't like the currently available pattern detection capabilities. There are so many packages available for analyzing static data that I don't think anyone would expect even one EP product to do everything.

Anyway, this discussion keeps covering the same topics over and over again. I did have a few thoughts about it that I haven't read recently, which I posted here.

[Tim Bass]

Hi Hans and David,

Well, it is good to see the boards up and running again!

Before addressing the topics of causality, order, partial order and posets, let's take a step back.

First, let's discuss the notion of "order" in sets with an example:

Let's take a set of 26 children's wooden building blocks. Each block has a nice letter of the (english) alphabet A-Z imprinted on the block.

I don't think anyone would argue that this is not a totally ordered set (TOSET).

Now, let's take the blocks and put them in a box and shake them up a bit.

Looking down on the blocks-in-the-box, are they still a TOSET?

I don't think anyone would argue, even Hans, that looking down on the blocks-in-the-box, we are still observing a TOSET.

Now, let's take those blocks and transfer them to another box, one at a time, with our eyes closed.

Without respect to the order of the blocks in the first box, we close our eyes and transfer the blocks a block-at-a-time, to another box.

The blocks arrive into the second box without respect to the order of the alphabet.

Now, over time, all 26 blocks are in the second box.

We look down into the box.

Do we still have a totally order set of blocks?

I think there might be somone who would passionate argue, in my opinion incorrectly, that because the blocks arrived in the second box out of linear sequence (not ABCDEFGHIJKLMNOPQRSTUVWXYZ), that the set of blocks is no longer a TOSET.

As many on these forums know, I have repeated asserted that the order-of-arrival does not define the relationship between the elements of the set, generally speaking, because the relationship of the elements of this set, in particular, is independent to the order-of-arrival when transferring the blocks from one box to another.

So, I stand by my assertion that regardless of the passion (and sometimes slander) of Han's arguments and debate points on this topic, that the order-of-arrival in transferring data (or events) from one place to another has very little to do with defining the properties of "order" within a set of event data.

Yours sincerely, Tim

(PS: Han's I really think you owe me an apology.)

[hgilde]

To extend this scenario, let us cover the letters on each block so that there is no longer a way to know what letter is on any block. We put the blocks in a box and shake it up. Now what kind of set do we have? Remember that there is no way to know the letter on each block.

Further, before we cover the letter, we write on a different side of the block a mark, where each mark corresponds to exactly one letter. Without knowing the letter, we have no way to deduce the ordering of these blocks. Now we cover the letter, so that the only identifier that we have for a block is this new mark. Now what kind of set do we have?

Now we write down a set of pairs (m,n) where m and n are one of these marks. We are free to write these pairs using any criteria, perhaps the colors of the blocks, and we have no access to know the letter on each block and so, no way to know this ordering of letters. We do require one criteria for writing these pairs, which is that if we write (m1,m2) and (m2,m3), we must also write (m1,m3). Remember that the m1,m2,m3 are these unknown marks.

Now for these blocks we have the letter, we have the mark and we have this set of pairs what was formed without knowing the letter and thus without knowing the "order".

So we uncover the letters. We now have a total ordering that comes from the letters and a partial order that comes from the list of pairs. And the partial ordering has no relation to the total ordering, because the partial ordering was created without knowledge of the total ordering. So are the blocks an ordered set or a partially ordered set?

Now let us relate this example to the kind of situation described by David in his book The Power of Events.

We have a set of events, each of which has the following fields:

• A GUID (globally unique ID).
• A sequence number (positive integer) that is unique and represents the order in which we observe the event.
• A payload field that can be considered the "value" of the event. This field is also numeric.
• A causality field, which contains the GUID of another event that can be interpreted to have caused this event.

Now remember that the causality field bears no relationship to the sequence number, perhaps the causality field was determined before we observed the event, while the sequence number was determined after we observed the event. Similarly, the payload field bears no relation to either the causality field or the sequence number.

Based on the sequence number, we can form a TOSET. And based on the payload, we can form a different TOSET. And based on the causality field, we can form a POSET.

Would you still say that because we have a sequence number, we should ignore the POSET? If we did not have the sequence number, would you say that because we can order the events based on the payload, that we still have a TOSET and not a POSET?

[hgilde]

Tim, I have reread all of my recent posts and the only thing that I could find to apologize for was some strong language. I considered apologizing for that language until I read your comment on this post on Opher's blog. Although some of my language criticizes your understanding of the issues involved, you have now repeatedly resorted to simply calling me names. I would understand if this happened only on Marc's blog, in the heat of a debate. But now several days later, you continue the tactic. You should be ashamed of this behavior.

I am happy to see that you have again engaged in a debate the facts, per your earlier post on this thread. I look forward to debating using only facts and if I'm wrong, you should have no problem in showing why.

Again, I will repeat my position:

• The ability to process a POSET is not a differentiating factor among EP software. Most or all EP software can easily represent and process POSETS.
• In order to handle causality between events, software must be able to form a POSET from events. Most EP software available today allows the user to form POSETS. Further, no EP software available today requires the user to form those causality POSETs based on the order in which events are observed or based on any other ordering. These points are demonstrated by my previous post where causality is determined by a field in the event.
• The argument that "modern EP software is not useful for advanced detection because they can not handle POSETs" is like saying "modern cars are not useful for flying because they are all painted red". Both of these statements are so close to being correct, but are still wrong.

[hgilde]

I recently posted a comment on Opher's blog that I like so much, I'm going to copy it here. To summarize where this comment came from:

(1) I posted a few bits about this topic. Opher wrote a comment on one of my posts indicating a certain amount of agreement on his part and referencing this entry on his blog expanding on the discussion.

(2) A couple of days later, Tim wrote a comment on that post on Opher's blog with two misunderstandings (AFAIK). First, Tim was under the impression that Opher was debunking my posts, rather than expanding on points that he agrees with. Second, and this part is the most confusing to me, Tim was under the impression that I am or have been arguing "that advanced EP applications do not exist".

(3) I posted the comment below, which I think sums up the situation nicely. I have inserted a minor addition in [brackets] that was not in the original comment.

It is a mystery to me why anyone would think that I argue that advanced EP applications do not exist. I argue simply that classifying "advanced applications" as those that process POSETs is wrong. Saying that "modern EP products do not have the features of more advanced inference engines because EP engines all fail to work with POSETs" is like saying "modern cars do not have the features of airplanes because modern cars are all made of wood."

The first part of each of these statements is correct, the second part makes the whole statement absurd. Processing and even forming POSETs is not what gives something advanced inference capability. Processing and forming POSETs is easy. The hard part is determining the criteria on which to [process or] form the POSET.

Also, not all "advanced" data analysis has the goal of forming a POSET. And not every POSET comes from a useful causal inference. Similarly with backward chaining, not all backward chaining produces useful causal inference and not all causal inference uses backward chaining.

So classifying anything as "simple" or "advanced" based on these one or two criteria of processing POSETs or using backward chaining is misleading. It gives the false impression that data analysis can be classified as "red" and "blue". Would you like the red or the blue analysis, sir?

"Advanced" data analysis has to be more accurate than other methods. Clearly there are cases where sophisticated techniques produce more accurate results. That really depends on the scenario and the kind of data being processed and not on whether those techniques involve POSETs or use backward chaining.

[Tim Bass]

Dear Folks,

Now that I have posted, in this thread, on totally ordered sets, and explained why arrival order is a red herring argument when discussing CEP; I will elaborate on hidden causality, Markov processes and how these concepts are central to the notion of complex event processing, what we refer to as CEP.

In addition, when I have time (sorry, I am simply very busy working cybersecurity issues in Asia at the moment - plus enjoying great food!), I will elaborate on why processing "complex events" implies that there is hidden, or unknown, causality in the event-set under observation; and why processing totally ordered sets and simply enriching (i.e. - performing time-series math), filtering, routing, performing "joins", or mediating that event data is not *really* complex event processing.

In the meantime, as food for thought, I will provide a simple illustrative example to think about, representing my views about CEP, as follows:

(1) Processing market data and running a VWAP against a stream of market data (in this example, a single stock symbol) IS NOT, in my opinion, processing "complex events", or CEP (regardless of how fast you do it.)

(2) Processing market data and trying to determine "why" the market moved in a particular direction IS processing a "complex event", or CEP.

In the future, I will elaborate on why, in my view, both (1) and (2) are both "event processing" however (1) IS NOT EQUAL TO CEP and (2) IS EQUAL TO CEP.

When I have a chance to elaborate (as I did at DEBS 2007 last year), I will NOT use the term "event cloud" or POSETS (as I did at DEBS); instead I will talk in terms of hidden, or undiscovered, causal relationships in a set of events -- events when combined together, form the components of "yet to be discovered" complex events.

[opher]

Since Tim has re-focused the discussion about "what is CEP" and not on the POSET issue - I have noted that there are different interpretations of CEP and summarized my view here: http://epthinking.blogspot.com/2008/05/on-three-meanings-of-cep.html
have fun.

Opher

[Tim Bass]

Hi Opher,

Good post on your blog. You also have an "opinion" so "welcome to the opinionated club"

The issue we all seem to have, or course, with the "glossary definition" is that the term "complex" is subject to wide interpretation.

I tend to interpret "complex" to mean "complex" as related to the term "complex system", since, I assume, "complex systems" produce "complex events". I also, at times, visualize "complex" in terms of "complexity theory":

http://en.wikipedia.org/wiki/Complex_system

http://en.wikipedia.org/wiki/Complexity_theory (refers back to "complex system" above)

Then, I "mentally merge" or synthesize, the intent of David's book on CEP, his subsequent posts on the topic, and his background at Stanford (and DARPA) in debugging "complex distributed systems" as the second guide post in my "foggy, opinionated"reasoning, along with the idea of "complex systems".

Finally, I confirm this technical "opinion" (at least in my own mind) based my hands-on operational experience in IT security and network management where processing streaming events with rule-based systems have serious limitations and scaleability issues (they simply do not work very well at all, except in the most primitive situations), as applied to "complex systems" and other classes of "complex problems".

For example, when fighting cyberattacks a number of years ago at Langley Air Force Base, we used rules, and they worked to a certain degree, but required constant resource intensive maintanence to keep updating the rule base based on the changing dynamices of these classes or problems. That is when I began to apply the JDL to event processing in cyberspace.

You, on the other hand, seem to have a different "opinion" -- "complex" is much lower on the knowledge-value chain, and you say that what I am describing is "intelligent event" processing.

However, on my examination of the established definition of "complex systems" or even "complexithy theory" in my "opinion" you (and others) have picked a definition (or visual concept) for "complexity" and "complex" that is inconsistant with the use of the term "complex" and also missing the "intend" of what David' was hoping to share with the world in his book on CEP.

Cheers.