Mutability and aggregations

[woolfel]

A rule in a "CEP system or engine" is not required to "have events in it" or to be triggered by events.
It can be, e.g., purely a constraint on the state of a system, or an invariant on the values of some set of variables.
The "CEP approach" can be in some situations the same as any other approach.
If events are irrelevant, then don't use them!
I'll look at you example of the trader some more because maybe the timing and sliding time windows is what is at issue.

.

I agree if events are not relevant, it shouldn't be used. The reason I ask is this seemingly "simple" example is more difficult than one would expect. Looking at how existing production systems and business rule engines handle it, this type of scenario isn't supported by existing rule languages. Normally, ECA and production rules only define the action when the conditions match. They don't define action when modifications occur. In other words, the rule language would need to treat modification logic as first class concept. The general form of an aggregation rule might be

if/then/modify


if
- the transaction ticker is IBM
- the transaction type is "buy"
- the time is between 9:30 AM and 4:00 PM
then
- calculate the average shares per transaction every 15 minutes
modify
- subtract previous share quantity
- add new share quantity
- calculate average

The modify logic would be optional, since some rules may not need it. Behind the scenes, the rule engine or application could compile the business rule into several rules. When I worked on trading systems, we would recalculate the aggregate from the dataset. Other pieces of data could have changed, so we couldn't just subtract, add, divide.

I was hoping those working in the CEP space have come across the same problem and found other techniques that work as well or better. My solution was to design a custom rule language for the business user and write a custom rule compiler to generate the appropriate executable rules. thanks to everyone for their response, it helps get a better understanding.

peter

[jdwootton]

Aleri has AFAIK a unique view on this. We allow events to explicitly update/change or delete information reported by previous events. We use keys to associate the change/delete to the original (or more precisely to the current known state). So in Peter’s original example, the model can recognize a “cancel” (aka delete) operation and remove the open order from the aggregation. The aggregation would thus represent the aggregate value of actual orders based on all current information – i.e. after having applied all changes and cancellations received to date.

Now at first glance, you might interpret this as meaning that events are mutable in Aleri, which would cause people to point out that events are immutable. But…

Clearly events are immutable. As David puts it “You cannot delete or reverse history”. But are we talking about the event itself or the message that reported the event? In the EP community and in discussions around the EP glossary, we have discussed extensively the overloading of the term “event” where there is the actual event that occurred in the real-world and the message or object or, as Opher and Peter (Niblett) call it – the programming entity that represents the event. What if a real-world event is incorrectly reported. When you become aware of the fact that it was incorrectly reported, what do you do with that information? It’s going to be situation dependent- i.e. depends on what you are doing with the information. In one situation, you might care about reported events as they were reported. In others, you might only care about what really happened and want the erroneously reported events removed from your analysis. So on the surface it’s hard to disagree with David’s statement, but it’s also not as absolute as it may seem. Which “events” represent the reality you care about? The events that were reported to you or the events that really happened?

But it’s not limited to erroneously reported events. The same question could be applied to which events you are interested in looking at. Back to Peter’s example and the erroneous order. Both the order creation event and the order deletion events were actual events that can’t be denied. The question is: How are you using this information? The full history of events that create new orders, delete open orders and change open orders might be useful to you. Chances are, however, all you really care about is the current set of open orders, in which case a system that applies changes and deletes so that your aggregate view always represents the current set of orders is very useful.

So the fact that Aleri recognizes change and delete events doesn’t make events themselves mutable, but it does give you flexibility in how you analyze the events that you care about.

[woolfel]

Aleri has AFAIK a unique view on this. We allow events to explicitly update/change or delete information reported by previous events. We use keys to associate the change/delete to the original (or more precisely to the current known state). So in Peter’s original example, the model can recognize a “cancel” (aka delete) operation and remove the open order from the aggregation. The aggregation would thus represent the aggregate value of actual orders based on all current information – i.e. after having applied all changes and cancellations received to date.

Now at first glance, you might interpret this as meaning that events are mutable in Aleri, which would cause people to point out that events are immutable. But…

Clearly events are immutable. As David puts it “You cannot delete or reverse history”. But are we talking about the event itself or the message that reported the event? In the EP community and in discussions around the EP glossary, we have discussed extensively the overloading of the term “event” where there is the actual event that occurred in the real-world and the message or object or, as Opher and Peter (Niblett) call it – the programming entity that represents the event. What if a real-world event is incorrectly reported. When you become aware of the fact that it was incorrectly reported, what do you do with that information? It’s going to be situation dependent- i.e. depends on what you are doing with the information. In one situation, you might care about reported events as they were reported. In others, you might only care about what really happened and want the erroneously reported events removed from your analysis. So on the surface it’s hard to disagree with David’s statement, but it’s also not as absolute as it may seem. Which “events” represent the reality you care about? The events that were reported to you or the events that really happened?

But it’s not limited to erroneously reported events. The same question could be applied to which events you are interested in looking at. Back to Peter’s example and the erroneous order. Both the order creation event and the order deletion events were actual events that can’t be denied. The question is: How are you using this information? The full history of events that create new orders, delete open orders and change open orders might be useful to you. Chances are, however, all you really care about is the current set of open orders, in which case a system that applies changes and deletes so that your aggregate view always represents the current set of orders is very useful.

So the fact that Aleri recognizes change and delete events doesn’t make events themselves mutable, but it does give you flexibility in how you analyze the events that you care about.

thanks for taking time to respond. this is my bias take on the definitions, but having two separate terms would help reduce confusion. From linguistics there's a theory named Sapir-Whorf hypothesis (http://en.wikipedia.org/wiki/Sapir%E2%8 ... hypothesis). The idea is that language creates thought. A common example used by Sapir-Whorf hypothesis is snow. The inuit have specific words of each kind of snow, where as english only has 1. The lack of clear and distinct terms for a concept "can" lead to confusion and muddled thinking. To me overloading "event" to mean multiple things is problematic and detrimental. A descriptive definition would help newbies like myself get a decent high level understanding with minimal pain.

peter

[marco]

...

Related to the issue of mutability is temporal versioning. If the user wants to see the cause of any given event at any given time, the system must provide temporal versioning. If it doesn't, it wouldn't provide accurate information. How many CEP engines address these issues?

I ask because the last 8 months I've been building a multi-dimensional temporal versioning system in my day job. I'm curious to hear if other people have ventured into these issues in the context of event processing.

peter

My guess is that most systems dealing with events has some feature(s) to deal with this. It's so fundamental to event processing so I think most systems would have some way of keeping track of things. (here's one example that I know of personally, where the CausedBy tag is the interesting one)

If your CEP system is automatically keeping track of what event caused another one, there you have a fundamental piece needed in order to build some functionality to find the root cause of some particular event.

Keeping the caused-by chain intact is the easy part. To present this information in a manner which makes it usable is another matter. I would suggest to try to create something domain specific in this area as any generic root cause display/analysis tool can be a bit difficult to understand.

[woolfel]

...

Related to the issue of mutability is temporal versioning. If the user wants to see the cause of any given event at any given time, the system must provide temporal versioning. If it doesn't, it wouldn't provide accurate information. How many CEP engines address these issues?

I ask because the last 8 months I've been building a multi-dimensional temporal versioning system in my day job. I'm curious to hear if other people have ventured into these issues in the context of event processing.

peter

My guess is that most systems dealing with events has some feature(s) to deal with this. It's so fundamental to event processing so I think most systems would have some way of keeping track of things. (here's one example that I know of personally, where the CausedBy tag is the interesting one)

If your CEP system is automatically keeping track of what event caused another one, there you have a fundamental piece needed in order to build some functionality to find the root cause of some particular event.

Keeping the caused-by chain intact is the easy part. To present this information in a manner which makes it usable is another matter. I would suggest to try to create something domain specific in this area as any generic root cause display/analysis tool can be a bit difficult to understand.

Thanks for the link. That covers causality, but it doesn't cover temporal versioning?

<rc:EventHeader xmlns:rc="http://www.rulecore.com/2008/user">
<rc:SecurityInfo>
<rc:Credentials>12345678-1234-1204-1234-123456789abc</rc:Credentials>
</rc:SecurityInfo>

<rc:CausedBy>
<rc:Event eventId="e75cc700-d36f-11dc-95ff-0800200c9a66" eventType="SOAP_CALL_put" index="1"/>
<rc:Event eventId="a88cc945-d36f-44dc-95ff-2423423c9a66" eventType="SOAP_CALL_get" index="2"/>
</rc:CausedBy>
</rc:EventHeader>

In the situation where a "derived event" changes, having the eventId isn't sufficient. For exploration sake, say an event with id "a88cc945-d36f-44dc-95ff-2423423c9a66" was produced by something in the real world. Something else happens in the real world, which ends up affecting the event with the same id. To track exactly which version of a given event was involved in the result, you'd need the version information.

Does rulecore support temporal version, bi-temporal versioning or multi-dimensional temporal versioning? Do any CEP products provide support for temporal versioning out of the box?

[guyos]

In the situation where a "derived event" changes, having the eventId isn't sufficient. For exploration sake, say an event with id "a88cc945-d36f-44dc-95ff-2423423c9a66" was produced by something in the real world. Something else happens in the real world, which ends up affecting the event with the same id. To track exactly which version of a given event was involved in the result, you'd need the version information.

You kind of suggest an implementation approach based on backward chaining method where once "something else happens" you have to find out the version or the correct events that are relevant to compute a pattern. How about a forward approach where you start computing\building the patterns you are looking for as you receive events and if "something else happens" you may "revise" a previous affect of an event on the computation. I am bringing this up because to provide such scenarios as given in this thread you do not necessarily need temporal versioning.
However, in the forward approach there is difficulty in supporting new introduced patterns or revised patterns that need to "look back" at events that have occurred before introducing this pattern. That is another discussion though.

[woolfel]

In the situation where a "derived event" changes, having the eventId isn't sufficient. For exploration sake, say an event with id "a88cc945-d36f-44dc-95ff-2423423c9a66" was produced by something in the real world. Something else happens in the real world, which ends up affecting the event with the same id. To track exactly which version of a given event was involved in the result, you'd need the version information.

You kind of suggest an implementation approach based on backward chaining method where once "something else happens" you have to find out the version or the correct events that are relevant to compute a pattern. How about a forward approach where you start computing\building the patterns you are looking for as you receive events and if "something else happens" you may "revise" a previous affect of an event on the computation. I am bringing this up because to provide such scenarios as given in this thread you do not necessarily need temporal versioning.
However, in the forward approach there is difficulty in supporting new introduced patterns or revised patterns that need to "look back" at events that have occurred before introducing this pattern. That is another discussion though.

The approach I described in the previous post isn't backward chaining. It's truth maintenance from production systems and expert systems. There's plenty of papers that go into great detail on different methods of implementing truth maintenance. Even with truth maintenance, that doesn't solve the problem completely. If the derived event is still within the engine, it should be able to find it and modify it. If the derived event has left the engine, and gone to another system, we may not be able to do anything about it.

This is my bias perspective, but these issues have to be addressed at the language level. As long as it's not defined by the rule language or event processing language, there's going to be a huge gaping hole. That tends to result in users roaming blindly and struggling to find a logical and consistent way of handling it. One thought is that any consumer of derived events needs to use a callback mechanism. This way, the consumer is notified the result is stale. In many ways, it's similar to concurrency management in a distributed system.

Back when I worked on health care systems, we needed to notify all users a given account has been modified. I feel the same thing applies in this case. I like to point out, this also applies to ECA rules. I've implemented this type of functionality a few different ways in the past and none of them are ideal. Usually, it means 1 logical business rules produces several executable rules. As the number of rules grows, the cost of maintenance grows until it's unmanageable.

peter

[DLuckham]

I've not had time to read all of this correspondence.
But given this

Back when I worked on health care systems, we needed to notify all users a given account has been modified. I feel the same thing applies in this case. I like to point out, this also applies to ECA rules. I've implemented this type of functionality a few different ways in the past and none of them are ideal. Usually, it means 1 logical business rules produces several executable rules. As the number of rules grows, the cost of maintenance grows until it's unmanageable.

I'll make a guess that the correspondence is NOT all about events and mutability of events, but about events and state.
In a lot of applications events change the system state. And other events undo those changes.
There's nothing logically wrong with that except that if you dont manage the state (e.g., make it a critical region) you can wind up with events that compete and non deterministic results. And in most such systems there will be race conditions.
So, e.g., one could have a health care system where there are competing notifications to users that their accounts have been modified.
Nothing logically wrong with that. Just bad system design, or bad use of the system.
But if you tried to have some notification events "undone" or "deleted" by other notification events, I do not have an adequate adjective for what kind of system that would be, that can be published here!
Anyway, I'm guessing that some of this discussion is about state change. Right or wrong?
- David

[hgilde]

This is my bias perspective, but these issues have to be addressed at the language level. As long as it's not defined by the rule language or event processing language, there's going to be a huge gaping hole. That tends to result in users roaming blindly and struggling to find a logical and consistent way of handling it.

This is one potential feature set of a language, but we do have to realize that it implies many features that are not right for every use case. Again pointing to the difficulties in providing a single best language in any area.

[woolfel]

I've not had time to read all of this correspondence.
But given this

Back when I worked on health care systems, we needed to notify all users a given account has been modified. I feel the same thing applies in this case. I like to point out, this also applies to ECA rules. I've implemented this type of functionality a few different ways in the past and none of them are ideal. Usually, it means 1 logical business rules produces several executable rules. As the number of rules grows, the cost of maintenance grows until it's unmanageable.

I'll make a guess that the correspondence is NOT all about events and mutability of events, but about events and state.
In a lot of applications events change the system state. And other events undo those changes.
There's nothing logically wrong with that except that if you dont manage the state (e.g., make it a critical region) you can wind up with events that compete and non deterministic results. And in most such systems there will be race conditions.
So, e.g., one could have a health care system where there are competing notifications to users that their accounts have been modified.
Nothing logically wrong with that. Just bad system design, or bad use of the system.
But if you tried to have some notification events "undone" or "deleted" by other notification events, I do not have an adequate adjective for what kind of system that would be, that can be published here!
Anyway, I'm guessing that some of this discussion is about state change. Right or wrong?
- David

thanks for responding. sorry for the vague and confusing response in the previous post.

The use cases I saw was centered around case management. At any given time, there may be multiple people viewing the same account. When the user pulls up an account, they're given a message like "this account is being modified by user X". If and when someone makes a change, the other sessions viewing the same account are notified. The user can decide if they want to refresh or ignore it.

I wasn't thinking the notification would "delete" or "undo" something. What the user or application does with that knowledge varies, but I feel the notification functionality should be part of the picture. I do tend to think of it as states, but not necessarily "events" + "states". It depends on the use case.

peter