What does event abstraction hierarchy mean?

[woolfel]

I still don't really understand event abstraction hierarchies, and I couldn't think of a good way to describe my confusion. Instead, I'll provide an example from OMS system and see how others would apply the technique to the problem. Several years back, I was fortunate enough to work on a pre-trade compliance engine for an Order Management System. A pre-trade compliance system is used to check all trades before they're executed to prevent costly mistakes and regulatory violations. Generally speaking, there's 4 different people that will use the system: fund manager, trader, compliance officer and compliance manager.

If any transaction violates a rule, the trade cannot proceed until the violation is resolved, or overridden. In this specific situation, all of them use the same exact tool and the page looks the same to all of them. What differs is the security and authorization. A trader can only see certain trades. The fund manager can see all trades for a fund, which could include multiple traders. The fund manager might manage multiple funds. The compliance officer is assigned specific funds and accounts. Any violations for those will show up on his screen. The compliance manager sees all violations.

In terms of authority, the fund manager can revise any transaction for his fund. He can't override compliance violations. The compliance officer can override certain types of violations. The compliance manager can override any violations, or reject an override.

On page 55 of The Power of Events, it states "In fact, it is usually true that two stockbrokers want different views of what is going on in a financial trading system."

Where I get confused is the definition of "view". When I think of view, I think of Model View Controller. Using that definition, all traders have the same view, what differs is the transactions they see. In some cases, all traders for the same fund see all the open, what-if and pending transctions for that fund. Usually this is handled with user groups. One thing that throws me off is diagram 3.4, which shows "Financial Trades level and Transaction Protocol Level." The diagram gives me an impression that hierarchical viewing is about event aggregation. One or more protocol events aggregate up to a financial event. If that's what the book describes, I don't see or understand the value.

The OMS system I worked on used FIX and SWIFT protocol to execute trades. Both of those protocols are complex and difficult to understand. If a software engineer has a hard time understanding FIX and SWIFT, it doesn't make sense to me to expose those details to the trader. Instead, they should just see predefined status codes like: pending, cancelled, executing, etc. I suppose we can show "transaction failed 2-phase commit" to the trader, but does that really make sense? What value does abstract hierarchy give me in this specific situation? The trading system I know of handle transaction status natively and don't expose that as event aggregations to an external system.

On page 135, Table 7.2 shows 2 different levels. My experience with real traders differs from the example in the book.

One thing aggregation would help with is showing portfolio performance. For example, many funds benchmark against S&P500 and Dow. For a large SP500 fund, there's 450-500 stocks, so displaying the weight of each stock gets tricky. One way to visualize that data is to show just the sectors at the highest level. As the user zooms in, they see the industryGroup weights. If they zoom all the way in, they see the sub-industry weights. The closest example to this type of visualization is google charts and google finance.

Was the intent of abstraction hierarchy similar to performance visualization example? If that's the case, I see the value of defining different levels. In this case though, it doesn't have anything to do with events per say, but the technique could be applied to event processing. I just haven't seen it myself, so I have a hard time making that leap. I hope this attempt as explaining my confusion is less obtuse than the previous.

peter

[hgilde]

I am not clear of you are questioning the specific examples raised in the book, or the value of an abstraction hierarchy.

If you are questioning the specific examples raised in the book, David will have hard time answering you as he is not a trading expert and did not intend his examples to be argued as such. So I will take a shot:

The OMS is a good example of an event hierarchy. Or at least some are.

A transaction protocol is mapped to a business intention. That intention may be communicated to the compliance engine for verification or may be passed directly. The intention is compared to the state of one or more orders and mapped to a transaction. The transaction is then executed. Each of those is an event, each is derived from another event (plus state).

There are many basic use cases involving different levels of viewing, for example:

A buy-side trader gets a call from the fund manager asking why the portfolio doesn't look the way it should. It is off by some shares. The trader now must look back through the trades to determine whether (1)the fund manager entered the wrong instructions to alter the portfolio, (2) the traders screwed up when executing the order or (3) something more complex needs troubleshooting. In the third case, either the trader or a support person will continue following this trail back through the transaction and protocol level events. We see that different roles look at different levels of events and of state.

A more simple example of hierarchy in viewing events: Some traders want to see every execution. Some want to see when their orders are filled. Some want an update every few minutes on the states of their orders. The fund manager may want a much less granular view of executions and might be content to look every day at static data.

There are plenty of event abstractions and levels of viewing in trading, you use them and think about them every day.

[woolfel]

I am not clear of you are questioning the specific examples raised in the book, or the value of an abstraction hierarchy.

If you are questioning the specific examples raised in the book, David will have hard time answering you as he is not a trading expert and did not intend his examples to be argued as such. So I will take a shot:

The OMS is a good example of an event hierarchy. Or at least some are.

A transaction protocol is mapped to a business intention. That intention may be communicated to the compliance engine for verification or may be passed directly. The intention is compared to the state of one or more orders and mapped to a transaction. The transaction is then executed. Each of those is an event, each is derived from another event (plus state).

There are many basic use cases involving different levels of viewing, for example:

A buy-side trader gets a call from the fund manager asking why the portfolio doesn't look the way it should. It is off by some shares. The trader now must look back through the trades to determine whether (1)the fund manager entered the wrong instructions to alter the portfolio, (2) the traders screwed up when executing the order or (3) something more complex needs troubleshooting. In the third case, either the trader or a support person will continue following this trail back through the transaction and protocol level events. We see that different roles look at different levels of events and of state.

A more simple example of hierarchy in viewing events: Some traders want to see every execution. Some want to see when their orders are filled. Some want an update every few minutes on the states of their orders. The fund manager may want a much less granular view of executions and might be content to look every day at static data.

There are plenty of event abstractions and levels of viewing in trading, you use them and think about them every day.

Thanks for the response. The system we worked on has the concept of filters, which gave each user the ability to filter their view. An example might be all transaction for sector Financial services. Lets say the fund manager knows they're going to trade in financial sector today, so he creates a filter. If he decides he wants to see all trading activity for the fund, he simply unchecks the filter box. That's how we approached it, I'm sure others handle it differently.

From an end user perspective, I agree that could be defined as a view. In our case though, it is an adhoc view defined by the end user. That view is just applying sorting or filtering of the same view everyone sees on the transaction screen. If someone designed a trading terminal differently, I could see it really being multiple table/views with differing data (aka columns or object properties). In my case, say we have a trade restriction rule for junior traders that says "no transaction over 500,000". If john doe junior trader makes a typo and adds an extra zero for "5,000,000", the compliance system would prevent that and we know exactly which rule was violated. We used a table tree to display the information. Any transaction that had a violation had a red color. The user clicks the "+" icon to expand that transaction to see exactly which rule was violated and additional details.

I'm bias, but to me this is more about good user interface design and less about event processing or event abstraction hierarchies. I know that some older OMS systems had different screens and tabs for showing the information, but they were difficult to use because the user couldn't see the whole picture easily. They had to jump between multiple screens and applications. I guess my question is this.

Is this really an event abstraction hierarchy problem, or an old fashion user interface design issue? I tend to think it's an UI issue.

peter

[DLuckham]

Peter,
there is no short and easy answer to your post, which is why I have been slow to answer. First, thanks to Hans for answering.
Secondly, the book is under revision, indeed it is being rewritten, so comments and questions are welcome. It needs revision.
So keep the comments coming!

Let me take a few of your points now and maybe deal with others later.

The OMS system I worked on used FIX and SWIFT protocol to execute trades. Both of those protocols are complex and difficult to understand. If a software engineer has a hard time understanding FIX and SWIFT, it doesn't make sense to me to expose those details to the trader.Instead, they should just see predefined status codes like: pending, cancelled, executing, etc.

Well exactly this is what an event hierarchy is about... hiding event activity from those who dont need to know. A good trading hierarchy would hide SWIFT activity from a trader. That's a lower level of activity. Figure 3.4 shows what the IT dept would use, not what the trader would use. The book should have made that clear!

On page 135, Table 7.2 shows 2 different levels. My experience with real traders differs from the example in the book.

Table 7.2 is hypothetical. Anything "real" wouldn't fit on a single page.
But it seems that you are certainly in a position to build your own hierarchies.
The only thing is whether you would think it worthwhile!
There are industry standard hierarchies (of functional operators) that you can look at, say, for protocols like TCP. Or the famous OMG 7 layer messaging hierarchy. Corresponding event hierarchies mirror the activities of performing the functional operators at each level.

More on other points later.
- David

[woolfel]

Peter,
there is no short and easy answer to your post, which is why I have been slow to answer. First, thanks to Hans for answering.
Secondly, the book is under revision, indeed it is being rewritten, so comments and questions are welcome. It needs revision.
So keep the comments coming!

Let me take a few of your points now and maybe deal with others later.

The OMS system I worked on used FIX and SWIFT protocol to execute trades. Both of those protocols are complex and difficult to understand. If a software engineer has a hard time understanding FIX and SWIFT, it doesn't make sense to me to expose those details to the trader.Instead, they should just see predefined status codes like: pending, cancelled, executing, etc.

Well exactly this is what an event hierarchy is about... hiding event activity from those who dont need to know. A good trading hierarchy would hide SWIFT activity from a trader. That's a lower level of activity. Figure 3.4 shows what the IT dept would use, not what the trader would use. The book should have made that clear!

On page 135, Table 7.2 shows 2 different levels. My experience with real traders differs from the example in the book.

Table 7.2 is hypothetical. Anything "real" wouldn't fit on a single page.
But it seems that you are certainly in a position to build your own hierarchies.
The only thing is whether you would think it worthwhile!
There are industry standard hierarchies (of functional operators) that you can look at, say, for protocols like TCP. Or the famous OMG 7 layer messaging hierarchy. Corresponding event hierarchies mirror the activities of performing the functional operators at each level.

More on other points later.
- David

thanks for taking time to respond. I'll have to re-read it several times and make sure i'm reading it correctly.

peter

[marco]

Just to give another example of hierarchies and why I think they are a good idea, here's how we use them:

See this as just one application of hierarchies, I'm sure the concept can be used in a number of different ways. But this usage fits nicely into our problem domain...

So, we do CEP with rules which are focused on tracking anything that moves and produces events from GPS, AIS or anything that can emit its location. In order to track different aspects and detect all kind of interesting behavior we have defined a number of rules.

We ended up using a hierarchy for a number of reasons:

- In order to understand how things work we have structured the rules into a hierarchy. So it's a way for us to create a understandable structure to begin with.
- The other feature of having a hierarchy is that rules can be added to the different levels without affecting other rules, rules communicate only using events. Pusing events to level n+1 and getting events from n-1.
- It's easy to add new levels
- Rules at a high level can use lower level rules as building blocks. So with a number of cleverly designed rule levels it should be easy (easier) to define rather complex rules without shooting yourself in the foot.

For example, our levels (specifically in the ruleCore geofence solution) look something like this:

6 - Timing restrictions of vehicle movement among zones.
5 - Zones visited in specific order
4 - Too many vehicles in a zone?
3 - Enter/Exit - Detect the action of entering or exiting a zone
2 - Boundaries - Detect if entity inside/outside of defines zones.
1 - GPS or other Location data
0 - Raw event data.

What is important that each level can emit events when rules trigger. As you can listen to events from a specific level, you can easily listen to situations happening of different complexity.

For example:
- At level 2 you get events when a vehicle enters a zone. A pretty simple situation.
- Listening at level 6, you get notifications when a vehicle moves too slowly between multiple waypoints and in addition visits the waypoints in correct order plus spends (exits in time) no more than x minutes at every waypoint. A bit more complex situation to detect.

So in order to create a new type of rule, the rule can use events from lower levels and it becomes rather easy to create complex rules as their inputs are already refined high level events. If every rule had to deal with raw GPS events they would be insanely complex and hard to understand.

[woolfel]

Just to give another example of hierarchies and why I think they are a good idea, here's how we use them:

See this as just one application of hierarchies, I'm sure the concept can be used in a number of different ways. But this usage fits nicely into our problem domain...

So, we do CEP with rules which are focused on tracking anything that moves and produces events from GPS, AIS or anything that can emit its location. In order to track different aspects and detect all kind of interesting behavior we have defined a number of rules.

We ended up using a hierarchy for a number of reasons:

- In order to understand how things work we have structured the rules into a hierarchy. So it's a way for us to create a understandable structure to begin with.
- The other feature of having a hierarchy is that rules can be added to the different levels without affecting other rules, rules communicate only using events. Pusing events to level n+1 and getting events from n-1.
- It's easy to add new levels
- Rules at a high level can use lower level rules as building blocks. So with a number of cleverly designed rule levels it should be easy (easier) to define rather complex rules without shooting yourself in the foot.

For example, our levels (specifically in the ruleCore geofence solution) look something like this:

6 - Timing restrictions of vehicle movement among zones.
5 - Zones visited in specific order
4 - Too many vehicles in a zone?
3 - Enter/Exit - Detect the action of entering or exiting a zone
2 - Boundaries - Detect if entity inside/outside of defines zones.
1 - GPS or other Location data
0 - Raw event data.

What is important that each level can emit events when rules trigger. As you can listen to events from a specific level, you can easily listen to situations happening of different complexity.

For example:
- At level 2 you get events when a vehicle enters a zone. A pretty simple situation.
- Listening at level 6, you get notifications when a vehicle moves too slowly between multiple waypoints and in addition visits the waypoints in correct order plus spends (exits in time) no more than x minutes at every waypoint. A bit more complex situation to detect.

So in order to create a new type of rule, the rule can use events from lower levels and it becomes rather easy to create complex rules as their inputs are already refined high level events. If every rule had to deal with raw GPS events they would be insanely complex and hard to understand.

thanks for taking time to respond. Your example makes much more sense to me than the ones "the power of events". having worked on mobile platforms in the past, I've done something similar to that. In my case, I used a business rule engine and divided the rules into modules by functionality. It wasn't so much a hierarchy and was more in line with classic rule chaining/rule management approach. The challenge I see with rigid hierarchies is that it "could" break down.

If we look at aspect oriented programming, one could organize it as aspects without forcing an hierarchy. We can also look at AI, production systems and BRMS and think of it in terms of modules. From a rule editing perspective, it may be desirable to write the rule in a custom rule language. The tools and application would then compile it down to one or more low level rules, which "could" fit a well defined hierarchy. The OMS project I worked on, we designed a compliance language based on thousands of real compliance rules. I then wrote a custom rule compiler to convert the business rule to several executable rules for JESS. I've written several natural language editing tools over the last 8 years, so my own bias and experience makes it difficult for me to make the mental leap.

peter

[DLuckham]

Marco,

Can you tell us what the purpose of your geofence application is? E.g., Is it road traffic control, or airport traffic control?

- In order to understand how things work we have structured the rules into a hierarchy. So it's a way for us to create a understandable structure to begin with.
- The other feature of having a hierarchy is that rules can be added to the different levels without affecting other rules, rules communicate only using events. Pusing events to level n+1 and getting events from n-1.
- It's easy to add new levels
- Rules at a high level can use lower level rules as building blocks. So with a number of cleverly designed rule levels it should be easy (easier) to define rather complex rules without shooting yourself in the foot.

Exactly! Both the events and the rules are structured into layers. And the hierarchy is flexible. It can be changed to fit different purposes and provide different views by changing the event types (adding new ones, deleting old ones etc.) and changing the rules accordingly.
- David

[marco]

Marco,
...
Can you tell us what the purpose of your geofence application is? E.g., Is it road traffic control, or airport traffic control?
...

The two most popular uses are for security and efficiency, both for tracking fleets of vehicles and their cargo and drivers.

For security, it gives a fleet manager a helping hand in monitoring a large fleet of vehicles.

There are a number of rules, regulations and internal policies which a fleet must follow. The problem is that different rules apply to different vehicles at different times and depending on a number of factors, it's context dependent as we would say in the CEP world. So keeping track of constantly moving vehicles, their cargo and drivers, and map that to a set of ever changing rules is something that is valuable to automate.

So instead of looking periodically at their nice map with all the vehicles in order to try to understand if the all behave correctly, the fleet manager gets an alert from ruleCore:Geofence immediately when a vehicle does something it's not supposed to.

The efficiency improving case is about saving precious minutes, provide early warning to fleet operating crews.

In the security case it's all about detecting bad things that might happen. In efficiency case it's about detecting things we expect to happen several times a day, we just don't know exactly when they happen so we need an alert when they do, or rather a couple of minutes before they do. For example if you run a port and expect tankers which require some special setup when they arrive you can save 14 minutes each time one arrives - if you can get an early alert of an arriving tanker. The alert allows for the crew at the port to walk a couple of hundreds of meters and do some things, so that when the tanker arrives it can start unloading at once instead of idling when waiting for the port operators to show up.


The motivation of automatic rule based tracking of vehicles is pretty simple, tracking constantly moving vehicles and cargo is hard (you need lots of people) to do manually when you have to consider a rather large number of context dependent rules which might apply to the vehicles and cargo depending on a number of factors. ruleCore:Geofence automates this tracking and allows the fleet manager to spend more time in making decisions and less time doing rather boring tracking.

Another example is in multi-modal transport. Different rules might apply depending on your mode of transport. Moving something from sea to rail or from rail to road might change the applicable rule sets depending on the cargo type.


This might be one of the simplest applications of CEP, no rocket science involved and it's in a market where there's no big money to be made. But still a solution to a real problem so its a good example of CEP usage.