Complex Event Processing Community

HAPPY NEW YEAR, EVERYONE!

What are your predictions for CEP in 2009?
What is likely to happen next?
And ... what do you want to happen next?

"Forrester estimated that more than $100 million was spent in 2007 on CEP software and services,
indicating a market in the 'early stage of development'.
Forrester expects the CEP market to continue to grow and even accelerate as business analysts begin to realize how many nontraditional events are available for CEP analysis. This acceleration will be driven by the breadth of usage of CEP across a wide range of applications and industries"

Some estimates for the market in 2008 reach $150 million.

• Do you expect the market to expand?
• Does the CEP technology currently offered by products on the market need improvement?
  If so, what improvements do you want to see?
• Are there problems out there that customers are willing to pay to solve, but the current generation of CEP products cannot solve? What are they?
• Has the time come to talk about standards or interoperability of CEP products?
• And discuss any other CEP questions that come to mind!

- David

There's no way to know the time line for this, but I have a few predictions of upcoming developments:

• The collapse of one or more of the independent CEP vendors. Clearly this is very likely in 2009. Of course they may not officially fold, but get sold at pennies on the dollar of capital investment, as happened to Syndera (not exactly a CEP vendor) recently.
• Use of patterns in CEP logic. Patterns are wide spread at both a high level (ESBs, pub/sub, queues, message channels, message processing agents) and low level (real-time processing) but not at a level that helps discuss the implementation of business logic.
• Discussion of use cases in terms of these patterns.
• Evaluation and evolution of CEP languages based on the ability to work with and implement these patterns. This will allow an understandable translation from language features to business requirements and use cases.

--H

DLuckham wrote:

• Do you expect the market to expand?
• Does the CEP technology currently offered by products on the market need improvement?
  If so, what improvements do you want to see?
• Are there problems out there that customers are willing to pay to solve, but the current generation of CEP products cannot solve? What are they?
• Has the time come to talk about standards or interoperability of CEP products?
• And discuss any other CEP questions that come to mind!

- David

Hi David - happy new year to you too!I started a blog on the same topic, but then saw you had already started a conversation and thought it better to add thoughts here rather than delve into TIBCO-specific thoughts ...

1. The CEP "market" will expand, but some of that will be existing tools claiming (partly correctly, in some cases) to be capable of CEP. So probably EPTS will need to define what is "CEP technology" (e.g. processing generic patterns of generic events in a continuous fashion).

2. All software technology is capable of improvement! Certainly we should expect the infrastructure players to continue integrating their CEP acquisitions this year with their other technologies.

3. There are problems customers want to solve and will pay for, but may be dissuaded from doing so by existing IT department capabilities and knowledge. Identifying and proving CEP solutions will continue to be the order of business for CEP companies, and which is also an explanation for Hans' comment on CEP vendor consolidation (smaller vendors always have a more difficult job penetrating accounts).

4. Standards are always useful in the right context (e.g you are using a QWERTY keyboard, right?), and can help adoption (consider BPMN in the BPM industry for example). However, whether there is sufficient concensus for "a" single CEP standard is another matter and unlikely IMHO this year without some major customer (US Govt, etc) interest.

5. On other topics: "event models and management" and "event-based computing" may become more prevalent in businesses and learning/teaching establishments respectively.

Hans,
With regard to your wishes for event patterns in "CEP logic" do you have a simple example by way of illustration of what you'd like to be able to do?

regards
- David

Hi David,

You asked,

* Do you expect the market to expand?

No. The entire software market, for the most part, will have a very tough year in 2009, due to a very serious global economic crisis. Firms that have "bet the farm" on capital markets will suffer even more, as these firms are in a deep recession and IT spending always suffers in these environments. We might see some "smoke-and-mirrors" in CEP marketing-land, claiming an expanding market, but these claims will be unfounded and unsupported. As a side note, the largest software companies (in the CEP space) tend to bundle their CEP sales into enterprise software license deals, so it is nearly impossible to correlate actual CEP software sales and usage with what we read from these large vendors (and their reports are the lion's share of the market).

* Does the CEP technology currently offered by products on the market need improvement?
If so, what improvements do you want to see?

Of course they need improvements! If the current generation products on the market actually could detect (difficult to detect) complex events in real-time with high confidence, the market would be much, much larger than it is today. I have discussed this in great detail in 2008, and will spare you all the pain and frustration of reading it again. Cheers!

* Are there problems out there that customers are willing to pay to solve, but the current generation of CEP products cannot solve? What are they?

Yes of course there are plenty of "complex event" classes of (difficult and complex) detection-oriented problems people are willing to pay for. As I said before, if the software on the market actually did true "complex event processing" (not a watered down version of "everything is marketable as CEP" as we have seen in 2008) then software would be selling like cold water in the hot, dry desert. This is not rocket science. Proprietary stream processing (continuous query) engines have limited commercial value in the overall market, and the market is overly crowded with these vendors! This situation is unsustainable, especially in this (terrible) economy. As someone else opined, these many "doing the same thing" (ESP) vendors cannot all survive in such a negative business climate with so little new actual value-added technical (detection) capability.

* Has the time come to talk about standards or interoperability of CEP products?

Software is already interoperable at the message / transport layer, and messaging is rapidly becoming a service-oriented commodity (see my recent post on Amazon SQS). Interoperability needs to happen at the "complex event modelling layer", and this is not likely to happen in the foreseeable future, since we have so few "CEP vendors" in this space actually doing what I would call CEP. Most vendors are focused on routing, scheduling and simple (not complex) detection scenarios, a software market which is not sustainable, in my opinion.

* And discuss any other CEP questions that come to mind!

Yes! But, I'm busy working on a few cybersecurity predictions and community (Web 2.0) software tasks for a major forum, so "that's all for now on CEP!"

Happy New Year!

Yours faithfully, Tim

PS: Also posted here, Predictions for the 2009 CEP Market.

I'll respond about patterns when I have a minute to write about it. As you say, this is more of a wish than a prediction.

Another interesting factor, IMO, will be whether vendors can break the barrier from project-by-project deployments to strategic platforms. This has been discussed before, but there are plenty of shops with many real-time applications that could benefit from migration to a unified platform. In theory, this would move components like locking, threading, data distribution, data storage and retrieval, server management and such from the developer to the infrastructure. The result (again, in theory) would be lower costs to maintain and administer applications over time. This would seem to be a way for CEP software to grow even in a bad business climate. However, vendors have not yet been able to break this barrier, so who can say whether it will happen in 2009 (unlikely, maybe).

--H

I wrote a bit about what I think will happen in 2009 in my last blog post. But it got a bit long and as the web has the patience of a ferret on speed, here's a short summary:

• During 2009 many vendors will eventually find the sweet spot for their products, or vanish...
• As a result of this, marketing messages will change. Smaller vendors might re-brand and bigger ones starts to understand how to sell their CEP stuff, do I hear CEP-SOA?
• At least one or two CEP vendors will go belly up. Discreetly as a camouflaged acquisition or in plain sight as a result of lack of VC cash.
• On the positive side, more users will find CEP and understand what kind of problems it solves and make lot of cash while doing so...

On the product/technology side I don't see many revolutions. Just the current products getting even better. One interesting development I predict is that at least one major RDBMS vendor will add the capability to run plain old SQL queries (perhaps with some CEP extensions) continuously inside their database.

Marco predicted as follows:

* During 2009 many vendors will eventually find the sweet spot for their products, or vanish...
* As a result of this, marketing messages will change. Smaller vendors might re-brand and bigger ones starts to understand how to sell their CEP stuff, do I hear CEP-SOA?
* At least one or two CEP vendors will go belly up. Discreetly as a camouflaged acquisition or in plain sight as a result of lack of VC cash.
* On the positive side, more users will find CEP and understand what kind of problems it solves and make lot of cash while doing so...

These are pretty safe bets, I think. Gartner counts 16 vendors in the high end CEP market at the moment, and another 70 in related markets where CEP is part of a larger offering. So shake out is an inevitable part of "maturing markets".
There are research reports out there that claim the CEP market is being recession-proof in 2008-09. But I haven't read them because they want you to pay for them!
However ... what about "new" technology in 2009? Will we see any new CEP capabilities entering commercial offerings?

Hans says:

# Use of patterns in CEP logic. Patterns are wide spread at both a high level (ESBs, pub/sub, queues, message channels, message processing agents) and low level (real-time processing) but not at a level that helps discuss the implementation of business logic.
# Discussion of use cases in terms of these patterns.
# Evaluation and evolution of CEP languages based on the ability to work with and implement these patterns. This will allow an understandable translation from language features to business requirements and use cases.

Sure would like to see an example of these, even a small example?

>>>> There are research reports out there that claim the CEP market is being recession-proof in 2008-09.

I would not put any credence in many of these reports. Their growth figures for CEP were computed before the current economic recession and IT slowdown hit. I have insight into a number of the CEP companies, and things are not that rosy. Many of the financial services companies have put major, non-critical purchases on hold. The number of hedge funds has dramatically decreased (as have the number of investment banks), and hedge funds were a prime marketplace for CEP systems.

In fact, as you well know, at the recent Gartner CEP conference, the number of vendors FAR outweighed the number of customers. Most of the vendors told me that, from a business standpoint, the conference was not worth the effort of attending.

It will be interesting to see what will happen to the VC-backed CEP vendors, as the VC's start to pull back on some investments in favor of the investments that offer a greater ROI. The shakeout will be good for the CEP industry, as the marginal players will have been weeded out, and there will be greater clarity around the remaining product offerings.

There is a new product on the scene that has recently been launched. Out of respect to the anti-spam etiquette, I will refrain from mentioning the product name.

It is positioned as a BEM-lite solution with fast time-to-value; specifically low-cost, small footprint, simple installation. It does not have a lot of bells and whistles, but the essential features to define events and start monitoring events the same day that the product is intsalled.

It is at the database/file level for data sources.
You write SQL statements and then schedule the query to run periodically.
When the query is satisfied, start event processing, send email, run other queries, etc.

There is a scorecard which displays the portfolio of defined events and their activity.

The product is intended for the SMB market where there is little-to-none formal enterprise architecture groups.

What say ye?