Could CEP help follow the Tweets?

[DLuckham]

I have been wondering for some time if it is possible to define a set of event patterns, or more generally a pattern language, that would be of any help whatsoever to those who wish to troll twitter clutter for useful tweets. Is there a specialized event pattern language with linear time matching for twitter trolling?
After all, it is an SMS format of sorts, so it might have something in common with Stock Market ticker tape monitoring, which has been a specialized area of pattern matching for some years.
What would the pattern operators be?
So far the only thing I've seen depends upon simple keyword detection and message counting (e.g., for flu trends, http://www.flutweet.com/).
That wont work, I think, for more general kinds of tweet detection.
Anyway, any ideas?
- David

[hgilde]

This is a very broad topic, I do not think you'll find a single unifying pattern language.

I wonder why you say that counts are not good enough. Plenty of very fancy techniques are based on counts.

[DLuckham]

Include counting by all means. But what else?
- David

[marco]

What about automatic creation of categories into which the tweet belong and automatic classification based on that?

[hgilde]

There are plenty of books and papers on text patterns in various languages. I think one would need to be somewhat of a specialist in this area to properly define an event language. Could be an opportunity for cross-discipline cooperation.

[DLuckham]

Here's the point ... this is a great potential area of application of event processing and CEP-like techniques.
And there's money in it!!!
The regular press agencies are beginning to use twitter to get early leads to news stories.
And Google.org is investigating the use of twitter and similar SMS sites for early detection of emerging epidemics - in great secrecy I might add!
Here you go ...

Can media organizations leverage the social web to get story tips faster than they could through traditional methods? A number of news aggregators believe so and are looking to Twitter for tips.

See http://www.readwriteweb.com/archives/ca ... xperie.php

Lets hope they beat Google to it ... I detest that company's secrecy ... and ".org" is the "philanthropic" (read not for profit) arm of Google too ... run by the so named "Larry Brilliant"! (Yeah ... I'll say no more, but I'm sure you can read between the lines here!)

There's one PH.D thesis in progress on this ... I hope the guy gets to post here when we let him in as a member ...sooooooooon!!! (Julian!!! )

Marco's suggestion of automated categories (ie pigeon holing that reduces topic search) seems right track ... but how??
Need more ideas people ...
cheers
- David

[AndreasB]

Hi,

things like flutweet.com are interesting, but I consider this only as ESP.

To really achieve measurable value from tweets you have to correlate more tweets by relating them through their inReplyTo-Ids, by aggregating spatial and timely closed tweets (therefore use the location and createdAt infos) or by aggregating tweets with certain signal words, i.d. not counting like flutweet but aggregating.

Then you could apply text mining techniques on this "higher level" tweets - as these contain much more valuable text than 140 charachters-, formalize the results (map them to a meta model, that is to be defined -, add context information (news or weather information) and feed them into a cep engine.

The patterns for detecting certain human initiated complex events in this stream should also differ from the ones used for "hard" data. Here you could use results from opinion mining and sentiment analysis research.

I think the combination of these techniques can offer a real benefit.

The guys from Psydex obviously have already done something like that (http://www.psydex.com , http://havemacwillblog.com/2009/03/18/p ... elligence/)


Cheers

[Tim Bass]

Folks are already using advanced statistical methods to classify massive amount of text-based events, like RSS news and spam filtering. '

There are a number of Bayesian classifiers available that process massive amounts of RSS news items. At the center of Google's Gmail anti-spam service is a Bayesian classifier. The same technologies are commonly used to filter and classify any text-based messaging service. Most text indexing, search and retrieval algorithms use similar Bayesian classifiers.

Rule-based technologies (alone) for classifying text-based messages were rapidly becoming obsolete a decade ago.

Hence, I am confused when David asked "could CEP follow the tweets?"

What do you mean by CEP? Do you mean "CEP the rule-based systems being sold as CEP by software companies?" Or do you mean "CEP, a concept to process complex events and detect opportunities and threats in real time?" Or do you mean "CEP" the general acronym like EDA, SOA, BPM, that is used by marketing folks in the most general (and confusing) way possible?

So far, the dominate technology in the market place calling themselves "CEP" are forward-chaining rule-based systems. There are also some RETE-based forward-chaining systems marketed under the "CEP" banner. These types of algorithms for filtering and classifying massive amounts of text-based messages have been obsolete for at least a decade in almost technology areas.

It follows that "CEP", as a concept or group of technologies, is becoming quite insular because the term "CEP" is used to go-to-market with a set of like rule-based technologies sold by a group of companies working together to market their software.

The real work and processing (filtering, classification, recommendation engines, etc) in complex event processing is happening elsewhere. Rule-based text classification methods have been obsolete for quite a long time.

[DLuckham]

Tim,
For a short overview of CEP concepts see
http://complexevents.com/?p=399

CEP was never intended to be a stand alone technology. It is a set of event processing concepts, and the abstraction concepts are yet to be seen in commercial offerings to date.

One can always incorporate statistical functions into event patterns and into abstraction mappings.
So for example, a higher level event might be a statistical measure on a set of tweets.
Question is, what would you measure?
Suppose the example is twitter about possible bird flu outbreaks in SE Asia.
Its "boots on the ground twittering", and highly diverse styles. Ignore the multilingual aspect, assume English.
Each tweeter has a personality, e.g., preferences for particular words and word combinations.
How do you recognize agreement across personalities?
I thought perhaps the pattern language might be different from the usual expression languages.
Maybe the "tweet abstraction hierarchy" would be based totally on statistical measures? That's possible.
But the concept of "tweet hierarchy" should be defined.
Anyway, worth a discussion.

Take a look at the post by AndreasB.!!

As for

rule-based systems

what do you mean?
Rules can be defined to be any algorithm including probabilistic ones involving coin tossing.
True, simple examples are usually of the E-C-A variety, but there are other kinds of rules.

One of the issues I thought might be how to measure concepts like "agreement", and "trends" in a set of tweets, maybe across different groups. Is there "group personality?

Also, you talk a lot about Bayesian Analysis. Can you give a simple example applied to a small set of tweets please.

- David

[DLuckham]

See also,

A Twitter-Based Graphing Tool

http://www.cmu.edu/homepage/computing/2 ... nner.shtml

- David