What is CEP?
CEP is typical queries run against data residing in a relational database. These queries could target recent data, such as customer transactions, or it could be older data that has been aggregated in a data warehouse. If you have a requirement to query data within milliseconds after it is generated, then you have a challenge. With such low latency requirements, it is just not feasible to use traditional methods like inserting the data into a database and then querying it. CEP solves this challenge d by providing tools to query streams of the data as the data is being generated. Where does the complexity come into play? As you will see in the example below, a single piece of data may not mean anything. But in the context of data from other events or sources, it is possible to create event driven applications that keep enterprises informed of significant events as they happen.
Examples of CEP
The first example we will look at shows how multiple events in a very short time could indicate a serious problem.
Imagine a car that has various sensors which generate data on a continuous basis. Let’s consider just three sources of data:
- The speed of the vehicle.
- The air pressure in each tire.
- Whether someone is sitting in the driver’s seat.
Data from a slow leak may be indicated by dropping tire pressure over a period of hours. But a tire blowout could be indicated by a rapid reduction in tire pressure over a short period of time. If there was a tire blowout and that event was accompanied by data indicating a rapid decrease in vehicle speed, it could indicate that the driver needs help. But if an additional event occurred simultaneously, indicating that the driver was no longer in his seat, this could indicate a serious accident.
Another example is fraud detection. Historically, health insurance fraud detection has been a manual process. In-house auditing processes reveal potentially fraudulent claims. There’s no guarantee that if fraud is detected, an insurance company will be able to retrieve their lost money. The effort might prove too costly to be worth it and this means the expense could be passed on to the other policy holder in higher premiums. In the end, the company may only have a new case of “what to look out for.” What if insurance fraud could be detected in real-time? As claims are entered, CEP could be used to identify fraud based on patterns used by manual auditing processes. This could provide streamlined fraud detection and cost savings simply by catching fraudulent claims before they are processed.
Business Activity Monitoring and Complex Event Processing: Fraud Detection
Industries using CEP
The Financial industry may have been the first adopters of CEP. The importance of knowing – in real-time – changes and trends in the financial markets, drove the need for CEP solutions. Professional traders need to know what is happening as soon as possible after events occur. CEP enables and allows this, and facilitates systems to respond automatically to events in order to protect or take advantage of financial events.
Oil and Gas is another industry that relies on CEP. With huge amounts of data coming in from operations, continuous intelligence is required. The data sources can vary from torque and rpm data coming from drilling rigs to sensors along the pipelines. Expensive resources like heavy equipment can be monitored for engine and drive train problems. Decisions on the best course of action can be made before equipment failure occurs.
Another area is clickstream analysis. CEP allows real-time analysis of user activity to help drive site layout and respond with appropriate advertising to the user based on their activity on the website. In the past, clickstream data was saved to a database to be analyzed and then when (or if) a user returned to the website a few days later, actions were taken to customize the site’s layout and advertising. A little too late if you were looking to order pizza online during your first visit.
CEP Vendors
With the release of SQL Server 2008 R2, Microsoft introduced StreamInsight which utilizes .Net with LINQ technologies. The MSDN website provides ample information in the form of videos, whitepapers and working .Net projects. There are many other vendors; some of them have a longer history than others.
Final Thoughts
With large development vendors like Microsoft adding CEP tools to their offerings, the ability to create event driven applications that address these unique requirements are now within reach of every IT department. The challenge is to know what exactly you are looking for in your data streams
Next Steps
Take a look at the following websites. They are great resources for getting to know more about Complex Event Processing:
- A great portal for all things CEP – http://www.complexevents.com
- Microsoft’s StreamInsight – http://www.microsoft.com/sqlserver/en/us/solutions-technologies/business-intelligence/complex-event-processing.aspx
The following post was originally done by my colleague Brain Peasey.
Brian has been working as a Database Analyst with Online Business Systems since 2008. He has over a decade of database development and administration experience in various industries and has worked on numerous platforms. Brian enjoys supporting clients and development teams in their database needs and regularly attends conferences to stay current in his field.
No comments:
Post a Comment