{"id":1575,"date":"2017-02-11T17:08:53","date_gmt":"2017-02-11T17:08:53","guid":{"rendered":"http:\/\/data-flair.training\/blogs\/?p=1575"},"modified":"2021-05-09T13:23:29","modified_gmt":"2021-05-09T07:53:29","slug":"flink-cep","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/flink-cep\/","title":{"rendered":"Flink CEP – Complex Event Processing with Flink"},"content":{"rendered":"

So, in this tutorial on Complex Event Processing with Apache Flink<\/strong> will help you in understanding Flink CEP library, how Flink CEP programs are written using Pattern API.<\/p>\n

Moreover, we will see various Flink CEP pattern operations with syntax, Pattern detection in CEP and advantages of CEP operations in Flink. Also, we will learn Flink Complex Event Processing use cases and examples to get in-depth knowledge of Complex Event Processing for Flink.<\/p>\n

What is Complex Event Processing with Apache Flink<\/h2>\n

With the increasing size of data and smart devices continuously collecting more and more data, there is a challenge to analyze this growing stream of data in near real-time for reacting quickly to changing trends or for delivering up to date business intelligence which can decide company\u2019s success or failure.<\/p>\n

Detection of event patterns in data streams is a key problem in real-time processing.<\/p>\n

Flink handles this problem through Complex event processing (CEP) library that addresses this problem of matching the incoming events against a pattern to produce complex events which are derived from the input events.<\/p>\n

CEP executes relevant data on a stored query unlike traditional RDBMSs and discards irrelevant data. This enables CEP queries to be applied on a potentially infinite stream of data and also enables inputs to be processed immediately.<\/p>\n

This aspect effectively leads to CEP\u2019s real-time analytics capability. This gives the opportunity to quickly get hold of what\u2019s really important in data. In this manner, Flink CEP is 1 of the key component of Apache Flink ecosystem<\/strong>.<\/p>\n

\"Flink<\/a>

Complex Event Processing with Apache Flink<\/p><\/div>\n

Apache Flink is a natural fit for CEP workloads due to its true streaming nature and its capabilities for low latency as well as high throughput stream processing.<\/p>\n

Consequently, CEP has found application in a wide variety of use cases as described below and has provided several features of Apache Flink<\/strong> that have created a huge difference between Apache Flink, Hadoop <\/strong>and Apache Spark<\/strong>.<\/p>\n

Pattern API in Flink<\/h2>\n

Flink CEP program can be written using the pattern API that allows defining complex event patterns. So, each pattern consists of multiple stages or states. The pattern needs to start with initial state and then to go from one state to the next, the user can specify conditions.<\/p>\n

Now, each state must have a unique name to identify the matched events later on.\u00a0 So, we can append states to detect complex patterns.<\/p>\n

\"Flink<\/a>

Flink CEP – Pattern API in Apache Flink<\/p><\/div>\n

Do you know different pattern operations? Let us see some of the most commonly used CEP pattern operations:<\/p>\n

a. Begin<\/h3>\n

It defines pattern starting state and is written as below:
\n[php]Pattern<Event, ?> start = Pattern.<Event>begin(“start”);[\/php]<\/p>\n

b. Next<\/h3>\n

It appends a new pattern state and matching event need to succeed the previous matching pattern as below:
\n[php]Pattern<Event, ?> next = start.next(“next”);[\/php]<\/p>\n

c. FollowedBy<\/h3>\n

It appends a new pattern state but here other events can occur between 2 matching events as below:
\n[php]Pattern<Event, ?> followedBy = start.followedBy(“next”);[\/php]<\/p>\n

d. Where<\/h3>\n

It defines a filter condition for current pattern state and if the event passes the filter, it can match the state as below:
\n[php]patternState.where(new FilterFunction <Event>() {
\n@Override
\npublic boolean filter(Event value) throws Exception {
\nreturn … \/\/ some condition
\n}
\n});[\/php]<\/p>\n

e. Or<\/h3>\n

It adds a new filter condition which is ORed with an existing filter condition. Only if an event passes the filter condition, it can match the state.<\/p>\n

f. Within<\/h3>\n

It defines the maximum time interval for an event sequence to match the pattern post which it discards. We can write it as:
\n[php]patternState.within(Time.seconds(10));[\/php]<\/p>\n

Pattern Detection<\/h2>\n

So, we need to create pattern stream to run stream of events as below using input stream and a pattern:
\n[php]DataStream <Event> input = …
\nPattern <Event, ?> pattern = …
\nPatternStream <Event> patternStream = CEP.pattern(input, pattern);[\/php]<\/p>\n

Use cases for Flink CEP<\/h2>\n

Apache Flink CEP is used for the large number of applications like for financial applications such as stock market trend, credit card fraud detection and RFID-based tracking and monitoring. It also finds its usage in detecting network intrusion by specifying patterns of suspicious user behavior.<\/p>\n

Refer Flink use case tutorial to get real-time use cases of Apache Flink and how industries are using Flink for their various purposes.<\/p>\n

Conclusion – Flink CEP<\/h2>\n

Flink CEP includes various challenges like:<\/p>\n

    \n
  • It has the ability to achieve high throughput and low latency processing<\/li>\n
  • Ability to produce the results as soon as the input event stream is available<\/li>\n
  • Ability to provide aggregation over time, timeout between two events of interest and other computations<\/li>\n
  • To provide real-time alerts & notifications on detection of complex event patterns<\/li>\n<\/ul>\n

    So, this was all in Flink Complex Event Processing. Still, if you face any issue in learning Flink CEP, you can freely ask us through comments.<\/p>\n","protected":false},"excerpt":{"rendered":"

    So, in this tutorial on Complex Event Processing with Apache Flink will help you in understanding Flink CEP library, how Flink CEP programs are written using Pattern API. Moreover, we will see various Flink...<\/p>\n","protected":false},"author":6,"featured_media":41415,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[8],"tags":[753,2429,16566,2797,4745,4760,16568,16567,16569],"class_list":["post-1575","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-flink","tag-apache-flink-cep","tag-cep-flink","tag-complex-event-processing","tag-complex-event-processing-for-flink","tag-flink-cep","tag-flink-ecosystem-cep","tag-pattern-api-in-flink","tag-pattern-detection","tag-use-cases-for-flink-cep"],"yoast_head":"\nFlink CEP - Complex Event Processing with Flink - DataFlair<\/title>\n<meta name=\"description\" content=\"Learn Flink CEP library-complex event processing in Flink,CEP programs,CEP pattern operations with syntax,CEP Pattern detection,CEP advantages,CEP use cases\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/flink-cep\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Flink CEP - Complex Event Processing with Flink - DataFlair\" \/>\n<meta property=\"og:description\" content=\"Learn Flink CEP library-complex event processing in Flink,CEP programs,CEP pattern operations with syntax,CEP Pattern detection,CEP advantages,CEP use cases\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/flink-cep\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2017-02-11T17:08:53+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-05-09T07:53:29+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2017\/02\/Flink-Complex-Event-Processing-CEP-01.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Flink CEP - Complex Event Processing with Flink - DataFlair","description":"Learn Flink CEP library-complex event processing in Flink,CEP programs,CEP pattern operations with syntax,CEP Pattern detection,CEP advantages,CEP use cases","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/flink-cep\/","og_locale":"en_US","og_type":"article","og_title":"Flink CEP - Complex Event Processing with Flink - DataFlair","og_description":"Learn Flink CEP library-complex event processing in Flink,CEP programs,CEP pattern operations with syntax,CEP Pattern detection,CEP advantages,CEP use cases","og_url":"https:\/\/data-flair.training\/blogs\/flink-cep\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2017-02-11T17:08:53+00:00","article_modified_time":"2021-05-09T07:53:29+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2017\/02\/Flink-Complex-Event-Processing-CEP-01.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/flink-cep\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/flink-cep\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89"},"headline":"Flink CEP – Complex Event Processing with Flink","datePublished":"2017-02-11T17:08:53+00:00","dateModified":"2021-05-09T07:53:29+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/flink-cep\/"},"wordCount":842,"commentCount":1,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/flink-cep\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2017\/02\/Flink-Complex-Event-Processing-CEP-01.jpg","keywords":["Apache Flink CEP","CEP Flink","Complex Event Processing","Complex event processing for flink","Flink CEP","Flink ecosystem CEP","Pattern API in Flink","Pattern Detection","Use cases for Flink CEP"],"articleSection":["Apache Flink Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/flink-cep\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/flink-cep\/","url":"https:\/\/data-flair.training\/blogs\/flink-cep\/","name":"Flink CEP - Complex Event Processing with Flink - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/flink-cep\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/flink-cep\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2017\/02\/Flink-Complex-Event-Processing-CEP-01.jpg","datePublished":"2017-02-11T17:08:53+00:00","dateModified":"2021-05-09T07:53:29+00:00","description":"Learn Flink CEP library-complex event processing in Flink,CEP programs,CEP pattern operations with syntax,CEP Pattern detection,CEP advantages,CEP use cases","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/flink-cep\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/flink-cep\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/flink-cep\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2017\/02\/Flink-Complex-Event-Processing-CEP-01.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2017\/02\/Flink-Complex-Event-Processing-CEP-01.jpg","width":1200,"height":628,"caption":"Flink CEP - Complex Event Processing with Flink"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/flink-cep\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Apache Flink Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/flink\/"},{"@type":"ListItem","position":3,"name":"Flink CEP – Complex Event Processing with Flink"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"The DataFlair Team provides industry-driven content on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our expert educators focus on delivering value-packed, easy-to-follow resources for tech enthusiasts and professionals.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam2\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/1575","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=1575"}],"version-history":[{"count":1,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/1575\/revisions"}],"predecessor-version":[{"id":94116,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/1575\/revisions\/94116"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/41415"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=1575"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=1575"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=1575"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}