Start the Flink Shell:<\/strong>
\nBefore starting Apache Flink shell you need to install Flink, to install Flink follow this installation tutorial.
\nApache Flink is shipped with interactive command shell or Flink-scala shell prompt, to start Flink shell:
\n[php] $bin\/start-scala-shell.sh local [\/php]<\/p>\ni. Create DataSet<\/h3>\n
It will create a DataSet with name \u201cdata\u201d. On DataSet you can perform various transformation operations to process data:
\n[php] val data = benv.readTextFile(“..\/about-dataflair.txt”) [\/php]
\nYou can download the data file from here.<\/p>\n
ii. Print elements of dataset<\/h3>\n
It will print all the elements of DataSet:
\n[php] data.print [\/php]<\/p>\n
iii. Count the number of elements<\/h3>\n
To count the number of elements of DataSet:
\n[php] data.count [\/php]<\/p>\n
iv. Filter Operation<\/h3>\n
Perform the filter operation based on custom business logic, based on the condition in the filter() data will be filtered and the result will be stored in dfData:
\n[php] val dfData = data.filter(_.contains(“DataFlair”))
\ndfData.print
\ndfData.count [\/php]<\/p>\n
v. Map Operation<\/h3>\n
Process the DataSet using Map operation, Map transformation takes one input at a time and returns 1 element. We can write custom business logic in the Map operation. Here we are converting data to upper case:
\n[php] val uData = data.map(_.toUpperCase)
\nuData.print [\/php]<\/p>\n
Log Mining Use case Example in Flink<\/h2>\n
In this use case, we will analyze the log data of Hadoop and generate various insights. You can download the data from here.
\ni. Create DataSet<\/strong>
\nCreate DataSet of an input file. DataSet is the basic abstraction in Flink.
\n[php] val logs = benv.readTextFile(“..\/hadoop_nm_logs”) [\/php]
\nii. Print Log Records<\/strong>
\n[php] logs.print [\/php]
\niii. Count the Number of Lines<\/strong>
\n[php] logs.count [\/php]
\niv. Error Analysis<\/strong>
\nAnalyze the logs and find out how many errors\/exceptions are there. We can analyze the logs and find out the exceptions to take corrective measures-
\na. Filter the Errors<\/strong>
\n[php] val err_logs = logs.filter(_.toLowerCase.contains(“exception”))
\nerr_logs.count
\nerr_logs.print [\/php]
\nb. Filter the Warnings<\/strong>
\n[php] val warn_logs = logs.filter(_.toLowerCase.contains(“warn”))
\nwarn_logs.count
\nwarn_logs.print [\/php]
\nTo study more Apache Flink real world use case follow this flink use-cases guide.<\/p>\nWordcount Example in Flink<\/h2>\n
In this section of Apache Flink shell commands tutorial, Classic Wordcount example is explained to run in Flink\u2019s Scala shell\/prompt. Count the number of times a specific word has occurred in the dataset-
\ni. Create DataSet<\/strong>
\n[php] val data = benv.readTextFile(“..\/about-dataflair.txt “) [\/php]
\nii. FlatMap Operation \u2013 Convert line into words<\/strong>
\n[php] val words = data.flatMap { _.toLowerCase.split(” “) } [\/php]
\niii. Map Operation \u2013 Convert word into (word,1) pair<\/strong>
\n[php] val wordPair = words.map { (_, 1) } [\/php]
\niv. GroupBy \u2013 Group similar words together and sum the counts<\/strong>
\n[php] val result = wordPair.groupBy(0).sum(1) [\/php]
\nv. Print the result<\/strong>
\n[php] result.print [\/php]
\nCreate your first Apache Flink Application in java with eclipse using this Flink guide.<\/p>\nConclusion – Flink Shell Commands<\/h2>\n
Hence, in this Apache Flink Shell Commands tutorial, we discussed this with the help of example such ad log mining and Flink wordcount. Still, if you have any query regarding Apache Flink Shell Commands, ask in the comment tab.<\/p>\n","protected":false},"excerpt":{"rendered":"
This Apache Flink quickstart tutorial will take you through various apache Flink shell commands. Apache Flink provides an interactive shell \/ Scala prompt where the user can run Flink commands for different transformation operations...<\/p>\n","protected":false},"author":6,"featured_media":43053,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[8],"tags":[750,759,4738,4749,4750,4780,4782,4784,4789],"class_list":["post-1223","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-flink","tag-apache-flink","tag-apache-flink-quickstart","tag-flink","tag-flink-command-tutorial","tag-flink-commands","tag-flink-quickstart","tag-flink-scala-command","tag-flink-shell-commands","tag-flink-tutorial"],"yoast_head":"\n
Apache Flink Shell Commands Tutorial - A Quickstart For beginners - DataFlair<\/title>\n<meta name=\"description\" content=\"Apache Flink Shell Commands-List of Flink operations to interact with flink shell.Create dataset,print elements, Flink filter operation, Flink map operation\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/apache-flink-shell-commands\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Apache Flink Shell Commands Tutorial - A Quickstart For beginners - DataFlair\" \/>\n<meta property=\"og:description\" content=\"Apache Flink Shell Commands-List of Flink operations to interact with flink shell.Create dataset,print elements, Flink filter operation, Flink map operation\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/apache-flink-shell-commands\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2016-11-19T18:13:01+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-05-09T07:53:45+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/11\/Apache-Flink-Shell-Commands-01.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Apache Flink Shell Commands Tutorial - A Quickstart For beginners - DataFlair","description":"Apache Flink Shell Commands-List of Flink operations to interact with flink shell.Create dataset,print elements, Flink filter operation, Flink map operation","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/apache-flink-shell-commands\/","og_locale":"en_US","og_type":"article","og_title":"Apache Flink Shell Commands Tutorial - A Quickstart For beginners - DataFlair","og_description":"Apache Flink Shell Commands-List of Flink operations to interact with flink shell.Create dataset,print elements, Flink filter operation, Flink map operation","og_url":"https:\/\/data-flair.training\/blogs\/apache-flink-shell-commands\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2016-11-19T18:13:01+00:00","article_modified_time":"2021-05-09T07:53:45+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/11\/Apache-Flink-Shell-Commands-01.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/apache-flink-shell-commands\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/apache-flink-shell-commands\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89"},"headline":"Apache Flink Shell Commands Tutorial – A Quickstart For beginners","datePublished":"2016-11-19T18:13:01+00:00","dateModified":"2021-05-09T07:53:45+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/apache-flink-shell-commands\/"},"wordCount":605,"commentCount":1,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/apache-flink-shell-commands\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/11\/Apache-Flink-Shell-Commands-01.jpg","keywords":["apache flink","apache flink quickstart","flink","flink command tutorial","flink commands","flink quickstart","flink scala command","flink shell commands","flink tutorial"],"articleSection":["Apache Flink Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/apache-flink-shell-commands\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/apache-flink-shell-commands\/","url":"https:\/\/data-flair.training\/blogs\/apache-flink-shell-commands\/","name":"Apache Flink Shell Commands Tutorial - A Quickstart For beginners - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/apache-flink-shell-commands\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/apache-flink-shell-commands\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/11\/Apache-Flink-Shell-Commands-01.jpg","datePublished":"2016-11-19T18:13:01+00:00","dateModified":"2021-05-09T07:53:45+00:00","description":"Apache Flink Shell Commands-List of Flink operations to interact with flink shell.Create dataset,print elements, Flink filter operation, Flink map operation","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/apache-flink-shell-commands\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/apache-flink-shell-commands\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/apache-flink-shell-commands\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/11\/Apache-Flink-Shell-Commands-01.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/11\/Apache-Flink-Shell-Commands-01.jpg","width":1200,"height":628,"caption":"Apache Flink Shell Commands Tutorial - A Quickstart For beginners"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/apache-flink-shell-commands\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Apache Flink Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/flink\/"},{"@type":"ListItem","position":3,"name":"Apache Flink Shell Commands Tutorial – A Quickstart For beginners"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"The DataFlair Team provides industry-driven content on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our expert educators focus on delivering value-packed, easy-to-follow resources for tech enthusiasts and professionals.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam2\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/1223","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=1223"}],"version-history":[{"count":1,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/1223\/revisions"}],"predecessor-version":[{"id":94125,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/1223\/revisions\/94125"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/43053"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=1223"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=1223"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=1223"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}