{"id":1125,"date":"2016-08-28T04:19:06","date_gmt":"2016-08-28T04:19:06","guid":{"rendered":"http:\/\/data-flair.training\/blogs\/?p=1125"},"modified":"2021-05-09T13:23:48","modified_gmt":"2021-05-09T07:53:48","slug":"flink-real-world-use-case","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/flink-real-world-use-case\/","title":{"rendered":"Apache Flink Real World Use Case – Crime Data Analysis Part II"},"content":{"rendered":"

This is the second part of Apache Flink <\/strong>real world Use case – Crime Data Analysis, for details about the case study follow the first part<\/strong>. In this blog we will use various Apache Flink APIs like readTextFile, readCsvFile, include fields, groupBy, FlatMap, Map etc. to analyze the crime report use-case.<\/p>\n

Platform<\/h2>\n
    \n
  • Operating system: Windows \/\u00a0Linux \/ Mac<\/li>\n
  • Apache Flink<\/li>\n
  • Java 7.x or higher<\/li>\n
  • IDE: Eclipse<\/li>\n<\/ul>\n

    Steps to make project and Add dependencies<\/h2>\n

     <\/p>\n

    Sample Data<\/h2>\n

    Following is the sample dataset for the use-case:
    \n[php]
    \ncdatetime,address,district,beat,grid,crimedescr,ucr_ncic_code,latitude,longitude
    \n1\/1\/2006 0:00,3108 OCCIDENTAL DR,3,3C,1115,484 PC PETTY THEFT\/INSIDE,2404,38.47350069,-121.4901858
    \n1\/1\/2006 0:00,4 PALEN CT,2,2A,212,459 PC BURGLARY BUSINESS,2299,38.47350069,-121.4901858
    \n1\/1\/2006 0:00,3547 P ST,3,3C,853,484 PC PETTY THEFT\/INSIDE,2404,38.47350069,-121.4901858
    \n1\/1\/2006 0:00,3421,AUBURN BLVD,2,2A,508 459 PC BURGLARY BUSINESS,2299,38.47350069,-121.4901858
    \n1\/1\/2006 0:00,6351 DRIFTWOOD ST,4,4C,1261,SUSP PERS-NO CRIME \u2013 I RPT,7000,38.47350069,-121.4901858
    \n1\/1\/2006 0:01,7001 EAST PKWY,6,6C ,1427,503 PC EMBEZZLEMENT,2799,38.51436734,-121.3854286
    \n1\/1\/2006 0:01,918 LAKE FRONT DR,4,4C,1294,TELEPEST -I RPT,7000,38.47948616,-121.5215057
    \n1\/1\/2006 0:01,4851 KOKOMO DR,1,1A ,123,487(A) GRAND THEFT-INSIDE,2308,38.65994218,-121.5259008
    \n1\/1\/2006 0:01,2377 OAK HARBOUR DR,1,1B ,440,653M(A) PC OBSCENE\/THREAT CALL,5309,38.60893745,-121.5187927
    \n1\/1\/2006 0:01,1823 P ST,3,3B ,766,484 PETTY THEFT\/LICENSE PLATE,2399,38.57084621,-121.484429
    \n1\/1\/2006 0:01,1100 14TH ST,3,3M,745,FOUND PROPERTY – I RPT,7000,38.57815667,-121.4876958
    \n1\/1\/2006 0:01,3301 ARENA BLVD,1,1A,303,530.5 PC USE PERSONAL ID INFO,2604,38.64378844,-121.5341593
    \n1\/1\/2006 0:01,1088 PREGO WAY,1,1B,404,530.5 PC USE PERSONAL ID INFO,2604,38.62798948,-121.4857344
    \n1\/1\/2006 0:01,3259 SPINNING ROD WAY,1,1B,475,484J PC PUBLISH CARD INFO,2605,38.61153542,-121.5370613
    \n1\/1\/2006 0:01,1010 J ST,3,3M,744,THREATS – I RPT,7000,38.57999251,-121.4930384
    \n1\/1\/2006 0:01,400 BANNON ST,3,3A,704,530 PC FALSE PERS. REC PROP,2604,38.59540732,-121.4978798
    \n[\/php]<\/p>\n

    Flink Real World Use Case – Problem & Solution<\/h2>\n

    In this section of Apache Flink real world use case, two different problem statements are given below along with their solution through Apache Flink program in java and output.<\/p>\n

    i. Problem Statement – 1<\/h3>\n

    In this use case, we will analyze the data and find out the Hour of the day when a maximum crime occurs (Highest Crime Hour Analysis) for each day.<\/p>\n

    a. Apache-Flink program<\/h4>\n

    [php]
    \nimport java.text.SimpleDateFormat;
    \nimport java.util.Date;
    \nimport org.apache.flink.api.common.functions.MapFunction;
    \nimport org.apache.flink.api.java.DataSet;
    \nimport org.apache.flink.api.java.ExecutionEnvironment;
    \nimport org.apache.flink.api.java.tuple.Tuple1;
    \nimport org.apache.flink.api.java.tuple.Tuple3;
    \npublic class Maximum_Crime_Hour {
    \npublic static void main(String[] args) throws Exception {
    \nExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
    \nDataSet < Tuple1 < String >> rawdata = env.readCsvFile(“E:\\\\SacramentocrimeJanuary2006.csv”)
    \n.includeFields(“1000000”).ignoreFirstLine()
    \n.types(String.class);
    \nrawdata.map(new TimeExtractor()) \/\/map the data according to the MM\/dd\/yyyy HH
    \n.groupBy(0, 1) \/\/group the data according to date & hour
    \n.sum(2) \/\/ sum on the field 2 to count
    \n.groupBy(0) \/\/group the data according to field date
    \n.maxBy(0,2) \/\/to find out the maximum on the basis of per day
    \n.print(); \/\/print the result
    \n}
    \npublic static class TimeExtractor implements MapFunction< Tuple1 < String >, Tuple3<String, String, Integer>> {
    \n@Override
    \npublic Tuple3<String, String, Integer> map(Tuple1<String> time) throws Exception {
    \nSimpleDateFormat formatter = new SimpleDateFormat(“MM\/dd\/yyyy HH”);
    \nSimpleDateFormat formatter2 = new SimpleDateFormat(“MM\/dd\/yyyy HH:mm”);
    \nString dateInString = time.f0;
    \nDate date = formatter2.parse(dateInString);
    \nString dateTokens[] = formatter.format(date).split(” “);
    \nreturn new Tuple3<>(dateTokens[0], dateTokens[1], 1);
    \n}
    \n}
    \n}
    \n[\/php]<\/p>\n

    b. Output<\/h4>\n

    [php]Date,Hour,No_Of_Crimes
    \n(01\/01\/2006,00,57)
    \n(01\/05\/2006,20,22)
    \n(01\/08\/2006,23,16)
    \n(01\/13\/2006,09,20)
    \n(01\/14\/2006,00,16)
    \n(01\/19\/2006,09,20)
    \n(01\/20\/2006,00,21)
    \n(01\/23\/2006,00,24)
    \n(01\/31\/2006,12,19)[\/php]<\/p>\n

    ii. Problem Statement – 2<\/h3>\n

    In this use case, we will analyze the data and find out the District with least number of crimes (Safest District).<\/p>\n

    a. Apache Flink Program<\/h4>\n

    [php]
    \nimport org.apache.flink.api.common.functions.FlatMapFunction;
    \nimport org.apache.flink.api.java.DataSet;
    \nimport org.apache.flink.api.java.ExecutionEnvironment;
    \nimport org.apache.flink.api.java.tuple.Tuple2;
    \nimport org.apache.flink.util.Collector;
    \npublic class SafeDistrict{
    \npublic static void main(String[] args) throws Exception {
    \nExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
    \nDataSet<String> rawdata = env.readTextFile(“E:\\\\SacramentocrimeJanuary2006.csv”);
    \nDataSet <Tuple2<String, Integer>> result = rawdata
    \n.flatMap(new Counter())\/\/map the data and as district,1
    \n.groupBy(0) \/\/ group the data according to district
    \n.sum(1) \/\/ to count no. of crimes in a district
    \n.minBy(1); \/\/to find out the minimum crime
    \nresult.print();
    \n}
    \npublic static class Counter implements FlatMapFunction<String, Tuple2<String, Integer>> {
    \n@Override
    \npublic void flatMap(String value, Collector<Tuple2<String, Integer>> out) {
    \nString[] tokens = value.split(“,”);
    \nif(tokens[2].contains(“district”))
    \n{
    \nreturn;
    \n}
    \nelse
    \n{
    \nout.collect(new Tuple2<String, Integer>(tokens[2], 1));
    \n}
    \n}
    \n}
    \n}
    \n[\/php]<\/p>\n

    b. Output<\/h4>\n

    [php] (District,Crime_Count) (1,868) [\/php]
    \nTo learn the comparison between Different big data technologies like- Apache Hadoop, Apache Spark<\/strong>, follow this comparison guide<\/strong>.<\/p>\n

    So, this was all in Apache Flink Real World Use Case. Hope you like our explanation.<\/p>\n

    Conclusion – Flink Use Cases<\/h2>\n

    So, in this tutorial we have completed the part 2 of Apache Flink real-world use case. Me have discussed it with the help of sample data and some problems and solutions of it. Still, if you have any query regarding Apache Flink Real World Use Case, ask in the comment tab.<\/p>\n","protected":false},"excerpt":{"rendered":"

    This is the second part of Apache Flink real world Use case – Crime Data Analysis, for details about the case study follow the first part. In this blog we will use various Apache...<\/p>\n","protected":false},"author":6,"featured_media":42794,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[8],"tags":[752,758,760,4781,16581,4791],"class_list":["post-1125","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-flink","tag-apache-flink-case-study","tag-apache-flink-project","tag-apache-flink-use-case","tag-flink-real-time-use-case","tag-flink-real-world-use-cases","tag-flink-use-cases"],"yoast_head":"\nApache Flink Real World Use Case - Crime Data Analysis Part II - DataFlair<\/title>\n<meta name=\"description\" content=\"Learn Apache Flink APIs-Understand Flink APIs-readTextFile,readCsvFile,includeFields,groupBy,FlatMap,Map to analyze Flink real world use case crime report\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/flink-real-world-use-case\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Apache Flink Real World Use Case - Crime Data Analysis Part II - DataFlair\" \/>\n<meta property=\"og:description\" content=\"Learn Apache Flink APIs-Understand Flink APIs-readTextFile,readCsvFile,includeFields,groupBy,FlatMap,Map to analyze Flink real world use case crime report\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/flink-real-world-use-case\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2016-08-28T04:19:06+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-05-09T07:53:48+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/08\/Apache-Flink-Real-World-Use-Case-01.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Apache Flink Real World Use Case - Crime Data Analysis Part II - DataFlair","description":"Learn Apache Flink APIs-Understand Flink APIs-readTextFile,readCsvFile,includeFields,groupBy,FlatMap,Map to analyze Flink real world use case crime report","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/flink-real-world-use-case\/","og_locale":"en_US","og_type":"article","og_title":"Apache Flink Real World Use Case - Crime Data Analysis Part II - DataFlair","og_description":"Learn Apache Flink APIs-Understand Flink APIs-readTextFile,readCsvFile,includeFields,groupBy,FlatMap,Map to analyze Flink real world use case crime report","og_url":"https:\/\/data-flair.training\/blogs\/flink-real-world-use-case\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2016-08-28T04:19:06+00:00","article_modified_time":"2021-05-09T07:53:48+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/08\/Apache-Flink-Real-World-Use-Case-01.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/flink-real-world-use-case\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/flink-real-world-use-case\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89"},"headline":"Apache Flink Real World Use Case – Crime Data Analysis Part II","datePublished":"2016-08-28T04:19:06+00:00","dateModified":"2021-05-09T07:53:48+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/flink-real-world-use-case\/"},"wordCount":814,"commentCount":0,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/flink-real-world-use-case\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/08\/Apache-Flink-Real-World-Use-Case-01.jpg","keywords":["apache flink case study","apache flink project","apache flink use case","flink real time use case","Flink Real world Use cases","flink use-cases"],"articleSection":["Apache Flink Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/flink-real-world-use-case\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/flink-real-world-use-case\/","url":"https:\/\/data-flair.training\/blogs\/flink-real-world-use-case\/","name":"Apache Flink Real World Use Case - Crime Data Analysis Part II - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/flink-real-world-use-case\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/flink-real-world-use-case\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/08\/Apache-Flink-Real-World-Use-Case-01.jpg","datePublished":"2016-08-28T04:19:06+00:00","dateModified":"2021-05-09T07:53:48+00:00","description":"Learn Apache Flink APIs-Understand Flink APIs-readTextFile,readCsvFile,includeFields,groupBy,FlatMap,Map to analyze Flink real world use case crime report","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/flink-real-world-use-case\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/flink-real-world-use-case\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/flink-real-world-use-case\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/08\/Apache-Flink-Real-World-Use-Case-01.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/08\/Apache-Flink-Real-World-Use-Case-01.jpg","width":1200,"height":628,"caption":"Apache Flink Real World Use Case - Crime Data Analysis Part II"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/flink-real-world-use-case\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Apache Flink Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/flink\/"},{"@type":"ListItem","position":3,"name":"Apache Flink Real World Use Case – Crime Data Analysis Part II"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"The DataFlair Team provides industry-driven content on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our expert educators focus on delivering value-packed, easy-to-follow resources for tech enthusiasts and professionals.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam2\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/1125","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=1125"}],"version-history":[{"count":1,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/1125\/revisions"}],"predecessor-version":[{"id":94128,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/1125\/revisions\/94128"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/42794"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=1125"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=1125"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=1125"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}