{"id":727,"date":"2016-08-22T12:40:03","date_gmt":"2016-08-22T12:40:03","guid":{"rendered":"http:\/\/data-flair.training\/blogs\/?p=727"},"modified":"2021-05-09T13:08:37","modified_gmt":"2021-05-09T07:38:37","slug":"apache-flume-installation-tutorial","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/apache-flume-installation-tutorial\/","title":{"rendered":"Apache Flume Installation Tutorial – A beginners guide"},"content":{"rendered":"

In this article, we will see how to install Apache Flume on Ubuntu. The article provides you the complete guide for installing Apache Flume 1.9.0 on Ubuntu.<\/p>\n

This article explains each step of the flume installation in detail along with the Screenshots. You will also know how to configure Flume for copying data into the Hadoop Distributed File System.<\/p>\n

Let us now start with the step-by-step guide for Apache Flume installation.<\/p>\n

Flume Installation Prerequisite<\/h2>\n

1. For installing Apache Flume we must have Hadoop installed and configured on our system.<\/p>\n

2. Java must also be installed on your system.<\/p>\n

If Hadoop is not installed on your system, then you can refer to the Hadoop 3 installation article to install Hadoop.<\/p>\n

Step-by-step guide for Flume installation on Ubuntu<\/h2>\n

1. First we have to download Apache Flume 1.9.0. So let’s Download Apache Flume<\/a>.<\/p>\n

2. Locate the tar file that you have downloaded.<\/p>\n

\".locating-flume-tar-flume-installation\"<\/a><\/p>\n

3. Extract the tar file using the below command:
\ntar xzf apache-flume-1.9.0-bin.tar.gz<\/strong><\/p>\n

\"extract-flume-tar\"<\/a><\/p>\n

Now we have successfully extracted apache-flume-1.9.0-bin.tar.gz. Use ls command to enlist files and directories.<\/p>\n

\"viewing<\/a><\/p>\n

4. Now we have to set the FLUME_HOME path in the .bashrc file. For this open .bashrc file in nano editor.<\/p>\n

\"opening<\/a><\/p>\n

Add below parameters in the .bashrc file.<\/p>\n

export FLUME_HOME=\/home\/dataflair\/apache-flume-1.9.0-bin<\/strong>
\nexport PATH=$PATH:$FLUME_HOME\/bin<\/strong><\/p>\n

\"adding<\/a><\/p>\n

Note: \u201c\/home\/dataflair\/apache-flume-1.9.0-bin\u201d is the path of my Flume directory and apache-flume-1.9.0-bin is the name of my flume directory. You must provide the path where your flume directory is located.<\/p>\n

Press Ctrl+O to save the changes and press Ctrl+X to exit.<\/p>\n

5. Refresh the .bashrc file by using the below command:
\nsource .bashrc<\/strong><\/p>\n

\"command<\/a><\/p>\n

6. The flume has now successfully been installed on our system. To verify Flume installation use below command:
\nflume-ng version<\/strong><\/p>\n

\"check<\/a><\/p>\n

\"verifying<\/a><\/p>\n

Congratulations!!!! We have successfully installed Apache Flume 1.9.0 on Ubuntu<\/strong>.<\/p>\n

Now, we have to Configure Apache Flume for copying data to HDFS. Follow the below steps to do so.<\/p>\n

Configuring Apache Flume<\/h2>\n

1. Locate the access log file in your local file system which you want to move to HDFS.<\/p>\n

\".locating<\/a><\/p>\n

2. Now, we have to create a file named flume.conf inside the \/home\/dataflair\/apache-flume-1.6.0-bin\/conf directory. Use the below-mentioned command for creating the file \u201cflume.conf\u201d:
\ntouch flume.conf<\/strong>
\n\"create<\/a>Open file in nano editor and add the below mentioned parameters in it:<\/p>\n

agent1.sources = tail
\nagent1.channels = Channel-2
\nagent1.sinks = sink-1<\/p>\n

agent1.sources.tail.type = exec
\nagent1.sources.tail.command = cat \/home\/dataflair\/access.log.2
\nagent1.sources.tail.channels = Channel-2<\/p>\n

agent1.sinks.sink-1.channel = Channel-2
\nagent1.sinks.sink-1.type = hdfs
\nagent1.sinks.sink-1.hdfs.path = hdfs:\/\/localhost:9000\/flume01
\nagent1.sinks.sink-1.hdfs.fileType = DataStream
\nagent1.sinks.sink-1.hdfs.rollInterval = 60
\nagent1.sinks.sink-1.hdfs.rollSize = 0
\nagent1.sinks.sink-1.hdfs.rollCount = 0<\/p>\n

agent1.channels.Channel-2.type = memory<\/p>\n

\"opening<\/a><\/p>\n

\"configuring<\/a><\/p>\n

Note: Here \u201c\/home\/dataflair\/access.log.2\u201d is the path for the data file which I am copying from the local file system to HDFS. You must provide the path of your \u201caccess.log\u201d file.<\/p>\n

3. Now we will start the Flume for copying files using the below command:<\/p>\n

bin\/flume-ng agent –conf .\/conf\/ -f conf\/flume.conf -n agent1 -Dflume.root.logger=DEBUG<\/strong><\/p>\n

\"Apache<\/a><\/p>\n

Now, we have successfully copied the data from the access log file to HDFS.<\/p>\n

4. We can check whether data gets copied in HDFS or not by either using a web console (http:\/\/localhost:9870) or view the files in HDFS by using the command prompt. On the web console you can view your files in the \u201c\/flume01\u201d directory.<\/p>\n

Summary<\/h2>\n

I hope now you have successfully installed and configured Apache Flume on your Ubuntu system. The article had explained and mentioned every single step you should go through for Apache Flume installation and configuration on Ubuntu.<\/p>\n

If you face any problem in installing and configuring Flume then you can ask us in the comment box.

In this article, we will see how to install Apache Flume on Ubuntu. The article provides you the complete guide for installing Apache Flume 1.9.0 on Ubuntu. This article explains each step of the...<\/p>\n","protected":false},"author":6,"featured_media":78422,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[21],"tags":[762,771,1907,1971,3416,3761,4799,4811,4822,4823,6767],"class_list":["post-727","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-flume","tag-apache-flume","tag-apache-flume-installation","tag-big-data","tag-big-data-training","tag-data-science","tag-deploy-flume","tag-flume","tag-flume-copy-data-into-hdfs","tag-flume-hdfs","tag-flume-installation","tag-install-flume"],"yoast_head":"\nApache Flume Installation Tutorial - A beginners guide - DataFlair<\/title>\n<meta name=\"description\" content=\"Learn Apache Flume installation-Learn to install & configure Apache Flume, setup Flume environment & run Flume to transport data into HDFS using Flume NG agent\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/apache-flume-installation-tutorial\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Apache Flume Installation Tutorial - A beginners guide - DataFlair\" \/>\n<meta property=\"og:description\" content=\"Learn Apache Flume installation-Learn to install & configure Apache Flume, setup Flume environment & run Flume to transport data into HDFS using Flume NG agent\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/apache-flume-installation-tutorial\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2016-08-22T12:40:03+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-05-09T07:38:37+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/08\/Apache-Flume-Installation.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Apache Flume Installation Tutorial - A beginners guide - DataFlair","description":"Learn Apache Flume installation-Learn to install & configure Apache Flume, setup Flume environment & run Flume to transport data into HDFS using Flume NG agent","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/apache-flume-installation-tutorial\/","og_locale":"en_US","og_type":"article","og_title":"Apache Flume Installation Tutorial - A beginners guide - DataFlair","og_description":"Learn Apache Flume installation-Learn to install & configure Apache Flume, setup Flume environment & run Flume to transport data into HDFS using Flume NG agent","og_url":"https:\/\/data-flair.training\/blogs\/apache-flume-installation-tutorial\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2016-08-22T12:40:03+00:00","article_modified_time":"2021-05-09T07:38:37+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/08\/Apache-Flume-Installation.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/apache-flume-installation-tutorial\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/apache-flume-installation-tutorial\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89"},"headline":"Apache Flume Installation Tutorial – A beginners guide","datePublished":"2016-08-22T12:40:03+00:00","dateModified":"2021-05-09T07:38:37+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/apache-flume-installation-tutorial\/"},"wordCount":638,"commentCount":2,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/apache-flume-installation-tutorial\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/08\/Apache-Flume-Installation.jpg","keywords":["Apache Flume","Apache flume installation","big data","big data training","data science","deploy flume","Flume","Flume Copy Data into HDFS","flume hdfs","flume installation","install flume"],"articleSection":["Flume Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/apache-flume-installation-tutorial\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/apache-flume-installation-tutorial\/","url":"https:\/\/data-flair.training\/blogs\/apache-flume-installation-tutorial\/","name":"Apache Flume Installation Tutorial - A beginners guide - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/apache-flume-installation-tutorial\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/apache-flume-installation-tutorial\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/08\/Apache-Flume-Installation.jpg","datePublished":"2016-08-22T12:40:03+00:00","dateModified":"2021-05-09T07:38:37+00:00","description":"Learn Apache Flume installation-Learn to install & configure Apache Flume, setup Flume environment & run Flume to transport data into HDFS using Flume NG agent","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/apache-flume-installation-tutorial\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/apache-flume-installation-tutorial\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/apache-flume-installation-tutorial\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/08\/Apache-Flume-Installation.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/08\/Apache-Flume-Installation.jpg","width":1200,"height":628,"caption":"Apache Flume Installation"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/apache-flume-installation-tutorial\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Flume Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/flume\/"},{"@type":"ListItem","position":3,"name":"Apache Flume Installation Tutorial – A beginners guide"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"The DataFlair Team provides industry-driven content on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our expert educators focus on delivering value-packed, easy-to-follow resources for tech enthusiasts and professionals.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam2\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/727","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=727"}],"version-history":[{"count":4,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/727\/revisions"}],"predecessor-version":[{"id":92708,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/727\/revisions\/92708"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/78422"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=727"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=727"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=727"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}