{"id":75361,"date":"2020-02-04T14:38:05","date_gmt":"2020-02-04T09:08:05","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=75361"},"modified":"2021-08-25T22:32:26","modified_gmt":"2021-08-25T17:02:26","slug":"hadoop-history","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/hadoop-history\/","title":{"rendered":"History of Hadoop – The complete evolution of Hadoop Ecosytem"},"content":{"rendered":"

Let\u2019s take a look at the history of Hadoop and its evolution in the last two decades and why it continues to be the backbone of the big data industry.<\/strong><\/p>\n

Hadoop is an open-source software<\/strong> framework for storing and processing large datasets ranging in size from gigabytes<\/strong> to petabytes<\/strong>. Hadoop was developed at the Apache Software Foundation.<\/p>\n

In 2008, Hadoop defeated the supercomputers<\/strong> and became the fastest system on the planet<\/strong> for sorting terabytes of data.<\/p>\n

This article describes the evolution of Hadoop over a period.<\/p>\n

History of Hadoop<\/h2>\n

\"Hadoop<\/a><\/p>\n

2002<\/strong><\/h4>\n

It all started in the year 2002 with the Apache Nutch project.<\/p>\n

In 2002, Doug Cutting and Mike Cafarella were working on Apache Nutch Project that aimed at building a web search engine that would crawl and index websites.<\/p>\n

After a lot of research, Mike Cafarella and Doug Cutting estimated that it would cost around $500,000 in hardware with a monthly running cost of $30,000 for a system supporting a one-billion-page index.<\/p>\n

This project proved to be too expensive and thus found infeasible for indexing billions of webpages. So they were looking for a feasible solution that would reduce the cost.<\/p>\n

2003<\/strong><\/h4>\n

Meanwhile, In 2003 Google released a search paper on Google distributed File System (GFS) that described the architecture for GFS that provided an idea for storing large datasets in a distributed environment. This paper solved the problem of storing huge files generated as a part of the web crawl and indexing process. But this is half of a solution to their problem.<\/p>\n

2004<\/strong><\/h4>\n

In 2004, Nutch\u2019s developers set about writing an open-source implementation, the Nutch Distributed File System (NDFS).<\/p>\n

In 2004, Google introduced MapReduce to the world by releasing a paper on MapReduce. This paper provided the solution for processing those large datasets. It gave a full solution to the Nutch developers.<\/p>\n

Google provided the idea for distributed storage and MapReduce. Nutch developers implemented MapReduce in the middle of 2004.<\/p>\n

2006<\/strong><\/h4>\n

The Apache community realized that the implementation of MapReduce and NDFS could be used for other tasks as well. In February 2006, they came out of Nutch and formed an independent subproject of Lucene called “Hadoop<\/strong>” (which is the name of Doug’s kid’s yellow elephant).<\/p>\n

As the Nutch project was limited to 20 to 40 nodes cluster, Doug Cutting in 2006 itself joined Yahoo to scale the Hadoop<\/a> project to thousands of nodes cluster.<\/p>\n

2007<\/strong><\/h4>\n

In 2007, Yahoo started using Hadoop on 1000 nodes cluster.<\/p>\n

2008<\/strong><\/h4>\n

In January 2008, Hadoop confirmed its success by becoming the top-level project at Apache.<\/p>\n

By this time, many other companies like Last.fm, Facebook, and the New York Times started using Hadoop.<\/p>\n

Hadoop Defeated supercomputers<\/h3>\n

In April 2008, Hadoop defeated supercomputers and became the fastest system on the planet by sorting an entire terabyte of data.<\/p>\n

In November 2008, Google reported that its Mapreduce implementation sorted 1 terabyte in 68 seconds.<\/p>\n

In April 2009, a team at Yahoo used Hadoop to sort 1 terabyte in 62 seconds, beaten Google MapReduce implementation.<\/p>\n

Various Release of Hadoop<\/h3>\n

2011 – 2012<\/strong><\/h4>\n

On 27 December 2011, Apache released Hadoop version 1.0 that includes support for Security, Hbase, etc.<\/p>\n

On 10 March 2012, release 1.0.1 was available. This is a bug fix release for version 1.0.<\/p>\n

On 23 May 2012, the Hadoop 2.0.0-alpha version was released. This release contains YARN.<\/p>\n

The second (alpha) version in the Hadoop-2.x series with a more stable version of YARN was released on 9 October 2012.<\/p>\n

2017 – now<\/strong><\/h4>\n

On 13 December 2017, release 3.0.0 was available<\/p>\n

On 25 March 2018, Apache released Hadoop 3.0.1, which contains 49 bug fixes in Hadoop 3.0.0.<\/p>\n

On 6 April 2018, Hadoop release 3.1.0 came that contains 768 bug fixes, improvements, and enhancements since 3.0.0.<\/p>\n

Later, in May 2018, Hadoop 3.0.3 was released.<\/p>\n

On 8 August 2018, Apache 3.1.1 was released.<\/p>\n

Hadoop 3.1.3 is the latest version of Hadoop.<\/p>\n

Wondering to install Hadoop 3.1.3?<\/strong> Follow the Step-by-step Installation<\/strong><\/a> tutorial and install it now!<\/p>\n

Summary<\/h2>\n

That’s the History of Hadoop in brief points.<\/p>\n

I hope after reading this article, you understand Hadoop\u2019s journey and how Hadoop confirmed its success and became the most popular big data analysis tool.<\/p>\n

Now it is your turn to take a ride and evolve yourself in the Big Data industry with the Hadoop course<\/strong><\/a>.<\/p>\n

Keep Executing!!

Let\u2019s take a look at the history of Hadoop and its evolution in the last two decades and why it continues to be the backbone of the big data industry. Hadoop is an open-source...<\/p>\n","protected":false},"author":7,"featured_media":75420,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[22],"tags":[21816,21815,21806,21805,21807,21814,21813],"class_list":["post-75361","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-hadoop","tag-apache-hadoop-history","tag-brief-history-of-hadoop","tag-evolution-of-hadoop","tag-hadoop-history","tag-hadoop-version-history","tag-history-of-hadoop-in-big-data","tag-mapreduce-history"],"yoast_head":"\nHistory of Hadoop - The complete evolution of Hadoop Ecosytem - DataFlair<\/title>\n<meta name=\"description\" content=\"Know the history of Hadoop and understand when it all started. Delve into key technologies and practices that have been playing since last two decades\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/hadoop-history\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"History of Hadoop - The complete evolution of Hadoop Ecosytem - DataFlair\" \/>\n<meta property=\"og:description\" content=\"Know the history of Hadoop and understand when it all started. Delve into key technologies and practices that have been playing since last two decades\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/hadoop-history\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2020-02-04T09:08:05+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-08-25T17:02:26+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2020\/02\/history-of-Hadoop.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"802\" \/>\n\t<meta property=\"og:image:height\" content=\"420\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"History of Hadoop - The complete evolution of Hadoop Ecosytem - DataFlair","description":"Know the history of Hadoop and understand when it all started. Delve into key technologies and practices that have been playing since last two decades","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/hadoop-history\/","og_locale":"en_US","og_type":"article","og_title":"History of Hadoop - The complete evolution of Hadoop Ecosytem - DataFlair","og_description":"Know the history of Hadoop and understand when it all started. Delve into key technologies and practices that have been playing since last two decades","og_url":"https:\/\/data-flair.training\/blogs\/hadoop-history\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2020-02-04T09:08:05+00:00","article_modified_time":"2021-08-25T17:02:26+00:00","og_image":[{"width":802,"height":420,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2020\/02\/history-of-Hadoop.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/hadoop-history\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/hadoop-history\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/beb0cab24b7aa54423a3b50e669a9dcd"},"headline":"History of Hadoop – The complete evolution of Hadoop Ecosytem","datePublished":"2020-02-04T09:08:05+00:00","dateModified":"2021-08-25T17:02:26+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/hadoop-history\/"},"wordCount":669,"commentCount":0,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/hadoop-history\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2020\/02\/history-of-Hadoop.jpg","keywords":["apache hadoop history","brief history of hadoop","evolution of hadoop","hadoop history","hadoop version history","history of hadoop in big data","mapreduce history"],"articleSection":["Hadoop Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/hadoop-history\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/hadoop-history\/","url":"https:\/\/data-flair.training\/blogs\/hadoop-history\/","name":"History of Hadoop - The complete evolution of Hadoop Ecosytem - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/hadoop-history\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/hadoop-history\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2020\/02\/history-of-Hadoop.jpg","datePublished":"2020-02-04T09:08:05+00:00","dateModified":"2021-08-25T17:02:26+00:00","description":"Know the history of Hadoop and understand when it all started. Delve into key technologies and practices that have been playing since last two decades","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/hadoop-history\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/hadoop-history\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/hadoop-history\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2020\/02\/history-of-Hadoop.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2020\/02\/history-of-Hadoop.jpg","width":802,"height":420,"caption":"hadoop history"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/hadoop-history\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Hadoop Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/hadoop\/"},{"@type":"ListItem","position":3,"name":"History of Hadoop – The complete evolution of Hadoop Ecosytem"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/beb0cab24b7aa54423a3b50e669a9dcd","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/c322416204232f4dd97ef3901b0a499a5d34d7ba7fe333f4bfe53a907873d293?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/c322416204232f4dd97ef3901b0a499a5d34d7ba7fe333f4bfe53a907873d293?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c322416204232f4dd97ef3901b0a499a5d34d7ba7fe333f4bfe53a907873d293?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"DataFlair Team specializes in creating clear, actionable content on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Backed by industry expertise, we make learning easy and career-oriented for beginners and pros alike.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam3\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/75361","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=75361"}],"version-history":[{"count":17,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/75361\/revisions"}],"predecessor-version":[{"id":75554,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/75361\/revisions\/75554"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/75420"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=75361"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=75361"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=75361"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}