《Hadoop The Definitive Guide》ch06 How MapReduce Works

1. MapReduce的工作原理

1) 客户端提交MapReduce作业。

2) jobtracker 协调作业的运行。 jobtracker是一个Java应用程序，它的主类是JobTracker。

3) tasktracker 运行作业划分后的任务。tasktracker是一个Java应用程序，它的主类是TaskTracker。

4) 分布式文件系统（一般为HDFS），用来在其他实体间共享作业文件。

2. JobClient的submitJob()方法所实现的作业提交过程如下

a. Asks the jobtracker for a new job ID (by calling getNewJobId() on JobTracker) (step 2).
b. Checks the output specification of the job. For example, if the output directory has not been specified or it already exists, the job is not submitted and an error is
thrown to the MapReduce program.
c. Computes the input splits for the job. If the splits cannot be computed, because the input paths don’t exist, for example, then the job is not submitted and an error
is thrown to the MapReduce program.
d. Copies the resources needed to run the job, including the job JAR file, the configuration file, and the computed input splits, to the jobtracker’s filesystem in a
directory named after the job ID. The job JAR is copied with a high replication factor (controlled by the mapred.submit.replication property, which defaults to
10) so that there are lots of copies across the cluster for the tasktrackers to access when they run tasks for the job (step 3).
e. Tells the jobtracker that the job is ready for execution (by calling submitJob() onJobTracker) (step 4).

3. tasktracker中执行的流和管道及其子进程的关系

来源：网络

智能推荐

Hadoop: the definitive guide 第三版拾遗第十二章之Hive初步

Hive简介 Hive是建立在 Hadoop 上的数据仓库基础构架。它提供了一系列的工具，可以用来进行数据提取转化加载（ETL），这是一种可以存储、查询和分析存储在 Hadoop 中的大规模数据的机制。Hive 定义了简单的类 SQL 查询语言，称为 HQL，它允许熟悉 SQL 的用户查询数据。同时，这个语言也允许熟悉 MapReduce 开发者的开发自定义的 mapper 和 reducer 来...

Build in the Cloud: How the Build System works

This is the second in a four part series describing how we use the cloud to scale building and testing of software at Google. This series elaborates on a presentation given during the Pre-GTAC 2010 ev...

The Definitive Guide to AdonisJs 免积分下载

Apress The Definitive Guide to AdonisJs 电子书下载 Apress The Definitive Guide to AdonisJs Book Description Learn everything you need to master the AdonisJs framework, including topics such as interac...

Spark:The Definitive Guide--读书小结（二）

个人读后小结，如有不对，欢迎批评指正前言第Ⅰ部分大数据与Spark概述第Ⅱ部分结构化API--DataFrame、SQL和Dataset 第Ⅲ部分低级API 第Ⅳ部分生产与应用第Ⅴ部分流处理第Ⅵ部分高级分析与机器学习第Ⅶ部分生态系统前言大概花了一个月（2020-09~2020-10）左右的时间，看完了整本书。笔者毕业后，主要从事机器学习、数据挖掘方向的相关工作。由于S...

Spark:The Definitive Guide--读书小结（四）

猜你喜欢

Spark:The Definitive Guide--读书小结（五）

《Closure_The Definitive Guide 》 Closure Libary 读书笔记

Closure Libary 是一个广泛的、经过良好测试的、模块化的、跨浏览器的JavaScript库。您可以从大量可重用的UI小部件和控件中提取所需的内容，也可以从用于DOM操作、服务器通信、动画、数据结构、单元测试、富文本编辑等较低级别的实用程序中提取所需的内容。 https://developers.google.com/closure/library...

How JavaScript works: an overview of the engine, the runtime, and the call stack

As JavaScript is getting more and more popular, teams are leveraging its support on many levels in their stack - front-end, back-end, hybrid apps, embedded devices and much more. This post is meant to...

How Tomcat Works: Introduction

How Servlet Container works (1) 为一个servlet处理一个request, 主要做三件事 I. 创建一个request 对象（包含parameters,headers,cookies,query string,URI等信息）. request 对象是javax.servlet.ServletRequest 接口或该接口的实现实...

原型对象，原型链

函数都有prototype属性，它指向原型对象。实例对象有__proto__属性，它指向对象原型每一个原型对象都有constructor输赢，指向构造函数，每一个原型对象又具有__proto__属性，这个指向Object.prototype.在这里插入图片描述...