MapReduce 编程

Hadoop

Hadoop教程 Hadoop - 简介 Hadoop HDFS HDFS 写文件 HDFS 读文件 HDFS 可靠性 HDFS 命令工具 Hadoop YARN YARN ResourceManager YARN NodeManager YARN ApplicationMaster YARN Container YARN Failover Hadoop MapReduce MapReduce 读取数据 MapReduce Mapper MapReduce Shuffle MapReduce 编程 Hadoop IO Hadoop 测试 Hadoop 安装 Hadoop配置 Hadoop 监控

MapReduce 编程

在线练习

http://cloudcomputing.ruc.edu.cn

处理

select：直接分析输入数据，取出需要的字段数据即可
where: 也是对输入数据处理的过程中进行处理，判断是否需要该数据
aggregation:min, max, sum
group by: 通过Reducer实现
sort
join: map join, reduce join

Third-Party Libraries

export LIBJARS=$MYLIB/commons-lang-2.3.jar, hadoop jar prohadoop-0.0.1-SNAPSHOT.jar org.aspress.prohadoop.c3. WordCountUsingToolRunner -libjars $LIBJARS

hadoop jar prohadoop-0.0.1-SNAPSHOT-jar-with-dependencies.jar org.aspress.prohadoop.c3. WordCountUsingToolRunner The dependent libraries are now included inside the application JAR file

一般还是上面的好，指定依赖可以利用Public Cache，如果是包含依赖，则每次都需要拷贝

参考书籍

MapReduce Design Patterns

上一篇:MapReduce Shuffle

下一篇:Hadoop IO

我要发贴

Hadoop