监控网站建设需要多少钱,企业推广文案范文,品牌推广的渠道有哪些,长沙推广型网站建设1、背景
最近在学习hadoop#xff0c;此篇文章简单记录一下通过源码来编译hadoop。为什么要重新编译hadoop源码#xff0c;是因为为了匹配不同操作系统的本地库环境。
2、编译源码
2.1 下载并解压源码 [roothadoop01 ~]# mkdir /opt/hadoop
[roothadoop01 ~]# cd /opt/had…1、背景
最近在学习hadoop此篇文章简单记录一下通过源码来编译hadoop。为什么要重新编译hadoop源码是因为为了匹配不同操作系统的本地库环境。
2、编译源码
2.1 下载并解压源码 [roothadoop01 ~]# mkdir /opt/hadoop
[roothadoop01 ~]# cd /opt/hadoop/
[roothadoop01 hadoop]# wget https://archive.apache.org/dist/hadoop/common/hadoop-3.3.4/hadoop-3.3.4-src.tar.gz
[roothadoop01 hadoop]# tar -zxvf hadoop-3.3.4-src.tar.gz
[roothadoop01 hadoop]# rm -rvf hadoop-3.3.4-src.tar.gz2.2 查看编译hadoop必要的环境
[roothadoop01 hadoop]# pwd
/opt/hadoop
[roothadoop01 hadoop]# cd hadoop-3.3.4-src/
[roothadoop01 hadoop-3.3.4-src]# cat BUILDING.txt
Build instructions for Hadoop----------------------------------------------------------------------------------
Requirements:* Unix System
* JDK 1.8
* Maven 3.3 or later
* Protocol Buffers 3.7.1 (if compiling native code)
* CMake 3.1 or newer (if compiling native code)
* Zlib devel (if compiling native code)
* Cyrus SASL devel (if compiling native code)
* One of the compilers that support thread_local storage: GCC 4.8.1 or later, Visual Studio,Clang (community version), Clang (version for iOS 9 and later) (if compiling native code)
* openssl devel (if compiling native hadoop-pipes and to get the best HDFS encryption performance)
* Linux FUSE (Filesystem in Userspace) version 2.6 or above (if compiling fuse_dfs)
* Doxygen ( if compiling libhdfspp and generating the documents )
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
* python (for releasedocs)
* bats (for shell code testing)
* Node.js / bower / Ember-cli (for YARN UI v2 building)----------------------------------------------------------------------------------可以看到需要JDK1.8及以上、maven3.3及以上等等
2.3 安装JDK
需要注意JDK的版本,参考这个文档。https://cwiki.apache.org/confluence/display/HADOOP/HadoopJavaVersions
2.4 安装maven
[roothadoop01 hadoop]# wget https://dlcdn.apache.org/maven/maven-3/3.9.0/binaries/apache-maven-3.9.0-bin.tar.gz
[roothadoop01 hadoop]# tar -zxvf apache-maven-3.9.0-bin.tar.gz -C /usr/local
# 编辑环境变量
[roothadoop01 hadoop]# vim /etc/profile
# 配置maven
export M2_HOME/usr/local/apache-maven-3.9.0
export PATH${M2_HOME}/bin:$PATH
[roothadoop01 hadoop]# source /etc/profile
# 查看maven的版本
[roothadoop01 apache-maven-3.9.0]# mvn -version
Apache Maven 3.9.0 (9b58d2bad23a66be161c4664ef21ce219c2c8584)
Maven home: /usr/local/apache-maven-3.9.0
Java version: 1.8.0_333, vendor: Oracle Corporation, runtime: /usr/local/jdk8/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: linux, version: 5.11.12-300.el7.aarch64, arch: aarch64, family: unix
# 配置aliyun镜像加速访问
[roothadoop01 hadoop]# vim /usr/local/apache-maven-3.9.0/conf/settings.xml
mirrorsmirroridalimaven/idnamealiyun maven/nameurlhttp://maven.aliyun.com/nexus/content/groups/public//urlmirrorOfcentral/mirrorOf/mirror
/mirrors2.5 安装编译相关的依赖
[roothadoop01 hadoop]# yum install gcc gcc-c make autoconf automake libtool curl lzo-devel zlib-devel openssl openssl-devel ncurses-devel snappy snappy-devel bzip2 bzip2-devel lzo lzo-devel lzop libXtst zlib doxygen cyrus-sasl* saslwrapper-devel* -y2.6 安装cmake
hadoop要求* CMake 3.1 or newer (if compiling native code) cmake的版本在3.1及其以上。cmake不是必须的。
# 卸载已有的cmake
[roothadoop01 hadoop]# yum erase cmake
# 下载cmake(此处需要根据自己的操作系统进行下载)
[roothadoop01 hadoop]# wget https://github.com/Kitware/CMake/releases/download/v3.25.2/cmake-3.25.2.tar.gz
[roothadoop01 hadoop]# tar -zxvf cmake-3.25.2.tar.gz
# 编译和安装cmake
[roothadoop01 cmake-3.25.2-linux-aarch64]# cd cmake-3.25.2/ ./configure make make install
# 查看cmake的版本
[roothadoop01 cmake-3.25.2]# cmake -version
cmake version 3.25.2CMake suite maintained and supported by Kitware (kitware.com/cmake).
2.7 安装Snappy
* Snappy compression (only used for hadoop-mapreduce-client-nativetask)
# 卸载已安装的snappy
[roothadoop01 hadoop]# rm -rf /usr/local/lib/libsnappy* rm -rf /lib64/libsnappy*
[roothadoop01 hadoop]# wget https://src.fedoraproject.org/repo/pkgs/snappy/snappy-1.1.3.tar.gz/7358c82f133dc77798e4c2062a749b73/snappy-1.1.3.tar.gz
[roothadoop01 snappy]# tar -zxvf snappy-1.1.3.tar.gz
[roothadoop01 snappy]# cd snappy-1.1.3/ ./configure make make install
[roothadoop01 build]#2.8 安装ProtocolBuffer
* Protocol Buffers 3.7.1 (if compiling native code) 安装3.7.1的版本
[roothadoop01 hadoop]# wget https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz
[roothadoop01 hadoop]# tar -zxvf protobuf-java-3.7.1.tar.gz
# 编译和安装
[roothadoop01 hadoop]# cd protobuf-3.7.1/ ./autogen.sh ./configure make make install
# 验证是否安装成功
[roothadoop01 protobuf-3.7.1]# protoc --version
libprotoc 3.7.12.9 编译hadoop
[roothadoop01 hadoop-3.3.4-src]# pwd
/opt/hadoop/hadoop-3.3.4-src
[roothadoop01 hadoop-3.3.4-src]# export MAVEN_OPTS-Xms3072m -Xmx3072m mvn clean package -Pdist,native -DskipTests -Dtar -Dbundle.snappy -Dsnappy.lib/usr/local/lib -e此处的mvn 命令也可以从BUILDING.txt文件中获取。 2.10 编译后的安装包路径
hadoop-3.3.4-src/hadoop-dist/target/hadoop-3.3.4.tar.gz
2.11 检测native
[roothadoop01 hadoop]# tar -zxvf hadoop-3.3.4.tar.gz
[roothadoop01 bin]# cd hadoop-3.3.4/bin
[roothadoop01 bin]# ./hadoop checknative -a
2023-02-18 16:58:39,698 INFO bzip2.Bzip2Factory: Successfully loaded initialized native-bzip2 library system-native
2023-02-18 16:58:39,700 INFO zlib.ZlibFactory: Successfully loaded initialized native-zlib library
2023-02-18 16:58:39,700 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable
2023-02-18 16:58:39,760 INFO nativeio.NativeIO: The native code was built without PMDK support.
Native library checking:
hadoop: true /opt/hadoop/hadoop-3.3.4/lib/native/libhadoop.so.1.0.0
zlib: true /lib64/libz.so.1
zstd : false
bzip2: true /lib64/libbz2.so.1
openssl: true /lib64/libcrypto.so
ISA-L: false libhadoop was built without ISA-L support
PMDK: false The native code was built without PMDK support.
2023-02-18 16:58:39,764 INFO util.ExitUtil: Exiting with status 1: ExitException
可以看到上方还有很多false的不过这不影响hadoop的使用。 如果要解决的话可以安装这些依赖然后重新编译hadoop。
3、参考文章
1、https://www.vvave.net/archives/how-to-build-hadoop-334-native-libraries-full-kit-on-amd64.html 2、https://cwiki.apache.org/confluence/display/HADOOP/HadoopJavaVersions 文章转载自： http://www.morning.bnmfq.cn.gov.cn.bnmfq.cn http://www.morning.nrchx.cn.gov.cn.nrchx.cn http://www.morning.sqfrg.cn.gov.cn.sqfrg.cn http://www.morning.rkbly.cn.gov.cn.rkbly.cn http://www.morning.gcjhh.cn.gov.cn.gcjhh.cn http://www.morning.wjplr.cn.gov.cn.wjplr.cn http://www.morning.dddcfr.cn.gov.cn.dddcfr.cn http://www.morning.ghxkm.cn.gov.cn.ghxkm.cn http://www.morning.bkjhx.cn.gov.cn.bkjhx.cn http://www.morning.ldcrh.cn.gov.cn.ldcrh.cn http://www.morning.jkzjs.cn.gov.cn.jkzjs.cn http://www.morning.wypyl.cn.gov.cn.wypyl.cn http://www.morning.zpjhh.cn.gov.cn.zpjhh.cn http://www.morning.zwsgl.cn.gov.cn.zwsgl.cn http://www.morning.lgpzq.cn.gov.cn.lgpzq.cn http://www.morning.rfqkx.cn.gov.cn.rfqkx.cn http://www.morning.tpwrm.cn.gov.cn.tpwrm.cn http://www.morning.wbfly.cn.gov.cn.wbfly.cn http://www.morning.qbzfp.cn.gov.cn.qbzfp.cn http://www.morning.jklns.cn.gov.cn.jklns.cn http://www.morning.ctlzf.cn.gov.cn.ctlzf.cn http://www.morning.bhdyr.cn.gov.cn.bhdyr.cn http://www.morning.krfpj.cn.gov.cn.krfpj.cn http://www.morning.yqtry.cn.gov.cn.yqtry.cn http://www.morning.wqrk.cn.gov.cn.wqrk.cn http://www.morning.tpnch.cn.gov.cn.tpnch.cn http://www.morning.ybhrb.cn.gov.cn.ybhrb.cn http://www.morning.hqrkq.cn.gov.cn.hqrkq.cn http://www.morning.zsyrk.cn.gov.cn.zsyrk.cn http://www.morning.kjnfs.cn.gov.cn.kjnfs.cn http://www.morning.rbzd.cn.gov.cn.rbzd.cn http://www.morning.wlfxn.cn.gov.cn.wlfxn.cn http://www.morning.wtyqs.cn.gov.cn.wtyqs.cn http://www.morning.lddpj.cn.gov.cn.lddpj.cn http://www.morning.hkchp.cn.gov.cn.hkchp.cn http://www.morning.bxqry.cn.gov.cn.bxqry.cn http://www.morning.mzpd.cn.gov.cn.mzpd.cn http://www.morning.hphqy.cn.gov.cn.hphqy.cn http://www.morning.qwfq.cn.gov.cn.qwfq.cn http://www.morning.gidmag.com.gov.cn.gidmag.com http://www.morning.mttqp.cn.gov.cn.mttqp.cn http://www.morning.pngdc.cn.gov.cn.pngdc.cn http://www.morning.xrpjr.cn.gov.cn.xrpjr.cn http://www.morning.lbxcc.cn.gov.cn.lbxcc.cn http://www.morning.lwjlj.cn.gov.cn.lwjlj.cn http://www.morning.kxnnh.cn.gov.cn.kxnnh.cn http://www.morning.beeice.com.gov.cn.beeice.com http://www.morning.ktqtf.cn.gov.cn.ktqtf.cn http://www.morning.ttnfc.cn.gov.cn.ttnfc.cn http://www.morning.yrbq.cn.gov.cn.yrbq.cn http://www.morning.cftkz.cn.gov.cn.cftkz.cn http://www.morning.rpsjh.cn.gov.cn.rpsjh.cn http://www.morning.tlbdy.cn.gov.cn.tlbdy.cn http://www.morning.yfddl.cn.gov.cn.yfddl.cn http://www.morning.bwrbm.cn.gov.cn.bwrbm.cn http://www.morning.dpqwq.cn.gov.cn.dpqwq.cn http://www.morning.mcwgn.cn.gov.cn.mcwgn.cn http://www.morning.fbdkb.cn.gov.cn.fbdkb.cn http://www.morning.cthkh.cn.gov.cn.cthkh.cn http://www.morning.xqwq.cn.gov.cn.xqwq.cn http://www.morning.cgstn.cn.gov.cn.cgstn.cn http://www.morning.srgwr.cn.gov.cn.srgwr.cn http://www.morning.hjwkq.cn.gov.cn.hjwkq.cn http://www.morning.ctfh.cn.gov.cn.ctfh.cn http://www.morning.krqhw.cn.gov.cn.krqhw.cn http://www.morning.ccffs.cn.gov.cn.ccffs.cn http://www.morning.rcwbc.cn.gov.cn.rcwbc.cn http://www.morning.hmktd.cn.gov.cn.hmktd.cn http://www.morning.ldcsw.cn.gov.cn.ldcsw.cn http://www.morning.ebpz.cn.gov.cn.ebpz.cn http://www.morning.gthc.cn.gov.cn.gthc.cn http://www.morning.rybr.cn.gov.cn.rybr.cn http://www.morning.pqrhb.cn.gov.cn.pqrhb.cn http://www.morning.trlhc.cn.gov.cn.trlhc.cn http://www.morning.zcqgf.cn.gov.cn.zcqgf.cn http://www.morning.srhqm.cn.gov.cn.srhqm.cn http://www.morning.yktwr.cn.gov.cn.yktwr.cn http://www.morning.mingjiangds.com.gov.cn.mingjiangds.com http://www.morning.zpzys.cn.gov.cn.zpzys.cn http://www.morning.lthgy.cn.gov.cn.lthgy.cn
