环境VMware虚拟机 Rocky Linux10.2内存2.5G远程工具MobaXterm Personal 23.0全部操作使用root用户 注意每一步均为MobaXterm SSH终端执行vim编辑按i进入编辑改完按Esc输入:wq回车保存退出。前置准备VMware打开Rocky Linux10.2虚拟机开机。MobaXterm新建SSH会话Session → SSHRemote host填虚拟机IPusername填root输入密码登录。在MobaXterm终端输入查看本机IP记下IP后面配置hosts要用ip a找到ens160的inet地址示例192.168.137.129步骤1安装 OpenJDK21dnf install -y java-21-openjdk java-21-openjdk-devel验证安装java -version看到openjdk 21版本即成功。查看java真实安装路径update-alternatives --list java路径/usr/lib/jvm/java-21-openjdk配置系统环境变量vim /etc/profile文件最末尾添加export JAVA_HOME/usr/lib/jvm/java-21-openjdk export PATH$PATH:$JAVA_HOME/bin:wq保存。 生效环境变量source /etc/profile校验JAVA_HOMEecho $JAVA_HOME步骤2关闭防火墙与SELinuxsystemctl stop firewalld systemctl disable firewalld setenforce 0步骤3设置主机名、hosts映射设置主机名为masterhostnamectl set-hostname master修改hosts文件把下面IP替换成你ip a查到的虚拟机IPvim /etc/hosts文件末尾增加一行192.168.137.129 master:wq保存。步骤4配置SSH免密登录Hadoop启动依赖ssh-keygen -t rsa一路回车不要输入任何密码。cat ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys chmod 600 ~/.ssh/authorized_keys测试免密登录ssh master不需要输入密码即为成功输入exit退回原终端。步骤5下载、解压Hadoop‑3.3.5进入/opt目录cd /opt下载安装包清华镜像速度快wget https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-3.3.5/hadoop-3.3.5.tar.gz解压并重命名tar -zxvf hadoop-3.3.5.tar.gz mv hadoop-3.3.5 hadoop配置Hadoop环境变量vim /etc/profile文件末尾追加export HADOOP_HOME/opt/hadoop export PATH$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:wq保存生效source /etc/profile验证hadoop版本hadoop version步骤6修改Hadoop5个核心配置文件配置文件目录/opt/hadoop/etc/hadoop① hadoop‑env.shvim /opt/hadoop/etc/hadoop/hadoop-env.sh文件末尾粘贴export JAVA_HOME/usr/lib/jvm/java-21-openjdk export HDFS_NAMENODE_USERroot export HDFS_DATANODE_USERroot export HDFS_SECONDARYNAMENODE_USERroot export YARN_RESOURCEMANAGER_USERroot export YARN_NODEMANAGER_USERroot② core‑site.xmlvim /opt/hadoop/etc/hadoop/core-site.xmlconfiguration property namefs.defaultFS/name valuehdfs://master:9000/value /property property namehadoop.tmp.dir/name value/opt/hadoop/tmp/value /property /configuration③ hdfs‑site.xmlvim /opt/hadoop/etc/hadoop/hdfs-site.xmlconfiguration property namedfs.replication/name value1/value /property /configuration④ mapred‑site.xmlcp /opt/hadoop/etc/hadoop/mapred-site.xml.template /opt/hadoop/etc/hadoop/mapred-site.xmlvim /opt/hadoop/etc/hadoop/mapred-site.xmlconfiguration property namemapreduce.framework.name/name valueyarn/value /property /configuration⑤ yarn‑site.xml适配2.5G虚拟机内存vim /opt/hadoop/etc/hadoop/yarn-site.xmlconfiguration property nameyarn.nodemanager.aux-services/name valuemapreduce_shuffle/value /property property nameyarn.nodemanager.resource.memory-mb/name value1536/value /property property nameyarn.scheduler.maximum-allocation-mb/name value1280/value /property property nameyarn.scheduler.minimum-allocation-mb/name value256/value /property /configuration步骤7格式化HDFS 只执行一次禁止多次运行hdfs namenode -format看到successfully formatted表示格式化成功。步骤8启动Hadoop集群start-dfs.sh start-yarn.sh查看进程验证集群启动jps需要看到NameNode、DataNode、SecondaryNameNode、ResourceManager、NodeManager网页访问Windows浏览器 HDFS页面http://虚拟机IP:9870YARN页面http://虚拟机IP:8088如果进程缺失查看Hadoop日志命令tail -200 /opt/hadoop/logs/hadoop-root-namenode-master.log tail -200 /opt/hadoop/logs/hadoop-root-datanode-master.log步骤9安装Spark‑3.5.0支持JDK21cd /opt wget https://mirrors.tuna.tsinghua.edu.cn/apache/spark/spark-3.5.0/spark-3.5.0-bin-hadoop3.tgz tar -zxvf spark-3.5.0-bin-hadoop3.tgz mv spark-3.5.0-bin-hadoop3 spark配置Spark环境变量vim /etc/profile末尾增加export SPARK_HOME/opt/spark export PATH$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin:wq生效source /etc/profile修改spark配置文件cd /opt/spark/conf cp spark-env.sh.template spark-env.sh vim spark-env.sh末尾粘贴export JAVA_HOME/usr/lib/jvm/java-21-openjdk export HADOOP_CONF_DIR/opt/hadoop/etc/hadoop export YARN_CONF_DIR/opt/hadoop/etc/hadoop写入主机名cp workers.template workers ​​​​​​​vim workersmaster环境搭建完成校验jps查看Hadoop五个进程正常运行。输入命令检验spark版本spark-submit --version能输出版本号即为环境搭建全部完成。环境启停常用命令启动集群start-dfs.sh start-yarn.sh关闭集群stop-yarn.sh stop-dfs.sh查看进程jpsHadoop查看日志排查故障tail -200 /opt/hadoop/logs/hadoop-root-namenode-master.log tail -200 /opt/hadoop/logs/hadoop-root-datanode-master.log重要注意事项hdfs namenode -format只执行一次多次格式化会造成NameNode与DataNode的clusterID不一致DataNode无法启动。对于虚拟机内存比较小的yarn‑site.xml内存参数不要调大。hosts里面IP必须和ip a查到的ens33IP保持一致主机名固定为master。防火墙必须关闭否则Windows浏览器打不开9870、8088网页。