BigData Testing Environment based on Apache Ambari 2.7.6 Source Code, build & install on CentOS7 and record details and errors as follow.

Ambari是 Apache软件基金会的一个顶级项目，截止2021年12月份最新的发布版本是 2.7.6。Ambari工具的主要作用是用于大数据环境下，创建、管理、监视 Hadoop 集群。这里的 Hadoop 指广义上Hadoop 整个生态圈(例如 Hive，Hbase，Sqoop，Zookeeper 等)，而并不仅是特指 Hadoop工具本身。简而言之 Ambari 就是为了让 Hadoop 以及相关的大数据工具安装与管理更加容易而使用的一个工具。

大数据测试环境建立，使用Apache Ambari 2.7.6版本编译并安装，记录操作过程与问题如下。通过PVE的LCT容器模拟集群，建立CentOS7系统，资源分配为1G Memory, 100G Harddisk - Master - Slave1 - Slave2

Step1: Install prerequirements on Master system

安装以下工具，，国内使用可更新为国内源，提高下载速度。

wget
curl
npm
rpmbuild (rpm-build package)
g++ (gcc-c++ package)
git
python-devel

$ yum install -y wget, curl
$ yum install -y rpm-build
$ yum install -y gcc-c++
$ yum install -y git python-devel rpm-build gcc-c++
$ yum install -y npm

Step2: Update Maven and edit Maven config with two mirrors for repository

Maven升级， Centos7默认安装为3.0版本，Ambari2.7编译需要3.3.9版本以下，升级Maven

$ yum install maven # 安装默认版本
$ yum erase maven # 删除已安装版本

下载Maven，修改系统设置

$ wget http://mirrors.hust.edu.cn/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
$ tar -xvf  apache-maven-3.3.9-bin.tar.gz
$ mv -f apache-maven-3.3.9 /usr/local/

# 编辑/etc/profile文件，在文件末尾添加如下代码：

$ vi /etc/profile
$ export MAVEN_HOME=/usr/local/apache-maven-3.3.9 # maven 目录
$ export PATH=${PATH}:${MAVEN_HOME}/bin
$ source /etc/profile # 设置生效
$ mvn -v

将Maven中config设置修改为以下，提升各类包下载速度；镜像1中心源，镜像2为中心以外的其它Package下载；正常科学上网的环境下，整个安装过程中全部第三方Package均可正常下载。

<settings>
  <mirrors>
    <mirror>
      <id>huaweicloud</id>
      <mirrorOf>central</mirrorOf>
      <url>https://mirrors.huaweicloud.com/repository/maven/</url>
    </mirror>

    <mirror>
      <id>nexus-hortonworks</id>
      <mirrorOf>*,!central</mirrorOf>
      <name>Nexus hortonworks</name>
      <url>https://repo.hortonworks.com/content/groups/public/</url>
    </mirror>
  </mirrors>
</settings>

Step3: Download Ambari-2.7.6

下载Ambari-2.7.6版本，准备编译工作

$ wget https://www-eu.apache.org/dist/ambari/ambari-2.7.6/apache-ambari-2.7.6-src.tar.gz 
$ tar xfvz apache-ambari-2.7.6-src.tar.gz 
$ cd apache-ambari-2.7.6-src
$ mvn versions:set -DnewVersion=2.7.6.0.0
$ pushd ambari-metrics
$ mvn versions:set -DnewVersion=2.7.6.0.0
$ popd

Step4: Building Source on RHEL (CentOS 7) & SUSE (SLES 12 SP2 & SP3):

通过以下几条命令进行编译，注意不同参数的使用

clean # 重新编译过程，可不加此参数，提升速度
-Drat.skip=true # 跳过Licence检测
-X # 显示编译过程上的详细信息
-rf :ambari-metrics-timelineservice # 指定从ambari-metrics-timelineservice开始编译，之前编译完成的保留

$ mvn -B clean install rpm:rpm -DnewVersion=2.7.6.0.0 -DbuildNumber=388e072381e71c7755673b7743531c03a4d61be8 -DskipTests -Dpython.ver="python >= 2.6"
或
$ mvn -B clean install rpm:rpm -DnewVersion=2.7.6.0.0 -DbuildNumber=388e072381e71c7755673b7743531c03a4d61be8 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true -X
或
$ mvn -B install rpm:rpm -DnewVersion=2.7.6.0.0 -DbuildNumber=388e072381e71c7755673b7743531c03a4d61be8 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true -X

# 编译中断后，再次编译
$ mvn -B install rpm:rpm -DnewVersion=2.7.6.0.0 -DbuildNumber=388e072381e71c7755673b7743531c03a4d61be8 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true -rf :ambari-metrics-timelineservice
或
$ mvn -B install rpm:rpm -DnewVersion=2.7.6.0.0 -DbuildNumber=388e072381e71c7755673b7743531c03a4d61be8 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true -rf :ambari-metrics-grafana

# 如Maven的mirros做了切换，每次切换镜像要先执行以下，清除文件； 另外，每次编译，先删除以下文件，再重新编译   
$ find ~/.m2/repository/ -name "*.lastUpdated" -exec rm -rf {} \; # 清理lastUpdated文件
$ find ~/.m2/repository/ -name "*.lock" -exec rm -rf {} \;        # # 清理lock文件

Building on Debian，以下是Debian系统中的编译命令，仅文件格式不同；

$ mvn -B install jdeb:jdeb -DnewVersion=2.7.6.0.0 -DbuildNumber=388e072381e71c7755673b7743531c03a4d61be8 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true -X

Step5: Fix Error and Rebuild

编译过程中会出现以下问题，处理后重新编译

可用以下命令，检查依赖关系：

$ mvn dependency:resolve

Error:

Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.4:yarn (yarn install) on project am          bari-admin: Failed to run task: 'yarn install --ignore-engines --pure-lockfile' failed. org.apache.commons.exe          c.ExecuteException: Process exited with an error: 1

Maven源问题，按Step2内容处理

Error:

Failed to execute goal org.apache.rat:apache-rat-plugin:0.12:check (default) on project ambari: Too many files with unapproved license:

Licence Checking问题，编译中增加 -Drat.skip=true 参数，跳过licensing 检查

Error：

Download Ambari Grafana:
      [get] Getting: https://dl.grafana.com/oss/release/grafana-6.7.4.linux-amd64.tar.gz
      [get] To: /home/temp/apache-ambari-2.7.6-src/ambari-metrics/ambari-metrics-grafana/target/grafana/grafana.tgz
      [get] Not modified - so not downloaded
    [untar] Expanding: /home/temp/apache-ambari-2.7.6-src/ambari-metrics/ambari-metrics-grafana/target/grafana/grafana.tgz into /home/temp/apache-ambari-2.7.6-src/ambari-metrics/ambari-metrics-grafana/target/grafana

网络问题，可手机下载对应的工具包，保存在对应的位置；可调整网络，重新编译

Error:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (default) on project ambari-metrics-grafana: An Ant BuildException has occured: Error while expanding /home/temp/apache-ambari-2.7.6-src/ambari-metrics/ambari-metrics-grafana/target/grafana/grafana.tgz
[ERROR] java.io.EOFException: Unexpected end of ZLIB input stream
[ERROR] around Ant part ...<untar src="/home/temp/apache-ambari-2.7.6-src/ambari-metrics/ambari-metrics-grafana/target/grafana/grafana.tgz" dest="/home/temp/apache-ambari-2.7.6-src/ambari-metrics/ambari-metrics-grafana/target/grafana" compression="gzip"/>... @ 6:231 in /home/temp/apache-ambari-2.7.6-src/ambari-metrics/ambari-metrics-grafana/target/antrun/build-Download Ambari Grafana.xml
[ERROR] -> [Help 1]

网络问题，重新编译

Step6: Building Success and Display as follows

编译成功后，显示如下

[INFO] Ambari Metrics Grafana 2.7.6.0.0 ................... SUCCESS [ 13.037 s]
[INFO] Ambari Metrics Host Aggregator 2.7.6.0.0 ........... SUCCESS [ 21.074 s]
[INFO] Ambari Metrics Assembly 2.7.6.0.0 .................. SUCCESS [02:50 min]
[INFO] Ambari Service Advisor 1.0.0.0-SNAPSHOT ............ SUCCESS [  1.452 s]
[INFO] Ambari Server 2.7.6.0.0 ............................ SUCCESS [07:34 min]
[INFO] Ambari Functional Tests 2.7.6.0.0 .................. SUCCESS [  6.812 s]
[INFO] Ambari Agent 2.7.6.0.0 ............................. SUCCESS [02:00 min]
[INFO] ambari-logsearch 2.7.6.0.0 ......................... SUCCESS [  0.962 s]
[INFO] Ambari Logsearch Appender 2.7.6.0.0 ................ SUCCESS [  3.850 s]
[INFO] Ambari Logsearch Config Api 2.7.6.0.0 .............. SUCCESS [  0.667 s]
[INFO] Ambari Logsearch Config JSON 2.7.6.0.0 ............. SUCCESS [  0.521 s]
[INFO] Ambari Logsearch Config Solr 2.7.6.0.0 ............. SUCCESS [ 10.829 s]
[INFO] Ambari Logsearch Config Zookeeper 2.7.6.0.0 ........ SUCCESS [  1.431 s]
[INFO] Ambari Logsearch Config Local 2.7.6.0.0 ............ SUCCESS [  0.498 s]
[INFO] Ambari Logsearch Log Feeder Plugin Api 2.7.6.0.0 ... SUCCESS [  4.807 s]
[INFO] Ambari Logsearch Log Feeder Container Registry 2.7.6.0.0 SUCCESS [  6.833 s]
[INFO] Ambari Logsearch Log Feeder 2.7.6.0.0 .............. SUCCESS [ 46.136 s]
[INFO] Ambari Logsearch Web 2.7.6.0.0 ..................... SUCCESS [03:52 min]
[INFO] Ambari Logsearch Server 2.7.6.0.0 .................. SUCCESS [02:49 min]
[INFO] Ambari Logsearch Assembly 2.7.6.0.0 ................ SUCCESS [  1.554 s]
[INFO] Ambari Logsearch Integration Test 2.7.6.0.0 ........ SUCCESS [01:42 min]
[INFO] ambari-infra 2.7.6.0.0 ............................. SUCCESS [  0.493 s]
[INFO] Ambari Infra Solr Client 2.7.6.0.0 ................. SUCCESS [ 10.080 s]
[INFO] Ambari Infra Solr Plugin 2.7.6.0.0 ................. SUCCESS [ 10.115 s]
[INFO] Ambari Infra Manager 2.7.6.0.0 ..................... SUCCESS [01:15 min]
[INFO] Ambari Infra Assembly 2.7.6.0.0 .................... SUCCESS [  1.000 s]
[INFO] Ambari Infra Manager Integration Tests 2.7.6.0.0 ... SUCCESS [ 41.036 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  25:10 min
[INFO] Finished at: 2021-12-09T17:08:51Z
[INFO] ------------------------------------------------------------------------

Step7: Install Ambari rpm package just compiled.

编译成功，rpm文件保存以下位置：

$ # 服务端文件
$ apache-ambari-2.7.6-src/ambari-server/target/rpm/ambari-server/RPMS/x86_64/ambari-server-2.7.6.0-0.x86_64.rpm
$ # 客户端文件
$ apache-ambari-2.7.6-src/ambari-agent/target/rpm/ambari-agent/RPMS/x86_64/ambari-agent-2.7.6.0-0.x86_64.rpm

进入服务端文件目录，安装rpm文件

$ yun install -y ambari-server-2.7.6.0-0.x86_64.rpm
$ # 安装完成后，查看版本信息
$ ambari-server --version
$ 2.7.6.0-0

安装Mysql5.7版本，用于保存ambari设置信息

添加供ambari使用的用户名、密码、数据库名等信息，开通对应权限； Mysql默认密码设置为1或2，ambari不支持复杂密码，可通过以下命令修改Mysql密码要求

alter user 'root'@'localhost' identified by 'new password';
set global validate_password_policy=0; # 复杂度，0为8位数字
set global validate_password_length=4; # 默认长度，可设置为4位

安装ambari库文件，建立各类表

$ mysql>source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

设置ambari-server，大部分均可选择默认项，java可按提示安装oracle版本，数据库部分选Mysql即可；

$ ambari-server setup

==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3
Hostname (localhost): 
Port (3306):
Database name (ambari):
Username (ambari):
Enter Database Password (bigdata):bigdata911 # 与实际Mysql中用户名，密码一致
Configuring ambari database...
Enter full path to custom jdbc driver:

启动服务端，如启动失败，可通过Log日志查看具体问题，通常为数据库问题

$ ambari-server start
$ cat /var/log/ambari-server/ambari-server.log
$ ambari-server setup --jdbc-db=mysql--jdbc-driver=/usr/share/java/mysql-connector-java-5.1.47.jar # 设置jdbc的jar包位置

Error:

WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: 
/var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

没建表，按指定文件建表

$ mysql>source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

Step8: Finished Ambari Server Setup and Login

安装完成，通过Brower访问，默认登录用户名与密码为：admin/admin

http:ipaddress:8080

整个安装过程并不复杂，主要问题集中在以下两点，根据提示耐心处理，均可解决 - 网络问题，稳定的科学上网是基本条件；编译失败后再多试几次基本都可解决 - 依赖包问题，按第一步中设置；同时设置maven的Mirror

自在工坊

走在代码边缘

Apache Ambari 2.7.6 源码编译安装记录

Step1: Install prerequirements on Master system

Step2: Update Maven and edit Maven config with two mirrors for repository

Step3: Download Ambari-2.7.6

Step4: Building Source on RHEL (CentOS 7) & SUSE (SLES 12 SP2 & SP3):

Step5: Fix Error and Rebuild

Error:

Error:

Error:

Step6: Building Success and Display as follows

Step7: Install Ambari rpm package just compiled.

Error:

Step8: Finished Ambari Server Setup and Login

Links:

Comments