Hortonworks 2.4 설치 - 1/2 - Ambari서버 설치까지

설치방법 4가지
1. Automated (with Ambari 2.2)
2. Manual (RPMs)
3. Cloud (with Cloudbreak 1.2)
4. Windows


Ambari를 활용한 Automated방식이 뭔가 가장 쉬워보인다... 이걸로 시작
참조: http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_Installing_HDP_AMB/content/ch_Getting_Ready.html


OS 는 CentOS 7으로 선택 다운로드시 CentOS-7-x86_64-Everything-1611.iso 같이 everything이 들어있는 미디어로 받았다.
참조: https://www.centos.org/download/
설치는 일단 최소 설치로 하였다.

Software Requirements 가 있다.
yum and rpm (RHEL/CentOS/Oracle Linux)
scp, curl, unzip, tar, and wget
OpenSSL (v1.01, build 16 or later)
Python (For CentOS 7, Ubuntu 12, Ubuntu 14, and Debian 7: Python 2.7.*)

위 항목 설치를 위해 yum repository구축하였다.
https://drunkendwcraft.blogspot.kr/2017/02/ambari-local-repository.html

내부 yum repository를 가르키도록 설정
. etc에 내부 yum repository를 참조하도록 repo설정 파일을 해당 경로로 지정한다.

# vi /etc/yum.repos.d /birepos.repo [birepos] name=birepos Internal Repository baseurl=http://10.203.9.208/repos/centos/iso/ enabled=1 gpgcheck=0

. yum repolist 목록에 뜨면 성공!

# yum repolist ... birepos                        birepos Internal Repository    9,363 ... # mkdir -p /etc/yum.repos.d/ ~/etc/yum.repos.d/ [root@bidevkylinm1 yum.repos.d]# mv C*  ~/etc/yum.repos.d/ [root@bidevkylinm1 yum.repos.d]# yum repolist Loaded plugins: fastestmirror Determining fastest mirrors repo id                                                                                                                   repo name                                                                                                                                      status birepos                                                                                                                   birepos Internal Repository                                                                                                                    9,363 repolist: 9,363

. repolist를 한개만 남겨두고 지워버리자

# mkdir -p /etc/yum.repos.d/ ~/etc/yum.repos.d/ # mv C* ~/etc/yum.repos.d/ # yum repolist Loaded plugins: fastestmirror Determining fastest mirrors repo id            repo name          status birepos            birepos              Internal Repository       9,363 repolist: 9,363

참조: https://www.lesstif.com/pages/viewpage.action?pageId=7634963

. 필수 software 설치

# yum -y install unzip # yum -y install wget # yum -y install openssl* # yum -y install bind-utils

. jdk 1.8과 jce설치

# mkdir -p /usr/java # cd /usr/java # tar zxf ~/jdk-8u121-linux-x64.tar.gz # ln -s /usr/java/jdk1.8.0_121/ /usr/java/default # vi /etc/bashrc export JAVA_HOME=/usr/java/default export PATH=$JAVA_HOME/bin:$PATH # source /etc/bashrc # java -version java version "1.8.0_121" Java(TM) SE Runtime Environment (build 1.8.0_121-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode) # cd ~ # unzip -o -j -q jce_policy-8.zip -d /usr/java/jdk1.8.0_121/jre/lib/security/

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.3/bk_installing_manually_book/content/meet-min-system-requirements.html#oracle-jdk-17-getting-started
http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

. 노드 간의 passwordless-ssh 통신 구축
step1. Ambari Server host(m2 노드)에서 SSH key 생성

# ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Created directory '/root/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: 35:89:26:c7:0d:6e:1a:86:bd:d3:81:dc:f6:5b:d2:07 root@bidevkylinm1.bikylinclust The key's randomart image is: +--[ RSA 2048]----+ |        .        | |     + = + .     | |    . B @ = E    | |     . @ + o .   | |      + S o o .  | |       .   + .   | |          .      | |                 | |                 | +-----------------+

step2. 타겟 호스트(m1~3, d1~d3)의 ~/.ssh폴더에 id_rsa.pub파일을 복사한 후 파일을 이름을 "authorized_keys"로 바꿔 복사하는 작업이 필요한데 ssh-copy-id 명령으로 가능하다.

# ssh-copy-id -i ~/.ssh/id_rsa.pub root@10.203.9.218 /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@bidevkylind1's password: Number of key(s) added: 1 Now try logging into the machine, with:   "ssh 'root@bidevkylind1'" and check to make sure that only the key(s) you wanted were added.

step3. Ambari Server에서 ssh를 사용해서 클러스터 호스트에 암호입력없이 접속 가능 확인

[root@bidevkylinm2 ~]# ssh root@10.203.9.218 Last login: Thu Feb  2 00:13:38 2017 from 10.149.186.145 [root@bidevkylind3 ~]#

http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_Installing_HDP_AMB/content/_set_up_password-less_ssh.html

. dns서버설정 및 정방향 역방향 조회 확인
dns서버 구축한 후
링크: https://drunkendwcraft.blogspot.kr/2017/02/centos-7-dns-nscd.html

dns1을 설정하고서 network재시작 후 정방향 역방향 조회가 되면 성공

# vi /etc/sysconfig/network-scripts/ifcfg-eth0 ... DNS1=10.203.9.208 ... # systemctl restart network # nslookup bidevkylinm1 Server:         10.203.9.208 Address:        10.203.9.208#53 Name:   bidevkylinm1.bikylinclust Address: 10.203.9.209 # nslookup 10.203.9.209 Server:         10.203.9.208 Address:        10.203.9.208#53 209.9.203.10.in-addr.arpa       name = bidevkylinm1.bikylinclust.

. nscd 서비스 설치

# yum -y install nscd # systemctl enable nscd Created symlink from /etc/systemd/system/multi-user.target.wants/nscd.service to /usr/lib/systemd/system/nscd.service. Created symlink from /etc/systemd/system/sockets.target.wants/nscd.socket to /usr/lib/systemd/system/nscd.socket. # systemctl start nscd

. ntp 서비스 설치

# yum -y install ntp # systemctl enable ntpd # systemctl start ntpd

  .firewall 서비스 stop

# systemctl stop firewalld # systemctl disable firewalld

  .firewall 서비스를 사용한다면 필요한 허용 포트를 아래와 같이 추가

# firewall-cmd --zone=public --add-port=1521/tcp

 .selinux 끄기

# setenforce 0 # vi /etc/selinux/config SELINUX=disabled

 . umask 0022 먹이기 (기본 생성 파일권한 755) 

# vi /etc/profile umask 0022

. Check the Maximum Open File Descriptors 10000개 이상

# ulimit -Sn 1024 # ulimit -Hn 4096 # ulimit -n 10000 # ulimit -Sn 10000 # ulimit -Hn 10000

. 각종 metastore로 postgresql 9.2를 선택하여 설치
master 1~3번기 중 2번기에 설치하기로 했다.
설치:
9.2 https://drunkendwcraft.blogspot.kr/2017/04/centos-7-postgresql-92.html
9.4 https://drunkendwcraft.blogspot.kr/2017/06/centos-7-postgresql-94.html

. postgress jdbc 설치

# yum -y install postgresql-jdbc* Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile Installed:   postgresql-jdbc.noarch 0:9.2.1002-5.el7                          postgresql-jdbc-javadoc.noarch 0:9.2.1002-5.el7 Dependency Installed:   copy-jdk-configs.noarch 0:1.2-1.el7                              fontconfig.x86_64 0:2.10.95-10.el7   fontpackages-filesystem.noarch 0:1.44-8.el7                      giflib.x86_64 0:4.1.6-9.el7   java-1.8.0-openjdk.x86_64 1:1.8.0.102-4.b14.el7                  java-1.8.0-openjdk-headless.x86_64 1:1.8.0.102-4.b14.el7   javapackages-tools.noarch 0:3.4.1-11.el7                         libICE.x86_64 0:1.0.9-2.el7   libSM.x86_64 0:1.2.2-2.el7                                       libX11.x86_64 0:1.6.3-3.el7   libX11-common.noarch 0:1.6.3-3.el7                               libXau.x86_64 0:1.0.8-2.1.el7   libXcomposite.x86_64 0:0.4.4-4.1.el7                             libXext.x86_64 0:1.3.3-3.el7   libXfont.x86_64 0:1.5.1-2.el7                                    libXi.x86_64 0:1.7.4-2.el7   libXrender.x86_64 0:0.9.8-2.1.el7                                libXtst.x86_64 0:1.2.2-2.1.el7   libfontenc.x86_64 0:1.1.2-3.el7                                  libjpeg-turbo.x86_64 0:1.2.90-5.el7   libpng.x86_64 2:1.5.13-7.el7_2                                   libxcb.x86_64 0:1.11-4.el7   libxslt.x86_64 0:1.1.28-5.el7                                    lksctp-tools.x86_64 0:1.0.17-2.el7   python-javapackages.noarch 0:3.4.1-11.el7                        python-lxml.x86_64 0:3.2.1-4.el7   ttmkfdir.x86_64 0:3.0.9-42.el7                                   tzdata-java.noarch 0:2016g-2.el7   xorg-x11-font-utils.x86_64 1:7.5-20.el7                          xorg-x11-fonts-Type1.noarch 0:7.5-9.el7 Complete! # chmod 644 /usr/share/java/postgresql-jdbc.jar # ambari-server setup --jdbc-db=postgres --jdbc-driver=/usr/share/java/postgresql-jdbc.jar Using python  /usr/bin/python Setup ambari-server Copying /usr/share/java/postgresql-jdbc.jar to /var/lib/ambari-server/resources JDBC driver was successfully initialized. Ambari Server 'setup' completed successfully.

. Ambari용 디비, 사용자 생성 

# sudo -u postgres psql psql (9.2.18) Type "help" for help. postgres=# CREATE DATABASE ambaridb; CREATE DATABASE postgres=# CREATE USER ambariuser WITH PASSWORD '<AMBARIPASSWORD>'; CREATE ROLE postgres=# GRANT ALL PRIVILEGES ON DATABASE ambaridb TO ambariuser; GRANT postgres=# \connect ambaridb; You are now connected to database "ambaridb" as user "postgres". ambaridb=# CREATE SCHEMA ambari AUTHORIZATION ambariuser; CREATE SCHEMA ambaridb=# ALTER SCHEMA ambari OWNER TO ambariuser; ALTER SCHEMA ambaridb=# ALTER ROLE ambariuser SET search_path to ‘ambari’, 'public'; ALTER ROLE

. Hive용 디비, 사용자 생성 

# sudo psql -U postgres Password for user postgres: psql (9.2.18) Type "help" for help. postgres=# CREATE DATABASE hivedb; CREATE DATABASE postgres=# CREATE USER hiveuser WITH PASSWORD '<HIVEPASSWORD>'; CREATE ROLE postgres=# GRANT ALL PRIVILEGES ON DATABASE hivedb TO hiveuser; GRANT postgres=# \connect hivedb; You are now connected to database "hivedb" as user "postgres". hivedb=# CREATE SCHEMA hive AUTHORIZATION hiveuser; CREATE SCHEMA hivedb=# ALTER SCHEMA hive OWNER TO hiveuser; ALTER SCHEMA hivedb=# ALTER ROLE hiveuser SET search_path to hive, 'public'; ALTER ROLE

참조: http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_ambari_reference_guide/content/_using_hive_with_postgresql.html

. Oozie용 디비, 사용자 생성 

# sudo psql -U postgres Password for user postgres: psql (9.2.18) Type "help" for help. postgres=# CREATE DATABASE ooziedb; CREATE DATABASE postgres=# CREATE USER oozieuser WITH PASSWORD '<OOZIEPASSWORD>'; CREATE ROLE postgres=# GRANT ALL PRIVILEGES ON DATABASE ooziedb TO oozieuser; GRANT postgres=# \connect ooziedb; You are now connected to database "ooziedb" as user "postgres". ooziedb=# CREATE SCHEMA oozie AUTHORIZATION oozieuser; CREATE SCHEMA ooziedb=# ALTER SCHEMA oozie OWNER TO oozieuser; ALTER SCHEMA ooziedb=# ALTER ROLE oozieuser SET search_path to oozie, 'public'; ALTER ROLE ooziedb=#

참조: http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_ambari_reference_guide/content/_using_oozie_with_postgresql.html

. 인터넷 연결이 없을때 Ambari Repositories를 구축해야 한다.
참조: https://drunkendwcraft.blogspot.kr/2017/02/ambari-local-repository.html?zx=c4557801edaeb21b

. Ambari Server 설치
다음 파일을 다운로드 받는다
http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.2.2.0/ambari.repo

내용을 다음과 같이 편집

#VERSION_NUMBER=2.2.2.0-460 [Updates-ambari-2.2.2.0] name=ambari-2.2.2.0 - Updates #baseurl=http://<web.server>/ambari-2.2.2.0/<OS> baseurl=http://10.203.9.208/ambari-2.2.2.0/centos7/2.2.2.0-460/ gpgcheck=1 gpgkey=http://10.203.9.208/ambari-2.2.2.0/centos7/2.2.2.0-460/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1

/etc/yum.repos.d/ambari.repo 위치에 복사하여 yum repolist에 뜨는지 살펴본다.

# yum repolist Loaded plugins: fastestmirror Updates-ambari-2.2.2.0                                                                                               | 2.9 kB  00:00:00 Updates-ambari-2.2.2.0/primary_db                                                                                    | 6.3 kB  00:00:00 Loading mirror speeds from cached hostfile repo id                                                         repo name                                                             status Updates-ambari-2.2.2.0                                          ambari-2.2.2.0 - Updates                                                  8 birepos                                                         birepos Internal Repository                                           9,363 repolist: 9,371

설치 시작하면 일단 postgresql도 같이 깔린다.... 무시

# yum install ambari-server Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile Resolving Dependencies --> Running transaction check ---> Package ambari-server.x86_64 0:2.2.2.0-460 will be installed --> Processing Dependency: postgresql-server >= 8.1 for package: ambari-server-2.2.2.0-460.x86_64 --> Running transaction check ---> Package postgresql-server.x86_64 0:9.2.18-1.el7 will be installed --> Processing Dependency: postgresql-libs(x86-64) = 9.2.18-1.el7 for package: postgresql-server-9.2.18-1.el7.x86_64 --> Processing Dependency: postgresql(x86-64) = 9.2.18-1.el7 for package: postgresql-server-9.2.18-1.el7.x86_64 --> Processing Dependency: libpq.so.5()(64bit) for package: postgresql-server-9.2.18-1.el7.x86_64 --> Running transaction check ---> Package postgresql.x86_64 0:9.2.18-1.el7 will be installed ---> Package postgresql-libs.x86_64 0:9.2.18-1.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================  Package                             Arch                     Version                        Repository                                Size ============================================================================================================================================ Installing:  ambari-server                       x86_64                   2.2.2.0-460                    Updates-ambari-2.2.2.0                   409 M Installing for dependencies:  postgresql                          x86_64                   9.2.18-1.el7                   birepos                                  3.0 M  postgresql-libs                     x86_64                   9.2.18-1.el7                   birepos                                  232 k  postgresql-server                   x86_64                   9.2.18-1.el7                   birepos                                  3.8 M Transaction Summary ============================================================================================================================================ Install  1 Package (+3 Dependent packages)

ambaridb db스키마 생성문을 수행한다.

# psql -U ambariuser -d ambaridb Password for user ambariuser: psql (9.2.18) Type "help" for help. ambaridb=> \i /var/lib/ambari-server/resources/Ambari-DDL-Postgres-CREATE.sql

참조: http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_ambari_reference_guide/content/_using_ambari_with_postgresql.html

yum install 이 끝나면 setup을 시작한다.

#  ambari-server setup --java-home=$JAVA_HOME Using python  /usr/bin/python Setup ambari-server Checking SELinux... SELinux status is 'disabled' Customize user account for ambari-server daemon [y/n] (n)? n Adjusting ambari-server permissions and ownership... Checking firewall status... Redirecting to /bin/systemctl status  iptables.service Unit iptables.service could not be found. Checking JDK... WARNING: JAVA_HOME /usr/java/default must be valid on ALL hosts WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts. Completing setup... Configuring database... Enter advanced database configuration [y/n] (n)? y Configuring database... ============================================================================== Choose one of the following options: [1] - PostgreSQL (Embedded) [2] - Oracle [3] - MySQL [4] - PostgreSQL [5] - Microsoft SQL Server (Tech Preview) [6] - SQL Anywhere ============================================================================== Enter choice (1): 4 Hostname (localhost): 127.0.0.1 Port (5432): Database name (ambaridb): ambaridb  Postgres schema (ambari): ambari Username (ambari): ambariuser Enter Database Password (bigdata): Configuring ambari database... Configuring remote database connection properties... WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-Postgres-CREATE.sql Proceed with configuring remote database connection properties [y/n] (y)? y Extracting system views... ambari-admin-2.2.2.0.460.jar ...... Adjusting ambari-server permissions and ownership... Ambari Server 'setup' completed successfully.

. Ambari서버 시작
postgresql을 metastore로 설정했으므로 start파라미터가 붙는다

# ambari-server start --jdbc-db=postgres --jdbc-driver=/path/to/postgres/postgresql.jar Using python  /usr/bin/python Starting ambari-server Ambari Server running with administrator privileges. Organizing resource files at /var/lib/ambari-server/resources... Server PID at: /var/run/ambari-server/ambari-server.pid Server out at: /var/log/ambari-server/ambari-server.out Server log at: /var/log/ambari-server/ambari-server.log Waiting for server start.................... Ambari Server 'start' completed successfully. # ambari-server status Using python  /usr/bin/python Ambari-server status Ambari Server running Found Ambari Server PID: 3661 at: /var/run/ambari-server/ambari-server.pid

브라우저에서 접근해보자 로그인 창이 뜨면 성공!

http://ambarihost:8080/
기본 관리자 계정: admin / admin

CentOS 7 에서 dns서버 구축 및 nscd설치

Hadoop이 dns에 많이 의존하기 때문에 dns서버를 구축해서 관리하거나
서버별로 모든 서버에 대해서 /etc/host설정을 해야 한다...

그래서 dns서버를 구축하고 각 노드들은 nscd를 설치해서 dns호출을 cache하도록 한다.
안그러면 dns서버에 과부하가 갈 수 있다고 한다.

참조: http://egloos.zum.com/dukuduku/v/7085276

. 서비스 설치

# yum -y install bind bind-libs bind-utils bind-chroot --disablerepo=* --enablerepo=local-repo

. named.conf 설정

# vi /etc/named.conf options {         listen-on port 53 { any; };         listen-on-v6 port 53 { ::1; };         directory       "/var/named";         dump-file       "/var/named/data/cache_dump.db";         statistics-file "/var/named/data/named_stats.txt";         memstatistics-file "/var/named/data/named_mem_stats.txt";         allow-query     { any; };         /*          - If you are building an AUTHORITATIVE DNS server, do NOT enable recursion.          - If you are building a RECURSIVE (caching) DNS server, you need to enable            recursion.          - If your recursive DNS server has a public IP address, you MUST enable access            control to limit queries to your legitimate users. Failing to do so will            cause your server to become part of large scale DNS amplification            attacks. Implementing BCP38 within your network would greatly            reduce such attack surface         */         recursion yes;         dnssec-enable yes;         dnssec-validation yes;         /* Path to ISC DLV key */         bindkeys-file "/etc/named.iscdlv.key";         managed-keys-directory "/var/named/dynamic";         pid-file "/run/named/named.pid";         session-keyfile "/run/named/session.key"; }; # systemctl start named # systemctl enable named Created symlink from /etc/systemd/system/multi-user.target.wants/named.service to /usr/lib/systemd/system/named.service.

서비스가 잘 시작되었다.

. dns 도메인 zone 등록

# vi /etc/named.rfc1912.zone 아래 내용 추가 # 정방향 zone "bikylinclust" IN {         type master;         file "bikylinclust.zone";         allow-update { none; }; }; # 역방향 zone "9.203.10.in-addr.arpa" IN {          type master;          file "bikylinclust.rev";          allow-update { none; };  };

. 정방향 zone파일 생성

# vi /var/named/bikylinclust.zone $TTL 3H @       IN      SOA     ns.bikylinclust. root (                                         160509  ; serial                                         1D      ; refresh                                         1H      ; retry                                         1W      ; expire                                         3H )    ; minimum         IN      NS      ns.bikylinclust.         IN      A       10.203.9.208 ns      IN      A       10.203.9.208 www     IN      A       10.203.9.208 bidevkylinm1 IN A       10.203.9.209 bidevkylinm2 IN A       10.203.9.210 bidevkylinm3 IN A       10.203.9.211 bidevkylind1 IN A       10.203.9.216 bidevkylind2 IN A       10.203.9.217 bidevkylind3 IN A       10.203.9.218 # chown root.named /var/named/bikylinclust.zone

. 역방향 rev파일 생성

# vi /var/named/bikylinclust.rev $TTL 3H @       IN      SOA     ns.bikylinclust.       root.bikylinclust.     (                                         0       ; serial                                         1D      ; refresh                                         1H      ; retry                                         1W      ; expire                                         3H )    ; minimum          IN      NS      ns.bikylinclust. 208      IN      PTR     ns.bikylinclust. 208      IN      PTR     www.bikylinclust. 209      IN      PTR     bidevkylinm1.bikylinclust. 210      IN      PTR     bidevkylinm2.bikylinclust. 211      IN      PTR     bidevkylinm3.bikylinclust. 216      IN      PTR     bidevkylind1.bikylinclust. 217      IN      PTR     bidevkylind2.bikylinclust. 218      IN      PTR     bidevkylind3.bikylinclust. # chown root.named /var/named/bikylinclust.rev

. 설정 정합성 체크 및 서비스 재시작

# named-checkconf -z /etc/named.conf zone localhost.localdomain/IN: loaded serial 0 zone localhost/IN: loaded serial 0 zone 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa/IN: loaded serial 0 zone 1.0.0.127.in-addr.arpa/IN: loaded serial 0 zone 0.in-addr.arpa/IN: loaded serial 0 zone bikylinclust/IN: loaded serial 160509 zone 10.203.9.in-addr.arpa/IN: loaded serial 0 # named-checkzone ns.bikylinclust /var/named/bikylinclust.zone zone ns.bikylinclust/IN: loaded serial 160509 OK # named-checkzone www.bikylinclust /var/named/bikylinclust.zone zone www.bikylinclust/IN: loaded serial 160509 OK # named-checkzone bidevkylind1.bikylinclust /var/named/bikylinclust.zone zone bidevkylind1.bikylinclust/IN: loaded serial 160509 OK # systemctl restart named

. nslookup 정방향 역방향 조회

# nslookup bidevkylinm1 Server:         10.203.9.208 Address:        10.203.9.208#53 Name:   bidevkylinm1.bikylinclust Address: 10.203.9.209 # nslookup 10.203.9.209 Server:         10.203.9.208 Address:        10.203.9.208#53 209.9.203.10.in-addr.arpa       name = bidevkylinm1.bikylinclust.

CentOS 7 에 MySQL 5.6 설치

Bigdata환경에서 Hive, HCatalog, Oozie, Ambari 등의 콤포넌트들의 모두 metastore공간으로 db를 요구한다. 그때마다 모두 dbms를 따로 설치할 수는 없는 노릇이라 1개의 dbms를 공유하기로 했다.

mysql 5.6을 선택

참조: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.3/bk_installing_manually_book/content/meet-min-system-requirements.html

https://dev.mysql.com/downloads/mysql/5.6.html#downloads
여기서 다운로드하기로...

설치 참조는 아래 링크로 진행해본다.
https://dev.mysql.com/doc/refman/5.6/en/linux-installation-rpm.html#mysql-installation-layout-linuxrpm

. MySQL 5.6깔기 전에 필요한 perl-Data-Dumper와 net-tools를 설치하고 메일보내는 postfix와 mariadb-libs를 삭제했다.

# yum -y install net-tools # yum -y install perl-Data-Dumper # rpm -e --allmatches postfix-2:2.10.1-6.el7.x86_64 # rpm -e --allmatches mariadb-libs-5.5.52-1.el7.x86_64

. server와 client 설치

# rpm -i MySQL-server-5.6.35-1.el7.x86_64.rpm # rpm -i MySQL-client-5.6.35-1.el7.x86_64.rpm

 . data폴더 변경하여 설치하기

# mkdir -p /data01/mysql # cd /data01/mysql # chown mysql . # chgrp mysql . # cd /usr/bin # ./mysql_install_db --datadir=/data01/mysql --user=mysql

. 기본 설정 파일 변경

# vi /usr/my.cnf [mysqld] ... datadir = /data01/mysql character_set_server=utf8 # cat /root/.mysql_secret

. 초기 root 비밀번호 설정

# systemctl start mysql # ./mysql_secure_installation NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MySQL       SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY! In order to log into MySQL to secure it, we'll need the current password for the root user.  If you've just installed MySQL, and you haven't set the root password yet, the password will be blank, so you should just press enter here. Enter current password for root (enter for none): OK, successfully used password, moving on... Setting the root password ensures that nobody can log into the MySQL root user without the proper authorisation. Set root password? [Y/n] y New password: Re-enter new password: Password updated successfully! Reloading privilege tables..  ... Success! By default, a MySQL installation has an anonymous user, allowing anyone to log into MySQL without having to have a user account created for them.  This is intended only for testing, and to make the installation go a bit smoother.  You should remove them before moving into a production environment. Remove anonymous users? [Y/n] y  ... Success! Normally, root should only be allowed to connect from 'localhost'.  This ensures that someone cannot guess at the root password from the network. Disallow root login remotely? [Y/n] n  ... skipping. By default, MySQL comes with a database named 'test' that anyone can access.  This is also intended only for testing, and should be removed before moving into a production environment. Remove test database and access to it? [Y/n] y  - Dropping test database...  ... Success!  - Removing privileges on test database...  ... Success! Reloading the privilege tables will ensure that all changes made so far will take effect immediately. Reload privilege tables now? [Y/n] y  ... Success! All done!  If you've completed all of the above steps, your MySQL installation should now be secure. Thanks for using MySQL! Cleaning up... 혹시 로그인 실패 시 최초 비밀번호는 아래 파일에 담겨져 있다. # cat /root/.mysql_secret

https://dev.mysql.com/doc/refman/5.6/en/data-directory-initialization.html
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.3/bk_installing_manually_book/content/meet-min-system-requirements.html

. 원격접속 허용하기 

# mysql -p Enter password: Welcome to the MySQL monitor.  Commands end with ; or \g. Your MySQL connection id is 3 Server version: 5.6.35 MySQL Community Server (GPL) Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> select host,user from mysql.user; +---------------------------+------+ | host                      | user | +---------------------------+------+ | 127.0.0.1                 | root | | ::1                       | root | | bidevkylint1.bikylinclust | root | | localhost                 | root | +---------------------------+------+ 4 rows in set (0.00 sec) mysql> CREATE USER 'root'@'%' IDENTIFIED BY '비밀번호'; mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%'; mysql> FLUSH PRIVILEGES;

http://zetawiki.com/wiki/MySQL_%EC%9B%90%EA%B2%A9_%EC%A0%91%EC%86%8D_%ED%97%88%EC%9A%A9

Hyper-V에서 Linux 가상컴퓨터 복사

1. 복사하고자 하는 가상컴퓨터를 "내보내기"

2. Hyper-V 관리자에서 "가상 컴퓨터 가져오기"

3. 가져오기 형식 선택에서 "가상 컴퓨터 복사"

4. 대상 선택 위치를 새로운 가상 컴퓨터 루트 폴더로 지정

5. 저장소 폴더 선택은 Virtual Hard Disks 폴더 지정

6. 이름 변경

7. 가상스위치를 연결되지 않음으로 돌린다.
IP충돌이 날 수 있으므로 가상스위치를 연결되지 않음으로 돌린 후 IP를 변경하고서 다시 연결합니다. (mac어드레스 충돌은 디폴트로 동적MAC 설정이 되어 있어서 충돌나지 않게 되어 있음)

 8. 부팅 후 ip변경

# vi /etc/sysconfig/network-scripts/ifcfg-eth0 IPADDR=10.203.9.216 # systemctl restart network # systemctl restart NetworkManager.service 재시작 후

8. 부팅 후 hostname 변경

# hostnamectl set-hostname bidevkylind1.bikylinclust # nmcli general hostname bidevkylind1.bikylinclust # hostname bidevkylind1.bikylinclust # uname -a Linux bidevkylind1.bikylinclust 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Ambari Local Repository 구축

Ambari Local Repository 구축

. yum repository랑 같은 서버에 구축하려 한다.
yum구축은 아래 링크 참조

https://drunkendwcraft.blogspot.kr/2017/01/centos-yum-repository.html

. 인터넷 연결이 없을때 Ambari Repositories를 구축해야 한다.

참조: http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_Installing_HDP_AMB/content/_ambari_repositories.html

다운로드 주소
. hadoop
https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-installation/content/hdp_26_repositories.html

. ambari
https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-installation/content/ambari_repositories.html

자 이제 다운로드 받은 tarball 3개를 풀어보자

최종 링크는 아래와 같이 나와야 한다.

Ambari Base URL: http://<web.server>/ambari-2.2.2.0/<OS>
HDP Base URL: http://<web.server>/hdp/HDP/<OS>/2.x/updates/<latest.version>
HDP-UTILS Base URL: http://<web.server>/hdp/HDP-UTILS-<version>/repos/<OS>

# tar -xzvf ambari-2.2.2.0-centos7.tar.gz -C /data01 # tar -xzvf HDP-2.4.2.0-centos7-rpm.tar.gz -C /data01 # tar -xzvf HDP-UTILS-1.1.0.20-centos7.tar.gz -C /data01 # ln -s /data01/AMBARI-2.2.2.0/ ambari-2.2.2.0 # mkdir hdp # cd hdp # ln -s /data01/HDP HDP # ln -s /data01/HDP-UTILS-1.1.0.20 HDP-UTILS-1.1.0.20

. configuration
ambari를 설치할 서버에서 다음 파일을 다운로드 받는다
http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.2.2.0/ambari.repo

내용을 다음과 같이 편집

#VERSION_NUMBER=2.2.2.0-460 [Updates-ambari-2.2.2.0] name=ambari-2.2.2.0 - Updates #baseurl=http://<web.server>/ambari-2.2.2.0/<OS> baseurl=http://10.203.9.208/ambari-2.2.2.0/centos7/2.2.2.0-460/ gpgcheck=1 gpgkey=http://10.203.9.208/ambari-2.2.2.0/centos7/2.2.2.0-460/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1

/etc/yum.repos.d/ambari.repo 위치에 복사하면 끝