greenplum集群安装
文章目录
安装greenplum集群
集成 PXF（无 Kerberos）
集成Bitmap
集成HLL
安装greenplum集群
服务器greenplum集群节点角色规划
用户 主机名 内网IP SSH端口 系统 rke 角色
gpadmin gp-master 192.168.2.131 22 CentOS Linux release 7.6.1810 (Core) Master
gpadmin gp-segment-01 192.168.2.132 22 CentOS Linux release 7.6.1810 (Core) Segment
gpadmin gp-segment-02 192.168.2.133 22 CentOS Linux release 7.6.1810 (Core) Segment
系统准备（所有节点 用root 用户操作）
安装一些个人常用的基础安装包
yum -y install epel-release.noarch
yum -y install psmisc gcc gcc-c++ texinfo wget unzip zip gcc libticonv-devel libcurl-devel curl nmap iotop dstat tree mlocate ntpdate openssh-clients net-tools vim ntsysv nmap curl lrzsz sysstat libselinux-python pcre pcre-devel zlib zlib-devel openssl openssl-devel readline-devel bzip2 httpd-devel python-devel python-pip python-setuptools lsof sqlite-devel nscd bind-utils telnet rsync tcpdump expect nc ntp lftp bash-completion ipset ipvsadm
更改服务hosts文件
cat >> /etc/hosts << EOF
192.168.2.131 gp-master
192.168.2.132 gp-segment-01
192.168.2.133 gp-segment-02
EOF
关闭防火墙
systemctl stop firewalld
systemctl stop iptables
systemctl disable firewalld
systemctl disable iptables
关闭selinux
sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/sysconfig/selinux
sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
setenforce 0
三台时间同步要一致
systemctl start ntpd
systemctl enable ntpd
关闭swap分区
swapoff -a
sed -i '/swap/d' /etc/fstab
mount -a
然后修改/etc/fstab，把swap分区相关的配置注释掉
内核参数调整
cat >> /etc/sysctl.conf <<EOF
fs.file-max = 2442652
net.ipv4.ip_local_port_range = 1024 65535
vm.swappiness=0
kernel.sem = 500 5120000 2500 9000
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 500000000
kernel.shmall = 4000000000
kernel.shmmni = 4096
vm.overcommit_memory = 2
vm.overcommit_ratio=95
EOF
sysctl -p
配置进程文件资源限制
cat > /etc/security/limit.conf << EOF
* soft nofile 655360
* hard nofile 655360
* soft nproc 131072
* hard nproc 131072
EOF
cat > /etc/security/limits.d/20-nproc.conf << EOF
* soft nofile 655360
* hard nofile 655360
* soft nproc 131072
* hard nproc 131072
EOF
IPC对象删除，防止Greenplum数据库实用程序gpinitsystem失败并出现信号错误
vim /etc/systemd/logind.conf
RemoveIPC=no
创建gpadmin 用户并设置密码
useradd gpadmin
echo "password" | passwd --stdin gpadmin
下载rpm安装包并安装并创建gp的数据目录
链接: https://pan.baidu.com/s/1UDV2Ki7eLtgS_t549dZwIg 提取码: p5qv
yum install greenplum-db-6.9.1-rhel7-x86_64.rpm -y
mkdir -p /data/gpdata
chown -R gpadmin.gpadmin /usr/local/greenplum-db
chown -R gpadmin.gpadmin /data/gpdata
配置Greenplum DB(用gpadmin用户操作)
两个segment节点 创建数据目录
mkdir -p /data/gpdata/gp{1,2,m1,m2}
gp-master主机上创建数据目录
mkdir -p /data/gpdata/gpmaster
在gp-master 创建一个all_hosts文件包含所有节点,创建一个seg_hosts文件包含所有segment节点
cat >/home/gpadmin/all_hosts <<EOF
gp-master
gp-segment-01
gp-segment-02
cat >/home/gpadmin/seg_hosts <<EOF
gp-segment-01
gp-segment-02
在gp-master上配置ssh 免密码登录
ssh-keygen -t rsa
ssh-copy-id -i /home/gpadmin/.ssh/id_rsa.pub gpadmin@gp-master
ssh-copy-id -i /home/gpadmin/.ssh/id_rsa.pub gpadmin@gp-segment-01
ssh-copy-id -i /home/gpadmin/.ssh/id_rsa.pub gpadmin@gp-segment-02
source /usr/local/greenplum-db/greenplum_path.sh
gpssh-exkeys -f /home/gpadmin/all_hosts
初始化前检查连通性
source /usr/local/greenplum-db/greenplum_path.sh
gpcheckperf -f /home/gpadmin/all_hosts -r N -d /tmp
配置初始化集群需要的配置文件
cp /usr/local/greenplum-db/docs/cli_help/gpconfigs/gpinitsystem_config ~/gpinitsystem_config
cat > ~/gpinitsystem_config << EOF
ARRAY_NAME="Greenplum Data Platform"
SEG_PREFIX=gpseg
PORT_BASE=6000
declare -a DATA_DIRECTORY=(/data/gpdata/gp1 /data/gpdata/gp2 )
MASTER_HOSTNAME=gp-master
MASTER_DIRECTORY=/data/gpdata/gpmaster
MASTER_PORT=5432
TRUSTED_SHELL=ssh
CHECK_POINT_SEGMENTS=8
ENCODING=UNICODE
MIRROR_PORT_BASE=7000
declare -a MIRROR_DATA_DIRECTORY=(/data/gpdata/gpm1 /data/gpdata/gpm2 )
DATABASE_NAME=test
EOF
初始化集群
gpinitsystem -c ~/gpinitsystem_config -h ~/seg_hosts
也可以指定standby 节点对master节点实现高可用(需要在standby节点创建/data/gpdata/gpmaster目录)
gpinitsystem -c ~/gpinitsystem_config -h ~/seg_hosts -s standby_host_ip
配置shell 终端
cat > ~/.bashrc << EOF
source /usr/local/greenplum-db/greenplum_path.sh
export MASTER_DATA_DIRECTORY=/data/gpdata/gpmaster/gpseg-1
export PGPORT=5432
export PGUSER=gpadmin
export PGDATABASE=postgres
export LD_PRELOAD=/lib64/libz.so.1
export PS1="[\u@\h \W]\$"
EOF
source ~/.bashrc
启动Greenplum: gpstart
停止Greenplum: gpstop
加载Greenplum配置文件: gpstop -u
查看集群状态: gpstate
集成 PXF（无 Kerberos）
export PXF_CONF=/usr/local/greenplum-db/pxf/conf
export GPHOME=/usr/local/greenplum-db
#初始化
$GPHOME/pxf/bin/pxf cluster init
cp -ap /etc/hadoop/conf/hdfs-site.xml ${GPHOME}/conf/servers/default
cp -ap /etc/hadoop/conf/core-site.xml ${GPHOME}/conf/servers/default
cp -ap /etc/hadoop/conf/yarn-site.xml ${GPHOME}/conf/servers/default
cp -ap /etc/hadoop/conf/mapred-site.xml ${GPHOME}/conf/servers/default
同步配置文件到其他pxf 节点（Greenplum 节点）
$GPHOME/pxf/bin/pxf cluster sync
启动 pxf
$GPHOME/pxf/bin/pxf cluster start
添加 pxf 插件
psql cdp
=# CREATE EXTENSION pxf;
=# \dx
问题： User: xxx is not allowed to impersonate xxx
解决方案： /usr/local/greenplum-db/pxf/conf/conf/pxf-env.sh 中追加export PXF_USER_IMPERSONATION=false 重启pxf 服务
集成Bitmap
tar -zxvf roaringbitmap_v0.2.66.tar.gz
cd gpdb-roaringbitmap-0.2.66
make && make install
创建 bitmap插件
# CREATE EXTENSION roaringbitmap;
集成HLL
tar -zxvf postgresql-hll-2.14.tar.gz
cd postgresql-hll-2.14
make && make install
创建 hll 插件
# CREATE EXTENSION hll;
本文来源：https://blog.csdn.net/qq_23191379/article/details/123993177
相关文章