greatdb集群的搭建
GreatDB是一款基于mysql的分布式关系型数据库，它采用Shared-Nothing原生分布式架构，主要面向大数据量、高并发场景下的结构化数据存储和事务处理。
GreatDB可以通过ansible工具搭建，也可手动搭建，本实验参考greatdb管理手册演示通过手动方式来搭建greatdb集群的过程。
一、系统规划
以搭建三个计算节点、九个数据节点为例，系统规划如下：
因资源限制，数据节点主机进行了复用，用不同端口加以区分同一主机上不同数据节点，搭建完成后系统架构如下图所示：
二、sshpass检查和ssh策略调正
将sshpass.tar.gz 包上传至/opt/目录下，
[root@sqlnode2 ~]# whereis sshpass
sshpass:[root@sqlnode2 ~]#
[root@datanode6 ~]# cd /opt/
-rw-r--r--. 1 root root 20480 Feb 17 11:06 sshpass.tar.gz
[root@datanode6 opt]# ll
total 20
-rw-r--r--. 1 root root 20480 Feb 17 11:06 sshpass.tar.gz
[root@datanode6 opt]# tar xf sshpass.tar.gz
[root@datanode6 opt]# ll
total 36
-rwxr-xr-x. 1 root root 15736 Nov 28 10:32 sshpass
-rw-r--r--. 1 root root 20480 Feb 17 11:06 sshpass.tar.gz
[root@datanode6 opt]# cp sshpass /usr/local/bin/sshpass
[root@datanode6 opt]# whereis sshpass
sshpass: /usr/local/bin/sshpass
ssh策略调整，设置root用户允许远程登录
[root@sqlnode2 opt]# cat /etc/ssh/sshd_config |grep PermitRootLogin
#PermitRootLogin yes
# the setting of "PermitRootLogin without-password".
[root@sqlnode2 opt]# vi /etc/ssh/sshd_config
PermitRootLogin yes
[root@sqlnode2 opt]#systemctl restart sshd
三、用户和用户组的创建
所有计算节点和数据节点创建管理用户、用户组.
[root@sqlnode1 ~]# groupadd greatdb
[root@sqlnode1 ~]# useradd -g greatdb greatdb
[root@sqlnode1 ~]# echo "greatdb" | passwd --stdin greatdb
Changing password for user greatdb.
passwd: all authentication tokens updated successfully.
四、规划目录
计算节点和数据节点均需要创建配置⽂件和数据文件⽬录。计算节点的数据⽬录⽤于存放集群元数据，数据节点数据⽬录⽤于存放⽤户数据。分别创建/greatdb/config 和 /greatdb/datas目录为例 ，/greatdb/config 目录用于存放配置文件，/greatdb/datas目录用于存放数据。
[root@sqlnode1 ~]# mkdir -p /greatdb/config
[root@sqlnode1 ~]# mkdir -p /greatdb/datas
[root@sqlnode1 ~]# chown -R greatdb.greatdb /greatdb
五、计算和数据节点配置文件创建
使用如下脚本生成各个节点配置文件。
[greatdb@sqlnode1 ~]$ vi create_config.sh
#!/bin/bash
set -ex
ip=$1
port=$2
echo "[mysqld]
datadir=/greatdb/datas/$port
socket=/greatdb/datas/$port/greatdb.sock
user=greatdb
port=$2
server_id=$RANDOM
max_connections=1000
report-host=$1
## group replication configuration
binlog-checksum=NONE
gtid-mode=ON
enforce-gtid-consistency=on
loose-group_replication_start_on_boot=OFF
loose-group_replication_recovery_get_public_key=ON
loose-group_replication_local_address=\"$1:1$2\""
5.1 计算节点配置文件生成
#生成192.168.3.31节点配置文件
bash create_config.sh 192.168.3.31 3306 > /greatdb/config/3306.cnf
#生成192.168.3.32节点配置文件
bash create_config.sh 192.168.3.32 3306 > /greatdb/config/3306.cnf
#生成192.168.3.33节点配置文件
bash create_config.sh 192.168.3.33 3306 > /greatdb/config/3306.cnf
[root@sqlnode1 greatdb-cluster]# cat /greatdb/config/3306.cnf
[mysqld]
datadir=/greatdb/datas/3306
socket=/greatdb/datas/3306/greatdb.sock
user=greatdb
port=3306
server_id=22195
max_connections=1000
report-host=192.168.3.31
## group replication configuration
binlog-checksum=NONE
gtid-mode=ON
enforce-gtid-consistency=on
loose-group_replication_start_on_boot=OFF
loose_group_replication_recovery_get_public_key=ON
loose-group_replication_local_address="192.168.3.31:13306"
[root@sqlnode1 greatdb-cluster]#
5.2 数据节点配置文件生成
#生成192.168.3.34节点配置文件
bash create_config.sh 192.168.3.34 3307 > /greatdb/config/3307.cnf
bash create_config.sh 192.168.3.34 3308 > /greatdb/config/3308.cnf
bash create_config.sh 192.168.3.34 3309 > /greatdb/config/3309.cnf
#生成192.168.3.35节点配置文件
bash create_config.sh 192.168.3.35 3307 > /greatdb/config/3307.cnf
bash create_config.sh 192.168.3.35 3308 > /greatdb/config/3308.cnf
bash create_config.sh 192.168.3.35 3309 > /greatdb/config/3309.cnf
#生成192.168.3.36节点配置文件
bash create_config.sh 192.168.3.36 3307 > /greatdb/config/3307.cnf
bash create_config.sh 192.168.3.36 3308 > /greatdb/config/3308.cnf
bash create_config.sh 192.168.3.36 3309 > /greatdb/config/3309.cnf
六、使用二进制包安装greatdb
将二进制包上传至各个主机。
root@sqlnode1 opt]# cd /usr/local/
[root@sqlnode1 local]# tar xvzf greatdb-cluster-5.0.7-fbb08fbed60-Linux-glibc2.17-x86_64.tar.gz
[root@sqlnode1 local]# ln -s greatdb-cluster-5.0.7-fbb08fbed60-Linux-glibc2.17-x86_64 greatdb-cluster
[root@sqlnode1 local]# cd greatdb-cluster
[root@sqlnode1 greatdb-cluster]# pwd
/usr/local/greatdb-cluster
[root@sqlnode1 greatdb-cluster]# chown greatdb:greatdb -R .
[root@sqlnode1 greatdb-cluster]# ll
total 628
drwxr-xr-x 2 greatdb greatdb 4096 Sep 1 17:56 bin
drwxr-xr-x 2 greatdb greatdb 69 Sep 1 17:50 cmake
-rw-r--r-- 1 greatdb greatdb 1703 Sep 1 17:50 COPYING-jemalloc
drwxr-xr-x 2 greatdb greatdb 115 Sep 1 17:50 docs
drwxr-xr-x 6 greatdb greatdb 4096 Sep 1 17:50 include
drwxr-xr-x 8 greatdb greatdb 4096 Sep 1 17:51 lib
-rw-r--r-- 1 greatdb greatdb 274942 Sep 1 02:24 LICENSE
-rw-r--r-- 1 greatdb greatdb 45333 Sep 1 02:24 LICENSE.router
-rw-r--r-- 1 greatdb greatdb 274942 Sep 1 02:24 LICENSE-test
drwxr-xr-x 4 greatdb greatdb 30 Sep 1 17:50 man
-rw-r--r-- 1 greatdb greatdb 1623 Sep 1 17:40 mysqlrouter-log-rotate
-rw-r--r-- 1 greatdb greatdb 685 Sep 1 02:24 README
-rw-r--r-- 1 greatdb greatdb 679 Sep 1 02:24 README.router
-rw-r--r-- 1 greatdb greatdb 685 Sep 1 02:24 README-test
drwxrwxr-x 2 greatdb greatdb 6 Sep 1 17:50 run
drwxr-xr-x 28 greatdb greatdb 4096 Sep 1 17:50 share
drwxr-xr-x 2 greatdb greatdb 77 Sep 1 17:50 support-files
drwxr-xr-x 3 greatdb greatdb 17 Sep 1 17:50 var
配置greatdb用户下的环境变量，方便后期一些命令的使用。
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
GHOME=/usr/local/greatdb-cluster/bin
PATH=$PATH:$HOME/.local/bin:$HOME/bin:$GHOME
export PATH
6.1 计算节点的初始化
三个计算节点分别通过greatdb_init 工具来初始化。
[root@sqlnode1 greatdb-cluster]# bin/greatdb_init --defaults-file=/greatdb/config/3306.cnf --cluster-user=greatdb --cluster-host=% --cluster-password=greatdb --node-type=sqlnode
2022-02-17 16:13:36 [Stage 1] initialize server
2022-02-17T08:13:37.086365Z 0 [System] [MY-013169] [Server] /usr/local/greatdb-cluster/bin/greatsqld (mysqld 8.0.25-15-greatdbcluster5.0.7-GA) initializing of server in progress as process 5986
2022-02-17T08:13:37.095532Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-02-17T08:13:38.315549Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-02-17T08:13:39.530373Z 0 [Warning] [MY-000067] [Server] unknown variable 'loose-group_replication_start_on_boot=OFF'.
2022-02-17T08:13:39.530694Z 0 [Warning] [MY-000067] [Server] unknown variable 'loose_group_replication_recovery_get_public_key=ON'.
2022-02-17T08:13:39.530925Z 0 [Warning] [MY-000067] [Server] unknown variable 'loose-group_replication_local_address=192.168.3.31:13306'.
2022-02-17T08:13:39.532068Z 6 [Warning] [MY-010453] [Server] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
2022-02-17 16:13:43 [Stage 2] start server instance
2022-02-17 16:13:43 [Stage 3] login and create cluster_user
2022-02-17 16:13:43 [Stage 3.1] get server port from defaults_file
2022-02-17 16:13:43 [Stage 3.2] wait server instance online
2022-02-17 16:13:43 waitting ...
2022-02-17 16:13:45 [INFO] server start
2022-02-17 16:13:45 [Stage 3.3] login and create user
2022-02-17 16:13:45 [INFO] create user greatdb@% success
2022-02-17 16:13:45 [INFO] create user greatdb@localhost success
2022-02-17 16:13:45 [INFO] set password for user root@localhost success
2022-02-17 16:13:45 ===============Finish===================
[root@sqlnode1 greatdb-cluster]#
6.2 数据节点的初始化
在每个数据节点都执行以下命令进行数据节点初始化。
bin/greatdb_init --defaults-file=/greatdb/config/3307.cnf --cluster-user=greatdb --cluster-host=% --cluster-password=greatdb --node-type=datanode
bin/greatdb_init --defaults-file=/greatdb/config/3308.cnf --cluster-user=greatdb --cluster-host=% --cluster-password=greatdb --node-type=datanode
bin/greatdb_init --defaults-file=/greatdb/config/3309.cnf --cluster-user=greatdb --cluster-host=% --cluster-password=greatdb --node-type=datanode
七、集群创建
集群创建在计算节点一个节点执行即可，本实验选择全部在计算一节点执行。
7.1 计算集群的创建
在计算一节点初始化集群
GreatDB Cluster[(none)]> call mysql.greatdb_init_cluster('greatdb_cluster','greatdb','greatdb');
Query OK, 1 row affected (13.38 sec)
分别将规划的的计算二节点和三节点加入集群
####添加二节点
GreatDB Cluster[(none)]> call mysql.greatdb_add_sqlnode('192.168.3.32',3306);
Query OK, 0 rows affected (3.41 sec)
####添加三节点
GreatDB Cluster[(none)]> call mysql.greatdb_add_sqlnode('192.168.3.33',3306);
Query OK, 0 rows affected (2.76 sec)
7.2数据节点的添加和初始化
添加shared1的数据节点并初始化；
call mysql.greatdb_add_datanode('shard1', 'sd1_dn1', '192.168.3.34',3307,'NODE_MGR');
call mysql.greatdb_add_datanode('shard1', 'sd1_dn2', '192.168.3.35',3307,'NODE_MGR');
call mysql.greatdb_add_datanode('shard1', 'sd1_dn3', '192.168.3.36',3307,'NODE_MGR');
call mysql.greatdb_init_shard('shard1');
添加shared2的数据节点并初始化；
call mysql.greatdb_add_datanode('shard2', 'sd2_dn1', '192.168.3.34',3308,'NODE_MGR');
call mysql.greatdb_add_datanode('shard2', 'sd2_dn2', '192.168.3.35',3308,'NODE_MGR');
call mysql.greatdb_add_datanode('shard2', 'sd2_dn3', '192.168.3.36',3308,'NODE_MGR');
call mysql.greatdb_init_shard('shard2');
添加shared3的数据节点并初始化。
call mysql.greatdb_add_datanode('shard3', 'sd3_dn1', '192.168.3.34',3309,'NODE_MGR');
call mysql.greatdb_add_datanode('shard3', 'sd3_dn2', '192.168.3.35',3309,'NODE_MGR');
call mysql.greatdb_add_datanode('shard3', 'sd3_dn3', '192.168.3.36',3309,'NODE_MGR');
call mysql.greatdb_init_shard('shard3');
八、集群健康状态检查
分别检查集群计算节点、数据节点、shared状态
[greatdb@sqlnode1 ~]$ greatsql -p -P3306 -h127.0.0.1 -ugreatdb
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1367
Server version: 8.0.25-15-greatdbcluster5.0.7-GA GreatDB Cluster, Release GA, Revision fbb08fbed60
Copyright (c) 2009-2021 BEIJING GREAT OPENSOURCE SOFTWARE TECHNOLOGY CO.,LTD.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
GreatDB Cluster[(none)]>
GreatDB Cluster[(none)]> select * from information_schema.greatdb_sqlnodes;
+--------------------------------------+--------------+------+--------+---------------+--------------------------+
| UUID | HOST | PORT | STATUS | EXECUTED_GTID | Received_transaction_set |
+--------------------------------------+--------------+------+--------+---------------+--------------------------+
| 532b2f0d-8fcc-11ec-b9c0-08002737fd33 | 192.168.3.32 | 3306 | ONLINE | | |
| 831670fa-8fc9-11ec-92a3-080027591892 | 192.168.3.31 | 3306 | ONLINE | | |
| 944abab5-8fcc-11ec-938f-0800270905a5 | 192.168.3.33 | 3306 | ONLINE | | |
+--------------------------------------+--------------+------+--------+---------------+--------------------------+
3 rows in set (3.00 sec)
GreatDB Cluster[(none)]> select * from information_schema.greatdb_datanodes;
+---------+-----------+----------+------------+--------------+------+-----------+---------------+------------------------------------------+------------------------------------------+
| NODE_ID | NODE_NAME | SHARD_ID | SHARD_NAME | HOST | PORT | NODE_TYPE | NODE_STATE | EXECUTED_GTID | RECEIVED_TRANSACTION_SET |
+---------+-----------+----------+------------+--------------+------+-----------+---------------+------------------------------------------+------------------------------------------+
| 6 | sd1_dn1 | 6 | shard1 | 192.168.3.34 | 3307 | NODE_MGR | STATE_ACTIVE | 36000000-0000-0000-0000-000000000000:1-2 | 36000000-0000-0000-0000-000000000000:1-2 |
| 7 | sd1_dn2 | 6 | shard1 | 192.168.3.35 | 3307 | NODE_MGR | STATE_STANDBY | 36000000-0000-0000-0000-000000000000:1-2 | 36000000-0000-0000-0000-000000000000:1-2 |
| 8 | sd1_dn3 | 6 | shard1 | 192.168.3.36 | 3307 | NODE_MGR | STATE_STANDBY | 36000000-0000-0000-0000-000000000000:1-2 | 36000000-0000-0000-0000-000000000000:1-2 |
| 12 | sd2_dn1 | 12 | shard2 | 192.168.3.34 | 3308 | NODE_MGR | STATE_ACTIVE | 31320000-0000-0000-0000-000000000000:1-2 | 31320000-0000-0000-0000-000000000000:1-2 |
| 13 | sd2_dn2 | 12 | shard2 | 192.168.3.35 | 3308 | NODE_MGR | STATE_STANDBY | 31320000-0000-0000-0000-000000000000:1-2 | 31320000-0000-0000-0000-000000000000:1-2 |
| 14 | sd2_dn3 | 12 | shard2 | 192.168.3.36 | 3308 | NODE_MGR | STATE_STANDBY | 31320000-0000-0000-0000-000000000000:1-2 | 31320000-0000-0000-0000-000000000000:1-2 |
| 16 | sd3_dn1 | 16 | shard3 | 192.168.3.34 | 3309 | NODE_MGR | STATE_ACTIVE | 31360000-0000-0000-0000-000000000000:1-2 | 31360000-0000-0000-0000-000000000000:1-2 |
| 17 | sd3_dn2 | 16 | shard3 | 192.168.3.35 | 3309 | NODE_MGR | STATE_STANDBY | 31360000-0000-0000-0000-000000000000:1-2 | 31360000-0000-0000-0000-000000000000:1-2 |
| 18 | sd3_dn3 | 16 | shard3 | 192.168.3.36 | 3309 | NODE_MGR | STATE_STANDBY | 31360000-0000-0000-0000-000000000000:1-2 | 31360000-0000-0000-0000-000000000000:1-2 |
+---------+-----------+----------+------------+--------------+------+-----------+---------------+------------------------------------------+------------------------------------------+
9 rows in set (0.01 sec)
GreatDB Cluster[(none)]> select * from information_schema.greatdb_shards;
+----------+------------+--------------+---------+
| SHARD_ID | SHARD_NAME | SHARD_STATE | SUSPEND |
+----------+------------+--------------+---------+
| 6 | shard1 | SHARD_ONLINE | OFF |
| 12 | shard2 | SHARD_ONLINE | OFF |
| 16 | shard3 | SHARD_ONLINE | OFF |
+----------+------------+--------------+---------+
3 rows in set (0.00 sec)相关文章