ScyllaDB(4.4.4)生产环境集群搭建

# 1、在可以连接公网的服务器上，设置ScyllaDB指定版本的yum源curl -o /etc/yum.repos.d/scylla.repo -L http://repositories.scylladb.com/scylla/repo/7ed9bb21-52de-46c1-b569-1bc6776913d8/centos/scylladb-4.4.repo# 2、根据yum源只下载相关依赖包、软件包，不安装cd /rootmkdir scylla-4.4.4yum install --downloadonly --downloaddir=/root/scylla-4.4.4 scylla-4.4.4

cd /root/scylla-4.4.4yum remove abrt* -yyum localinstall * -y

cd /root/scylla-4.4.4# 执行初始化脚本scylla_setup

[10.xx.xx.xxx:root@bj-jd-xxx-xxxxxx-prod-0005:/root/scylla-4.4.4]# scylla_setupSkip any of the following steps by answering 'no'Do you want to run Amazon EC2 configuration check?Yes - runs a script to verify that this instance is optimized for running Scylla. No - skips the configuration check.[YES/no]no  #是否进行Amazon EC2型云服务器配置检测，我们是京东云，不检测，填写noDo you want to run check your kernel version?Yes - runs a  script to verify that the kernel for this instance qualifies to run Scylla. No - skips the kernel check.[YES/no]YES #检查内核是否支持ScyllaWARN 2022-01-18 14:21:07,927 [shard ] iotune - Available space on filesystem at /var/tmp/mnt: 124 MB: is less than recommended: 10 GBINFO 2022-01-18 14:21:07,927 [shard ] iotune - /var/tmp/mnt passed sanity checksThis is a supported kernel version.Do you want to verify the ScyllaDB packages are installed?Yes - runs a script to confirm that ScyllaDB is installed. No - skips the installation check.[YES/no]YES #检查Scylla是否安装Do you want the Scylla server service to automatically start when the Scylla node boots?Yes - Scylla server service automatically starts on Scylla node boot. No - skips this step. Note you will have to start the Scylla Server service manually.[YES/no]YES #设置Scylla自启动Created symlink from /etc/systemd/system/multi-user.target.wants/scylla-server.service to /usr/lib/systemd/system/scylla-server.service.Do you want to enable Scylla to check if there is a newer version of Scylla available?Yes - start the Scylla-housekeeping service to check for a newer version. This check runs periodically. No - skips this step.[YES/no]no  #是否检查更新，不检查，填写noCreated symlink from /etc/systemd/system/scylla-housekeeping-daily.timer to /dev/null.Created symlink from /etc/systemd/system/scylla-housekeeping-restart.timer to /dev/null.Unable to retrieve version informationDo you want to disable SELinux?Yes - disables SELinux. Choosing Yes greatly improves performance. No - keeps SELinux activated.[YES/no]YES #是否关闭SELinuxDo you want to setup Network Time Protocol(NTP) to auto-synchronize the current time on the node?Yes - enables time-synchronization. This keeps the correct time on the node. No - skips this step.[YES/no]YES #是否设置NTP18 Jan 14:21:38 ntpdate[7993]: no server suitable for synchronization foundCreated symlink from /etc/systemd/system/multi-user.target.wants/ntpd.service to /usr/lib/systemd/system/ntpd.service.Do you want to setup RAID0 and XFS?It is recommended to use RAID0 and XFS for Scylla data. If you select yes, you will be prompted to choose the unmounted disks to use for Scylla data. Selected disks are formatted as part of the process.Yes - choose a disk/disks to format and setup for RAID0 and XFS. No - skip this step.[YES/no]YES #是否选择设备设置软RAID0Are you sure you want to setup RAID0 and XFS?If you choose Yes, the selected drive will be reformated, erasing all existing data in the process.[YES/no]YES #填写YES，开始设置Please select unmounted disks from the following list: /dev/nvme0n1, /dev/nvme1n1, /dev/nvme2n1, /dev/nvme3n1type 'cancel' to cancel RAID/XFS setup.type 'done' to finish selection. Selected:> /dev/nvme0n1 #选中第1个裸设备type 'cancel' to cancel RAID/XFS setup.type 'done' to finish selection. Selected: /dev/nvme0n1> /dev/nvme1n1 #选中第2个裸设备type 'cancel' to cancel RAID/XFS setup.type 'done' to finish selection. Selected: /dev/nvme0n1, /dev/nvme1n1> /dev/nvme2n1 #选中第3个裸设备type 'cancel' to cancel RAID/XFS setup.type 'done' to finish selection. Selected: /dev/nvme0n1, /dev/nvme1n1, /dev/nvme2n1> /dev/nvme3n1 #选中第4个裸设备type 'cancel' to cancel RAID/XFS setup.type 'done' to finish selection. Selected: /dev/nvme0n1, /dev/nvme1n1, /dev/nvme2n1, /dev/nvme3n1 #选中的裸设备列表> done #确认选中的裸设备列表，填写done完成选择/dev/md0 will be used to setup a RAIDCreating RAID0 for scylla using 4 disk(s): /dev/nvme0n1,/dev/nvme1n1,/dev/nvme2n1,/dev/nvme3n1mdadm: Defaulting to version 1.2 metadatamdadm: array /dev/md0 started.meta-data=/dev/md0 isize=512    agcount=32, agsize=61043200 blks         = sectsz=512   attr=2, projid32bit=1         = crc=1        finobt=, sparse=data = bsize=4096   blocks=1953382400, imaxpct=5         = sunit=256    swidth=1024 blksnaming =version 2              bsize=4096   ascii-ci= ftype=1log      =internal log           bsize=4096   blocks=521728, version=2         = sectsz=512   sunit=8 blks, lazy-count=1realtime =none extsz=4096   blocks=, rtextents=Created symlink from /etc/systemd/system/multi-user.target.wants/var-lib-scylla.mount to /etc/systemd/system/var-lib-scylla.mount.Do you want to enable coredumps?Yes - sets up coredump to allow a post-mortem analysis of the Scylla state just prior to a crash. No - skips this step.[YES/no]YES #是否设置coredumpCreated symlink from /etc/systemd/system/multi-user.target.wants/var-lib-systemd-coredump.mount to /etc/systemd/system/var-lib-systemd-coredump.mount.kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %p %u %g %s %t %e"Generating coredump to test systemd-coredump...PID: 8252 (bash)           UID:  (root)           GID:  (root)        Signal: 11 (SEGV)     Timestamp: Tue 2022-01-18 14:22:10 CST (3s ago)  Command Line: /bin/bash /tmp/tmpr5m6lvd9    Executable: /usr/bin/bash Control Group: /user.slice/user-.slice/session-2391.scope          Unit: session-2391.scope         Slice: user-.slice       Session: 2391     Owner UID:  (root)       Boot ID: 47cd570e8e774c7a8dba1ebda4ec06ab    Machine ID: e658d54604f59aa28275af74dce7c2cf      Hostname: bj-jd-xxx-xxxxxx-prod-0005.xxxxxx.com      Coredump: /var/lib/systemd/coredump/core.bash.0.47cd570e8e774c7a8dba1ebda4ec06ab.8252.1642486930000000       Message: Process 8252 (bash) of user  dumped core.                Stack trace of thread 8252:                #  x00007f466f762607 kill (libc.so.6)                #1  x0000000000440736 kill_pid (bash)                #2  x0000000000472d2e kill_builtin (bash)                #3  x000000000042f2ef execute_builtin.isra.2 (bash)                #4  x00000000004313d9 execute_simple_command (bash)                #5  x00000000004322e3 execute_command_internal (bash)                #6  x0000000000433cce execute_command (bash)                #7  x000000000041e305 reader_loop (bash)                #8  x000000000041c96e main (bash)                #9  x00007f466f74e505 __libc_start_main (libc.so.6)                #10 x000000000041d40a _start (bash)systemd-coredump is working finely.Do you want to setup a system-wide customized configuration for Scylla?Yes - setup the sysconfig file. No - skips this step.[YES/no]YES #是否创建自定义的系统配置文件Do you want to enable Network Interface Card (NIC) and disk(s) optimization?Yes - optimize the NIC queue and disks settings. Selecting Yes greatly improves performance. No - skip this step.[yes/NO]YES #是否优化NIC(Network Interface Card)设置The clocksource is the physical device that Linux uses to take time measurements. In most cases Linux chooses the fastest available clocksource device as long as it is accurate. In some situations, however, Linux errs in the side of caution and does not choose the fastest available clocksource despite it being accurate enough. If you know your hardwares fast clocksource is stable enough, choose "yes" here. The safest is the choose "no" (the default)Yes - enforce clocksource setting. No - keep current configuration.[yes/NO]no  #是否设置Linux时钟源，填写no为默认安全设置Do you want IOTune to study your disks IO profile and adapt Scylla to it? (*WARNING* Saying NO here means the node will not boot in production mode unless you configure the I/O Subsystem manually!)Yes - let iotune study my disk(s). Note that this action will take a few minutes. No - skip this step.[YES/no]YES #是否进行基准IO测试tuning /sys/devices/virtual/block/md0tuning: /sys/devices/virtual/block/md0/queue/nomerges 2tuning /sys/devices/pci0000:00/0000:00:06.0/nvme/nvme0/nvme0n1tuning: /sys/devices/pci0000:00/0000:00:06.0/nvme/nvme0/nvme0n1/queue/nomerges 2tuning /sys/devices/pci0000:00/0000:00:07.0/nvme/nvme1/nvme1n1tuning: /sys/devices/pci0000:00/0000:00:07.0/nvme/nvme1/nvme1n1/queue/nomerges 2tuning /sys/devices/pci0000:00/0000:00:08.0/nvme/nvme2/nvme2n1tuning: /sys/devices/pci0000:00/0000:00:08.0/nvme/nvme2/nvme2n1/queue/nomerges 2tuning /sys/devices/pci0000:00/0000:00:09.0/nvme/nvme3/nvme3n1tuning: /sys/devices/pci0000:00/0000:00:09.0/nvme/nvme3/nvme3n1/queue/nomerges 2tuning /sys/devices/virtual/block/md0tuning /sys/devices/virtual/block/md0tuning /sys/devices/virtual/block/md0tuning /sys/devices/virtual/block/md0INFO 2022-01-18 14:22:41,333 [shard ] iotune - /var/lib/scylla/saved_caches passed sanity checksINFO 2022-01-18 14:22:41,335 [shard ] iotune - Disk parameters: max_iodepth=4092 disks_per_array=4 minimum_io_size=512Starting Evaluation. This may take a while...Measuring sequential write bandwidth: 4983 MB/s #基准测试结果：顺序写带宽Measuring sequential read bandwidth: 8075 MB/s #基准测试结果：顺序读带宽Measuring random write IOPS: 1576952 IOPS #基准测试结果：随机写IOPSMeasuring random read IOPS: 2621466 IOPS #基准测试结果：随机读IOPSWriting result to /etc/scylla.d/io_properties.yamlWriting result to /etc/scylla.d/io.confDo you want to enable node exporter to export Prometheus data from the node? Note that the Scylla monitoring stack uses this dataYes - enable node exporter. No - skip this step.[YES/no]YES #是否设置监控agent(node exporter)Created symlink from /etc/systemd/system/multi-user.target.wants/scylla-node-exporter.service to /usr/lib/systemd/system/scylla-node-exporter.service.Do you want to set the CPU scaling governor to Performance level on boot?Yes - sets the CPU scaling governor to performance level. No - skip this step.[YES/no]YES #是否设置CPU大性能模式This computer doesn't supported CPU scaling configuration.Do you want to enable fstrim service?Yes - runs fstrim on your SSD. No - skip this step.[YES/no]YES #是否优化SSD fstrim配置Created symlink from /etc/systemd/system/timers.target.wants/scylla-fstrim.timer to /usr/lib/systemd/system/scylla-fstrim.timer.Will Scylla be the only service on this host?Answer yes to lock all memory to Scylla, to prevent swapout. Answer no to do nothing.[YES/no]YES #是否将机器内存只留给Scylla，专用Scylla服务器，所有填写YESNo swap is configured, it is highly recommended to setup swap on Scylla node.Do you want to configure swapfile?Answer yes to setup swapfile to Scylla, Answer no to do nothing.[YES/no]no #是否设置SWAP fileDo you want to configure rsyslog to send log to a remote repository?Answer yes to setup rsyslog to a remote server, Answer no to do nothing.[YES/no]no #是否设置远程rsyslogScyllaDB setup finished.scylla_setup accepts command line arguments as well! For easily provisioning in a similar environment than this, type:    scylla_setup \                 --disks /dev/nvme0n1,/dev/nvme1n1,/dev/nvme2n1,/dev/nvme3n1 \                 --nic eth0 --no-ec2-check --io-setup 1 \                 --no-version-check --no-swap-setup --no-rsyslog-setupAlso, to avoid the time-consuming I/O tuning you can add --no-io-setup and copy the contents of /etc/scylla.d/io*Only do that if you are moving the files into machines with the exact same hardware

# 修改集群名称，5个节点配置一致sed -i "s/\#cluster_name: 'Test Cluster'/cluster_name: 'prod-xx-xxxx-scylla'/g" /etc/scylla/scylla.yaml# 设置集群种子(seed)节点，5个节点配置一致sed -i 's/seeds: \"127.0.0.1\"/seeds: \"10.xx.xx.xxx,10.xx.xx.xxx\"/g' /etc/scylla/scylla.yaml# 设置监听地址为本机IPsed -i 's/listen_address: localhost/listen_address: '"$local_ip"'/g' /etc/scylla/scylla.yaml# 设置rpc_address地址为本机IPsed -i 's/rpc_address: localhost/rpc_address: '"$local_ip"'/g' /etc/scylla/scylla.yaml# 设置接口地址为本机IPsed -i 's/api_address: 127.0.0.1/api_address: '"$local_ip"'/g' /etc/scylla/scylla.yaml# 设置不自动拍摄快照sed -i 's/# auto_snapshot: true/auto_snapshot: false/g' /etc/scylla/scylla.yaml# 设置ScyllaDB不自动检测更新sed -i 's/check-version: True/check-version: False/g' /etc/scylla.d/housekeeping.cfg

systemctl start scylla-server

nodetool status