十大网站建设公司,开发软件需要什么学历,自己想做一个网站怎么做的,手把手指导做网站MHA简介
MHA#xff08;Master High Availability#xff09;目前在MySQL高可用方面是一个相对成熟的解决方案#xff0c;它由日本DeNA公司的youshimaton#xff08;现就职于Facebook公司#xff09;开发#xff0c;是一套优秀的作为MySQL高可用性环境下故障切换和主从提…MHA简介
MHAMaster High Availability目前在MySQL高可用方面是一个相对成熟的解决方案它由日本DeNA公司的youshimaton现就职于Facebook公司开发是一套优秀的作为MySQL高可用性环境下故障切换和主从提升的高可用软件。在MySQL故障切换过程中MHA能做到在0~30秒之内自动完成数据库的故障切换操作并且在进行故障切换的过程中MHA能在最大程度上保证数据的一致性以达到真正意义上的高可用。 MHA架构图 MHA工作原理总结为以下几条
从宕机崩溃的master保存二进制日志事件binlog events识别含有最新更新的slave应用差异的中继日志relay log到其他slave应用从master保存的二进制日志事件binlog events提升一个slave为新master使用其他的slave连接新的master进行复制。
MHA特点:
故障切换时间 10-30秒自动监控Master以及故障转移在线切换Master到异机
MHA搭建 基本环境
操作系统版本CentOS Linux release 7.8.2003 (Core) MySQL版本 5.7.31 VIP 10.31.1.241
主机信息: 配置主从
配置 hp3、hp4两台同步hp2的主从复制参考主从复制blog此处略过
安装Perl等依赖模块
用root用户在所有四个节点执行下面的操作。
# 安装一个epel源
wget -O /etc/yum.repos.d/epel-7.repo http://mirrors.aliyun.com/repo/epel-7.repo# 用yum安装依赖包
yum install perl-DBD-MySQL perl-Config-Tiny perl-Log-Dispatch perl-Parallel-ForkManager perl-Time-HiRes -y配置SSH登录无密码验证
在hdp1 10.31.1.119Monitor上用root用户执行
ssh-keygen -t rsa
ssh-copy-id -i /root/.ssh/id_rsa.pub root10.31.1.120
ssh-copy-id -i /root/.ssh/id_rsa.pub root10.31.1.121
ssh-copy-id -i /root/.ssh/id_rsa.pub root10.31.1.122在hdp2 10.31.1.120Master上用root用户执行
ssh-keygen -t rsa
ssh-copy-id -i /root/.ssh/id_rsa.pub root10.31.1.119
ssh-copy-id -i /root/.ssh/id_rsa.pub root10.31.1.121
ssh-copy-id -i /root/.ssh/id_rsa.pub root10.31.1.122在hdp3 10.31.1.121Candidate Master上用root用户执行
ssh-keygen -t rsa
ssh-copy-id -i /root/.ssh/id_rsa.pub root10.31.1.119
ssh-copy-id -i /root/.ssh/id_rsa.pub root10.31.1.120
ssh-copy-id -i /root/.ssh/id_rsa.pub root10.31.1.122在hdp4 10.31.1.122Slave上用root用户执行
ssh-keygen -t rsa
ssh-copy-id -i /root/.ssh/id_rsa.pub root10.31.1.119
ssh-copy-id -i /root/.ssh/id_rsa.pub root10.31.1.120
ssh-copy-id -i /root/.ssh/id_rsa.pub root10.31.1.121安装MHA Node
下载地址https://github.com/yoshinorim/mha4mysql-manager/wiki/Downloads 在hdp2、hdp3、hdp4上用root用户执行下面的操作。
rpm -ivh mha4mysql-node-0.56-0.el6.noarch.rpm安装完成后在/usr/bin/目录下有如下MHA相关文件
apply_diff_relay_logs
filter_mysqlbinlog
purge_relay_logs
save_binary_logs这些脚本工具通常由MHA Manager的脚本触发无需人为操作。脚本说明 apply_diff_relay_logs识别差异的中继日志事件并将其差异的事件应用于其它slave。 filter_mysqlbinlog去除不必要的ROLLBACK事件MHA已不再使用这个工具。 purge_relay_logs清除中继日志不会阻塞SQL线程。 save_binary_logs保存和复制master的二进制日志。
安装MHA Manager
下载地址https://github.com/yoshinorim/mha4mysql-manager/wiki/Downloads 在hdp1上用root用户执行下面的操作。
rpm -ivh mha4mysql-manager-0.56-0.el6.noarch.rpm安装报错
[root10-31-1-119 src]# rpm -ivh mha4mysql-manager-0.56-0.el6.noarch.rpm
错误依赖检测失败mha4mysql-node 0.54 被 mha4mysql-manager-0.56-0.el6.noarch 需要perl(MHA::BinlogManager) 被 mha4mysql-manager-0.56-0.el6.noarch 需要perl(MHA::NodeConst) 被 mha4mysql-manager-0.56-0.el6.noarch 需要perl(MHA::NodeUtil) 被 mha4mysql-manager-0.56-0.el6.noarch 需要perl(MHA::SlaveUtil) 被 mha4mysql-manager-0.56-0.el6.noarch 需要
[root10-31-1-119 src]#
[root10-31-1-119 src]#
[root10-31-1-119 src]# yum -y install mha4mysql-manager-0.56-0.el6.noarch.rpm
已加载插件fastestmirror, langpacks
正在检查 mha4mysql-manager-0.56-0.el6.noarch.rpm: mha4mysql-manager-0.56-0.el6.noarch
mha4mysql-manager-0.56-0.el6.noarch.rpm 将被安装
正在解决依赖关系
-- 正在检查事务
--- 软件包 mha4mysql-manager.noarch.0.0.56-0.el6 将被 安装
-- 正在处理依赖关系 mha4mysql-node 0.54它被软件包 mha4mysql-manager-0.56-0.el6.noarch 需要
Loading mirror speeds from cached hostfile
-- 正在处理依赖关系 perl(MHA::BinlogManager)它被软件包 mha4mysql-manager-0.56-0.el6.noarch 需要
-- 正在处理依赖关系 perl(MHA::NodeConst)它被软件包 mha4mysql-manager-0.56-0.el6.noarch 需要
-- 正在处理依赖关系 perl(MHA::NodeUtil)它被软件包 mha4mysql-manager-0.56-0.el6.noarch 需要
-- 正在处理依赖关系 perl(MHA::SlaveUtil)它被软件包 mha4mysql-manager-0.56-0.el6.noarch 需要
-- 解决依赖关系完成
错误软件包mha4mysql-manager-0.56-0.el6.noarch (/mha4mysql-manager-0.56-0.el6.noarch)需要mha4mysql-node 0.54
错误软件包mha4mysql-manager-0.56-0.el6.noarch (/mha4mysql-manager-0.56-0.el6.noarch)需要perl(MHA::NodeUtil)
错误软件包mha4mysql-manager-0.56-0.el6.noarch (/mha4mysql-manager-0.56-0.el6.noarch)需要perl(MHA::BinlogManager)
错误软件包mha4mysql-manager-0.56-0.el6.noarch (/mha4mysql-manager-0.56-0.el6.noarch)需要perl(MHA::NodeConst)
错误软件包mha4mysql-manager-0.56-0.el6.noarch (/mha4mysql-manager-0.56-0.el6.noarch)需要perl(MHA::SlaveUtil)您可以尝试添加 --skip-broken 选项来解决该问题您可以尝试执行rpm -Va --nofiles --nodigest
[root10-31-1-119 src]# 监控节点需要先安装node再安装manager
[root10-31-1-119 src]# rpm -ivh mha4mysql-node-0.56-0.el6.noarch.rpm
准备中... ################################# [100%]
正在升级/安装...1:mha4mysql-node-0.56-0.el6 ################################# [100%]
[root10-31-1-119 src]#
[root10-31-1-119 src]# rpm -ivh mha4mysql-manager-0.56-0.el6.noarch.rpm
准备中... ################################# [100%]
正在升级/安装...1:mha4mysql-manager-0.56-0.el6 ################################# [100%]
[root10-31-1-119 src]# 安装完成后在/usr/bin/目录下有如下MHA相关文件
masterha_check_repl
masterha_check_ssh
masterha_check_status
masterha_conf_host
masterha_manager
masterha_master_monitor
masterha_master_switch
masterha_secondary_check
masterha_stop
apply_diff_relay_logs
filter_mysqlbinlog
purge_relay_logs
save_binary_logs配置MHA
在hdp1上用root用户执行下面1、2、3的操作。 1建立配置文件目录
mkdir -p /etc/masterha2创建配置文件/etc/masterha/app1.cnf内容如下
[server default]
manager_log/var/log/masterha/app1/manager.log
manager_workdir/var/log/masterha/app1.log
master_binlog_dir/var/lib/mysql
master_ip_failover_script/usr/bin/master_ip_failover
master_ip_online_change_script/usr/bin/master_ip_online_change
passwordabc123
ping_interval1
remote_workdir/tmp
repl_passwordabc123
repl_userrepl
secondary_check_script/usr/bin/masterha_secondary_check -s hp2 -s hp3 --userroot --master_hosthp2 --master_ip10.31.1.120 --master_port3306
shutdown_script
ssh_userroot
userroot[server1]
hostname10.31.1.120
port3306[server2]
candidate_master1
check_repl_delay0
hostname10.31.1.121
port3306[server3]
hostname10.31.1.122
port3306server default段是manager的一些基本配置参数server1、server2、server3分别对应复制中的master、第一个slave、第二个slave。该文件的语法要求严格变量值后不要有多余的空格。主要配置项说明如下。
manager_log设置manager的日志文件。 manager_workdir设置manager的工作目录。 master_binlog_dir设置master保存binlog的位置以便MHA可以找到master的日志这里的也就是mysql的数据目录。 master_ip_failover_script设置自动failover时候的切换脚本。 master_ip_online_change_script设置手动切换时候的切换脚本。 password设置mysql中root用户的密码。 ping_interval设置监控主库发送ping包的时间间隔默认是3秒尝试三次没有回应的时候自动进行railover。 remote_workdir设置远端mysql在发生切换时binlog的保存位置。 repl_password设置复制用户的密码。 repl_user设置复制环境中的复制用户名 secondary_check_script一旦MHA到hdp4的监控之间出现问题MHA Manager将会尝试从hdp3登录到hdp4。 shutdown_script设置故障发生后关闭故障主机脚本。该脚本的主要作用是关闭主机放在发生脑裂这里没有使用。 ssh_user设置ssh的登录用户名。 user设置监控用户为root。 candidate_master设置为候选master。设置该参数以后发生主从切换以后将会将此从库提升为主库即使这个主库不是集群中事件最新的slave。 check_repl_delay默认情况下如果一个slave落后master 100M的relay logs的话MHA将不会选择该slave作为一个新的master因为对于这个slave的恢复需要花费很长时间通过设置check_repl_delay0MHA触发切换在选择一个新的master的时候将会忽略复制延时这个参数对于设置了candidate_master1的主机非常有用因为这个候选主在切换的过程中一定是新的master。 3建立软连接
-- 这一步如果 mysqlbinlog和mysql已经可以直接使用就没问题
ln -s /home/mysql/mysql-5.6.14/bin/mysqlbinlog /usr/bin/mysqlbinlog
ln -s /home/mysql/mysql-5.6.14/bin/mysql /usr/bin/mysql4设置复制中Slave的relay_log_purge参数 在hdp3和hdp4上用mysql用户执行
mysql -uroot -pabc123 -e set global relay_log_purge0注意MHA在发生切换的过程中从库的恢复过程中依赖于relay log的相关信息所以这里要将relay log的自动清除设置为OFF采用手动清除relay log的方式。默认情况下从服务器上的中继日志会在SQL线程执行完毕后被自动删除。但是在MHA环境中这些中继日志在恢复其他从服务器时可能会被用到因此需要禁用中继日志的自动删除功能。定期清除中继日志需要考虑到复制延时的问题。在ext3的文件系统下删除大的文件需要一定的时间会导致严重的复制延时。为了避免复制延时需要暂时为中继日志创建硬链接因为在linux系统中通过硬链接删除大文件速度会很快。在mysql数据库中删除大表时通常也采用建立硬链接的方式 创建相关脚本
创建定期清理relay脚本
在hdp3、hdp4两台slave上建立/root/purge_relay_log.sh文件内容如下
#!/bin/bash. /home/mysql/.bashrcuserroot
passwdabc123
port3306
log_dir/data
work_dir/data
purge/usr/bin/purge_relay_logsif [ ! -d $log_dir ]
thenmkdir $log_dir -p
fi$purge --user$user --password$passwd --disable_relay_log_purge --port$port --workdir$work_dir $log_dir/purge_relay_logs.log 21purge_relay_logs的参数说明
user mysqlMySQL用户名。 password mysqlMySQL用户密码。 portMySQL端口号。 workdir指定创建relay log的硬链接的位置默认是/var/tmp。由于系统不同分区创建硬链接文件会失败故需要执行硬链接具体位置成功执行脚本后硬链接的中继日志文件被删除。 disable_relay_log_purge默认情况下如果relay_log_purge1脚本会什么都不清理自动退出。通过设定这个参数当relay_log_purge1的情况下会将relay_log_purge设置为0。清理relay log之后最后将参数设置为OFF。 改模式为可执行
mkdir -p /data
chmod 755 purge_relay_log.sh添加到crontab中
0 4 * * * /bin/bash /root/purge_relay_log.sh创建自动failover脚本
在hdp1上创建/usr/bin/master_ip_failover文件内容如下
#!/usr/bin/env perl
use strict;
use warnings FATAL all;use Getopt::Long;my ($command, $ssh_user, $orig_master_host, $orig_master_ip,$orig_master_port, $new_master_host, $new_master_ip, $new_master_port
);my $vip 10.31.1.241;
my $key 1;
my $ssh_start_vip /sbin/ifconfig ens192:$key $vip/24;
my $ssh_stop_vip /sbin/ifconfig ens192:$key down;GetOptions(commands \$command,ssh_users \$ssh_user,orig_master_hosts \$orig_master_host,orig_master_ips \$orig_master_ip,orig_master_porti \$orig_master_port,new_master_hosts \$new_master_host,new_master_ips \$new_master_ip,new_master_porti \$new_master_port,
);exit main();sub main {print \n\nIN SCRIPT TEST$ssh_stop_vip$ssh_start_vip\n\n;if ( $command eq stop || $command eq stopssh ) {my $exit_code 1;eval {print Disabling the VIP on old master: $orig_master_host \n;stop_vip();$exit_code 0;};if ($) {warn Got Error: $\n;exit $exit_code;}exit $exit_code;}elsif ( $command eq start ) {my $exit_code 10;eval {print Enabling the VIP - $vip on the new master - $new_master_host \n;start_vip();$exit_code 0;};if ($) {warn $;exit $exit_code;}exit $exit_code;}elsif ( $command eq status ) {print Checking the Status of the script.. OK \n;exit 0;}else {usage();exit 1;}
}
sub start_vip() {ssh $ssh_user\$new_master_host \ $ssh_start_vip \;
}
# A simple system call that disable the VIP on the old_master
sub stop_vip() {ssh $ssh_user\$orig_master_host \ $ssh_stop_vip \;
}sub usage {printUsage: master_ip_failover --commandstart|stop|stopssh|status --orig_master_hosthost --orig_master_ipip --orig_master_portport --new_master_hosthost --new_master_ipip --new_master_portport\n;
}注意脚本中的vip部分
chmod ux /usr/bin/master_ip_failover创建手动failover脚本
在hdp1上创建/usr/bin/master_ip_online_change文件内容如下
#!/usr/bin/env perl## Note: This is a sample script and is notcomplete. Modify the script based on your environment.use strict;
use warnings FATAL all;use Getopt::Long;
use MHA::DBHelper;
use MHA::NodeUtil;
# use Time::HiRes qw( sleep gettimeofdaytv_interval );
use Time::HiRes qw(sleep gettimeofday tv_interval);
use Data::Dumper;my $_tstart;
my $_running_interval 0.1;
my ($command, $orig_master_host, $orig_master_ip,$orig_master_port, $orig_master_user,$new_master_host, $new_master_ip, $new_master_port,$new_master_user,
);my $vip 10.31.1.241; # Virtual IP
my $key 1;
my $ssh_start_vip /sbin/ifconfig ens32:$key $vip/24;
my $ssh_stop_vip /sbin/ifconfig ens160:$key down;
my $ssh_user root;
my $new_master_password 123456;
my $orig_master_password 123456;GetOptions(commands \$command,#ssh_users \$ssh_user, orig_master_hosts \$orig_master_host,orig_master_ips \$orig_master_ip,orig_master_porti \$orig_master_port,orig_master_users \$orig_master_user,#orig_master_passwords \$orig_master_password,new_master_hosts \$new_master_host,new_master_ips \$new_master_ip,new_master_porti \$new_master_port,new_master_users \$new_master_user,#new_master_passwords \$new_master_password,
);exit main();sub current_time_us {my ($sec, $microsec ) gettimeofday();my$curdate localtime($sec);return $curdate . . sprintf( %06d, $microsec);
}sub sleep_until {my$elapsed tv_interval($_tstart);if ($_running_interval $elapsed ) {sleep( $_running_interval - $elapsed );}
}sub get_threads_util {my$dbh shift;my$my_connection_id shift;my$running_time_threshold shift;my$type shift;$running_time_threshold 0 unless ($running_time_threshold);$type 0 unless($type);mythreads;my$sth $dbh-prepare(SHOW PROCESSLIST);$sth-execute();while ( my $ref $sth-fetchrow_hashref() ) {my$id $ref-{Id};my$user $ref-{User};my$host $ref-{Host};my$command $ref-{Command};my$state $ref-{State};my$query_time $ref-{Time};my$info $ref-{Info};$info ~ s/^\s*(.*?)\s*$/$1/ if defined($info);next if ( $my_connection_id $id );next if ( defined($query_time) $query_time $running_time_threshold );next if ( defined($command) $command eq Binlog Dump );next if ( defined($user) $user eq system user );nextif ( defined($command) $command eq Sleep defined($query_time) $query_time 1 );if( $type 1 ) {next if ( defined($command) $command eq Sleep );nextif ( defined($command) $command eq Connect );}if( $type 2 ) {next if ( defined($info) $info ~ m/^select/i );next if ( defined($info) $info ~ m/^show/i );}push threads, $ref;}return threads;
}sub main {if ($command eq stop ) {##Gracefully killing connections on the current master#1. Set read_only 1 on the new master#2. DROP USER so that no app user can establish new connections#3. Set read_only 1 on the current master#4. Kill current queries#* Any database access failure will result in script die.my$exit_code 1;eval {## Setting read_only1 on the new master (to avoid accident)my $new_master_handler new MHA::DBHelper();# args: hostname, port, user, password, raise_error(die_on_error)_or_not$new_master_handler-connect( $new_master_ip, $new_master_port,$new_master_user, $new_master_password, 1 );print current_time_us() . Set read_only on the new master..;$new_master_handler-enable_read_only();if ( $new_master_handler-is_read_only() ) {print ok.\n;}else {die Failed!\n;}$new_master_handler-disconnect();# Connecting to the orig master, die if any database error happensmy $orig_master_handler new MHA::DBHelper();$orig_master_handler-connect( $orig_master_ip, $orig_master_port,$orig_master_user, $orig_master_password, 1 );## Drop application user so that nobodycan connect. Disabling per-session binlog beforehand#$orig_master_handler-disable_log_bin_local();#print current_time_us() . Drpping app user on the origmaster..\n;#FIXME_xxx_drop_app_user($orig_master_handler);## Waiting for N * 100 milliseconds so that current connections can exitmy $time_until_read_only 15;$_tstart [gettimeofday];my threads get_threads_util( $orig_master_handler-{dbh},$orig_master_handler-{connection_id} );while ( $time_until_read_only 0 $#threads 0 ) {if ( $time_until_read_only % 5 0 ) {printf %s Waiting all running %d threads aredisconnected.. (max %d milliseconds)\n,current_time_us(), $#threads 1, $time_until_read_only * 100;if ( $#threads 5 ) {print Data::Dumper-new( [$_] )-Indent(0)-Terse(1)-Dump .\nforeach (threads);}}sleep_until();$_tstart [gettimeofday];$time_until_read_only--;threads get_threads_util( $orig_master_handler-{dbh},$orig_master_handler-{connection_id} );}## Setting read_only1 on the current master so that nobody(exceptSUPER) can writeprint current_time_us() . Set read_only1 on the orig master..;$orig_master_handler-enable_read_only();if ( $orig_master_handler-is_read_only() ) {print ok.\n;}else {die Failed!\n;}## Waiting for M * 100 milliseconds so that current update queries cancompletemy $time_until_kill_threads 5;threads get_threads_util( $orig_master_handler-{dbh},$orig_master_handler-{connection_id} );while ( $time_until_kill_threads 0 $#threads 0 ) {if ( $time_until_kill_threads % 5 0 ) {printf %s Waiting all running %d queries aredisconnected.. (max %d milliseconds)\n,current_time_us(), $#threads 1, $time_until_kill_threads * 100;if ( $#threads 5 ) {print Data::Dumper-new( [$_] )-Indent(0)-Terse(1)-Dump .\nforeach (threads);}}sleep_until();$_tstart [gettimeofday];$time_until_kill_threads--;threads get_threads_util( $orig_master_handler-{dbh},$orig_master_handler-{connection_id} );}print Disabling the VIPon old master: $orig_master_host \n;stop_vip(); ## Terminating all threadsprint current_time_us() . Killing all applicationthreads..\n;$orig_master_handler-kill_threads(threads) if ( $#threads 0);print current_time_us() . done.\n;#$orig_master_handler-enable_log_bin_local();$orig_master_handler-disconnect();## After finishing the script, MHA executes FLUSH TABLES WITH READ LOCK$exit_code 0;};if($) {warn Got Error: $\n;exit $exit_code;}exit $exit_code;}elsif ( $command eq start ) {##Activating master ip on the new master#1. Create app user with write privileges#2. Moving backup script if needed#3. Register new masters ip to the catalog database# We dont return error even thoughactivating updatable accounts/ip failed so that we dont interrupt slavesrecovery.
# If exit code is 0 or 10, MHA does notabortmy$exit_code 10;eval{my $new_master_handler new MHA::DBHelper();# args: hostname, port, user, password, raise_error_or_not$new_master_handler-connect( $new_master_ip, $new_master_port,$new_master_user, $new_master_password, 1 );## Set read_only0 on the new master#$new_master_handler-disable_log_bin_local();print current_time_us() . Set read_only0 on the newmaster.\n;$new_master_handler-disable_read_only();## Creating an app user on the new master#print current_time_us() . Creating app user on the newmaster..\n;#FIXME_xxx_create_app_user($new_master_handler);#$new_master_handler-enable_log_bin_local();$new_master_handler-disconnect();## Update master ip on the catalog database, etcprint Enabling the VIP -$vip on the new master - $new_master_host \n;start_vip();$exit_code 0;};if($) {warn Got Error: $\n;exit $exit_code;}exit $exit_code;}elsif ( $command eq status ) {#do nothingexit 0;}else{usage();exit 1;}
}# A simple system call that enable the VIPon the new master
sub start_vip() {ssh $ssh_user\$new_master_host \ $ssh_start_vip \;
}
# A simple system call that disable the VIPon the old_master
sub stop_vip() {ssh $ssh_user\$orig_master_host \ $ssh_stop_vip \;
}sub usage {print
Usage: master_ip_online_change --commandstart|stop|status--orig_master_hosthost --orig_master_ipip --orig_master_portport--new_master_hosthost --new_master_ipip --new_master_portport\n;die;
}授权
chmod ux /usr/bin/master_ip_online_change检查MHA配置
检查SSH配置
在hdp1上用root用户操作。
[root10-31-1-119 ~]# masterha_check_ssh --conf/etc/masterha/app1.cnf
Mon Aug 24 17:47:04 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Mon Aug 24 17:47:04 2020 - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Mon Aug 24 17:47:04 2020 - [info] Reading server configuration from /etc/masterha/app1.cnf..
Mon Aug 24 17:47:04 2020 - [info] Starting SSH connection tests..
Mon Aug 24 17:47:06 2020 - [debug]
Mon Aug 24 17:47:04 2020 - [debug] Connecting via SSH from root10.31.1.120(10.31.1.120:22) to root10.31.1.121(10.31.1.121:22)..
Mon Aug 24 17:47:05 2020 - [debug] ok.
Mon Aug 24 17:47:05 2020 - [debug] Connecting via SSH from root10.31.1.120(10.31.1.120:22) to root10.31.1.122(10.31.1.122:22)..
Mon Aug 24 17:47:06 2020 - [debug] ok.
Mon Aug 24 17:47:07 2020 - [debug]
Mon Aug 24 17:47:05 2020 - [debug] Connecting via SSH from root10.31.1.122(10.31.1.122:22) to root10.31.1.120(10.31.1.120:22)..
Mon Aug 24 17:47:06 2020 - [debug] ok.
Mon Aug 24 17:47:06 2020 - [debug] Connecting via SSH from root10.31.1.122(10.31.1.122:22) to root10.31.1.121(10.31.1.121:22)..
Mon Aug 24 17:47:07 2020 - [debug] ok.
Mon Aug 24 17:47:07 2020 - [debug]
Mon Aug 24 17:47:05 2020 - [debug] Connecting via SSH from root10.31.1.121(10.31.1.121:22) to root10.31.1.120(10.31.1.120:22)..
Mon Aug 24 17:47:06 2020 - [debug] ok.
Mon Aug 24 17:47:06 2020 - [debug] Connecting via SSH from root10.31.1.121(10.31.1.121:22) to root10.31.1.122(10.31.1.122:22)..
Mon Aug 24 17:47:06 2020 - [debug] ok.
Mon Aug 24 17:47:07 2020 - [info] All SSH connection tests passed successfully.检查整个复制环境状况
hp2上授权
create user root10.31.1.% identified by abc123;
grant all privileges on *.* to root10.31.1.%;
flush privileges;在hdp1上用root用户操作。
[root10-31-1-119 ~]# masterha_check_repl --conf/etc/masterha/app1.cnf
Mon Aug 24 17:57:22 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Mon Aug 24 17:57:22 2020 - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Mon Aug 24 17:57:22 2020 - [info] Reading server configuration from /etc/masterha/app1.cnf..
Mon Aug 24 17:57:22 2020 - [info] MHA::MasterMonitor version 0.56.
Mon Aug 24 17:57:23 2020 - [info] GTID failover mode 0
Mon Aug 24 17:57:23 2020 - [info] Dead Servers:
Mon Aug 24 17:57:23 2020 - [info] Alive Servers:
Mon Aug 24 17:57:23 2020 - [info] 10.31.1.120(10.31.1.120:3306)
Mon Aug 24 17:57:23 2020 - [info] 10.31.1.121(10.31.1.121:3306)
Mon Aug 24 17:57:23 2020 - [info] 10.31.1.122(10.31.1.122:3306)
Mon Aug 24 17:57:23 2020 - [info] Alive Slaves:
Mon Aug 24 17:57:23 2020 - [info] 10.31.1.121(10.31.1.121:3306) Version5.7.31-log (oldest major version between slaves) log-bin:enabled
Mon Aug 24 17:57:23 2020 - [info] Replicating from 10.31.1.120(10.31.1.120:3306)
Mon Aug 24 17:57:23 2020 - [info] Primary candidate for the new Master (candidate_master is set)
Mon Aug 24 17:57:23 2020 - [info] 10.31.1.122(10.31.1.122:3306) Version5.7.31-log (oldest major version between slaves) log-bin:enabled
Mon Aug 24 17:57:23 2020 - [info] Replicating from 10.31.1.120(10.31.1.120:3306)
Mon Aug 24 17:57:23 2020 - [info] Current Alive Master: 10.31.1.120(10.31.1.120:3306)
Mon Aug 24 17:57:23 2020 - [info] Checking slave configurations..
Mon Aug 24 17:57:23 2020 - [info] read_only1 is not set on slave 10.31.1.121(10.31.1.121:3306).
Mon Aug 24 17:57:23 2020 - [info] read_only1 is not set on slave 10.31.1.122(10.31.1.122:3306).
Mon Aug 24 17:57:23 2020 - [info] Checking replication filtering settings..
Mon Aug 24 17:57:23 2020 - [info] binlog_do_db , binlog_ignore_db
Mon Aug 24 17:57:23 2020 - [info] Replication filtering check ok.
Mon Aug 24 17:57:23 2020 - [info] GTID (with auto-pos) is not supported
Mon Aug 24 17:57:23 2020 - [info] Starting SSH connection tests..
Mon Aug 24 17:57:26 2020 - [info] All SSH connection tests passed successfully.
Mon Aug 24 17:57:26 2020 - [info] Checking MHA Node version..
Mon Aug 24 17:57:27 2020 - [info] Version check ok.
Mon Aug 24 17:57:27 2020 - [info] Checking SSH publickey authentication settings on the current master..
Mon Aug 24 17:57:27 2020 - [info] HealthCheck: SSH to 10.31.1.120 is reachable.
Mon Aug 24 17:57:27 2020 - [info] Master MHA Node version is 0.56.
Mon Aug 24 17:57:27 2020 - [info] Checking recovery script configurations on 10.31.1.120(10.31.1.120:3306)..
Mon Aug 24 17:57:27 2020 - [info] Executing command: save_binary_logs --commandtest --start_pos4 --binlog_dir/var/lib/mysql --output_file/tmp/save_binary_logs_test --manager_version0.56 --start_file10-31-1-120-bin.000001
Mon Aug 24 17:57:27 2020 - [info] Connecting to root10.31.1.120(10.31.1.120:22).. Creating /tmp if not exists.. ok.Checking output directory is accessible or not..ok.Binlog found at /var/lib/mysql, up to 10-31-1-120-bin.000001
Mon Aug 24 17:57:28 2020 - [info] Binlog setting check done.
Mon Aug 24 17:57:28 2020 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Mon Aug 24 17:57:28 2020 - [info] Executing command : apply_diff_relay_logs --commandtest --slave_userroot --slave_host10.31.1.121 --slave_ip10.31.1.121 --slave_port3306 --workdir/tmp --target_version5.7.31-log --manager_version0.56 --relay_log_info/var/lib/mysql/relay-log.info --relay_dir/var/lib/mysql/ --slave_passxxx
Mon Aug 24 17:57:28 2020 - [info] Connecting to root10.31.1.121(10.31.1.121:22).. Checking slave recovery environment settings..Opening /var/lib/mysql/relay-log.info ... ok.Relay log found at /var/lib/mysql, up to 10-31-1-121-relay-bin.000005Temporary relay log file is /var/lib/mysql/10-31-1-121-relay-bin.000005Testing mysql connection and privileges..mysql: [Warning] Using a password on the command line interface can be insecure.done.Testing mysqlbinlog output.. done.Cleaning up test file(s).. done.
Mon Aug 24 17:57:28 2020 - [info] Executing command : apply_diff_relay_logs --commandtest --slave_userroot --slave_host10.31.1.122 --slave_ip10.31.1.122 --slave_port3306 --workdir/tmp --target_version5.7.31-log --manager_version0.56 --relay_log_info/var/lib/mysql/relay-log.info --relay_dir/var/lib/mysql/ --slave_passxxx
Mon Aug 24 17:57:28 2020 - [info] Connecting to root10.31.1.122(10.31.1.122:22).. Checking slave recovery environment settings..Opening /var/lib/mysql/relay-log.info ... ok.Relay log found at /var/lib/mysql, up to 10-31-1-122-relay-bin.000005Temporary relay log file is /var/lib/mysql/10-31-1-122-relay-bin.000005Testing mysql connection and privileges..mysql: [Warning] Using a password on the command line interface can be insecure.done.Testing mysqlbinlog output.. done.Cleaning up test file(s).. done.
Mon Aug 24 17:57:29 2020 - [info] Slaves settings check done.
Mon Aug 24 17:57:29 2020 - [info]
10.31.1.120(10.31.1.120:3306) (current master)--10.31.1.121(10.31.1.121:3306)--10.31.1.122(10.31.1.122:3306)Mon Aug 24 17:57:29 2020 - [info] Checking replication health on 10.31.1.121..
Mon Aug 24 17:57:29 2020 - [info] ok.
Mon Aug 24 17:57:29 2020 - [info] Checking replication health on 10.31.1.122..
Mon Aug 24 17:57:29 2020 - [info] ok.
Mon Aug 24 17:57:29 2020 - [info] Checking master_ip_failover_script status:
Mon Aug 24 17:57:29 2020 - [info] /usr/bin/master_ip_failover --commandstatus --ssh_userroot --orig_master_host10.31.1.120 --orig_master_ip10.31.1.120 --orig_master_port3306 IN SCRIPT TEST/sbin/ifconfig ens192:1 down/sbin/ifconfig ens192:1 10.31.1.241/24Checking the Status of the script.. OK
Mon Aug 24 17:57:29 2020 - [info] OK.
Mon Aug 24 17:57:29 2020 - [warning] shutdown_script is not defined.
Mon Aug 24 17:57:29 2020 - [info] Got exit code 0 (Not master dead).MySQL Replication Health is OK.
[root10-31-1-119 ~]# 检查MHA Manager的状态
在hdp1上用root用户操作。
[root10-31-1-119 ~]# masterha_check_status --conf/etc/masterha/app1.cnf
app1 is stopped(2:NOT_RUNNING).
[root10-31-1-119 ~]# 显示NOT_RUNNING这代表MHA监控没有开启。执行下面的命令后台启动MHA。
mkdir -p /var/log/masterha/app1/
nohup masterha_manager --conf/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover /dev/null /var/log/masterha/app1/manager.log 21 启动参数说明 remove_dead_master_conf该参数代表当发生主从切换后老的主库的ip将会从配置文件中移除。 manger_log日志存放位置。 ignore_last_failover在缺省情况下如果MHA检测到连续发生宕机且两次宕机间隔不足8小时的话则不会进行Failover之所以这样限制是为了避免ping-pong效应。该参数代表忽略上次MHA触发切换产生的文件默认情况下MHA发生切换后会在日志目录也就是上面设置的/data产生app1.failover.complete文件下次再次切换的时候如果发现该目录下存在该文件将不允许触发切换除非在第一次切换后收到删除该文件。为了方便这里设置为–ignore_last_failover。
再次检查MHA Manager的状态
[root10-31-1-119 ~]# masterha_check_status --conf/etc/masterha/app1.cnf
app1 monitoring program is now on initialization phase(10:INITIALIZING_MONITOR). Wait for a while and try checking again.
[root10-31-1-119 ~]# 查看启动日志
在hdp1上用root用户操作。
[root10-31-1-119 ~]# tail -n20 /var/log/masterha/app1/manager.logCleaning up test file(s).. done.
Mon Aug 24 18:06:00 2020 - [info] Slaves settings check done.
Mon Aug 24 18:06:00 2020 - [info]
10.31.1.120(10.31.1.120:3306) (current master)--10.31.1.121(10.31.1.121:3306)--10.31.1.122(10.31.1.122:3306)Mon Aug 24 18:06:00 2020 - [info] Checking master_ip_failover_script status:
Mon Aug 24 18:06:00 2020 - [info] /usr/bin/master_ip_failover --commandstatus --ssh_userroot --orig_master_host10.31.1.120 --orig_master_ip10.31.1.120 --orig_master_port3306 IN SCRIPT TEST/sbin/ifconfig eth1:1 down/sbin/ifconfig eth1:1 10.31.1.241/24Checking the Status of the script.. OK
Mon Aug 24 18:06:00 2020 - [info] OK.
Mon Aug 24 18:06:00 2020 - [warning] shutdown_script is not defined.
Mon Aug 24 18:06:00 2020 - [info] Set master ping interval 1 seconds.
Mon Aug 24 18:06:00 2020 - [info] Set secondary check script: /usr/bin/masterha_secondary_check -s hdp2 -s hdp3 --userroot --master_hosthdp2 --master_ip10.31.1.120 --master_port3306
Mon Aug 24 18:06:00 2020 - [info] Starting ping health check on 10.31.1.120(10.31.1.120:3306)..
Mon Aug 24 18:06:00 2020 - [info] Ping(SELECT) succeeded, waiting until MySQL doesnt respond..功能测试
初始绑定VIP
在hdp2 10.31.1.112master上用root用户执行
/sbin/ifconfig ens192:1 10.31.1.241/24执行记录
[root10-31-1-120 src]# ip addr
1: lo: LOOPBACK,UP,LOWER_UP mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet6 ::1/128 scope host valid_lft forever preferred_lft forever
2: ens192: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc mq state UP group default qlen 1000link/ether 00:0c:29:54:f4:e5 brd ff:ff:ff:ff:ff:ffinet 10.31.1.120/24 brd 10.31.1.255 scope global noprefixroute ens192valid_lft forever preferred_lft foreverinet6 fe80::b3f8:a2f7:5ba0:b68d/64 scope link noprefixroute valid_lft forever preferred_lft forever
3: virbr0: NO-CARRIER,BROADCAST,MULTICAST,UP mtu 1500 qdisc noqueue state DOWN group default qlen 1000link/ether 52:54:00:c4:b9:a5 brd ff:ff:ff:ff:ff:ffinet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0valid_lft forever preferred_lft forever
4: virbr0-nic: BROADCAST,MULTICAST mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000link/ether 52:54:00:c4:b9:a5 brd ff:ff:ff:ff:ff:ff
[root10-31-1-120 src]#
[root10-31-1-120 src]#
[root10-31-1-120 src]# /sbin/ifconfig ens192:1 10.31.1.241/24
[root10-31-1-120 src]#
[root10-31-1-120 src]# ip addr
1: lo: LOOPBACK,UP,LOWER_UP mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet6 ::1/128 scope host valid_lft forever preferred_lft forever
2: ens192: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc mq state UP group default qlen 1000link/ether 00:0c:29:54:f4:e5 brd ff:ff:ff:ff:ff:ffinet 10.31.1.120/24 brd 10.31.1.255 scope global noprefixroute ens192valid_lft forever preferred_lft foreverinet 10.31.1.241/24 brd 10.31.1.255 scope global secondary ens192:1valid_lft forever preferred_lft foreverinet6 fe80::b3f8:a2f7:5ba0:b68d/64 scope link noprefixroute valid_lft forever preferred_lft forever
3: virbr0: NO-CARRIER,BROADCAST,MULTICAST,UP mtu 1500 qdisc noqueue state DOWN group default qlen 1000link/ether 52:54:00:c4:b9:a5 brd ff:ff:ff:ff:ff:ffinet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0valid_lft forever preferred_lft forever
4: virbr0-nic: BROADCAST,MULTICAST mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000link/ether 52:54:00:c4:b9:a5 brd ff:ff:ff:ff:ff:ff
[root10-31-1-120 src]# 也可以手工解除绑定
[root10-31-1-120 keepalived-2.0.15]# ip addr
1: lo: LOOPBACK,UP,LOWER_UP mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet6 ::1/128 scope host valid_lft forever preferred_lft forever
2: ens192: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc mq state UP group default qlen 1000link/ether 00:0c:29:54:f4:e5 brd ff:ff:ff:ff:ff:ffinet 10.31.1.120/24 brd 10.31.1.255 scope global noprefixroute ens192valid_lft forever preferred_lft foreverinet 10.31.1.241/24 brd 10.31.1.255 scope global secondary ens192:1valid_lft forever preferred_lft foreverinet6 fe80::b3f8:a2f7:5ba0:b68d/64 scope link noprefixroute valid_lft forever preferred_lft forever
3: virbr0: NO-CARRIER,BROADCAST,MULTICAST,UP mtu 1500 qdisc noqueue state DOWN group default qlen 1000link/ether 52:54:00:c4:b9:a5 brd ff:ff:ff:ff:ff:ffinet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0valid_lft forever preferred_lft forever
4: virbr0-nic: BROADCAST,MULTICAST mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000link/ether 52:54:00:c4:b9:a5 brd ff:ff:ff:ff:ff:ff
[root10-31-1-120 keepalived-2.0.15]#
[root10-31-1-120 keepalived-2.0.15]#
[root10-31-1-120 keepalived-2.0.15]# ifconfig ens192:10.31.1.241/24 down
SIOCSIFFLAGS: 无法指定被请求的地址
[root10-31-1-120 keepalived-2.0.15]#
[root10-31-1-120 keepalived-2.0.15]#
[root10-31-1-120 keepalived-2.0.15]# ifconfig ens192:1 down
[root10-31-1-120 keepalived-2.0.15]#
[root10-31-1-120 keepalived-2.0.15]# ip addr
1: lo: LOOPBACK,UP,LOWER_UP mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet6 ::1/128 scope host valid_lft forever preferred_lft forever
2: ens192: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc mq state UP group default qlen 1000link/ether 00:0c:29:54:f4:e5 brd ff:ff:ff:ff:ff:ffinet 10.31.1.120/24 brd 10.31.1.255 scope global noprefixroute ens192valid_lft forever preferred_lft foreverinet6 fe80::b3f8:a2f7:5ba0:b68d/64 scope link noprefixroute valid_lft forever preferred_lft forever
3: virbr0: NO-CARRIER,BROADCAST,MULTICAST,UP mtu 1500 qdisc noqueue state DOWN group default qlen 1000link/ether 52:54:00:c4:b9:a5 brd ff:ff:ff:ff:ff:ffinet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0valid_lft forever preferred_lft forever
4: virbr0-nic: BROADCAST,MULTICAST mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000link/ether 52:54:00:c4:b9:a5 brd ff:ff:ff:ff:ff:ff
[root10-31-1-120 keepalived-2.0.15]# 测试自动切换
通过vip登陆mysql
[rootuattest ~]# mysql -uroot -p -h10.31.1.241
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 27
Server version: 5.7.31-log MySQL Community Server (GPL)Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.Type help; or \h for help. Type \c to clear the current input statement.mysql 停止主库hp1的mysql服务器
service mysqld stop
ip addr[root10-31-1-120 src]# service mysqld stop
Redirecting to /bin/systemctl stop mysqld.service
[root10-31-1-120 src]#
[root10-31-1-120 src]#
[root10-31-1-120 src]#
-- 可以看到 10.31.1.241的vip已经不再hp1(master)上了
[root10-31-1-120 src]# ip addr
1: lo: LOOPBACK,UP,LOWER_UP mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet6 ::1/128 scope host valid_lft forever preferred_lft forever
2: ens192: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc mq state UP group default qlen 1000link/ether 00:0c:29:54:f4:e5 brd ff:ff:ff:ff:ff:ffinet 10.31.1.120/24 brd 10.31.1.255 scope global noprefixroute ens192valid_lft forever preferred_lft foreverinet6 fe80::b3f8:a2f7:5ba0:b68d/64 scope link noprefixroute valid_lft forever preferred_lft forever
3: virbr0: NO-CARRIER,BROADCAST,MULTICAST,UP mtu 1500 qdisc noqueue state DOWN group default qlen 1000link/ether 52:54:00:c4:b9:a5 brd ff:ff:ff:ff:ff:ffinet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0valid_lft forever preferred_lft forever
4: virbr0-nic: BROADCAST,MULTICAST mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000link/ether 52:54:00:c4:b9:a5 brd ff:ff:ff:ff:ff:ffhp2和hp3查看
hp2
-- 可以看到 10.31.1.241这个vip飘到hp2上了
[root10-31-1-121 src]# ip addr
1: lo: LOOPBACK,UP,LOWER_UP mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet6 ::1/128 scope host valid_lft forever preferred_lft forever
2: ens192: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc mq state UP group default qlen 1000link/ether 00:0c:29:e1:fe:b4 brd ff:ff:ff:ff:ff:ffinet 10.31.1.121/24 brd 10.31.1.255 scope global noprefixroute ens192valid_lft forever preferred_lft foreverinet 10.31.1.241/24 brd 10.255.255.255 scope global ens192:1valid_lft forever preferred_lft foreverinet6 fe80::9470:f61e:6e0e:48e4/64 scope link tentative noprefixroute dadfailed valid_lft forever preferred_lft foreverinet6 fe80::1aa1:d23e:e28a:fb62/64 scope link tentative noprefixroute dadfailed valid_lft forever preferred_lft foreverinet6 fe80::f974:3f03:d1f0:1672/64 scope link noprefixroute valid_lft forever preferred_lft forever
3: virbr0: NO-CARRIER,BROADCAST,MULTICAST,UP mtu 1500 qdisc noqueue state DOWN group default qlen 1000link/ether 52:54:00:c4:b9:a5 brd ff:ff:ff:ff:ff:ffinet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0valid_lft forever preferred_lft forever
4: virbr0-nic: BROADCAST,MULTICAST mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000link/ether 52:54:00:c4:b9:a5 brd ff:ff:ff:ff:ff:ff-- hp2已经由从库变为主库了
[root10-31-1-121 src]# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 53
Server version: 5.7.31-log MySQL Community Server (GPL)Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.Type help; or \h for help. Type \c to clear the current input statement.mysql
mysql show slave status\G
Empty set (0.00 sec)mysql show master status\G
*************************** 1. row ***************************File: 10-31-1-121-bin.000001Position: 507Binlog_Do_DB: Binlog_Ignore_DB:
Executed_Gtid_Set:
1 row in set (0.00 sec)
hp3
hp3变为了hp2的从库
[root10-31-1-122 bin]# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 48
Server version: 5.7.31-log MySQL Community Server (GPL)Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.Type help; or \h for help. Type \c to clear the current input statement.mysql show slave status\G
*************************** 1. row ***************************Slave_IO_State: Waiting for master to send eventMaster_Host: 10.31.1.121Master_User: replMaster_Port: 3306Connect_Retry: 60Master_Log_File: 10-31-1-121-bin.000001Read_Master_Log_Pos: 507Relay_Log_File: 10-31-1-122-relay-bin.000002Relay_Log_Pos: 679Relay_Master_Log_File: 10-31-1-121-bin.000001Slave_IO_Running: YesSlave_SQL_Running: YesReplicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0Last_Error: Skip_Counter: 0Exec_Master_Log_Pos: 507Relay_Log_Space: 892Until_Condition: NoneUntil_Log_File: Until_Log_Pos: 0Master_SSL_Allowed: NoMaster_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: NoLast_IO_Errno: 0Last_IO_Error: Last_SQL_Errno: 0Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 102Master_UUID: 5ad1f5aa-e2cb-11ea-ba89-000c29e1feb4Master_Info_File: /var/lib/mysql/master.infoSQL_Delay: 0SQL_Remaining_Delay: NULLSlave_SQL_Running_State: Slave has read all relay log; waiting for more updatesMaster_Retry_Count: 86400Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: Executed_Gtid_Set: Auto_Position: 0Replicate_Rewrite_DB: Channel_Name: Master_TLS_Version:
1 row in set (0.00 sec)还原环境
还原数据库复制
-- 在hdp2、hdp3、hdp4上重置master、slave
stop slave;
reset master;
reset slave all;-- 在hdp3、hdp4上重新指向hdp2为master
change master to
master_host10.31.1.120,
master_port3306,
master_userrepl,
master_passwordabc123,
master_log_file10-31-1-120-bin.000001,
master_log_pos154;start slave;
show slave status\G还原VIP绑定
# 在hdp3上用root用户执行
/sbin/ifconfig ens192:1 down# 在hdp2上用root用户执行
/sbin/ifconfig ens192:1 10.31.1.241/24还原配置文件 编辑在hdp1上/etc/masterha/app1.cnf将[server1]段添加回去。
启动MHA Manage
# 在hdp1上用root用户执行
nohup masterha_manager --conf/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover /dev/null /var/log/masterha/app1/manager.log 21 至此环境还原完毕。
测试手工切换
停止MHA Manage
在hp1用root用户操作
masterha_stop --conf/etc/masterha/app1.cnf关闭master
在hdp2上用root用户操作。
service mysqld stop执行手工切换
在hdp1上用root用户操作。
masterha_master_switch --master_statedead --conf/etc/masterha/app1.cnf --dead_master_host10.31.1.120 --dead_master_port3306 --new_master_host10.31.1.121 --new_master_port3306 --ignore_last_failover执行记录
[root10-31-1-119 ~]# masterha_master_switch --master_statedead --conf/etc/masterha/app1.cnf --dead_master_host10.31.1.120 --dead_master_port3306 --new_master_host10.31.1.121 --new_master_port3306 --ignore_last_failover
--dead_master_ipdead_master_ip is not set. Using 10.31.1.120.
Tue Aug 25 16:18:55 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Tue Aug 25 16:18:55 2020 - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Tue Aug 25 16:18:55 2020 - [info] Reading server configuration from /etc/masterha/app1.cnf..
Tue Aug 25 16:18:55 2020 - [info] MHA::MasterFailover version 0.56.
Tue Aug 25 16:18:55 2020 - [info] Starting master failover.
Tue Aug 25 16:18:55 2020 - [info]
Tue Aug 25 16:18:55 2020 - [info] * Phase 1: Configuration Check Phase..
Tue Aug 25 16:18:55 2020 - [info]
Tue Aug 25 16:18:56 2020 - [info] GTID failover mode 0
Tue Aug 25 16:18:56 2020 - [info] Dead Servers:
Tue Aug 25 16:18:56 2020 - [info] 10.31.1.120(10.31.1.120:3306)
Tue Aug 25 16:18:56 2020 - [info] Checking master reachability via MySQL(double check)...
Tue Aug 25 16:18:56 2020 - [info] ok.
Tue Aug 25 16:18:56 2020 - [info] Alive Servers:
Tue Aug 25 16:18:56 2020 - [info] 10.31.1.121(10.31.1.121:3306)
Tue Aug 25 16:18:56 2020 - [info] 10.31.1.122(10.31.1.122:3306)
Tue Aug 25 16:18:56 2020 - [info] Alive Slaves:
Tue Aug 25 16:18:56 2020 - [info] 10.31.1.121(10.31.1.121:3306) Version5.7.31-log (oldest major version between slaves) log-bin:enabled
Tue Aug 25 16:18:56 2020 - [info] Replicating from 10.31.1.120(10.31.1.120:3306)
Tue Aug 25 16:18:56 2020 - [info] Primary candidate for the new Master (candidate_master is set)
Tue Aug 25 16:18:56 2020 - [info] 10.31.1.122(10.31.1.122:3306) Version5.7.31-log (oldest major version between slaves) log-bin:enabled
Tue Aug 25 16:18:56 2020 - [info] Replicating from 10.31.1.120(10.31.1.120:3306)
Master 10.31.1.120(10.31.1.120:3306) is dead. Proceed? (yes/NO): yes
Tue Aug 25 16:19:01 2020 - [info] Starting Non-GTID based failover.
Tue Aug 25 16:19:01 2020 - [info]
Tue Aug 25 16:19:01 2020 - [info] ** Phase 1: Configuration Check Phase completed.
Tue Aug 25 16:19:01 2020 - [info]
Tue Aug 25 16:19:01 2020 - [info] * Phase 2: Dead Master Shutdown Phase..
Tue Aug 25 16:19:01 2020 - [info]
Tue Aug 25 16:19:01 2020 - [info] HealthCheck: SSH to 10.31.1.120 is reachable.
Tue Aug 25 16:19:01 2020 - [info] Forcing shutdown so that applications never connect to the current master..
Tue Aug 25 16:19:01 2020 - [info] Executing master IP deactivation script:
Tue Aug 25 16:19:01 2020 - [info] /usr/bin/master_ip_failover --orig_master_host10.31.1.120 --orig_master_ip10.31.1.120 --orig_master_port3306 --commandstopssh --ssh_userroot IN SCRIPT TEST/sbin/ifconfig ens192:1 down/sbin/ifconfig ens192:1 10.31.1.241/24Disabling the VIP on old master: 10.31.1.120
Tue Aug 25 16:19:02 2020 - [info] done.
Tue Aug 25 16:19:02 2020 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Tue Aug 25 16:19:02 2020 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Tue Aug 25 16:19:02 2020 - [info]
Tue Aug 25 16:19:02 2020 - [info] * Phase 3: Master Recovery Phase..
Tue Aug 25 16:19:02 2020 - [info]
Tue Aug 25 16:19:02 2020 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Tue Aug 25 16:19:02 2020 - [info]
Tue Aug 25 16:19:02 2020 - [info] The latest binary log file/position on all slaves is 10-31-1-120-bin.000001:154
Tue Aug 25 16:19:02 2020 - [info] Latest slaves (Slaves that received relay log files to the latest):
Tue Aug 25 16:19:02 2020 - [info] 10.31.1.121(10.31.1.121:3306) Version5.7.31-log (oldest major version between slaves) log-bin:enabled
Tue Aug 25 16:19:02 2020 - [info] Replicating from 10.31.1.120(10.31.1.120:3306)
Tue Aug 25 16:19:02 2020 - [info] Primary candidate for the new Master (candidate_master is set)
Tue Aug 25 16:19:02 2020 - [info] 10.31.1.122(10.31.1.122:3306) Version5.7.31-log (oldest major version between slaves) log-bin:enabled
Tue Aug 25 16:19:02 2020 - [info] Replicating from 10.31.1.120(10.31.1.120:3306)
Tue Aug 25 16:19:02 2020 - [info] The oldest binary log file/position on all slaves is 10-31-1-120-bin.000001:154
Tue Aug 25 16:19:02 2020 - [info] Oldest slaves:
Tue Aug 25 16:19:02 2020 - [info] 10.31.1.121(10.31.1.121:3306) Version5.7.31-log (oldest major version between slaves) log-bin:enabled
Tue Aug 25 16:19:02 2020 - [info] Replicating from 10.31.1.120(10.31.1.120:3306)
Tue Aug 25 16:19:02 2020 - [info] Primary candidate for the new Master (candidate_master is set)
Tue Aug 25 16:19:02 2020 - [info] 10.31.1.122(10.31.1.122:3306) Version5.7.31-log (oldest major version between slaves) log-bin:enabled
Tue Aug 25 16:19:02 2020 - [info] Replicating from 10.31.1.120(10.31.1.120:3306)
Tue Aug 25 16:19:02 2020 - [info]
Tue Aug 25 16:19:02 2020 - [info] * Phase 3.2: Saving Dead Masters Binlog Phase..
Tue Aug 25 16:19:02 2020 - [info]
Tue Aug 25 16:19:02 2020 - [info] Fetching dead masters binary logs..
Tue Aug 25 16:19:02 2020 - [info] Executing command on the dead master 10.31.1.120(10.31.1.120:3306): save_binary_logs --commandsave --start_file10-31-1-120-bin.000001 --start_pos154 --binlog_dir/var/lib/mysql --output_file/tmp/saved_master_binlog_from_10.31.1.120_3306_20200825161855.binlog --handle_raw_binlog1 --disable_log_bin0 --manager_version0.56Creating /tmp if not exists.. ok.Concat binary/relay logs from 10-31-1-120-bin.000001 pos 154 to 10-31-1-120-bin.000001 EOF into /tmp/saved_master_binlog_from_10.31.1.120_3306_20200825161855.binlog ..Binlog Checksum enabledDumping binlog format description event, from position 0 to 154.. ok.Dumping effective binlog data from /var/lib/mysql/10-31-1-120-bin.000001 position 154 to tail(177).. ok.Binlog Checksum enabledConcat succeeded.
saved_master_binlog_from_10.31.1.120_3306_20200825161855.binlog 100% 177 140.4KB/s 00:00
Tue Aug 25 16:19:03 2020 - [info] scp from root10.31.1.120:/tmp/saved_master_binlog_from_10.31.1.120_3306_20200825161855.binlog to local:/var/log/masterha/app1.log/saved_master_binlog_from_10.31.1.120_3306_20200825161855.binlog succeeded.
Tue Aug 25 16:19:04 2020 - [info] HealthCheck: SSH to 10.31.1.121 is reachable.
Tue Aug 25 16:19:05 2020 - [info] HealthCheck: SSH to 10.31.1.122 is reachable.
Tue Aug 25 16:19:05 2020 - [info]
Tue Aug 25 16:19:05 2020 - [info] * Phase 3.3: Determining New Master Phase..
Tue Aug 25 16:19:05 2020 - [info]
Tue Aug 25 16:19:05 2020 - [info] Finding the latest slave that has all relay logs for recovering other slaves..
Tue Aug 25 16:19:05 2020 - [info] All slaves received relay logs to the same position. No need to resync each other.
Tue Aug 25 16:19:05 2020 - [info] 10.31.1.121 can be new master.
Tue Aug 25 16:19:05 2020 - [info] New master is 10.31.1.121(10.31.1.121:3306)
Tue Aug 25 16:19:05 2020 - [info] Starting master failover..
Tue Aug 25 16:19:05 2020 - [info]
From:
10.31.1.120(10.31.1.120:3306) (current master)--10.31.1.121(10.31.1.121:3306)--10.31.1.122(10.31.1.122:3306)To:
10.31.1.121(10.31.1.121:3306) (new master)--10.31.1.122(10.31.1.122:3306)Starting master switch from 10.31.1.120(10.31.1.120:3306) to 10.31.1.121(10.31.1.121:3306)? (yes/NO): yes
Tue Aug 25 16:19:09 2020 - [info] New master decided manually is 10.31.1.121(10.31.1.121:3306)
Tue Aug 25 16:19:09 2020 - [info]
Tue Aug 25 16:19:09 2020 - [info] * Phase 3.3: New Master Diff Log Generation Phase..
Tue Aug 25 16:19:09 2020 - [info]
Tue Aug 25 16:19:09 2020 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
Tue Aug 25 16:19:09 2020 - [info] Sending binlog..
saved_master_binlog_from_10.31.1.120_3306_20200825161855.binlog 100% 177 176.2KB/s 00:00
Tue Aug 25 16:19:10 2020 - [info] scp from local:/var/log/masterha/app1.log/saved_master_binlog_from_10.31.1.120_3306_20200825161855.binlog to root10.31.1.121:/tmp/saved_master_binlog_from_10.31.1.120_3306_20200825161855.binlog succeeded.
Tue Aug 25 16:19:10 2020 - [info]
Tue Aug 25 16:19:10 2020 - [info] * Phase 3.4: Master Log Apply Phase..
Tue Aug 25 16:19:10 2020 - [info]
Tue Aug 25 16:19:10 2020 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed.
Tue Aug 25 16:19:10 2020 - [info] Starting recovery on 10.31.1.121(10.31.1.121:3306)..
Tue Aug 25 16:19:10 2020 - [info] Generating diffs succeeded.
Tue Aug 25 16:19:10 2020 - [info] Waiting until all relay logs are applied.
Tue Aug 25 16:19:10 2020 - [info] done.
Tue Aug 25 16:19:10 2020 - [info] Getting slave status..
Tue Aug 25 16:19:10 2020 - [info] This slave(10.31.1.121)s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(10-31-1-120-bin.000001:154). No need to recover from Exec_Master_Log_Pos.
Tue Aug 25 16:19:10 2020 - [info] Connecting to the target slave host 10.31.1.121, running recover script..
Tue Aug 25 16:19:10 2020 - [info] Executing command: apply_diff_relay_logs --commandapply --slave_userroot --slave_host10.31.1.121 --slave_ip10.31.1.121 --slave_port3306 --apply_files/tmp/saved_master_binlog_from_10.31.1.120_3306_20200825161855.binlog --workdir/tmp --target_version5.7.31-log --timestamp20200825161855 --handle_raw_binlog1 --disable_log_bin0 --manager_version0.56 --slave_passxxx
Tue Aug 25 16:19:10 2020 - [info]
MySQL client version is 5.7.31. Using --binary-mode.
Applying differential binary/relay log files /tmp/saved_master_binlog_from_10.31.1.120_3306_20200825161855.binlog on 10.31.1.121:3306. This may take long time...
Applying log files succeeded.
Tue Aug 25 16:19:10 2020 - [info] All relay logs were successfully applied.
Tue Aug 25 16:19:10 2020 - [info] Getting new masters binlog name and position..
Tue Aug 25 16:19:10 2020 - [info] 10-31-1-121-bin.000001:154
Tue Aug 25 16:19:10 2020 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST10.31.1.121, MASTER_PORT3306, MASTER_LOG_FILE10-31-1-121-bin.000001, MASTER_LOG_POS154, MASTER_USERrepl, MASTER_PASSWORDxxx;
Tue Aug 25 16:19:10 2020 - [info] Executing master IP activate script:
Tue Aug 25 16:19:10 2020 - [info] /usr/bin/master_ip_failover --commandstart --ssh_userroot --orig_master_host10.31.1.120 --orig_master_ip10.31.1.120 --orig_master_port3306 --new_master_host10.31.1.121 --new_master_ip10.31.1.121 --new_master_port3306 --new_master_userroot --new_master_passwordabc123
Unknown option: new_master_user
Unknown option: new_master_passwordIN SCRIPT TEST/sbin/ifconfig ens192:1 down/sbin/ifconfig ens192:1 10.31.1.241/24Enabling the VIP - 10.31.1.241 on the new master - 10.31.1.121
Tue Aug 25 16:19:10 2020 - [info] OK.
Tue Aug 25 16:19:10 2020 - [info] ** Finished master recovery successfully.
Tue Aug 25 16:19:10 2020 - [info] * Phase 3: Master Recovery Phase completed.
Tue Aug 25 16:19:10 2020 - [info]
Tue Aug 25 16:19:10 2020 - [info] * Phase 4: Slaves Recovery Phase..
Tue Aug 25 16:19:10 2020 - [info]
Tue Aug 25 16:19:10 2020 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..
Tue Aug 25 16:19:10 2020 - [info]
Tue Aug 25 16:19:10 2020 - [info] -- Slave diff file generation on host 10.31.1.122(10.31.1.122:3306) started, pid: 14489. Check tmp log /var/log/masterha/app1.log/10.31.1.122_3306_20200825161855.log if it takes time..
Tue Aug 25 16:19:11 2020 - [info]
Tue Aug 25 16:19:11 2020 - [info] Log messages from 10.31.1.122 ...
Tue Aug 25 16:19:11 2020 - [info]
Tue Aug 25 16:19:10 2020 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
Tue Aug 25 16:19:11 2020 - [info] End of log messages from 10.31.1.122.
Tue Aug 25 16:19:11 2020 - [info] -- 10.31.1.122(10.31.1.122:3306) has the latest relay log events.
Tue Aug 25 16:19:11 2020 - [info] Generating relay diff files from the latest slave succeeded.
Tue Aug 25 16:19:11 2020 - [info]
Tue Aug 25 16:19:11 2020 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase..
Tue Aug 25 16:19:11 2020 - [info]
Tue Aug 25 16:19:11 2020 - [info] -- Slave recovery on host 10.31.1.122(10.31.1.122:3306) started, pid: 14491. Check tmp log /var/log/masterha/app1.log/10.31.1.122_3306_20200825161855.log if it takes time..
saved_master_binlog_from_10.31.1.120_3306_20200825161855.binlog 100% 177 170.8KB/s 00:00
Tue Aug 25 16:19:13 2020 - [info]
Tue Aug 25 16:19:13 2020 - [info] Log messages from 10.31.1.122 ...
Tue Aug 25 16:19:13 2020 - [info]
Tue Aug 25 16:19:11 2020 - [info] Sending binlog..
Tue Aug 25 16:19:12 2020 - [info] scp from local:/var/log/masterha/app1.log/saved_master_binlog_from_10.31.1.120_3306_20200825161855.binlog to root10.31.1.122:/tmp/saved_master_binlog_from_10.31.1.120_3306_20200825161855.binlog succeeded.
Tue Aug 25 16:19:12 2020 - [info] Starting recovery on 10.31.1.122(10.31.1.122:3306)..
Tue Aug 25 16:19:12 2020 - [info] Generating diffs succeeded.
Tue Aug 25 16:19:12 2020 - [info] Waiting until all relay logs are applied.
Tue Aug 25 16:19:12 2020 - [info] done.
Tue Aug 25 16:19:12 2020 - [info] Getting slave status..
Tue Aug 25 16:19:12 2020 - [info] This slave(10.31.1.122)s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(10-31-1-120-bin.000001:154). No need to recover from Exec_Master_Log_Pos.
Tue Aug 25 16:19:12 2020 - [info] Connecting to the target slave host 10.31.1.122, running recover script..
Tue Aug 25 16:19:12 2020 - [info] Executing command: apply_diff_relay_logs --commandapply --slave_userroot --slave_host10.31.1.122 --slave_ip10.31.1.122 --slave_port3306 --apply_files/tmp/saved_master_binlog_from_10.31.1.120_3306_20200825161855.binlog --workdir/tmp --target_version5.7.31-log --timestamp20200825161855 --handle_raw_binlog1 --disable_log_bin0 --manager_version0.56 --slave_passxxx
Tue Aug 25 16:19:13 2020 - [info]
MySQL client version is 5.7.31. Using --binary-mode.
Applying differential binary/relay log files /tmp/saved_master_binlog_from_10.31.1.120_3306_20200825161855.binlog on 10.31.1.122:3306. This may take long time...
Applying log files succeeded.
Tue Aug 25 16:19:13 2020 - [info] All relay logs were successfully applied.
Tue Aug 25 16:19:13 2020 - [info] Resetting slave 10.31.1.122(10.31.1.122:3306) and starting replication from the new master 10.31.1.121(10.31.1.121:3306)..
Tue Aug 25 16:19:13 2020 - [info] Executed CHANGE MASTER.
Tue Aug 25 16:19:13 2020 - [info] Slave started.
Tue Aug 25 16:19:13 2020 - [info] End of log messages from 10.31.1.122.
Tue Aug 25 16:19:13 2020 - [info] -- Slave recovery on host 10.31.1.122(10.31.1.122:3306) succeeded.
Tue Aug 25 16:19:13 2020 - [info] All new slave servers recovered successfully.
Tue Aug 25 16:19:13 2020 - [info]
Tue Aug 25 16:19:13 2020 - [info] * Phase 5: New master cleanup phase..
Tue Aug 25 16:19:13 2020 - [info]
Tue Aug 25 16:19:13 2020 - [info] Resetting slave info on the new master..
Tue Aug 25 16:19:13 2020 - [info] 10.31.1.121: Resetting slave info succeeded.
Tue Aug 25 16:19:13 2020 - [info] Master failover to 10.31.1.121(10.31.1.121:3306) completed successfully.
Tue Aug 25 16:19:13 2020 - [info] ----- Failover Report -----app1: MySQL Master failover 10.31.1.120(10.31.1.120:3306) to 10.31.1.121(10.31.1.121:3306) succeededMaster 10.31.1.120(10.31.1.120:3306) is down!Check MHA Manager logs at 10-31-1-119 for details.Started manual(interactive) failover.
Invalidated master IP address on 10.31.1.120(10.31.1.120:3306)
The latest slave 10.31.1.121(10.31.1.121:3306) has all relay logs for recovery.
Selected 10.31.1.121(10.31.1.121:3306) as a new master.
10.31.1.121(10.31.1.121:3306): OK: Applying all logs succeeded.
10.31.1.121(10.31.1.121:3306): OK: Activated master IP address.
10.31.1.122(10.31.1.122:3306): This host has the latest relay log events.
Generating relay diff files from the latest slave succeeded.
10.31.1.122(10.31.1.122:3306): OK: Applying all logs succeeded. Slave started, replicating from 10.31.1.121(10.31.1.121:3306)
10.31.1.121(10.31.1.121:3306): Resetting slave info succeeded.
Master failover to 10.31.1.121(10.31.1.121:3306) completed successfully.
[root10-31-1-119 ~]# 验证VIP漂移到10.31.1.121
[root10-31-1-121 ~]# ip addr
1: lo: LOOPBACK,UP,LOWER_UP mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet6 ::1/128 scope host valid_lft forever preferred_lft forever
2: ens192: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc mq state UP group default qlen 1000link/ether 00:0c:29:e1:fe:b4 brd ff:ff:ff:ff:ff:ffinet 10.31.1.121/24 brd 10.31.1.255 scope global noprefixroute ens192valid_lft forever preferred_lft foreverinet 10.31.1.241/24 brd 10.31.1.255 scope global secondary ens192:1valid_lft forever preferred_lft foreverinet6 fe80::9470:f61e:6e0e:48e4/64 scope link tentative noprefixroute dadfailed valid_lft forever preferred_lft foreverinet6 fe80::1aa1:d23e:e28a:fb62/64 scope link tentative noprefixroute dadfailed valid_lft forever preferred_lft foreverinet6 fe80::f974:3f03:d1f0:1672/64 scope link noprefixroute valid_lft forever preferred_lft forever
3: virbr0: NO-CARRIER,BROADCAST,MULTICAST,UP mtu 1500 qdisc noqueue state DOWN group default qlen 1000link/ether 52:54:00:c4:b9:a5 brd ff:ff:ff:ff:ff:ffinet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0valid_lft forever preferred_lft forever
4: virbr0-nic: BROADCAST,MULTICAST mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000link/ether 52:54:00:c4:b9:a5 brd ff:ff:ff:ff:ff:ff验证复制关系
[root10-31-1-122 bin]# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 81
Server version: 5.7.31-log MySQL Community Server (GPL)Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.Type help; or \h for help. Type \c to clear the current input statement.mysql show slave status\G
*************************** 1. row ***************************Slave_IO_State: Waiting for master to send eventMaster_Host: 10.31.1.121Master_User: replMaster_Port: 3306Connect_Retry: 60Master_Log_File: 10-31-1-121-bin.000001Read_Master_Log_Pos: 154Relay_Log_File: 10-31-1-122-relay-bin.000002Relay_Log_Pos: 326Relay_Master_Log_File: 10-31-1-121-bin.000001Slave_IO_Running: YesSlave_SQL_Running: YesReplicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0Last_Error: Skip_Counter: 0Exec_Master_Log_Pos: 154Relay_Log_Space: 539Until_Condition: NoneUntil_Log_File: Until_Log_Pos: 0Master_SSL_Allowed: NoMaster_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: NoLast_IO_Errno: 0Last_IO_Error: Last_SQL_Errno: 0Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 102Master_UUID: 5ad1f5aa-e2cb-11ea-ba89-000c29e1feb4Master_Info_File: /var/lib/mysql/master.infoSQL_Delay: 0SQL_Remaining_Delay: NULLSlave_SQL_Running_State: Slave has read all relay log; waiting for more updatesMaster_Retry_Count: 86400Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: Executed_Gtid_Set: Auto_Position: 0Replicate_Rewrite_DB: Channel_Name: Master_TLS_Version:
1 row in set (0.00 sec)mysql 验证客户端VIP访问
[rootuattest ~]# mysql -uroot -p -h10.31.1.241
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 99
Server version: 5.7.31-log MySQL Community Server (GPL)Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.Type help; or \h for help. Type \c to clear the current input statement.mysql show databases;
--------------------
| Database |
--------------------
| information_schema |
| mysql |
| performance_schema |
| sys |
--------------------
4 rows in set (0.00 sec)mysql
mysql 修复宕机的Master
通常情况下自动切换以后原master可能已经废弃掉待原master主机修复后如果数据完整的情况下可能想把原来master重新作为新主库的slave。这时我们可以借助当时自动切换时刻的MHA日志来完成对原master的修复。下面是提取相关日志的命令
grep -i All other slaves should start /var/log/masterha/app1/manager.log可以看到类似下面的信息
All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST10.31.1.120, MASTER_PORT3306, MASTER_LOG_FILE10-31-1-121-bin.000005, MASTER_LOG_POS120, MASTER_USERrepl, MASTER_PASSWORDabc123;意思是说如果Master主机修复好了可以在修复好后的Master执行CHANGE MASTER操作作为新的slave库。
参考
MySQL MHA高可用实战_只是甲的博客-CSDN博客
