• 1. 線上MYSQL同步報錯故障處理總結 賀春旸
  • 2. 在發生故障切換后,經常遇到的問題就是同步報錯,數據庫很小的時候,dump完再導入很簡單就處理好了,但線上的數據庫都150G-200G,如果用單純的這種方法,成本太高,故經過一段時間的摸索,總結了幾種處理方法。 前言
  • 3. 生產環境架構圖 目前現網的架構,保存著兩份數據,通過異步復制做的高可用集群,兩臺機器提供對外服務。在發生故障時,切換到slave上,并將其變成master,壞掉的機器反向同步新的master,在處理故障時,遇到最多的就是主從報錯。下面是我收錄下來的報錯信息。VIP
  • 4. 最常見的3種情況第一種:在master上刪除一條記錄,而slave上找不到。 Last_SQL_Error: Could not execute Delete_rows event on table hcy.t1; Can't find record in 't1', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log mysql-bin.000006, end_log_pos 254
  • 5. 最常見的3種情況第二種:主鍵重復。在slave已經有該記錄,又在master上插入了同一條記錄。 Last_SQL_Error: Could not execute Write_rows event on table hcy.t1; Duplicate entry '2' for key 'PRIMARY', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log mysql-bin.000006, end_log_pos 924
  • 6. 最常見的3種情況這3種情況是在HA切換時,由于是異步復制, 且sync_binlog=0,會造成一小部分binlog沒接收完導致 同步報錯。第三種:在master上更新一條記錄,而slave上找不到,丟失了數據。 Last_SQL_Error: Could not execute Update_rows event on table hcy.t1; Can't find record in 't1', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log mysql-bin.000010, end_log_pos 263
  • 7. 異步復制:簡單的說就是master把binlog發送過去,不管slave是否接收完,也不管是否執行完,這一動作就結束了. 半同步復制:簡單的說就是master把binlog發送過去,slave確認接收完,但不管它是否執行完,給master一個信號我這邊收到了,這一動作就結束了。(谷歌寫的代碼,5.5上正式應用。) 異步的劣勢:當master上寫操作繁忙時,當前POS點例如是10,而slave上IO_THREAD線程接收過來的是3,此時master宕機,會造成相差7個點未傳送到slave上而數據丟失。區別
  • 8. 特殊的情況出現頻率低這種情況SLAVE在宕機,或者非法關機,例如電源故障、 主板燒了等,造成中繼日志損壞,同步停掉。第四種:slave的中繼日志relay-bin損壞。 Last_SQL_Error: Error initializing relay log position: I/O error reading the header from the binary log Last_SQL_Error: Error initializing relay log position: Binlog has bad magic number; It's not a binary log file that can be used by this version of MySQL
  • 9. 人為失誤需謹慎這種情況同步會一直延時,永遠也同步不完,error錯誤日 志里一直出現上面兩行信息。解決方法就是把server-id改 成不一致即可。第五種:多臺slave存在重復server-id。 Slave: received end packet from server, apparent master shutdown: Slave I/O thread: Failed reading log event, reconnecting to retry, log 'mysql-bin.000012' at postion 106
  • 10. 開始處理……
  • 11. 1、在master上刪除一條記錄,而slave上找不到。 Last_SQL_Error: Could not execute Delete_rows event on table hcy.t1; Can't find record in 't1', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log mysql-bin.000006, end_log_pos 254 解決方法:由于master要刪除一條記錄,而slave上找不到故報錯,這種情況主上都將其刪除了,那么從機可以直接跳過??捎妹睿簊top slave ;set global sql_slave_skip_counter=1;start slave; 如果這種情況很多,可用我寫的一個腳本skip_error_replcation.sh,默認跳過10個錯誤(只針對這種情況才跳,其他情況輸出錯誤結果,等待處理),這個腳本是參考maakit工具包的mk-slave-restart原理用shell寫的,功能上定義了一些自己的東西,不是無論什么錯誤都一律跳過。)故障處理
  • 12. 2、主鍵重復。在slave已經有該記錄,又在master上插入了同一條記錄。 Last_SQL_Error: Could not execute Write_rows event on table hcy.t1; Duplicate entry '2' for key 'PRIMARY', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log mysql-bin.000006, end_log_pos 924 解決方法: 在slave上desc hcy.t1; 先看下表結構, 故障處理
  • 13. mysql> desc hcy.t1; +-------+---------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------+---------+------+-----+---------+-------+ | id | int(11) | NO | PRI | 0 | | | name | char(4) | YES | | NULL | | +-------+---------+------+-----+---------+-------+ 2 rows in set (0.03 sec) 刪除重復的主鍵 mysql> delete from t1 where id=2; Query OK, 1 row affected (0.00 sec)故障處理
  • 14. mysql> start slave; Query OK, 0 rows affected (0.00 sec) mysql> show slave status\G; …… Slave_IO_Running: Yes Slave_SQL_Running: Yes …… mysql> select * from t1 where id=2; 在master上和slave上再分別確認一下。故障處理
  • 15. 在master上更新一條記錄,而slave上找不到,丟失了數據。 Last_SQL_Error: Could not execute Update_rows event on table hcy.t1; Can't find record in 't1', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log mysql-bin.000010, end_log_pos 794 解決方法: 在master上,用mysqlbinlog 分析下出錯的binlog日志在干什么。 [[email protected] data]# /usr/local/mysql/bin/mysqlbinlog --no-defaults -v -v --base64-output=DECODE-ROWS mysql-bin.000010 | grep -A '10' 794 故障處理
  • 16. [[email protected] data]# /usr/local/mysql/bin/mysqlbinlog --no-defaults -v -v --base64-output=DECODE-ROWS mysql-bin.000010 | grep -A '10' 794 #120302 12:08:36 server id 22 end_log_pos 794 Update_rows: table id 33 flags: STMT_END_F ### UPDATE hcy.t1 ### WHERE ### @1=2 /* INT meta=0 nullable=0 is_null=0 */ ### @2='bbc' /* STRING(4) meta=65028 nullable=1 is_null=0 */ ### SET ### @1=2 /* INT meta=0 nullable=0 is_null=0 */ ### @2='BTV' /* STRING(4) meta=65028 nullable=1 is_null=0 */ # at 794 #120302 12:08:36 server id 22 end_log_pos 821 Xid = 60 COMMIT/*!*/; DELIMITER ; # End of log file ROLLBACK /* added by mysqlbinlog */; /*!50003 SET [email protected]_COMPLETION_TYPE*/; [[email protected] data]# 故障處理
  • 17. 在slave上,查找下更新后的那條記錄,應該是不存在的。 mysql> select * from t1 where id=2; Empty set (0.00 sec) 然后再到master查看 mysql> select * from t1 where id=2; +----+------+ | id | name | +----+------+ | 2 | BTV | +----+------+ 1 row in set (0.00 sec) 故障處理
  • 18. 把丟失的數據在slave上填補,然后跳過報錯即可。 mysql> insert into t1 values (2,'BTV'); Query OK, 1 row affected (0.00 sec) mysql> select * from t1 where id=2; +----+------+ | id | name | +----+------+ | 2 | BTV | +----+------+ 1 row in set (0.00 sec)故障處理
  • 19. mysql> stop slave ;set global sql_slave_skip_counter=1;start slave; Query OK, 0 rows affected (0.01 sec) Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) mysql> show slave status\G; …… Slave_IO_Running: Yes Slave_SQL_Running: Yes ……故障處理
  • 20. 第四種:slave的中繼日志relay-bin損壞。 Last_SQL_Error: Error initializing relay log position: I/O error reading the header from the binary log Last_SQL_Error: Error initializing relay log position: Binlog has bad magic number; It's not a binary log file that can be used by this version of MySQL 解決方法:找到同步的binlog和POS點,然后重新做同步,這樣就可以有新的中繼日值了。 例子:故障處理
  • 21. mysql> show slave status\G; *************************** 1. row *************************** Master_Log_File: mysql-bin.000010 Read_Master_Log_Pos: 1191 Relay_Log_File: vm02-relay-bin.000005 Relay_Log_Pos: 253 Relay_Master_Log_File: mysql-bin.000010 Slave_IO_Running: Yes Slave_SQL_Running: No Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 1593 Last_Error: Error initializing relay log position: I/O error reading the header from the binary log Skip_Counter: 1 Exec_Master_Log_Pos: 821故障處理
  • 22. Slave_IO_Running :接收master的binlog信息 Master_Log_File Read_Master_Log_Pos Slave_SQL_Running:執行寫操作 Relay_Master_Log_File Exec_Master_Log_Pos 以執行寫的binlog和POS點為準。 Relay_Master_Log_File: mysql-bin.000010 Exec_Master_Log_Pos: 821故障處理
  • 23. mysql> stop slave; Query OK, 0 rows affected (0.01 sec) mysql> CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000010',MASTER_LOG_POS=821; Query OK, 0 rows affected (0.01 sec) mysql> start slave; Query OK, 0 rows affected (0.00 sec)故障處理
  • 24. mysql> show slave status\G; *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: 192.168.8.22 Master_User: repl Master_Port: 3306 Connect_Retry: 10 Master_Log_File: mysql-bin.000010 Read_Master_Log_Pos: 1191 Relay_Log_File: vm02-relay-bin.000002 Relay_Log_Pos: 623 Relay_Master_Log_File: mysql-bin.000010 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: 故障處理
  • 25. Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 1191 Relay_Log_Space: 778 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: 故障處理
  • 26. 故障處理各種大招都用上了,無奈slave數據丟失過多,ibbackup(需要銀子)該你登場了。
  • 27. Ibbackup熱備份工具,是付費的。xtrabackup是免費的,功能上一樣。 Ibbackup備份期間不鎖表,備份時開啟一個事務(相當于做一個快照),然后會記錄一個點,之后數據的更改保存在ibbackup_logfile文件里,恢復時把ibbackup_logfile 變化的數據再寫入到ibdata里。 Ibbackup 只備份數據( ibdata、.ibd ),表結構.frm不備份。 下面一個演示例子:故障處理
  • 28. 備份:ibbackup /bak/etc/my_local.cnf /bak/etc/my_bak.cnf 恢復: ibbackup --apply-log /bak/etc/my_bak.cnf [[email protected] etc]# more my_local.cnf datadir =/usr/local/mysql/data innodb_data_home_dir = /usr/local/mysql/data innodb_data_file_path = ibdata1:10M:autoextend innodb_log_group_home_dir = /usr/local/mysql/data innodb_buffer_pool_size = 100M innodb_log_file_size = 5M innodb_log_files_in_group=2 [[email protected] etc]#故障處理
  • 29. [[email protected] etc]# more my_bak.cnf datadir =/bak/data innodb_data_home_dir = /bak/data innodb_data_file_path = ibdata1:10M:autoextend innodb_log_group_home_dir = /bak/data innodb_buffer_pool_size = 100M innodb_log_file_size = 5M innodb_log_files_in_group=2 [[email protected] etc]#故障處理
  • 30. [[email protected] etc]# ibbackup /bak/etc/my_local.cnf /bak/etc/my_bak.cnf InnoDB Hot Backup version 3.0.0; Copyright 2002-2005 Innobase Oy License A21488 is granted to vm01 ([email protected]) (--apply-log works in any computer regardless of the hostname) Licensed for use in a computer whose hostname is 'vm01' Expires 2012-5-1 (year-month-day) at 00:00 See //www.innodb.com for further information Type ibbackup --license for detailed license terms, --help for help Contents of /bak/etc/my_local.cnf: innodb_data_home_dir got value /usr/local/mysql/data innodb_data_file_path got value ibdata1:10M:autoextend datadir got value /usr/local/mysql/data innodb_log_group_home_dir got value /usr/local/mysql/data innodb_log_files_in_group got value 2 innodb_log_file_size got value 5242880 Contents of /bak/etc/my_bak.cnf: innodb_data_home_dir got value /bak/data innodb_data_file_path got value ibdata1:10M:autoextend故障處理
  • 31. datadir got value /bak/data innodb_log_group_home_dir got value /bak/data innodb_log_files_in_group got value 2 innodb_log_file_size got value 5242880 ibbackup: Found checkpoint at lsn 0 1636898 ibbackup: Starting log scan from lsn 0 1636864 120302 16:47:43 ibbackup: Copying log... 120302 16:47:43 ibbackup: Log copied, lsn 0 1636898 ibbackup: We wait 1 second before starting copying the data files... 120302 16:47:44 ibbackup: Copying /usr/local/mysql/data/ibdata1 ibbackup: A copied database page was modified at 0 1636898 ibbackup: Scanned log up to lsn 0 1636898 ibbackup: Was able to parse the log up to lsn 0 1636898 ibbackup: Maximum page number for a log record 0 120302 16:47:46 ibbackup: Full backup completed! [[email protected] etc]# [[email protected] etc]# cd /bak/data/ [[email protected] data]# ls ibbackup_logfile ibdata1故障處理
  • 32. [[email protected] data]# ibbackup --apply-log /bak/etc/my_bak.cnf InnoDB Hot Backup version 3.0.0; Copyright 2002-2005 Innobase Oy License A21488 is granted to vm01 ([email protected]) (--apply-log works in any computer regardless of the hostname) Licensed for use in a computer whose hostname is 'vm01' Expires 2012-5-1 (year-month-day) at 00:00 See //www.innodb.com for further information Type ibbackup --license for detailed license terms, --help for help Contents of /bak/etc/my_bak.cnf: innodb_data_home_dir got value /bak/data innodb_data_file_path got value ibdata1:10M:autoextend datadir got value /bak/data innodb_log_group_home_dir got value /bak/data innodb_log_files_in_group got value 2 innodb_log_file_size got value 5242880 120302 16:48:38 ibbackup: ibbackup_logfile's creation parameters: ibbackup: start lsn 0 1636864, end lsn 0 1636898, ibbackup: start checkpoint 0 1636898故障處理
  • 33. ibbackup: start checkpoint 0 1636898 InnoDB: Doing recovery: scanned up to log sequence number 0 1636898 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 Setting log file size to 0 5242880 Setting log file size to 0 5242880 ibbackup: We were able to parse ibbackup_logfile up to ibbackup: lsn 0 1636898 ibbackup: Last MySQL binlog file position 0 1191, file name ./mysql-bin.000010 ibbackup: The first data file is '/bak/data/ibdata1' ibbackup: and the new created log files are at '/bak/data/' 120302 16:48:38 ibbackup: Full backup prepared for recovery successfully! [[email protected] data]# ls ibbackup_logfile ibdata1 ib_logfile0 ib_logfile1 把ibdata1 ib_logfile0 ib_logfile1拷貝到從,把.frm也拷貝過去,啟動MYSQL后,做同步,那個點就是給出的紅色字。 CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000010',MASTER_LOG_POS=1191;故障處理
  • 34. //www.maatkit.org/ 簡介:maatkit是一個開源的工具包 包,為mysql日常管理提供了幫助。 目前,已被Percona公司收購并維護。 。其中mk-table-checksum是用來檢測 master和slave上的表結構和數據是 否一致。mk-table-sync是發生主從數據不一致時,來修復的。 這兩個工具包,沒有在現網實際操作的經驗,這里僅僅是新技術探討和學術交流,下面展示下如何使用。 //www.actionsky.com/products/mysql-others/maatkit.jsp Maatkit工具包
  • 35. [[email protected]]# mk-table-checksum h=vm01,u=admin,p=123456 h=vm02,u=admin,p=123456 -d hcy -t t1 Cannot connect to MySQL because the Perl DBI module is not installed or not found. Run 'perl -MDBI' to see the directories that Perl searches for DBI. If DBI is not installed, try: Debian/Ubuntu apt-get install libdbi-perl RHEL/CentOS yum install perl-DBI OpenSolaris pgk install pkg:/SUNWpmdbi [[email protected]]# 提示缺少perl-DBI???,那么直接 yum install perl-DBI。Maatkit工具包
  • 36. [[email protected] bin]# mk-table-checksum h=vm01,u=admin,p=123456 h=vm02,u=admin,p=123456 -d hcy -t t1 DATABASE TABLE CHUNK HOST ENGINE COUNT CHECKSUM TIME WAIT STAT LAG hcy t1 0 vm02 InnoDB NULL 1957752020 0 0 NULL NULL hcy t1 0 vm01 InnoDB NULL 1957752020 0 0 NULL NULL [[email protected] bin]# 如果表數據不一致,CHECKSUM的值是不相等的。Maatkit工具包
  • 37. 解釋下輸出的意思: DATABASE:數據庫名 TABLE:表名 CHUNK:checksum時的近似數值 HOST:MYSQL的地址 ENGINE:表引擎 COUNT:表的行數 CHECKSUM:校驗值 TIME:所用時間 WAIT:等待時間 STAT:MASTER_POS_WAIT()返回值 LAG:slave的延時時間 Maatkit工具包
  • 38. 如果你想過濾出不相等的都有哪些表,可以用mk-checksum-filter這個工具。 只要在后面加個管道符就行了。 [[email protected] ~]# mk-table-checksum h=vm01,u=admin,p=123456 h=vm02,u=admin,p=123456 -d hcy | mk-checksum-filter hcy t2 0 vm01 InnoDB NULL 1957752020 0 0 NULL NULL hcy t2 0 vm02 InnoDB NULL 1068689114 0 0 NULL NULL 知道有哪些表不一致,可以用mk-table-sync這個工具來處理。 注:在執行mk-table-checksum時會鎖表,表的大小取決于執行的快慢。Maatkit工具包
  • 39. MASTER上的t2表數據: SLAVE上的t2表數據: mysql> select * from t2; mysql> select * from t2; +----+------+ +----+------+ | id | name | | id | name | +----+------+ +----+------+ | 1 | a | | 1 | a | | 2 | b | | 2 | b | | 3 | ss | | 3 | ss | | 4 | asd | | 4 | asd | | 5 | ss | +----+------+ +----+------+ 4 rows in set (0.00 sec) 5 rows in set (0.00 sec) mysql> \! hostname; mysql> \! hostname; vm02 vm01 mysql> mysql>Maatkit工具包
  • 40. [[email protected] ~]# mk-table-sync --execute --print --no-check-slave --transaction --databases hcy h=vm01,u=admin,p=123456 h=vm02,u=admin,p=123456 INSERT INTO `hcy`.`t2`(`id`, `name`) VALUES ('5', 'ss') /*maatkit src_db:hcy src_tbl:t2 src_dsn:h=vm01,p=...,u=admin dst_db:hcy dst_tbl:t2 dst_dsn:h=vm02,p=...,u=admin lock:0 transaction:1 changing_src:0 replicate:0 bidirectional:0 pid:3246 user:root host:vm02*/; 它的工作原理是:先一行一行檢查主從庫的表是否一樣,如果哪里不一樣,就執行刪除,更新,插入等操作,使其達到一致。表的大小決定著執行的快慢。 If C<--transaction> is specified, C is not used. Instead, lock and unlock are implemented by beginning and committing transactions. The exception is if L<"--lock"> is 3. If C<--no-transaction> is specified, then C is used for any value of L<"--lock">. See L<"--[no]transaction">. When enabled, either explicitly or implicitly, the transaction isolation level is set C and transactions are started C Maatkit工具包
  • 41. 結束!

篮球比分188直播 www.703192.live 下載文檔到電腦,查找使用更方便

需要 10 金幣 [ 分享文檔獲得金幣 ] 1 人已下載

下載文檔

{ganrao} 上海十一选五任二遗漏 黑龙江快乐十分投注金额 指数投资网 黑龙江省福彩36选7的中奖号 彩票走势图上海11选分布 四川金7乐结果 佳永配资_股票配资平台_10倍杠杆配资 15选5推荐号码预测 陕西福彩快乐十分遗漏查询 股票配资平台ˉ选杨方配资靠谱 手机重庆时时开奖视频 黑龙江p62开奖 炒股用哪个app 股票指数如何计算 辽宁35选7图标 黑龙江快乐十分台子