Mysql service down with [ERROR] InnoDB: Page log sequence number is in the future



  • I have a Centos 7.9 server with MariaDB version 10.2.43. All of a sudden on May 2 2022, I have started receiving mysql down alerts. On checking I could see below log information in logs.

    Error 2:- May  2 22:09:03 server1 mysqld: 2022-05-02 22:09:03 140534819583744 [ERROR] InnoDB: Page [page id: space=57618, page number=185] log sequence number 6629091300064 is in the future! Current system log sequence number 6629091254594.
    May  2 22:09:03 server1 mysqld: 2022-05-02 22:09:03 140534819583744 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
    

    I have managed to recover the data using innodb_force_recovery to the value of 5, restore the databases from backups/dump. I have some doubts regarding the issue. I have checked entire log information for Mariadb and I could see below log entry repeatedly logged days before the InnoDB Crash.

    [ERROR] mysqld: Got error 'Could not get an exclusive lock; file is probably in use by another process' when trying to use aria control file '/var/lib/mysql/aria_log_control'
    

    I wanted to understand / Clarify few points regarding this issue.

    1. Is the aria_log_control error a critical error and is it connected to InnoDB Crash? Why this error?

    2. How come the Page log sequence number is in the future for Mariadb service? Is it related to some kind of disk corruption or memory issue?

    3. How to monitor this on production environment and prevent the same?

    4. Is this a bug with maridb 10.2.43? Will updating to latest stable 10.6 version will fix it?



  • This issue will usually happen after one of the following changes:

    • If the InnoDB log files were deleted and one or more of these events preceded that deletion (none of the events can on their own cause the error

    • it takes both one of the following events and deleting the log files):

    The hardware server crashed.

    The mysqld process was terminated with kill -9 or another reason.

    The data was restored from a hot backup.

    MySQL was shut down using while innodb_fast_shutdown = 2.

    • MySQL was restored from a backup where the InnoDB log files did not exist or is from a different backup.

    • MySQL was restored from an inconsistent backup.

    • MySQL was started with innodb_force_recovery = 6.

    Please visit the oracle official guide for this error.

    https://support.oracle.com/knowledge/Oracle%20Database%20Products/1416063_1.html#aref_section41




Suggested Topics

  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2