Very slow large stored procedure



  • I have a rather large stored procedure that processes over a hundred thousand records. This SP has been running beautifully until last week when the database server was hit with a ransomware attack. By a HUGE stroke of luck I took an offsite backup of the entire database the night before for testing. Historically the SP ran in under 3 minutes.

    We have a new server, same OS, same MySQL version(5.7) same processors and memory(It's a VM) as the corrupted server. The only thing that was lost is the my.ini. IT didn't have the database server in their backup plan. IT found an older copy of the my.ini file which I ran with. The stored proc in now EXTREMELY slow, like a calculated 30 hours to run rather than 3 minutes. Barring something wrong with the VM setup I'm doubting the my.ini I was given.

    Pertinent server specs(Where have I gone wrong?):

    • OS: Windows server 2012 R2 64 Bit

    • Available RAM: 9.41G

    • Available Virtual Memory: 6.67 GB

    • Processor:Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz, 2098 Mhz, 1 Core(s), 1 Logical Processor(s)

    • Page File Space: 1.81 GB

    my.ini:

    # Other default tuning values
    # MySQL Server Instance Configuration File
    # ----------------------------------------------------------------------
    # Generated by the MySQL Server Instance Configuration Wizard
    #
    #
    # Installation Instructions
    # ----------------------------------------------------------------------
    #
    # On Linux you can copy this file to /etc/my.cnf to set global options,
    # mysql-data-dir/my.cnf to set server-specific options
    # (@localstatedir@ for this installation) or to
    # ~/.my.cnf to set user-specific options.
    #
    # On Windows you should keep this file in the installation directory 
    # of your server (e.g. C:\Program Files\MySQL\MySQL Server X.Y). To
    # make sure the server reads the config file use the startup option 
    # "--defaults-file". 
    #
    # To run run the server from the command line, execute this in a 
    # command line shell, e.g.
    # mysqld --defaults-file="C:\Program Files\MySQL\MySQL Server X.Y\my.ini"
    #
    # To install the server as a Windows service manually, execute this in a 
    # command line shell, e.g.
    # mysqld --install MySQLXY --defaults-file="C:\Program Files\MySQL\MySQL Server X.Y\my.ini"
    #
    # And then execute this in a command line shell to start the server, e.g.
    # net start MySQLXY
    #
    #
    # Guildlines for editing this file
    # ----------------------------------------------------------------------
    #
    # In this file, you can use all long options that the program supports.
    # If you want to know the options a program supports, start the program
    # with the "--help" option.
    #
    # More detailed information about the individual options can also be
    # found in the manual.
    #
    # For advice on how to change settings please see
    # http://dev.mysql.com/doc/refman/5.7/en/server-configuration-defaults.html
    #
    #
    # CLIENT SECTION
    # ----------------------------------------------------------------------
    #
    # The following options will be read by MySQL client applications.
    # Note that only client applications shipped by MySQL are guaranteed
    # to read this section. If you want your own MySQL client program to
    # honor these values, you need to specify it as an option during the
    # MySQL client library initialization.
    #
    [client]
    

    pipe

    socket=0.0

    port=3306

    [mysql]
    no-beep

    default-character-set=utf8

    SERVER SECTION

    ----------------------------------------------------------------------

    The following options will be read by the MySQL Server. Make sure that

    you have installed the server correctly (see above) so it reads this

    file.

    server_type=2

    [mysqld]

    The next three options are mutually exclusive to SERVER_PORT below.

    skip-networking

    enable-named-pipe

    shared-memory

    shared-memory-base-name=MYSQL

    The Pipe the MySQL Server will use

    socket=MYSQL

    The TCP/IP Port the MySQL Server will listen on

    port=3306

    wait_timeout=2147483

    Path to installation directory. All paths are usually resolved relative to this.

    basedir="C:/Program Files/MySQL/MySQL Server 5.7/"

    Path to the database root

    datadir=C:/ProgramData/MySQL/MySQL Server 5.7/Data

    The default character set that will be used when a new schema or table is

    created and no character set is defined

    character-set-server=utf8

    The default storage engine that will be used when create new tables when

    default-storage-engine=INNODB

    Set the SQL mode to strict

    sql-mode="STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"

    * Query Cache Configuration

    query_cache_limit = 2M
    query_cache_min_res_unit = 512k
    query_cache_size = 200M

    General and Slow logging.

    log-output=FILE
    general-log=0
    general_log_file="PUPIL-PBP.log"
    slow-query-log=1
    slow_query_log_file="PUPIL-PBP-slow.log"
    long_query_time=10

    Binary Logging.

    log-bin

    Error Logging.

    log-error="PUPIL-PBP.err"

    Server Id.

    server-id=1

    Secure File Priv.

    secure-file-priv="C:/ProgramData/MySQL/MySQL Server 5.7/Uploads"

    The maximum amount of concurrent sessions the MySQL server will

    allow. One of these connections will be reserved for a user with

    SUPER privileges to allow the administrator to login even if the

    connection limit has been reached.

    max_connections=151

    The number of open tables for all threads. Increasing this value

    increases the number of file descriptors that mysqld requires.

    Therefore you have to make sure to set the amount of open files

    allowed to at least 4096 in the variable "open-files-limit" in

    section [mysqld_safe]

    table_open_cache=2000

    Maximum size for internal (in-memory) temporary tables. If a table

    grows larger than this value, it is automatically converted to disk

    based table This limitation is for a single table. There can be many

    of them.

    tmp_table_size=512M

    How many threads we should keep in a cache for reuse. When a client

    disconnects, the client's threads are put in the cache if there aren't

    more than thread_cache_size threads from before. This greatly reduces

    the amount of thread creations needed if you have a lot of new

    connections. (Normally this doesn't give a notable performance

    improvement if you have a good thread implementation.)

    thread_stack = 192K
    thread_cache_size = 100

    #*** MyISAM Specific options

    The maximum size of the temporary file MySQL is allowed to use while

    recreating the index (during REPAIR, ALTER TABLE or LOAD DATA INFILE.

    If the file-size would be bigger than this, the index will be created

    through the key cache (which is slower).

    myisam_max_sort_file_size=100G

    If the temporary file used for fast index creation would be bigger

    than using the key cache by the amount specified here, then prefer the

    key cache method. This is mainly used to force long character keys in

    large tables to use the slower key cache method to create the index.

    myisam_sort_buffer_size=8M

    Size of the Key Buffer, used to cache index blocks for MyISAM tables.

    Do not set it larger than 30% of your available memory, as some memory

    is also required by the OS to cache rows. Even if you're not using

    MyISAM tables, you should still set it to 8-64M as it will also be

    used for internal temporary disk tables.

    key_buffer_size=512M

    Size of the buffer used for doing full table scans of MyISAM tables.

    Allocated per thread, if a full scan is needed.

    read_buffer_size=0

    read_rnd_buffer_size=0

    #*** INNODB Specific options ***

    innodb_data_home_dir=0.0

    Use this option if you have a MySQL server with InnoDB support enabled

    but you do not plan to use it. This will save memory and disk space

    and speed up some things.

    skip-innodb

    If set to 1, InnoDB will flush (fsync) the transaction logs to the

    disk at each commit, which offers full ACID behavior. If you are

    willing to compromise this safety, and you are running small

    transactions, you may set this to 0 or 2 to reduce disk I/O to the

    logs. Value 0 means that the log is only written to the log file and

    the log file flushed to disk approximately once per second. Value 2

    means the log is written to the log file at each commit, but the log

    file is only flushed to disk approximately once per second.

    innodb_flush_log_at_trx_commit=0

    The size of the buffer InnoDB uses for buffering log data. As soon as

    it is full, InnoDB will have to flush it to disk. As it is flushed

    once per second anyway, it does not make sense to have it very large

    (even with long transactions).

    innodb_log_buffer_size=256M

    InnoDB, unlike MyISAM, uses a buffer pool to cache both indexes and

    row data. The bigger you set this the less disk I/O is needed to

    access data in tables. On a dedicated database server you may set this

    parameter up to 80% of the machine physical memory size. Do not set it

    too large, though, because competition of the physical memory may

    cause paging in the operating system. Note that on 32bit systems you

    might be limited to 2-3.5G of user level memory per process, so do not

    set it too high.

    innodb_buffer_pool_size=4G

    Size of each log file in a log group. You should set the combined size

    of log files to about 25%-100% of your buffer pool size to avoid

    unneeded buffer pool flush activity on log file overwrite. However,

    note that a larger logfile size will increase the time needed for the

    recovery process.

    innodb_log_file_size=1G

    Number of threads allowed inside the InnoDB kernel. The optimal value

    depends highly on the application, hardware as well as the OS

    scheduler properties. A too high value may lead to thread thrashing.

    innodb_thread_concurrency=8

    The increment size (in MB) for extending the size of an auto-extend InnoDB system tablespace file when it becomes full.

    innodb_autoextend_increment=64

    The number of regions that the InnoDB buffer pool is divided into.

    For systems with buffer pools in the multi-gigabyte range, dividing the buffer pool into separate instances can improve concurrency,

    by reducing contention as different threads read and write to cached pages.

    innodb_buffer_pool_instances=8

    Determines the number of threads that can enter InnoDB concurrently.

    innodb_concurrency_tickets=5000

    Specifies how long in milliseconds (ms) a block inserted into the old sublist must stay there after its first access before

    it can be moved to the new sublist.

    innodb_old_blocks_time=1000

    It specifies the maximum number of .ibd files that MySQL can keep open at one time. The minimum value is 10.

    innodb_open_files=300

    When this variable is enabled, InnoDB updates statistics during metadata statements.

    innodb_stats_on_metadata=0

    When innodb_file_per_table is enabled (the default in 5.6.6 and higher), InnoDB stores the data and indexes for each newly created table

    in a separate .ibd file, rather than in the system tablespace.

    innodb_file_per_table=1

    Use the following list of values: 0 for crc32, 1 for strict_crc32, 2 for innodb, 3 for strict_innodb, 4 for none, 5 for strict_none.

    innodb_checksum_algorithm=0

    The number of outstanding connection requests MySQL can have.

    This option is useful when the main MySQL thread gets many connection requests in a very short time.

    It then takes some time (although very little) for the main thread to check the connection and start a new thread.

    The back_log value indicates how many requests can be stacked during this short time before MySQL momentarily

    stops answering new requests.

    You need to increase this only if you expect a large number of connections in a short period of time.

    back_log=80

    If this is set to a nonzero value, all tables are closed every flush_time seconds to free up resources and

    synchronize unflushed data to disk.

    This option is best used only on systems with minimal resources.

    flush_time=0

    The minimum size of the buffer that is used for plain index scans, range index scans, and joins that do not use

    indexes and thus perform full table scans.

    join_buffer_size=256K

    The maximum size of one packet or any generated or intermediate string, or any parameter sent by the

    mysql_stmt_send_long_data() C API function.

    max_allowed_packet=32M

    If more than this many successive connection requests from a host are interrupted without a successful connection,

    the server blocks that host from performing further connections.

    max_connect_errors=100

    Changes the number of file descriptors available to mysqld.

    You should try increasing the value of this option if mysqld gives you the error "Too many open files".

    open_files_limit=4161

    If you see many sort_merge_passes per second in SHOW GLOBAL STATUS output, you can consider increasing the

    sort_buffer_size value to speed up ORDER BY or GROUP BY operations that cannot be improved with query optimization

    or improved indexing.

    sort_buffer_size=256K

    The number of table definitions (from .frm files) that can be stored in the definition cache.

    If you use a large number of tables, you can create a large table definition cache to speed up opening of tables.

    The table definition cache takes less space and does not use file descriptors, unlike the normal table cache.

    The minimum and default values are both 400.

    table_definition_cache=1400

    Specify the maximum size of a row-based binary log event, in bytes.

    Rows are grouped into events smaller than this size if possible. The value should be a multiple of 256.

    binlog_row_event_max_size=8K

    If the value of this variable is greater than 0, a replication slave synchronizes its master.info file to disk.

    (using fdatasync()) after every sync_master_info events.

    sync_master_info=10000

    If the value of this variable is greater than 0, the MySQL server synchronizes its relay log to disk.

    (using fdatasync()) after every sync_relay_log writes to the relay log.

    sync_relay_log=10000

    If the value of this variable is greater than 0, a replication slave synchronizes its relay-log.info file to disk.

    (using fdatasync()) after every sync_relay_log_info transactions.

    sync_relay_log_info=10000

    Load mysql plugins at start."plugin_x ; plugin_y".

    plugin_load="mysqlx"

    MySQL server's plugin configuration.

    loose_mysqlx_port=33060



  • Rate Per Second = RPS

    Suggestions to consider for your my.ini [mysqld] section to improve performance

    read_rnd_buffer_size=96k  # from 1 which is just nibbling at data
    read_buffer_size=256K  # from 8K to reduce handler_read_next RPS of 120,235
    innodb_lru_scan_depth=100  # from 1024 to conserve 90% of CPU cycles used for function
    innodb_thread_concurrency=0  # from 8 per dba.stackexchange.com Question 5666 answered by Rolando on 9/12/11 to allow OS management of concurrency limit. 
    net_buffer_length=96K  # from 16K to reduce packet counts in/out
    innodb_open_files=2000  # from 300 to match table_open_cache best practice
    

    Observation, 79 rollbacks were reported involved in 1 day with only one savepoint. This seems like an unreasonable combination. When time permits, we need to find out why. Rollbacks are usually expensive.

    There are many more opportunities to improve configuration. Please view profile for contact info and get in touch if additional assistance is needed.




Suggested Topics

  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2