MariaDB fail after three weeks

General
1

I’m a relatively new user to docker. I’ve built a server at Vultr. The server contains three wordpress sites, three mariadb’s and one static HTML site. After three weeks, one of the docker containers with mariaDB is failing. Not sure what’s going on. I do know the normal database has had no data added in the past three weeks. All three wordpress sites were set up exactly the same. The Vultr server is a single core CPU, RAM: 1024 MB, with Storage: 25 GB SSD. The site is also running fail2ban, mostly to protect against SSH login attempts…

docker compose extracts:

XXX_db:     
    image: mariadb
    volumes:
      - ./XXX_db_data:/var/lib/mysql
    ports:
      - 8082:3306
    environment:
      MYSQL_ROOT_PASSWORD: xxxpasswordxxx
    restart: on-failure:8
XXX_wp:
    image: wordpress:latest
    links:
      - XXX_db:mysql
    environment:
      WORDPRESS_DB_PASSWORD: xxxpasswordxxx
      VIRTUAL_HOST: mywebsiteXXX.com, www.mywebsiteXXX.com
    volumes:
      - ./XXX:/var/www/html
    expose:
      - 80
    depends_on:
      - XXX_db     # start services in dependency order`

website now suddenly shows: Error establishing a database connection

$ docker ps -a shows one of three databases Exited.

If I $ docker logs XXX_database I get a long log file.

Key extracts:

2017-07-06  8:14:48 140361347122944 [Warning] Access denied for user 'wordpress'@'172.18.0.10' (using password: YES)   <-- this was me attempting to access on day server was built.
2017-07-26 21:03:21 140339552581504 [Note] mysqld (mysqld 10.2.6-MariaDB-10.2.6+maria~jessie) starting as process 1 ...
2017-07-26 21:03:21 140339552581504 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-07-26 21:03:21 140339552581504 [Note] InnoDB: Uses event mutexes
2017-07-26 21:03:21 140339552581504 [Note] InnoDB: Compressed tables use zlib 1.2.8
2017-07-26 21:03:21 140339552581504 [Note] InnoDB: Using Linux native AIO
2017-07-26 21:03:21 140339552581504 [Note] InnoDB: Number of pools: 1
2017-07-26 21:03:21 140339552581504 [Note] InnoDB: Using SSE2 crc32 instructions
2017-07-26 21:03:21 140339552581504 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
2017-07-26 21:03:21 140339552581504 [ERROR] InnoDB: mmap(137560064 bytes) failed; errno 12
2017-07-26 21:03:21 140339552581504 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
2017-07-26 21:03:21 140339552581504 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2017-07-26 21:03:21 140339552581504 [Note] InnoDB: Starting shutdown...
*** Error in `mysqld': double free or corruption (out): 0x00007fa35c0e25b0 ***
170726 21:03:21 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

more

Server version: 10.2.6-MariaDB-10.2.6+maria~jessie
key_buffer_size=134217728
read_buffer_size=2097152
max_used_connections=0
max_threads=102
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 759887 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
(my_addr_resolve failure: fork)
mysqld(my_print_stacktrace+0x2e) [0x7f5812817a5e]
mysqld(handle_fatal_signal+0x305) [0x7f58122584f5]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf890) [0x7f58117b0890]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37) [0x7f580fad8067]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7f580fad9448]
/lib/x86_64-linux-gnu/libc.so.6(+0x731b4) [0x7f580fb161b4]
/lib/x86_64-linux-gnu/libc.so.6(+0x7898e) [0x7f580fb1b98e]
/lib/x86_64-linux-gnu/libc.so.6(+0x79696) [0x7f580fb1c696]
mysqld(+0x9419ac) [0x7f58125229ac]
mysqld(+0x93c4bd) [0x7f581251d4bd]
mysqld(+0x80125a) [0x7f58123e225a]
mysqld(ha_initialize_handlerton(st_plugin_int*)+0x64) [0x7f581225a7b4]
mysqld(+0x4e5e1b) [0x7f58120c6e1b]
mysqld(plugin_init(int*, char**, int)+0x8aa) [0x7f58120c852a]
mysqld(+0x43c4be) [0x7f581201d4be]
mysqld(mysqld_main(int, char**)+0x65e) [0x7f58120214ee]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f580fac4b45]
mysqld(+0x43400d) [0x7f581201500d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Fatal signal 11 while backtracing
2017-07-26 21:59:14 139757590316928 [Note] mysqld (mysqld 10.2.6-MariaDB-10.2.6+maria~jessie) starting as process 1 ...
2017-07-26 21:59:14 139757590316928 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-07-26 21:59:14 139757590316928 [Note] InnoDB: Uses event mutexes
2017-07-26 21:59:14 139757590316928 [Note] InnoDB: Compressed tables use zlib 1.2.8
2017-07-26 21:59:14 139757590316928 [Note] InnoDB: Using Linux native AIO
2017-07-26 21:59:14 139757590316928 [Note] InnoDB: Number of pools: 1
2017-07-26 21:59:14 139757590316928 [Note] InnoDB: Using SSE2 crc32 instructions
2017-07-26 21:59:14 139757590316928 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
2017-07-26 21:59:14 139757590316928 [ERROR] InnoDB: mmap(137560064 bytes) failed; errno 12
2017-07-26 21:59:14 139757590316928 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
2017-07-26 21:59:14 139757590316928 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2017-07-26 21:59:14 139757590316928 [Note] InnoDB: Starting shutdown...
*** Error in `mysqld': double free or corruption (out): 0x00007f1bdd44b5b0 ***
170726 21:59:14 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

I don’t understand what’s going on. Here is a file listing of XXX_db_data (which is a clone of :/var/lib/mysql)

# ls -all
total 219124
drwxr-xr-x   5 systemd-bus-proxy ssh_keys      4096 Jul 26 22:13 .
dr-xr-x---. 11 root              root          4096 Jul  7 00:23 ..
-rw-rw----   1 systemd-bus-proxy ssh_keys     16384 Jul  6 08:10 aria_log.00000001
-rw-rw----   1 systemd-bus-proxy ssh_keys        52 Jul  6 08:10 aria_log_control
-rw-------   1 systemd-bus-proxy ssh_keys 223567872 Jul 26 22:13 core.1
-rw-rw----   1 systemd-bus-proxy ssh_keys      2799 Jul  6 08:10 ib_buffer_pool
-rw-rw----   1 systemd-bus-proxy ssh_keys  79691776 Jul 26 17:54 ibdata1
-rw-rw----   1 systemd-bus-proxy ssh_keys  50331648 Jul 26 17:54 ib_logfile0
-rw-rw----   1 systemd-bus-proxy ssh_keys  50331648 Jul  6 08:10 ib_logfile1
-rw-rw----   1 systemd-bus-proxy ssh_keys  12582912 Jul  6 08:10 ibtmp1
-rw-rw----   1 systemd-bus-proxy ssh_keys         0 Jul  6 08:10 multi-master.info
drwx------   2 systemd-bus-proxy ssh_keys      4096 Jul  6 08:10 mysql
drwx------   2 systemd-bus-proxy ssh_keys      4096 Jul  6 08:10 performance_schema
-rw-rw----   1 systemd-bus-proxy ssh_keys     24576 Jul  6 08:10 tc.log
drwx------   2 systemd-bus-proxy ssh_keys      4096 Jul  7 19:21 wordpress

Ouch, that core.1 file is mighty large. What’s in there, and why is it growing? This database supports a simple Wordpress website that gets maybe 40 hits a day on average.

Anyone have ideas on what to look for to repair this database? Hints or tips?

2

I was able to get things working again by adding swap space, per this link

The one downside (from this link)

Although swap is generally recommended for systems utilizing traditional spinning hard drives, using swap with SSDs can cause issues with hardware degradation over time. Due to this consideration, we do not recommend enabling swap on DigitalOcean or any other provider that utilizes SSD storage. Doing so can impact the reliability of the underlying hardware for you and your neighbors.

If you need to improve the performance of your server, we recommend upgrading your Droplet. This will lead to better results in general and will decrease the likelihood of contributing to hardware issues that can affect your service.

I would still like to understand why this was even necessary. I’m serving data from that database, not really writing anything there. Content is relatively stable. Its not clear at all why I have growing files within the DB folder…