Each RUN line in a Dockerfile gets executed as the only process in a container. The command you are running, mysql_install_db --verbose && /usr/bin/mysqladmin -u root password 'password', attempts to communicate with an already running mysql/mariadb server. Since this is the only thing running in the container, and there is no mysql service running, it fails.
Normally this sort of stuff should get run at runtime in an ENTRYPOINT script instead of at build time. This is because you’ll want your mysql/mariadb files to be stored in a volume and not in the layered filesystem.
Take a look at the official mysql and mariadb images as a starting point: https://github.com/docker-library/mariadb/tree/master/10.1 and https://github.com/docker-library/mysql/tree/master/5.7