Understanding swarm secrets and rootless mode

germaggus · June 27, 2024, 7:44am

Hello everyone,

I am new to this forum and to the whole topic of Docker and everything related to it. I am trying to learn, understand, and implement the whole topic. Over the last few days, I have read and tried a lot, much of which has worked, but now I have reached a point where I am grateful for help from people who are already familiar with it.

My infrastructure / my goal:
I have a dedicated server in a data center running Proxmox. On it, I have an Ubuntu Server VM (24.04 LTS) running, which is supposed to become my future application server, providing various services via Docker and Portainer. I am a big fan of security, and the infrastructure is supposed to be the foundation for my future professional IT business. I have chosen a Docker rootless mode variant, which I have successfully installed, and now Portainer is running on it. So far, so good.

Now, I want to provide various containers using Portainer stacks and corresponding YAML files. For testing purposes, I started with, for example, a Teamspeak server to practice how everything works. What I want to avoid is having passwords and sensitive data displayed in plain text in the config files, and I am looking for best practice solutions for this.

After some research, I came across the topic of Docker Swarm and Secrets and tried to implement it, but it always failed due to some permission errors. Actually, I don’t currently need a cluster, and I would like to use the solution as a single node just for the Secrets feature. After days of trying and researching, I found a post that said Docker Swarm does not work in rootless mode (#overlay network?).

My question is therefore: Is this still the case? Can someone briefly explain why this is (at a somewhat beginner level)? What alternatives do I have regarding the use of Secrets? What would be recommended? Would it be advisable to use a rootful approach instead and utilize Docker Swarm’s secret features?

I have searched everywhere on this topic but either didn’t find the right thing or didn’t understand it. I would be infinitely grateful for solutions, explanations, or a pointer in the right direction.

Thank you in advance for your help.

Best regards

germaggus

bluepuma77 · June 27, 2024, 8:45am

Docker Swarm sadly doesn’t run in rootless mode.

The Swarm part was bought, it’s still supported as paid enterprise offering. it seems the company doesn’t care much about open source, lots of ready community patches are just sitting there, not being merged. (Github)

Pro users seem to use separate secret stores like Vault. But that company changed license, so people seem to migrate away to other solutions.

Lots of politics in software

One (thought) shower later:

I think Docker is fine for home and very small business users, but everyone seems to be using k8s nowadays. So if you want advance into that direction…

BUT : configs and secrets work with regular Docker and compose. No Swarm needed, just tested

germaggus · June 29, 2024, 11:04am

Hello bluepuma77,

Thank you very much for your response and for taking the time to answer.

Ok. So Docker Swarm is not a solution for my problem. I also came across Vault in my research. Thanks for the tip about the licenses. Then I probably don’t need to try it in that direction.

I have gained a small overview of the topic of k8s. The topic seems quite interesting, but it doesn’t seem to fit my requirements at all. For a single-node solution, it seems to be quite overkill and brings in an incredible amount of complexity, which in my opinion seems unnecessary.

To pick up on your statement “BUT : configs and secrets work with regular Docker and compose. No Swarm needed, just tested ”:

Could you or someone else perhaps help me - an absolute beginner - with a concrete example: The docker-compose.yml for example for TeamSpeak looks as follows:

yaml

Code kopieren

version: '3.1'
services:
  teamspeak:
    image: teamspeak
    restart: always
    ports:
      - 9987:9987/udp
      - 10011:10011/tcp
      - 30033:30033/tcp
    environment:
      TS3SERVER_DB_PLUGIN: ts3db_mariadb
      TS3SERVER_DB_SQLCREATEPATH: create_mariadb
      TS3SERVER_DB_HOST: db
      TS3SERVER_DB_USER: root
      TS3SERVER_DB_PASSWORD: example
      TS3SERVER_DB_NAME: teamspeak
      TS3SERVER_DB_WAITUNTILREADY: 30
      TS3SERVER_LICENSE: accept
  db:
    image: mariadb
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: example
      MYSQL_DATABASE: teamspeak

When I include it like this, everything seems to work as it should. But now I want to replace the user and password entries with these secrets. How do I proceed now concretely? What best practice options are there for this? And what are the advantages and disadvantages?

Thanks again for your help.

Best regards,

germaggus

meyay · June 29, 2024, 11:22am

The only thing secret about Swarm Secrets is that they are encrypted in the raft-logs, so the secret remains secret in the raft-logs. The source file is not encrypted, and if a secret is used in a container it is also not encrypted inside the container.

On a single node using secrets, instead of using a bind volume for a single file that is mounted read-only into the container, there is no benefit at all. It will do nothing for security. It will just harmonize where the mounted “secret” file is expected.

The opensource fork of HashiCorp Vault is OpenBao. I am not sure if it makes sense to use OpenBao on a single node.

What you are looking for is something like SOPS encryption of your env-files. You would still need a gpg or age key pair on your host to encrypt and decrypt the files.

Once you encrypt your files with sops (see doc link above), you can decrypt and interact with compose like this:

sops exec-file --no-fifo mysecrets.env "docker compose --env-file {} up -d"

I am sure the documentation will help when it comes to installing sops, creating a key pair and encrypting files.

germaggus · June 30, 2024, 11:35am

Hello meyay,

Thank you very much for your helpful input. I have delved into the topic and also found a helpful YouTube video by Techno Tim, which has been very useful to me. However, I am still stuck in some areas, which is probably because I have not yet fully understood everything.

What I have done:

Installed SOPS (version 3.9.0)
Installed AGE / AGE-KEYGEN (version 1.2.0)
Created the private and public key pair and stored it in a .txt file, which is located at /home/dockeruser/.sops/key.txt
Added the key file to the ~./bashrc
Created a secrets.env file under /home/dockeruser/secrets/secrets.env
The secrets.env file contains the following test entries:
TS3SERVER_DB_USER=root
TS3SERVER_DB_PASSWORD=example
MYSQL_ROOT_PASSWORD=example
Encrypted the file with the following command:

sops --encrypt --age $(cat /home/dockeruser/.sops/key.txt | grep -oP "public key: \K(.*)") -i /home/dockeruser/secrets/secrets.env

The file seems to have been encrypted successfully. The cat secrets.txt command produces the following result:

TS3SERVER_DB_PASSWORD=ENC[AES256_GCM,data:j22bHPelKQ==,iv:IBYeMhM+Dej8tth/g5q/hKx6shKXGHsuG6JyxPs82XA=,tag:utkMP6ww24aw4r68QN0d0A==,type:str]
MYSQL_ROOT_PASSWORD=ENC[AES256_GCM,data:0apgenYPoQ==,iv:Im/Pyy9Q8pDr82cIs26LtwAENp6Uzg3XVj3StyQ0FpI=,tag:/YMfGes/lJ3uL//yqbN6Kw==,type:str]
sops_age__list_0__map_enc=-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBCRnJxLzlOVm9ybEhjU3Ja\nZ1N0QzBiT3oyam42eFhEWElzaDZSMTJGZWpzCnV3ZUxuK0dXdVoxNGMva0EvMTBU\nUDl4cm5mMGY2V2VtOXp5ZjR0OHFqdFEKLS0tIFV1TEptOUNzZkZlT0gxQkV6Q1M2\namJKZjZiVWc3d0ozTytySFpGbDRXUEEKcZrCbf2NWlv8DGqxQHQKWsduXwrAwLby\nXuXFMxdexe7bnGf4AXSL9WcjeOks/OHh9tl0SfAUvxSW0pK3c5A4/w==\n-----END AGE ENCRYPTED FILE-----\n
sops_age__list_0__map_recipient=age12dsd6a88qv374cwdfzugmvdzm28a8l03zdneg9jwxzrue9e6dv7q6k4l4d
sops_lastmodified=2024-06-30T10:23:23Z
sops_mac=ENC[AES256_GCM,data:FlDAAhFVReuzzykKmrleEBzzlB5FSQOILSQYvSFZwIkyiNB8hs2LEzT66H1KYfFtmk6x+TuuQHf4+7Aa+cV+69dnuWMBLEU9JixwetAQNNzDMkzbb2PFvl4UrCfgrtGSjYiJwniCvOJRnb5kH/prTqoHYjtidwSUIWdNRBYBvmM=,iv:DtgTS9ZDIQ/ig6BjjAhl+gywakDgJsRtU9OEIwzb75U=,tag:M9kqz6RL6frgnPbtiAK07g==,type:str]
sops_unencrypted_suffix=_unencrypted
sops_version=3.9.0

So far, so good. The file can be manually encrypted and decrypted without any problems.
But from this point onward, my problems begin.

I have adjusted the config file as follows:

version: '3.1'

services:
  teamspeak:
    image: teamspeak
    restart: always
    ports:
      - 9987:9987/udp
      - 10011:10011
      - 30033:30033
    environment:
      TS3SERVER_DB_PLUGIN: ts3db_mariadb
      TS3SERVER_DB_SQLCREATEPATH: create_mariadb
      TS3SERVER_DB_HOST: db
      TS3SERVER_DB_USER: ${TS3SERVER_DB_USER}
      TS3SERVER_DB_PASSWORD: ${TS3SERVER_DB_PASSWORD}
      TS3SERVER_DB_NAME: teamspeak
      TS3SERVER_DB_WAITUNTILREADY: 30
      TS3SERVER_LICENSE: accept
    volumes:
      - teamspeak_data:/var/ts3server

  db:
    image: mariadb
    restart: always
    environment:
      MYSQL_DATABASE: teamspeak
      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
    volumes:
      - mariadb_data:/var/lib/mysql

volumes:
  teamspeak_data:
    name: teamspeak_data
  mariadb_data:
    name: mariadb_data

If I deploy the stack now, it technically works, meaning the containers are created, but I receive the following error message in my MariaDB container:

[ERROR] [Entrypoint]: Database is uninitialized and password option is not specified
You need to specify one of MARIADB_ROOT_PASSWORD, MARIADB_ROOT_PASSWORD_HASH, MARIADB_ALLOW_EMPTY_ROOT_PASSWORD and MARIADB_RANDOM_ROOT_PASSWORD

Of course. The credentials are encrypted, so obviously they cannot be used as is.
That’s where the ‘sops exec’ command comes in, if I understand correctly. I’ve adjusted it to point to both my encrypted ‘secrets.env’ and my ‘docker-compose.yml’ file paths:

sops exec-file --no-fifo /home/dockeruser/secrets/secrets.env "docker compose -f /home/dockeruser/.local/share/docker/volumes/portainer_data/_data/compose/9/v1/docker-compose.yml --env-file {} up -d"

When executing this command, however, I receive the following messages:

WARN[0000] /home/dockeruser/.local/share/docker/volumes/portainer_data/_data/compose/9/v1/docker-compose.yml: `version` is obsolete
WARN[0000] volume "teamspeak_data" already exists but was created for project "teamspeak" (expected "v1"). Use `external: true` to use an existing volume
WARN[0000] volume "mariadb_data" already exists but was created for project "teamspeak" (expected "v1"). Use `external: true` to use an existing volume
[+] Running 1/2
 ✔ Container v1-db-1         Started                                                                                                                                                                                                                                0.2s
 ⠙ Container v1-teamspeak-1  Starting                                                                                                                                                                                                                               0.2s
Error response from daemon: driver failed programming external connectivity on endpoint v1-teamspeak-1 (5b8c2b15915788fbceb997ccbba53b9c938d8eec03259a3e115122a1dff20a02): Bind for 0.0.0.0:9987 failed: port is already allocated
exit status 1

I must be doing something obviously wrong, in the wrong order perhaps, or I haven’t quite understood the dependencies correctly. I’m slowly getting frustrated with this mental block and would appreciate it if you or someone else could shed some light on this.

I also understand that this is a Docker forum, not Portainer, but it seems my current issues aren’t related to Portainer.

I sincerely thank you all for your help.

germaggus

meyay · June 30, 2024, 11:44am

Everything you did looks good to me. Thank you for sharing it in such a detail, I am sure it will help others as well. I think you sorted out the SOPS party beautifully.

The problem you experience, appears to be a simple one: port 9987 is already bound:

A port can only be bound once on an ip. The special ip 0.0.0.0 binds a port to every IP the host has. If the port is used on any of those ip’s, you get the error message you see above.

Please share the output of sudo netstat -tulpn | grep 9987, so we can see what process binds the port.

If the last column shows {process id}/docker-proxy or {process id}/dockerd, the another container already publishes this port. Check docker ps -a to see if a container exists that publishes a container port to this host port. Remove that container and it should work.

Update:
At a second glance, it looks like you have a compose project deployed, which already “occupy” the ports. Compose thinks that the project is managed by a different compose deployment (which is true). You need to specify the project name, to let compose know that it’s the same compose project:

sops exec-file --no-fifo /home/dockeruser/secrets/secrets.env "docker compose -f /home/dockeruser/.local/share/docker/volumes/portainer_data/_data/compose/9/v1/docker-compose.yml --env-file {} --project-name teamspeak up -d"

germaggus · June 30, 2024, 1:16pm

Thank you for the compliment. If I want to get help here, I need to provide detailed information. Otherwise, no one can understand what’s going on. However, I also hope that the solution will eventually work and help others in the future.

But back to the topic:
As I mentioned, I’m using Portainer. When I create the Docker Compose file and deploy the stack, everything runs directly.
The issue with the “sops exec-file” command seems to be that it ends up calling and executing Docker Compose again at the end of the command, alongside an already existing instance. That’s how I understand the problem.

The question now is how to solve this. Is there a way to integrate the sops exec command into the Docker Compose file somehow?

Where am I making a mistake or what haven’t I understood yet?

Thank you very much for your help.

germaggus

meyay · June 30, 2024, 1:40pm

I should have noticed earlier that you want to manage your containers in Portainer.
My bad! SOPS encryption/decryption will not work with Portainer, it will only work from the cli.

germaggus · June 30, 2024, 1:52pm

No problem
Maybe I need to fundamentally rethink the whole concept. It can’t be that I’m the first person on this planet who wants to use a Docker / Portainer infrastructure and also wants a higher security standard than what’s available out of the box.
So, if I want to pursue the rootless Docker approach (as described on the official Docker site) and also want to use Portainer, both Docker Swarm and SOPS are out of the picture. What options do I have left? Or what path should I take to achieve the best “high-level security” solution?
There are so many people in the world who are smarter than I am. There must be something out there. But if you don’t know what to ask for, searching becomes difficult.
Do you have any ideas or approaches?

Best regards,

germaggus

meyay · July 6, 2024, 10:51am

Some thoughts:

You get the most security out of it, if you put your Docker node in a private network, and restrict who has access to the node. Use rootless docker, or at least user namespace remapping. Never run your payload as privileged container - you can add individual capabilities if required (not sure if this applies to rootless docker). Don’t run the main process inside containers as root user (this does not apply to rootless docker). Use a ready-only filesystem if the image allows it, and mount volumes into each directory that needs to persist data. Require your host to use a http/https proxy for outgoing communication - use a proxy that allows whitelisting of target domains so in case of an exploit the attack can not load stuff from arbitrary sources. Use different UIDs:GIDs for the different containers and their volumes - make sure they do not align with a user on your host that can sudo and become root .

Furthermore, the forum search should yield a couple of useful topics, as this topic has been discussed a couple of times in the past years.

Topic		Replies	Views
Running Docker Swarm in rootless mode? General swarm	1	1884	June 18, 2023
Docker Secret concept - non cluster requirement General docker , swarm	0	471	February 28, 2022
Using Docker Secrets with Docker-Rootless & Docker Compose Compose	4	3203	March 12, 2022
Docker Swarm issues on one node General swarm , portainer	6	559	June 7, 2024
Docker swarm with rootless docker? Swarm docker , swarm	2	4237	October 9, 2020

Understanding swarm secrets and rootless mode

Related topics