Dear community,
I am doning my first steps with Docker (20.10.22) and Graylog an a Ubuntu 22.04.1 LTS. I get to the point, where I can access Graylog on the IP address of the ubuntu server. Now, the server and the Graylog should be in different VLANs. I have read a lot about ipvlan and macvlan.
I want to have a fixed IP for each docker container, some of them on “VLAN 30”, some of them in “VLAN 70”.
I have a Unifi environment and I should be able to have a single trunk switchport, to which I can connect my Server to. (Server is an old HP EliteDesk 800 G2.)
Graylog is configured with a docker-compose.yml. I think I have been able to set up the two VLANs in docker, but I struggle to configure it into the docker-compose.yml.
Here is my current, not working .yml:
version: '2'
services:
# MongoDB: https://hub.docker.com/_/mongo/
mongodb:
image: mongo:5.0.13
networks:
- ipvlan70
#DB in share for persistence
volumes:
- mongo_data:/data/db
restart: always
# Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/7.10/docker.html
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2
#data folder in share for persistence
volumes:
- es_data:/usr/share/elasticsearch/data
environment:
- http.host=0.0.0.0
- transport.host=localhost
- network.host=0.0.0.0
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
mem_limit: 1g
networks:
- ipvlan70
restart: always
# Graylog: https://hub.docker.com/r/graylog/graylog/
graylog:
image: graylog/graylog:5.0
#journal and config directories in local NFS share for persistence
volumes:
- graylog_journal:/usr/share/graylog/data/journal
environment:
# CHANGE ME (must be at least 16 characters)!
- GRAYLOG_PASSWORD_SECRET=[abcd]
# Password: admin
- GRAYLOG_ROOT_PASSWORD_SHA2=[efgh]
- GRAYLOG_HTTP_EXTERNAL_URI=http://192.168.1.57:9000/
- GRAYLOG_HTTP_ENABLE_CORS=true
entrypoint: /usr/bin/tini -- wait-for-it elasticsearch:9200 -- /docker-entrypoint.sh
networks:
- ipvlan70
links:
- mongodb:mongo
- elasticsearch
restart: always
depends_on:
- mongodb
- elasticsearch
ports:
# Graylog web interface and REST API
- 9000:9000
# Syslog TCP
- 1514:1514
# Syslog UDP
- 1514:1514/udp
# GELF TCP
- 12201:12201
# GELF UDP
- 12201:12201/udp
# Volumes for persisting data, see https://docs.docker.com/engine/admin/volumes/volumes/
volumes:
mongo_data:
driver: local
es_data:
driver: local
graylog_journal:
driver: local
networks:
ipvlan70:
driver: ipvlan
If I docker-compose up -d this yml and inspect the container, I get some internal IP, which is not what I want.
I don’t configure ipvlan often, but I think you should set the ipvlan parent interface name and some other parameters in the compose file. Currently you set only the driver.
I tried ipvlan in a virtual machine but not with compose for now, and it worked based on this guide:
Hi @rimelek,
thank you very much for your replay and the link to the documentation. I have read it before but I struggle to adapt my yml to it.
I thought of going for the version describt in this chapter: Use IPvlan networks | Docker Documentation → IPvlan 802.1q trunk L2 mode example usage
What I want to achieve, is this:
I have created the following networks in docker:
uadmin@ubuntu:~$ docker network list
NETWORK ID NAME DRIVER SCOPE
a5bb19d84c09 bridge bridge local
ec89e9852ecf graylog_graylog bridge local
4abc5a416842 graylog_ipvlan70 ipvlan local
989a7cc8de67 host host local
a970b778160f ipvlan30 ipvlan local <-- manually created
b213b5141b30 ipvlan70 ipvlan local <-- manually created
fb9e3c4b2724 none null local
uadmin@ubuntu:~$
Dear @rimelek,
thank you very much for your quick response.
I got this to work now: as long as I specify the same network for all three components, they get an IP out of the same VLAN and therefore can communicate to each other.
The ipvlan30 basically works as a DHCP. I do have to make sure that there are no dublicate IPs, if my router provides IPs as well.
Is there a way to define fixed IPs per container within the docker-compose.yml?
does not take effect, as the container gets the IP 192.168.30.2 and is reachable on this ip.
Comming back to the macvlan: how would this work? Would there be a unique mac-address per container on the network? Then, I could manage DHCP-reservations in my router, instead of using the ipvlan30 within docker?
The ip_range is one of the optional settings. Docker will use an own dhcp server on the network, as such, the subnet (or if the ip_range is defined then the ip_range) should not overlap with the ip-range of another dhcp server.
You can specify the ipv4/ipv6 address in the long syntax for the network declaration:
One more thing: docker compose v2 (cmd: docker compose) ignores the version. Though, afaik docker compose v1 (cmd: docker-compose) still uses it. Be careful with version ‘2’ as it does not mean the latest ‘2.x’, but instead equals to ‘2.0’.
Within the Docker on that Ubuntu Server, I would like to run several things:
Graylog: gathering of the logs of some systems - Server-VLAN70
HomeAssistant: currently running on a RaspberryPi - IoT-VLAN30
AdGuard: removing advertisements from webpages - Server-VLAN70
VPN-Server: Unifi does not support the new VPN protocolls, my new phone does not support the old ones - Management-LAN-VLAN1
some more cool stuff (any ideas are welcome)
…
If I can manage the IP-addresses of my docker-container within the Unifi-environment, it would be very easy. Every container would pop up as if it was a pysical machine on the network with its own mac-address.
If I do have to manage IP-ranges and fixed IP-adresses within docker, can I then stay with the now working ipvlan? Or is there a recommendation on when to use what?
BTW: I just tried to permanently acitivate the “promiscuous mode” on the eno1, but failed. How to do that?
I have seen in the documentation, that there are different versions. Well, is there a need to use the newest version? Do I have something in my yml, that would prevent me from just changing the version-number?
I was not able to find the full documentation on how to write a correct yml. It is all a bit try and error - and asking the pros like you two. (Could you post the link to the right place in the documentation? Thx)
I will only address the 2nd response now, and respond to the other one later.
If the schema version you use already supports what you need: no, there is not.
Since I would have to look it up myself, I will leave it to you to look it up: Compose file version 2 reference | Docker Docs. The long syntax for networks, might be a good candidate for not being supported by the schema version 2.0 (I might be wrong, I haven’t checked).
The link to the legacy v2 reference is above. The current compose file specification can be found here:
I would highly suggest using the docker compose cli plugin (v2.x) and follow the compose file specification, which will ignore the schema version and will always use the latest.
Looks like ulimits is still a direct child element of the service, like it was before, see ulimits.
Even though memlock is not mentioned, it doesn’t mean that it’s not supported.
On each linux system, you can find the settings in /etc/security/limits.conf:
#<type> can have the two values:
# - "soft" for enforcing the soft limits
# - "hard" for enforcing hard limits
#
#<item> can be one of the following:
# - core - limits the core file size (KB)
# - data - max data size (KB)
# - fsize - maximum filesize (KB)
# - memlock - max locked-in-memory address space (KB)
# - nofile - max number of open file descriptors
# - rss - max resident set size (KB)
# - stack - max stack size (KB)
# - cpu - max CPU time (MIN)
# - nproc - max number of processes
# - as - address space limit (KB)
# - maxlogins - max number of logins for this user
# - maxsyslogins - max number of logins on the system
# - priority - the priority to run user process with
# - locks - max number of file locks the user can hold
# - sigpending - max number of pending signals
# - msgqueue - max memory used by POSIX message queues (bytes)
# - nice - max nice priority allowed to raise to values: [-20, 19]
# - rtprio - max realtime priority
# - chroot - change root to directory (Debian-specific)
The setting in the compose file is just a convenient way to configure it inside the container.
Ulimits are used to control process limits on resources. memlock defines the max memory size that is held in memory and not paged out.
Indeed, -1 should be unlimited, so that the container will keep all allocated memory in ram.
But since a resource limit is set, the container will be OOM killed if the memory usage passes the limit.
ES is implemented in Java, which (if a recent enough Java JVM is used) respects CGROUP ressource limits and as such behaves like “the system” (from the container pov) only has 1gb. It influences how much memory java actually allocates and uses. if you experience an OOM kill, you might want to raise the memory limit until you find a sweat spot.
If have no idea, home much memory Graylog, MongoDB and Elasticsearch will need. My “Server” has 8GB of memory. I assume, I could allocated more memory.
There was a comand to see the CPU usage of a container. I assume there is one to see he memory usage as well.
Anyway, thank you very much for your very helpful inputs. I will try to go forward from here and post my final solution, as soon as I find one.
Afaik Docker networks always use IPAM and provide dhcp for the networks. Whenever you define an ip-range and gateway, it will be a subconfiguration of IPAM.
That indeed would be charming. Since ipvlan can be operated on layer2, I would be interested to know if this has an impact on how ipam/dhcp behaves.
If your switch is able to cope with ipvlan, I don’t see a reason agains using ipvlan. Some home grade devices are confused by ipvlan. Though, those devices typcially are also confused by carp or vrrp failover-ip’s
I have no idea. I do run my container engines in vm’s and only had to enable promiscous mode on the vswitch/bridge on the hypervisor level (esxi/proxmox).
I am still playing around with ipvlan vs. macvlan.
In both cases, I cannot have my Ubuntu Server, on which there is the docker engine, in the same vlan, as the docker containers. If I try to, I can access the Graylog-container, but I cannot ssh into the ubuntu server. If I put them in seperate vlan’s, everything is fine. I am by far no network expert. Is there someting I could config to make this possible?
I would like to have the
ubuntu on 192.168.70.2
Graylog-container on 192.168.70.3
HomeAssistant-container on 192.168.30.3
VPN-Server-container on 192.168.40.4
Or do I have to live with it, that host and containers can not share the same vlan, due to whatever.
The other thing: if I use ipvlan, the container is reachable, but is not visible in my network environment. If I use macvlan, the container and the server are visible in the network environment.
I am not sure, if I already wrote it or not… Due to kernel security restrictions a macvlan child interface is prohibited from direct communication with its parent interface - and vise versa. This is caused by the kernel and is not something docker does or can work around.
@ajeetraina wrote a blog post about how to enable container to host communication:
The trick is to create another macvlan child interface and attach it to the host. If a container needs to communicate with the host, it needs to use the ip of the host’s macvlan child interface, instead of the parent interfaces ip. As a result you have child interface to child interface communication.
@meyay
Thank you for posting that link. Well, I don’t need to have container <-> host communication. I only want them to be in the same ip-range. But as soon as I put them into the same range, the host is not accessable anymore. As if the container receivces all the tcp-packages and omits them, because the needed ports are not open on the container.
Oh, I got you wrong, I thought its about container to host communication. But apparently it is about the reachability of the host in general.
Are you experiencing this problem with macvlan as well?
I can see how this might be happening with ipvlan, if the switches get confused by multiple ip’s on the same mac address. Though, I would be surprised if the same happens for macvlan as well.
With my “prosumer” grade homelab equipment, additional macvlan ip’s appear in the neighbours list for the primary ip/mac combination (in my internet router), It looks pretty much the same, as the entry for my router for my private networks looks like, where each ip/mac of the other subnet’s are in the neighbours list for the router entry.
I actually couldn’t use ipvlan, as it freaked out my router/switches.I used to use macvlan. but I don’t use any of both anymore.
- Management-LAN-VLAN1