We have a second docker daemon running on a host to be used by a Jenkins server for dynamically creating docker-based build slaves. This daemon uses a manually created bridge network and I have also switched iptables handling off on that daemon and manually created the masquerading rules. Moreover, containers on this daemon do not use the docker DNS server. They directly communicate with external DNS servers.
What we observe is that whenever Jenkins builds are running and thus containers are created and removed frequently, other containers on the same daemon have sporadic problems with resolving host names via DNS. Looking at captured packages via wireshark shows that in these situations the UDP DNS requests are not rerouted to the external network interface of the server, which is the behavior for successful requests. Hence, the requests in the faulty situations never reach the external DNS servers and end in a timeout.
Does anyone know what could be causing this?