Splunk log driver gives "Forbidden"

General Discussions
1

I am trying to use the splunk logging driver, like this:

docker run --log-driver=splunk --log-opt splunk-token=my-hec-token --log-opt splunk-url="https://splunkheavy:8088" --log-opt splunk-index=docker --log-opt splunk-insecureskipverify=true -p 8000:80 nginx
docker: Error response from daemon: failed to initialize logging driver: Options https://splunkheavy:8088/services/collector/event/1.0: Forbidden.
ERRO[0000] error waiting for container: context canceled

I have verifed my splunk hec with the following command:

curl -k "https://splunkheavy:8088/services/collector/event/1.0" -H "Authorization: Splunk my-hec-token" -d '{"event": "Hello, world!"}'
{"text":"Success","code":0}

Docker version:

docker --version
Docker version 17.06.2-ee-16, build 9ef4f0a

So it seems my parameters are correct, network connectivity is OK … what have I missed?

2

I finally found the problem: I have a proxy defined in

/etc/systemd/system/docker.service.d/https-proxy.conf

l had to add my splunkheavy to the NO_PROXY environmental variable, because docker was trying to access my splunkheavy via the proxy (which is only for internet access). Finally my https-proxy.conf looks like this:

[Service]
Environment="HTTPS_PROXY=https://myproxy.acme.org:3128" "NO_PROXY=splunkheavy,localhost,127.0.0.1,docker-registry.acme.org"

Then reload the systemctl daemon and docker:

systemctl daemon-reload
systemctl restart docker

And all is fine.