Access denied when starting HttpListener in a Windows Docker container

When I start an HttpListener in a .NET Core 2.1 console application running in a Windows Docker container I get this exception:

System.Net.HttpListenerException: Access is denied

This is my C# code:

HttpListener httpListener = new HttpListener();
httpListener.Prefixes.Add("http://*:8080/test/");
httpListener.Start();

My Docker image is based on this Microsoft image: microsoft/dotnet:2.1-runtime

When I run my console application outside a Docker container it works if I start the application from a command prompt with Administrator rights. If I run it as a normal user outside a container it fails with the same Access Denied exception.

How can I avoid this exception when running in a Docker container? Can you run a container as an Administrator on Docker for Windows?