How to instantiate '3rd party' docker container from 'my' fastapi docker

General Discussions
1

I have a 3rd party docker image that I can run from command line as follows
$ lean backtest 'strategy'
(after I have pulled the image using $ docker pull quantconnect/lean)
This image executes the ‘strategy’ (a python module), dumps results and exit

if I were to run all this without docker, then on my computer then I would simply start fastapi server
$ uvicorn routes:app --reload
and on receiving the request, I would perform subprocess run
subprocess.run(['lean', 'backtest', 'strategy'], cwd=base_dir, capture_output=True, text=True)

I want to embed this in my fastapi docker server (to be able to finally run this on Azure)

If I create a docker image of fastapi and run the fastapi container then it complains that docker not running.
So as I understand within this fastapi container two things should happen 1. docker should be running 2. quanconnect/lean image should have been downloaded

Any ideas how to implement this in a way that I can later move this setup from my machine to Azure?

2

Of course you can run Docker in Docker and you can also run the Docker CLI inside your container and connect to your host to instantiate a parallel container, if you mount the host Docker socket into your container. But that’s all more theoretical.

I would create a new image, based on theirs, and add a small http wrapper, that receives http requests and then executes the script.

Python seems already installed (link), so that would be the obvious choice.

3

@bluepuma77 , thank you for your reply.
I am a bit new to this so please pardon my stupid question.
my current docker looks something like (inspired from internet)

FROM python:3.10-slim as compiler
...
RUN pip install -Ur requirements.txt
...
FROM python:3.10-slim as runner
WORKDIR /app/
COPY --from=compiler /app/venv /app/venv
...
ENV PATH="/app/venv/bin:$PATH"
ENV LEAN_PATH="/app/venv/bin/lean"
COPY . /app/
CMD python -m uvicorn --app-dir ./  --reload --host 0.0.0.0 --port 8000 routes:app

Are you saying something like

  1. do a git pull and then use their Dockerfile and insert my docker content? or
    or
  2. in my Docker file change
    FROM python:3.10-slim as compiler → FROM quantconnect/lean:latest
    and keep rest as it is?

If you mean 2nd option then is it cool especially if my subprocess.run works as it does currently.

Thank you for your guidance

4

I would have decoupled, FROM lean and add a simple http server to execute command.

But if you use Python anyway, you can try FROM lean for your regular app, too.

5

ok, thank you, let me try option 2 first (looks easier)… I will let you know …
(I do want to understand how to decouple without complex option like ‘Docker in Docker’ or Docker CLI you mentioned earlier)

6

While changing from FROM python to FROM quantconnect/lean worked in compilation …
the running did not work and gave this error
% docker run --rm -it -p 8000:8000 sappv01 bash

Welcome to .NET 6.0!

Could not execute because the specified command or file was not found.
Possible reasons for this include:

  • You misspelled a built-in dotnet command.
  • You intended to execute a .NET program, but dotnet-QuantConnect.Lean.Launcher.dll does not exist.
  • You intended to run a global tool, but a dotnet-prefixed executable with this name could not be found on the PATH.

It uses .NET underlying and the entrypoint is mentioned in the Dockerfile as
WORKDIR /Lean/Launcher/bin/Debug
ENTRYPOINT [ "dotnet", "QuantConnect.Lean.Launcher.dll" ]

Can you elaborate on your statement:

I would have decoupled, FROM lean and add a simple http server to execute command.

7

A Dockerfile always uses entrypoint (mostly a script) and cmd (as args) in combination. So you need to figure out how to combine that.

8

@bluepuma77 this was a useful tip…thank you

I added line ENTRYPOINT in my Dockerfile to prevent the base image (quantconnect/lean) ENTRYPOINT. This ran the fastapi server.
However, when I reached the stage in program
subprocess.run(['lean', 'backtest', 'strategy'], cwd=base_dir, capture_output=True, text=True)
the FastApi server complained:

Error: Please make sure Docker is installed and running

i.e. it does not realize that this container is already being called from within a container.
So back to my original problem. How would I run from fastapi docker container another container

9

You noted the original entrypoint

ENTRYPOINT [ "dotnet", "QuantConnect.Lean.Launcher.dll" ]

Do you run this inside the container?

lean backtest 'strategy'

Or is it a script outside, running Docker?

10

On my MacOS, If I am running my fastapi server manually from command line then flow is
$ uvicorn --app-dir . routes:app --reload
and this server on receiving the request calls lean
subprocess.run(['lean', 'backtest', 'strategy']).
Since Docker Desktop is running on my machine, the subprocess executes successfully.
Note: this lean process is ephemeral and exists after each run.

Now, from my viewpoint I only wrapped my fastapi in docker with command
CMD ["uvicorn", "--app-dir", "./", "routes:app", "--reload"]
and start this fastapi server from command line as
docker run --rm -it -p 8000:8000 sappv01

So lean backtest strategy is running inside the container and now if I think about your question the original entry point is is never being called.

11

It seams lean is a command installed by pip, which starts a Docker container (doc).