How to properly use docker/compose/v2 API in Go (Golang) code?

Hi everyone,

I’m trying to build a high-level tool on top of docker compose using Go.
I’m trying to figure out how to setup a docker compose client in the same manner that docker compose command does it.
This is my self-contained main.go:

package main

import (
	"context"
	"flag"
	"log"
	"path/filepath"

	"github.com/compose-spec/compose-go/v2/cli"
	"github.com/compose-spec/compose-go/v2/loader"
	"github.com/docker/cli/cli/command"
	"github.com/docker/cli/cli/flags"
	"github.com/docker/compose/v2/pkg/api"
	"github.com/docker/compose/v2/pkg/compose"
	"github.com/docker/docker/client"
)

func main() {
	filePath := "./docker-compose.yaml"
	flag.StringVar(&filePath, "path", filePath, "Path to the Compose file")
	flag.Parse()

	// Get absolute path
	filePath, err := filepath.Abs(filePath)
	if err != nil {
		log.Fatalf("Error getting absolute path for %s: %v", filePath, err)
	}
	projectName := loader.NormalizeProjectName(
		filepath.Base(filepath.Dir(filePath)),
	)

	// Parse project
	ctx := context.Background()
	options, err := cli.NewProjectOptions([]string{filePath},
		cli.WithOsEnv,
		cli.WithDotEnv,
		cli.WithName(projectName),
	)
	if err != nil {
		log.Fatalf("Error creating project options: %v", err)
	}

	project, err := options.LoadProject(ctx)
	if err != nil {
		log.Fatalf("Error loading project: %v", err)
	}

	// Initialize docker client
	client, err := client.NewClientWithOpts(client.FromEnv)
	if err != nil {
		log.Fatalf("Error initializing docker client: %v", err)
	}

	// Initialize docker cli
	cli, err := command.NewDockerCli(command.WithAPIClient(client))
	if err != nil {
		log.Fatalf("Error initializing docker cli: %v", err)
	}

	// Initialize docker cli context
	if err := cli.Initialize(flags.NewClientOptions()); err != nil {
		log.Fatalf("Error initializing docker cli context: %v", err)
		return
	}

	// Initialize docker-compose backend client
	backend := compose.NewComposeService(cli)
	if err != nil {
		log.Fatalf("Error initializing docker-compose backend: %v", err)
	}

	optsCreate := api.CreateOptions{
		Services:             []string{},
		Recreate:             api.RecreateDiverged,
		RecreateDependencies: api.RecreateDiverged,
		Inherit:              true,
	}
	optsStart := api.StartOptions{
		Project:        project,
		NavigationMenu: true,
	}
	optsUp := api.UpOptions{
		Create: optsCreate,
		Start:  optsStart,
	}

	project, err = project.WithServicesEnabled()
	if err != nil {
		log.Fatalf("Error enabling services: %v", err)
	}

	// Start the project
	if err := backend.Up(ctx, project, optsUp); err != nil {
		log.Fatalf("Error starting project: %v", err)
	}
}

This is the docker-compose.yaml file in the same directory (“…/compose-go/”):

services:
  example-service:
    image: ubuntu:20.04
    command: ["echo", "Hello, World!"]
    volumes:
      - example-volume:/example
    networks:
      - example-network

volumes:
  example-volume:
    name: example-volume

networks:
  example-network:
    name: example-network
    driver: bridge

When I execute go run main.go, I get the following error:

[+] Running 1/1
 ✔ Container compose-go-example-service-1  Created                                                                     0.1s 
2025/01/27 12:59:47 Error starting project: service "example-service" has no container to start
exit status 1

When I execute it again, I get a different error:

[+] Running 0/1
 ⠋ Container compose-go-example-service-1  Creating                                                                    0.0s 
2025/01/27 13:00:30 Error starting project: Error response from daemon: Conflict. The container name "/compose-go-example-se
rvice-1" is already in use by container "2f26df711ed6d702392caf6a97b64e64f5b747aefd39685988fce5364c58fcf2". You have to remo
ve (or rename) that container to be able to reuse that name.
exit status 1

So it seems that backend.Up successfully creates the container, but doesn’t “recognize” it as its own afterwards.

Does anyone know what I’m doing wrong here?

I don’t know, but have you tried comparing your solution with the source code of the original compose?

It’s okay if you are just sharing it so someone can help to find the difference, but if you haven’t checked, you can try.

Thanks for replying.

Yes, I’ve checked the original compose - in fact, I’ve derived my code from it.
Here’s my analysis of the code flow when running docker compose up with official implementation:

docker/compose/blob/7c7407672acc0d7ddf4613a8d12ea4b5651fc9a0/cmd/main.go#L38 calls plugin.Run from docker/cli/blob/master/cli-plugins/plugin/plugin.go#L87:

dockerCli, err := command.NewDockerCli()
// ...
makeCmd(dockerCli)

(Forgive the verbose references to files - forum doesn’t let me post many links - just add github.com/ in front of them to find the direct link to lines I’m referencing)

Here, makeCmd is the callback function passed to the plugin.Run in the docker/compose/blob/7c7407672acc0d7ddf4613a8d12ea4b5651fc9a0/cmd/main.go#L38, which does some additional setup:

backend := compose.NewComposeService(dockerCli).(commands.Backend)
cmd := commands.RootCommand(dockerCli, backend)
// ...

Afterwards, RunPlugin() is called with dockerCli as (I think) the most relevant argument in docker/cli/blob/master/cli-plugins/plugin/plugin.go#L33:

// ...
cmd.Execute()

The cmd.Execute() then executes *cobra.Command - which, if up arg is specified, runs command added in docker/compose/blob/7c7407672acc0d7ddf4613a8d12ea4b5651fc9a0/cmd/compose/compose.go#L605, and defined in docker/compose/blob/7c7407672acc0d7ddf4613a8d12ea4b5651fc9a0/cmd/compose/up.go#L110, which eventually calls runUp from the same file:

// ...
create := api.CreateOptions{
	Build:                build,
	Services:             services,
	RemoveOrphans:        createOptions.removeOrphans,
	IgnoreOrphans:        createOptions.ignoreOrphans,
	Recreate:             createOptions.recreateStrategy(),
	RecreateDependencies: createOptions.dependenciesRecreateStrategy(),
	Inherit:              !createOptions.noInherit,
	Timeout:              createOptions.GetTimeout(),
	QuietPull:            createOptions.quietPull,
	AssumeYes:            createOptions.AssumeYes,
}
// ...
opts := api.UpOptions{
	Create: create,
	Start: api.StartOptions{
		Project:        project,
		Attach:         consumer,
		AttachTo:       attach,
		ExitCodeFrom:   upOptions.exitCodeFrom,
		OnExit:         upOptions.OnExit(),
		Wait:           upOptions.wait,
		WaitTimeout:    timeout,
		Watch:          upOptions.watch,
		Services:       services,
		NavigationMenu: upOptions.navigationMenu && ui.Mode != "plain",
	},
}
// ...
return backend.Up(ctx, project, opts)

I’ve logged the api.UpOptions of both my implementation and compose implementation, and they are identical (except for the Create.Build which is nil in my implementation, but even after adding it, behavior doesn’t change) - equal to this:

{
  "Create": {
    "AssumeYes": false,
    "Build": {
      "Pull": false,
      "Push": false,
      "Progress": "auto",
      "Args": null,
      "NoCache": false,
      "Quiet": false,
      "Services": null,
      "Deps": false,
      "SSHs": null,
      "Memory": 0,
      "Builder": ""
    },
    "IgnoreOrphans": false,
    "Inherit": true,
    "QuietPull": false,
    "Recreate": "diverged",
    "RecreateDependencies": "diverged",
    "RemoveOrphans": false,
    "Services": [],
    "Timeout": null
  },
  "Start": {
    "Attach": null,
    "AttachTo": [],
    "ExitCodeFrom": "",
    "NavigationMenu": true,
    "OnExit": 0,
    "Project": {
      "name": "compose-go",
      "services": {
        "example-service": {
          "command": [
            "echo",
            "Hello, World!"
          ],
          "entrypoint": null,
          "image": "ubuntu:20.04",
          "networks": {
            "example-network": null
          },
          "volumes": [
            {
              "type": "volume",
              "source": "example-volume",
              "target": "/example",
              "volume": {}
            }
          ]
        }
      },
      "networks": {
        "example-network": {
          "name": "example-network",
          "driver": "bridge",
          "ipam": {}
        }
      },
      "volumes": {
        "example-volume": {
          "name": "example-volume"
        }
      }
    },
    "Services": [],
    "Wait": false,
    "WaitTimeout": 0,
    "Watch": false
  }
}

Since the arguments are exactly the same, I can only assume there is some behind-the-scenes stuff happening during initialization of *cobra.Command. I could use the Compose’s *cobra.Command itself, but my issue is it always calls os.Exit(1) on any error, which - unlike panic - cannot be caught and always crashes the program.

The command initialization code is hairy and all over the place, and I haven’t been able to pinpoint exact thing that makes compose work, and not my implementation.

I was hoping someone experienced with compose codebase could help me out with a hint.
In the meantime, I’m continuing my experiments. If I find a solution I’ll post it here for future readers.

Personally I don’t think you will find a compose developer here. Rather try compose Github.

Who knows. Even I worked with it when I tried to fix a bug, so I would gladly play with it again if I had time. But the compose github repo is mainly for issues rather than asking for help with the API.

@paskozdilar I usually don’t mention Slack, since an answer there helps only for the person who asked the question and maybe people who are there at the moment and had the same problem, which is not likely. But in this case you can try the #docker-compose Slack channel too: https://dockercommunity.slack.com//archives/C2X82D9PA