Host system is Arm64 ,docker container is amd x86

General Discussions General
1

Hi all,
I encountered a trouble.My host system is arm64/kernel 5.4.213 for openwrt ,and docker container is amd64 x86 for ubuntu 22.04 that kernel is 5.4.0-169-generic I think,but it will map to real host system 5.4.213.
I want to insmod a haipci driver from docker container,but it alway told me below:
/lib/modules/5.4.213# insmod hailo_pci.ko
insmod: ERROR: could not insert module hailo_pci.ko: Function not implemented
how come?
The environment as below:
Host Dokcer Container
Openwrt Ubuntu 22.04
Arm64 Amd64
Kernel:5.4.213 Kernel: 5.4.0-169-generic

Thanks.

2

I found that is ioctl restriction ,“Error in ioctl: Function not implemented” as run hailortcli benchmark to read/write /dev/hailo0 ,but I really don’t understand what’s problem.As below is running command:
docker run ‘–cap-add=SYS_RAWIO’ ‘–cap-add=cap_chown’ ‘–cap-add=cap_sys_admin’ ‘–cap-add=sys_nice’ ‘–cap-add=net_raw’ ‘–cap-add=cap_sys_rawio’ ‘–cap-add=cap_sys_module’ ‘–cap-add=NET_ADMIN’ --platform linux/amd64 ‘–net=host’ --device /dev/hailo0:/dev/hailo0:rw --volume /usr/bin/qemu-x86_64:/usr/bin/qemu-x86_64 -v /dev:/dev:rw -v /lib/modules:/lib/modules:rw -v /lib/firmware:/lib/firmware -v /lib/udev/rules.d:/lib/udev/rules.d -v /mnt/sda1/shared_with_docker/:/local/shared_with_docker:rw --name max1 -ti hailort:4.16.0
.I stuck the issue a couple of weeks.

3

Docker is for containers, they need to have the same architecture. You can’t just run x86 on arm.

--platform linux/amd64 usually just pulls the target architecture image, for executing you need a virtual machine (VM) which emulates the target architecture.

It might be possible to integrate qemu emulation with latest buildx, not sure if this can only be used for building images or also for running them.

4

Thanks,bluepuma,
Yes,I integrated a buildx into my docker platform also can work it,but I encounted ioctl issue as using "hailortcli benchmark " to read/write /dev/hailo0.I worried about maybe some missing in host or container,so I need to help it.Thanks
As below is detailed:

./hailortcli benchmark 
/resnet_v1_50.hef

Starting Measurements

Measuring FPS in hw_only mode
Error in ioctl: Function not implemented
[HailoRT] [error] Failed query driver info, errno 38
[HailoRT] [error] CHECK_SUCCESS_AS_EXPECTED failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_EXPECTED failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_EXPECTED failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_EXPECTED failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_EXPECTED failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_EXPECTED failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_EXPECTED failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_EXPECTED failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT CLI] [error] CHECK_EXPECTED failed with status=HAILO_DRIVER_FAIL(36) - Failed creating vdevice
[HailoRT CLI] [error] CHECK_EXPECTED_AS_STATUS failed with status=HAILO_DRIVER_FAIL(36) - hw_only measuring failed

5

Though, it is what many people with Docker Desktop for Mac do :slight_smile:
Just with the different that the utility vm that runs the Docker Engine for Docker Desktop comes with a kernel that has all required modules, and all required os packages to make it work.

Maybe the official documentation provides details you haven’t checked already?

Furthermore, I would suggest asking the maintainers of the openwrt docker package, as they likely know what needs to be done to run containers based on images of another cpu platform.

2 Likes
6

image
image1499×785 60.9 KB

7

image
image1615×520 23.7 KB

8

image
image1228×205 5.8 KB

9

Yeah, but Docker Desktop includes a VM :sunglasses:

10

It is not the virtual machine that helps, but two packages. qemu-user-static and binfmt-support. I have no idea how that can be installed on openwrt if it is possible at all. Emulation and virtualization is not the same thing. Only the same architecture can be virtualized as the host has, but qemu can emulate other architectures. It would not help with the missing kernel module though.

2 Likes
11

Oddly there is a page about qemu in the OpenWrt Wiki: [OpenWrt Wiki] OpenWrt as QEMU/KVM host server

I have a router with OpenWRT 23.05.03 on an aarch64/ARMv8 architecture cpu. When I try to find the packages with opkg find qemu*, nothing is found.

12

So,qemu-amd_x86 binary is typically used for user-mode emulation,not including the CPU, memory, peripherals, and devices.
I don’t know below work or not.
python hailort API(docker/amd64_x86)–> libhailort (docker/amd64_x86) <-mapping to ->libhailort (physical/arm64) <–>hailort Driver(physical/arm64)

13

image
image1041×89 4.54 KB

Can I use the same idea like /usr/bin/qemu-x86_64 ./hailort_service ./hailort_service in host arm64?
I mean like “/usr/bin/qemu-x86_64-static /container/path/to/libhailort.so”

14

Thanks,Unfortunely,Our HW platform implemented in OpenWRT 19.07.However,thanks so much.Maybe need to get it from another version and try to port it.

15

I can’t tell you how docker uses the under the hood. Though, I can say that on supported platforms with the installed required packages you just need to provide the parameters @bluepuma77 provided.

Like I wrote earlier: your best bet is to ask the maintainer of the OpenWRT dockerd package. Since it’s a legit docker use case, maybe you can convince them to implement support for what you are looking for.