TFTP "Destination Unreachable" Issue in Containerized Setup

General
1

I’m working on a project that involves flashing ONT (Optical Network Terminal) devices using a Java-based application running in Docker containers. The process requires communicating with the ONTs via telnet and TFTP. While the telnet communication works fine, I’m experiencing issues with the TFTP transfer, specifically when trying to send a file to the ONTs.

The setup works perfectly until the file transfer stage via TFTP. At this point, the ONT sends a TFTP Read Request to the TFTP server running on the host (not in the containers). The host receives the request, but no file is sent, and the transfer fails with a “Destination unreachable (port unreachable)” error.

Technical Stack:

  • Java Application containerized with Docker.
  • Selenium for automating device interactions.
  • Telnet and TFTP protocols for communication and file transfer.
  • TFTP-HPA running on the host system, not inside the containers.
  • Docker Macvlan network used to assign IPs to containers and communicate with ONTs.
  • TP-Link Switch with VLANs to separate traffic between different ONT devices.

Detailed Setup:

  1. Docker Containers: Containers run the Java app and communicate with ONT devices via telnet. The file transfer is initiated using TFTP.
  2. TFTP Server: The TFTP server is TFTP-HPA, running on the host machine, not inside the Docker containers.
  3. VLAN Setup: Each ONT device is on a separate VLAN, and containers are mapped to these VLANs via a macvlan network. Containers have individual IP addresses like 192.168.100.10, 192.168.100.11, etc.

Observed Behavior:

  • The ONT device sends a TFTP Read Request to the host (which is verified via Wireshark and tcpdump).
  • The host receives the request, but instead of responding with the file, it sends a Destination unreachable (port unreachable) error.
  • Here’s a sample Wireshark trace of the error:

yaml

Copy code

Internet Protocol Version 4, Src: 192.168.100.1, Dst: 192.168.100.11
User Datagram Protocol, Src Port: 49401, Dst Port: 69
Trivial File Transfer Protocol
    Opcode: Read Request (1)
    Source File: M1.bin
    Option: tsize = 0
    Option name: tsize
    Option value: 0
Internet Control Message Protocol
    Type: 3 (Destination unreachable)
    Code: 3 (Port unreachable)

image
image1282×414 144 KB

  • The TFTP transfer does not complete, and no files are transferred.

Steps Already Taken:

  1. Network Traffic Inspection:
  • Used Wireshark and tcpdump to confirm that the TFTP Read Requests are arriving at the host machine.
  • Confirmed the “Destination unreachable” error originates from the host.
  1. Firewall Disabled:
  • I disabled the firewall (UFW) on the host machine to rule out firewall interference. The issue persists even with UFW disabled.
  1. Docker Network Configuration:
  • Using a macvlan network to isolate traffic between containers and ONTs. Each container has its own IP and VLAN configuration to avoid conflicts.
  1. Port Binding:
  • The host machine listens on port 69 for TFTP transfers, and the containers expose ports for telnet (23) and TFTP (69).

Suspected Cause:

It seems that the port negotiation required for the TFTP transfer is not happening properly. TFTP uses a two-step process:

  1. The ONT sends a read request on port 69.
  2. The server responds by using a random ephemeral port to transfer the file.

I suspect that the response from the host machine (the TFTP server) is failing due to network misconfiguration, where the ephemeral port is not being routed back to the ONT device, resulting in the Destination unreachable (port unreachable) error.

Environment Details:

  • Host OS: Ubuntu 22.04
  • Docker Version: 24.0.2
  • TFTP Server: TFTP-HPA
  • Network: Using macvlan to isolate traffic between containers and ONTs
  • ONT Devices: Communicating via telnet and TFTP

Specific Questions:

  1. Has anyone encountered similar issues where the TFTP server running on the host system does not properly negotiate ports for file transfer in a containerized environment?
  2. Could this be a routing issue between the TFTP server’s ephemeral ports and the ONT device?
  3. How can I troubleshoot or configure the Docker macvlan network or host system to ensure that TFTP transfers complete successfully?
  4. Are there any alternative setups for handling multiple containers that need to use the same TFTP server?

Any insights or suggestions would be greatly appreciated!

2

Do I understand correctly that the containerized app instructs the ONT device to request the image from the TFTP directly?

Thus, the containerized app only controls the ONT device, but is not directly involved in the TFTP communication?