I am building a RAG using ollama in docker environment on Windows 11. There are some issues I am facing while binding the host directory with the container.
Following is the docker compose file:
version: '3.8'
services:
qdrant:
image: qdrant/qdrant:v1.12.1
container_name: qdrant
ports:
- "6333:6333" # Expose Qdrant on the default port
volumes:
#- qdrant_data:/qdrant/storage
- d:/qdrant_storage:/qdrant/storage # Map D:\qdrant_storage to /qdrant/storage
- "D:/Softwares/VS Code Files/DocBot APP/PDF_Database:/qdrant/pdf_storage" # Map PDF_Database to /qdrant/pdf_storage
networks:
- my_network # Connect qdrant to my_network
ollama:
image: ollama/ollama:latest # Replace with the actual Ollama server image
container_name: ollama
ports:
- "11434:11434" # Expose Ollama on the default port (change if needed)
environment:
- OLLAMA_MODEL=llama3.2:3b # Set the default model name for Ollama if needed
command: ollama serve # Start the Ollama server to listen for requests
volumes:
#- /d/myollamamodels:/models
#- ./myollamamodels:/root/.ollama
- /d/myollamamodels:/root/.ollama
networks:
- my_network
app:
image: docbot-app-v6.0:latest
build: .
container_name: docbot_v6
ports:
- "8501:8501" # Streamlit default port
environment:
- QDRANT_URL: http://qdrant:6333 # Use Qdrant service name from Docker Compose
- OLLAMA_URL: http://ollama:11434
#OLLAMA_MODEL: http://host.docker.internal:11434/llama3.2:3b # Point to Ollama on host
- UPLOAD_DRIVE: D:/
depends_on:
- qdrant
- ollama
volumes:
- ./models:/models # Mount the model directory for access
#- "D:/Softwares/VS Code Files/DocBot APP:/app" # Bind mount code folder for real-time updates
#- "D:/Softwares/VS Code Files/DocBot APP/PDF_Database:/app/PDF_Database" # Sync uploaded files in real-time
#- "D:/Softwares/VS Code Files/DocBot APP/PDF_Database:/app/PDF_Database"
- type: bind
#source: "/d/Softwares/VS Code Files/DocBot APP/PDF_Database"
source: "D:/Softwares/VS Code Files/DocBot APP/PDF_Database"
target: /app/PDF_Database
#- "D:/NH_Docbot_Logs:/app/Logs"
- type: bind
#source: "/d/Softwares/VS Code Files/DocBot APP/Logs"
source: "D:/Softwares/VS Code Files/DocBot APP/Logs"
target: /app/Logs
networks:
- my_network # Connect app to my_network
volumes:
qdrant_data:
networks:
my_network:
driver: bridge
#driver: overlay # Overlay network for Swarm
Now the issue comes when i try to upload the document on streamlit app and though the document is uploaded but its there in the container and NOT in the host directory which is in D Drive. I have confirmed by using âdocker exec -it container name bashâ and then ls⊠and the file uploaded is found there.
The streamlit code that I am using to upload is -
uploaded_files = st.file_uploader("đUpload PDFs to create embeddings", type=["pdf"], accept_multiple_files=True)
# Define absolute path to store PDFs
pdf_database_dir = "/app/PDF_Database"
if uploaded_files:
embeddings_manager = PDFDatabaseManager(
model_name="BAAI/bge-small-en",
#device="cpu",
device=get_available_device(), # Dynamic device assignment
encode_kwargs={"normalize_embeddings": True},
qdrant_url=qdrant_url,
collection_name="vector_db"
)
with st.spinner("đ Creating embeddings..."):
for uploaded_file in uploaded_files:
pdf_path = os.path.join(pdf_database_dir, uploaded_file.name)
with open(pdf_path, "wb") as f:
f.write(uploaded_file.getbuffer())
st.write(f"File saved to: {pdf_path}")
st.session_state['pdf_paths'].append(pdf_path)
result = embeddings_manager.add_pdf_to_database(pdf_path)
st.success(result)
# Update the chatbot manager after embeddings are created
if st.session_state['chatbot_manager'] is None:
st.session_state['chatbot_manager'] = ChatbotManager(
model_name="BAAI/bge-small-en",
#device="cpu",
device=get_available_device(), # Dynamic device assignment
encode_kwargs={"normalize_embeddings": True},
llm_model="llama3.2:3b",
llm_temperature=0.7,
qdrant_url=qdrant_url,
collection_name="vector_db",
ollama_url=ollama_url
)
I am using Docker Desktop latest version and back end is WSL 2 as is checked using docker info. Docker Installation and WSL are in D Drive. Please help me this as i have tried like bunch of things in docker documentation but nothing seems to be working.
I just tried to reproduce your problem with this simple compose file:
services:
alpine:
image: alpine:latest
command: sh -c 'date | tee /mnt/data/date.txt'
volumes:
- D:\dev\compose-tests:/mnt/data
It creates a container with a volume, prints the date and writes it into a file.
# folder only contains the compose.yml
PS D:\dev\compose-tests> ls
Directory: D:\dev\compose-tests
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 13.12.2024 19:50 155 compose.yml
PS D:\dev\compose-tests> docker compose up
[+] Running 2/2
â Network compose-tests_default Created 0.1s
â Container compose-tests-alpine-1 Created 0.1s
Attaching to alpine-1
alpine-1 | Fri Dec 13 18:51:19 UTC 2024
alpine-1 exited with code 0
PS D:\dev\compose-tests>
PS D:\dev\compose-tests> cat .\date.txt
Fri Dec 13 18:51:19 UTC 2024
The result is the same (of course with different timestamps), regardless whether the bind uses backslashes in the path D:\dev\compose-tests:/mnt/data, or slashes like this: D:/dev/compose-tests:/mnt/data
Did you try Windows host paths that donât have space characters in it? At least it would help to rule out whether they cause a problem.
Update: I just checked with a subfolder that has a space in it. It still works.
There is a docker container inspect command which can at least tell you whether the mount was interpreted correctly. If you see something incorrect in the output, then it could be caused by the terminal software like Git Bash. I also remember that you canât use a USB device even if you see it on the host as âDâ drive. But I think in that case you would have an error message, but Iâm not sure.
On using the Inspect command for the container, the mount label seems to be empty. I donât know why it showing like that in spite of command being right. The PDF file does get uploaded inside the container but not on host directory. I read it somewhere in the docker documentation that both the container and bind mount folder are in sync, so even if we upload on the container it will sync with the host sub directory, That is what is not happening.
Below is the inspection status:
Hi, i just checked the same issue is there with path name with spaces. I have even used double backward slashes for source path in docker compose file and it doesnât work.
I have resolved the issue related to host directory sync with the container. Basically was using âbuild.â which was not required and i manually mounted the container using command.
Thanks for the help.
But i am now struck in the issue related to docker swarm. I am now creating a replicas for ollama, qdrant and app containers for scaling up and initially i have pulled up a image of ollama and pulled the ML Model in the D drive âmyollamamodelsâ directory.
The issue is the replicas of ollama in swarm are not able to found the model and throwing up the error that model not found. following is the docker-stack.yml:
version: '3.8'
services:
qdrant:
image: qdrant/qdrant:v1.12.1 # Use the latest stable Qdrant version
container_name: qdrant
deploy:
replicas: 1 # Typically one replica is sufficient for Qdrant
resources:
limits:
cpus: '1' # Minimal CPU for Qdrant
memory: 1G
restart_policy:
condition: on-failure
ports:
- "6333:6333" # Expose Qdrant on the default port
volumes:
#- qdrant_data:/qdrant/storage
- d:/qdrant_storage:/qdrant/storage # Map D:\qdrant_storage to /qdrant/storage
networks:
- my_network # Connect qdrant to my_network
ollama:
image: ollama/ollama:latest # Replace with the actual Ollama server image
container_name: ollama
deploy:
replicas: 1 # Start with 2 replicas
resources:
limits:
cpus: '1' # Allocate 4 CPUs per replica
memory: 2G # More memory for large models
restart_policy:
condition: on-failure
environment:
#- NVIDIA_VISIBLE_DEVICES=all # Enable GPU visibility
ports:
- "11434:11434" # Expose Ollama on the default port (change if needed)
environment:
- OLLAMA_MODEL=llama3.2:3b # Set the default model name for Ollama if needed
#command: ollama serve # Start the Ollama server to listen for requests
volumes:
#- /d/myollamamodels:/models
#- ./myollamamodels:/root/.ollama
#- /d/myollamamodels:/root/.ollama
- ollama_model:/root/.ollama # Use Docker volume
networks:
- my_network
app:
image: docbot-app-v6.0:latest
ports:
- "8501:8501" # Streamlit default port
environment:
QDRANT_URL: http://qdrant:6333 # Use Qdrant service name from Docker Compose
OLLAMA_URL: http://ollama:11434
#OLLAMA_MODEL: http://host.docker.internal:11434/llama3.2:3b # Point to Ollama on host
depends_on:
- qdrant
- ollama
volumes:
- ./models:/models # Mount the model directory for access
- type: bind
source: "/d/Softwares/VS Code Files/DocBot APP/PDF_Database"
target: /app/PDF_Database
#- "D:/Softwares/VS Code Files/DocBot APP/PDF_Database:/app/PDF_Database" # Sync uploaded files in real-time
#- "D:/Docbot_Logs:/app/Logs"
- type: bind
source: "/d/Softwares/VS Code Files/DocBot APP/Logs"
target: /app/Logs
networks:
- my_network # Connect app to my_network
deploy:
replicas: 1 # To be Adjusted based on testing
resources:
limits:
cpus: '2' # Allocate 1 CPUs per app replica
memory: 1G # App memory
restart_policy:
condition: on-failure
labels:
- traefik.http.services.app.loadbalancer.sticky = true #Enable Sticky sessions.
volumes:
qdrant_data:
ollama_model: # This is a Docker-managed volume
#driver: local # Use the local driver (default)
#driver_opts:
# type: none
# device: "D:\\myollamamodels"
# o: bind # Use bind mount as part of volume configuration
networks:
my_network:
#driver: bridge
driver: overlay # Overlay network for Swarm
Tried using volumes and by specifying driver, type and bind as in above code but it throws the error:
(HTTP code 500) server error - error while mounting volume '/var/lib/docker/volumes/docbot_stack_ollama_model/_data': failed to mount local volume: mount D:/myollamamodels:/var/lib/docker/volumes/docbot_stack_ollama_model/_data, flags: 0x1000: no such file or directory
what might be the issue, can you help figure it out with docker swarm.
if binds are used, the host folder needs to exist on every node the swarm task (=container) get spawned and needs to have the same data. The same applies to named volumes backed by a bind.
if named volumes backed by a remote share are used, they must have the same configuration on all nodes (note: once created their configuration is imutable, so they need to be removed so they can be re-created with updated configuration)
Swarm will not replicate the content of a volume or bind amongst the nodes.
Thatâs why people usually use named volumes backed by cifs or preferably nfs4 remote shares, so that every node is able to access the share and use the files within.
If you walk down the bind path, you can use host labels and matching placement constraints to pin the swarm tasks to specific nodes. You would still have to make sure the same data is present in the same path on those hosts.
I wont be using multiple nodes (physical machines or workers), there will be ONLY ONE physical machine which will be using docker swarm to run several instances of ollama, app and qdrant, now please help how shall i configure the stack file . I dont think NFS will be needed then
I am trying to use the following but its not working.
version: "3.9"
services:
ollama:
image: ollama/ollama
deploy:
replicas: 3 # Run multiple instances of the service
volumes:
- ollama_model:/models # Mount the shared directory into the container
ports:
- "11434:11434"
environment:
- OLLAMA_MODEL_PATH=/models
volumes:
ollama_model:
driver: local
driver_opts:
type: "none"
o: "bind"
device: "D:/myollamamodels"