Docker Community Forums

Share and learn in the Docker community.

Mailcatcher not working properly with Haproxy


(Yourcursus) #1

For my staging environment I’m using mailcatcher.

In my stack file I have an lb node using the tutum haproxy image, which is linked to my application and to mailcatcher in order to access to web UI.
Then my application is linked to the mailcatcher instance so that it can send emails to it.

When I try to access the mailcatcher page, I can access it, but I have some calls failing with a 502 error. For instance, while the HMTL request worked fine (200), and same for the CSS request, the JS download is failing with the 502 error.
Next try, this time the CSS wont load (502) but the JS will.
Another try and only the HTML works, or even worst, nothing work, I have 502 error on the HTML download.

Before using Docker cloud and haproxy, I was using an nginx-proxy docker image, and docker-compose in order to orchestrate the app.
I have never seen this behavior before.

Mailcatcher is using websockets, so my first though was that the WS connection is the issue. But in the end it’s not, as first step would be to have the CSS and the JS loading correctly.
Also, I tried to change the image, and here I got it working fine, but the day after I had to redeploy the instance and I got the errors again…

I have tried to add the -v falg to the mailcatcher process, but I don’t have more logs and I don’t see anything special on the haproxy instance.

Can someone help me please ?


(Fernando Mayo) #2

I think in your case HAproxy is balancing requests between the two backend services in a round robin fashion. Have you specified virtual host configuration for HAproxy to route to the appropriate backend service?


(Yourcursus) #3

Thank you Fernando for your answer.

I’m not sure to understand what you meant with “HAproxy is balancing requests between the two backend services”.
And to reply to “Have you specified virtual host configuration” :

The LB instance is serving “staging.yourcursus.com” traffic to my Rails app, and mailcatcher.staging.yourcursus.com to the mailcatcher instance.
They each has the VIRTUAL_HOST environment variable defined with the given values.

While the Rails app is working like a charm, mailcatcher has this strange behavior.


(Tifayuki) #4

dockercloud/haproxy image is “service agnostic”. It should not behave differently on mailcatcher and rail apps.

Can you paste your stack file and the configuration of haproxy printed in the dockercloud/haproxy container logs(with any sensitive information removed)?


(Yourcursus) #5

Thank you @tifayuki for your reply.

Stack file


cursus:
image: 'yourcursus/cursus:0.10.0’
environment:
- CLOSED_BETA=true
- RAILS_ENV=staging
- VIRTUAL_HOST=staging.yourcursus.com
links:
- mailcatcher
- postgres
- redis
restart: always
tags:
- staging
target_num_containers: 2
lb:
image: 'dockercloud/haproxy:latest’
environment:
- 'HTTP_BASIC_AUTH=???:???'
links:
- cursus
- mailcatcher
ports:
- '80:80’
roles:
- global
tags:
- staging
mailcatcher:
image: 'schickling/mailcatcher:latest’
autoredeploy: true
command: 'mailcatcher --verbose --foreground --ip=0.0.0.0’
environment:
- VIRTUAL_HOST=mailcatcher.staging.yourcursus.com
ports:
- '1080’
restart: always
tags:
- staging
postgres:
image: 'postgres:9.5.1’
tags:
- staging
redis:
image: 'redis:3.0.7’
tags:
- staging

The dockercloud/haproxy container logs


2016-03-05T13:24:45.802060005Z INFO:haproxy:dockercloud/haproxy 1.1.1 has access to the cloud API - will reload list of backends in real-time
2016-03-05T13:24:45.802377976Z INFO:haproxy:HAProxy PID: 1
2016-03-05T13:24:46.269540841Z INFO:haproxy:==========BEGIN==========
2016-03-05T13:24:46.269793083Z INFO:haproxy:Websocket open
2016-03-05T13:24:47.568831006Z INFO:haproxy:Service links: CURSUS(7a9bd28d-4be0-4a79-9052-024427b6585b), MAILCATCHER(3c919184-107f-4eb5-b77a-abeb374eccf7)
2016-03-05T13:24:47.569151801Z INFO:haproxy:Container links: CURSUS_1(962b806e-8a0d-40b0-a357-9a922851c52b), MAILCATCHER_1(cae4d370-270e-41bf-9bbd-b0f6003bccc5)
2016-03-05T13:24:47.581684462Z INFO:haproxy:HAProxy configuration:
2016-03-05T13:24:47.581775174Z global
2016-03-05T13:24:47.581792778Z log 127.0.0.1 local0
2016-03-05T13:24:47.581802305Z log 127.0.0.1 local1 notice
2016-03-05T13:24:47.581810258Z log-send-hostname
2016-03-05T13:24:47.581817993Z maxconn 4096
2016-03-05T13:24:47.581825642Z pidfile /var/run/haproxy.pid
2016-03-05T13:24:47.581833318Z user haproxy
2016-03-05T13:24:47.581840897Z group haproxy
2016-03-05T13:24:47.581853244Z daemon
2016-03-05T13:24:47.581862268Z stats socket /var/run/haproxy.stats level admin
2016-03-05T13:24:47.581870444Z ssl-default-bind-options no-sslv3
2016-03-05T13:24:47.581880025Z ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:AES256-GCM-SHA384:AES256-SHA256:AES256-SHA:DHE-DSS-AES128-SHA:DES-CBC3-SHA
2016-03-05T13:24:47.581903015Z defaults
2016-03-05T13:24:47.581911031Z balance roundrobin
2016-03-05T13:24:47.581918570Z log global
2016-03-05T13:24:47.581926172Z mode http
2016-03-05T13:24:47.581936390Z option redispatch
2016-03-05T13:24:47.581949065Z option httplog
2016-03-05T13:24:47.581961222Z option dontlognull
2016-03-05T13:24:47.581973620Z option forwardfor
2016-03-05T13:24:47.581986719Z timeout connect 5000
2016-03-05T13:24:47.581998609Z timeout client 50000
2016-03-05T13:24:47.582006128Z timeout server 50000
2016-03-05T13:24:47.582013506Z listen stats
2016-03-05T13:24:47.582020978Z bind :1936
2016-03-05T13:24:47.582028507Z mode http
2016-03-05T13:24:47.582035772Z stats enable
2016-03-05T13:24:47.582043611Z timeout connect 10s
2016-03-05T13:24:47.582051130Z timeout client 1m
2016-03-05T13:24:47.582073272Z timeout server 1m
2016-03-05T13:24:47.582081805Z stats hide-version
2016-03-05T13:24:47.582106067Z stats realm Haproxy\ Statistics
2016-03-05T13:24:47.582116485Z stats uri /
2016-03-05T13:24:47.582124071Z stats auth stats:stats
2016-03-05T13:24:47.582131766Z userlist haproxy_userlist
2016-03-05T13:24:47.582139402Z user ??? insecure-password ???
2016-03-05T13:24:47.582146884Z frontend port_80
2016-03-05T13:24:47.582154456Z bind :80
2016-03-05T13:24:47.582162021Z acl is_websocket hdr(Upgrade) -i WebSocket
2016-03-05T13:24:47.582169764Z acl host_rule_1 hdr(host) -i mailcatcher.staging.yourcursus.com
2016-03-05T13:24:47.582177339Z acl host_rule_1_port hdr(host) -i mailcatcher.staging.yourcursus.com:80
2016-03-05T13:24:47.582201834Z use_backend SERVICE_MAILCATCHER if host_rule_1 or host_rule_1_port
2016-03-05T13:24:47.582209977Z acl host_rule_2 hdr(host) -i staging.yourcursus.com
2016-03-05T13:24:47.582217753Z acl host_rule_2_port hdr(host) -i staging.yourcursus.com:80
2016-03-05T13:24:47.582229793Z use_backend SERVICE_CURSUS if host_rule_2 or host_rule_2_port
2016-03-05T13:24:47.582238066Z backend SERVICE_MAILCATCHER
2016-03-05T13:24:47.582245425Z acl need_auth http_auth(haproxy_userlist)
2016-03-05T13:24:47.582253290Z http-request auth realm haproxy_basic_auth if !need_auth
2016-03-05T13:24:47.582261073Z server MAILCATCHER_1 10.7.0.2:1025 check
2016-03-05T13:24:47.582275893Z server MAILCATCHER_1 10.7.0.2:1080 check
2016-03-05T13:24:47.582283559Z backend SERVICE_CURSUS
2016-03-05T13:24:47.582291498Z acl need_auth http_auth(haproxy_userlist)
2016-03-05T13:24:47.582299427Z http-request auth realm haproxy_basic_auth if !need_auth
2016-03-05T13:24:47.582307453Z server CURSUS_1 10.7.0.8:3000 check
2016-03-05T13:24:47.582491586Z INFO:haproxy:Launching HAProxy
2016-03-05T13:24:47.590770720Z INFO:haproxy:HAProxy has been launched(PID: 23)
2016-03-05T13:24:47.590978461Z INFO:haproxy:===========END===========
2016-03-07T06:56:06.945868064Z INFO:haproxy:==========BEGIN==========
2016-03-07T06:56:06.946030085Z INFO:haproxy:Event: container 962b806e-8a0d-40b0-a357-9a922851c52b is terminated
2016-03-07T06:56:08.072417359Z INFO:haproxy:Service links: CURSUS(7a9bd28d-4be0-4a79-9052-024427b6585b), MAILCATCHER(3c919184-107f-4eb5-b77a-abeb374eccf7)
2016-03-07T06:56:08.072722973Z INFO:haproxy:Container links: CURSUS_1(5a9d15a1-ac47-42c6-972e-512c62258644), MAILCATCHER_1(cae4d370-270e-41bf-9bbd-b0f6003bccc5)
2016-03-07T06:56:08.079880485Z INFO:haproxy:HAProxy configuration remains unchanged
2016-03-07T06:56:08.080106112Z INFO:haproxy:===========END===========
2016-03-07T06:56:08.081445133Z INFO:haproxy:==========BEGIN==========
2016-03-07T06:56:08.081679464Z INFO:haproxy:Event: container 5a9d15a1-ac47-42c6-972e-512c62258644 is running
2016-03-07T06:56:08.439929092Z INFO:haproxy:Service links: CURSUS(7a9bd28d-4be0-4a79-9052-024427b6585b), MAILCATCHER(3c919184-107f-4eb5-b77a-abeb374eccf7)
2016-03-07T06:56:08.440365217Z INFO:haproxy:Container links: CURSUS_1(5a9d15a1-ac47-42c6-972e-512c62258644), MAILCATCHER_1(cae4d370-270e-41bf-9bbd-b0f6003bccc5)
2016-03-07T06:56:08.449723846Z INFO:haproxy:HAProxy configuration remains unchanged
2016-03-07T06:56:08.449981813Z INFO:haproxy:===========END===========
2016-03-07T06:58:09.121720508Z INFO:haproxy:==========BEGIN==========
2016-03-07T06:58:09.121784222Z INFO:haproxy:Event: container 5a9d15a1-ac47-42c6-972e-512c62258644 is terminated
2016-03-07T06:58:09.883602597Z INFO:haproxy:Service links: CURSUS(7a9bd28d-4be0-4a79-9052-024427b6585b), MAILCATCHER(3c919184-107f-4eb5-b77a-abeb374eccf7)
2016-03-07T06:58:09.883812602Z INFO:haproxy:Container links: CURSUS_1(b6c3f198-bd1a-4b49-9f72-b8b6cf5a3d9f), MAILCATCHER_1(cae4d370-270e-41bf-9bbd-b0f6003bccc5)
2016-03-07T06:58:09.893373189Z INFO:haproxy:HAProxy configuration remains unchanged
2016-03-07T06:58:09.893502046Z INFO:haproxy:===========END===========
2016-03-07T06:58:09.894685017Z INFO:haproxy:==========BEGIN==========
2016-03-07T06:58:09.894822686Z INFO:haproxy:Event: container b6c3f198-bd1a-4b49-9f72-b8b6cf5a3d9f is running
2016-03-07T06:58:10.244462548Z INFO:haproxy:Service links: CURSUS(7a9bd28d-4be0-4a79-9052-024427b6585b), MAILCATCHER(3c919184-107f-4eb5-b77a-abeb374eccf7)
2016-03-07T06:58:10.244628687Z INFO:haproxy:Container links: CURSUS_1(b6c3f198-bd1a-4b49-9f72-b8b6cf5a3d9f), MAILCATCHER_1(cae4d370-270e-41bf-9bbd-b0f6003bccc5)
2016-03-07T06:58:10.251904999Z INFO:haproxy:HAProxy configuration remains unchanged
2016-03-07T06:58:10.252052583Z INFO:haproxy:===========END===========
2016-03-07T07:00:37.060772705Z INFO:haproxy:==========BEGIN==========
2016-03-07T07:00:37.061297172Z INFO:haproxy:Event: container d156455f-6097-4171-b1d5-8e8ef5602205 is running
2016-03-07T07:00:37.976965044Z INFO:haproxy:Service links: CURSUS(7a9bd28d-4be0-4a79-9052-024427b6585b), MAILCATCHER(3c919184-107f-4eb5-b77a-abeb374eccf7)
2016-03-07T07:00:37.977359851Z INFO:haproxy:Container links: CURSUS_1(b6c3f198-bd1a-4b49-9f72-b8b6cf5a3d9f), CURSUS_2(d156455f-6097-4171-b1d5-8e8ef5602205), MAILCATCHER_1(cae4d370-270e-41bf-9bbd-b0f6003bccc5)
2016-03-07T07:00:37.993426993Z INFO:haproxy:HAProxy configuration:
2016-03-07T07:00:37.993456071Z global
2016-03-07T07:00:37.993463299Z log 127.0.0.1 local0
2016-03-07T07:00:37.993469389Z log 127.0.0.1 local1 notice
2016-03-07T07:00:37.993475411Z log-send-hostname
2016-03-07T07:00:37.993481256Z maxconn 4096
2016-03-07T07:00:37.993487129Z pidfile /var/run/haproxy.pid
2016-03-07T07:00:37.993493103Z user haproxy
2016-03-07T07:00:37.993498921Z group haproxy
2016-03-07T07:00:37.993504458Z daemon
2016-03-07T07:00:37.993510634Z stats socket /var/run/haproxy.stats level admin
2016-03-07T07:00:37.993531454Z ssl-default-bind-options no-sslv3
2016-03-07T07:00:37.993573567Z ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:AES256-GCM-SHA384:AES256-SHA256:AES256-SHA:DHE-DSS-AES128-SHA:DES-CBC3-SHA
2016-03-07T07:00:37.993583576Z defaults
2016-03-07T07:00:37.993588698Z balance roundrobin
2016-03-07T07:00:37.993593746Z log global
2016-03-07T07:00:37.993598803Z mode http
2016-03-07T07:00:37.993604065Z option redispatch
2016-03-07T07:00:37.993609393Z option httplog
2016-03-07T07:00:37.993614579Z option dontlognull
2016-03-07T07:00:37.993619736Z option forwardfor
2016-03-07T07:00:37.993624729Z timeout connect 5000
2016-03-07T07:00:37.993629676Z timeout client 50000
2016-03-07T07:00:37.993634500Z timeout server 50000
2016-03-07T07:00:37.993639450Z listen stats
2016-03-07T07:00:37.993644320Z bind :1936
2016-03-07T07:00:37.993649041Z mode http
2016-03-07T07:00:37.993653838Z stats enable
2016-03-07T07:00:37.993658634Z timeout connect 10s
2016-03-07T07:00:37.993663431Z timeout client 1m
2016-03-07T07:00:37.993668197Z timeout server 1m
2016-03-07T07:00:37.993677478Z stats hide-version
2016-03-07T07:00:37.993682495Z stats realm Haproxy\ Statistics
2016-03-07T07:00:37.993687561Z stats uri /
2016-03-07T07:00:37.993692436Z stats auth stats:stats
2016-03-07T07:00:37.993697419Z userlist haproxy_userlist
2016-03-07T07:00:37.993702336Z user ??? insecure-password ???
2016-03-07T07:00:37.993707311Z frontend port_80
2016-03-07T07:00:37.993712147Z bind :80
2016-03-07T07:00:37.993717155Z acl is_websocket hdr(Upgrade) -i WebSocket
2016-03-07T07:00:37.993722248Z acl host_rule_1 hdr(host) -i mailcatcher.staging.yourcursus.com
2016-03-07T07:00:37.993727314Z acl host_rule_1_port hdr(host) -i mailcatcher.staging.yourcursus.com:80
2016-03-07T07:00:37.993732493Z use_backend SERVICE_MAILCATCHER if host_rule_1 or host_rule_1_port
2016-03-07T07:00:37.993737592Z acl host_rule_2 hdr(host) -i staging.yourcursus.com
2016-03-07T07:00:37.993742585Z acl host_rule_2_port hdr(host) -i staging.yourcursus.com:80
2016-03-07T07:00:37.993747513Z use_backend SERVICE_CURSUS if host_rule_2 or host_rule_2_port
2016-03-07T07:00:37.993752612Z backend SERVICE_MAILCATCHER
2016-03-07T07:00:37.993757471Z acl need_auth http_auth(haproxy_userlist)
2016-03-07T07:00:37.993767694Z http-request auth realm haproxy_basic_auth if !need_auth
2016-03-07T07:00:37.993772980Z server MAILCATCHER_1 10.7.0.2:1025 check
2016-03-07T07:00:37.993778041Z server MAILCATCHER_1 10.7.0.2:1080 check
2016-03-07T07:00:37.993782994Z backend SERVICE_CURSUS
2016-03-07T07:00:37.993787844Z acl need_auth http_auth(haproxy_userlist)
2016-03-07T07:00:37.993792928Z http-request auth realm haproxy_basic_auth if !need_auth
2016-03-07T07:00:37.993797973Z server CURSUS_1 10.7.0.8:3000 check
2016-03-07T07:00:37.993802964Z server CURSUS_2 10.7.0.18:3000 check
2016-03-07T07:00:37.994457321Z INFO:haproxy:Reloading HAProxy
2016-03-07T07:00:38.005159419Z INFO:haproxy:HAProxy has been reloaded(PID: 90)
2016-03-07T07:00:38.005405984Z INFO:haproxy:===========END===========
2016-03-07T07:00:38.015887268Z INFO:haproxy:HAProxy(PID:23) has been terminated
2016-03-07T11:33:42.382361628Z INFO:haproxy:Websocket close
2016-03-07T11:33:42.837082029Z INFO:haproxy:==========BEGIN==========
2016-03-07T11:33:42.837370249Z INFO:haproxy:Websocket open
2016-03-07T11:33:44.532179656Z INFO:haproxy:Service links: CURSUS(7a9bd28d-4be0-4a79-9052-024427b6585b), MAILCATCHER(3c919184-107f-4eb5-b77a-abeb374eccf7)
2016-03-07T11:33:44.532428460Z INFO:haproxy:Container links: CURSUS_1(b6c3f198-bd1a-4b49-9f72-b8b6cf5a3d9f), CURSUS_2(d156455f-6097-4171-b1d5-8e8ef5602205), MAILCATCHER_1(cae4d370-270e-41bf-9bbd-b0f6003bccc5)
2016-03-07T11:33:44.545486191Z INFO:haproxy:HAProxy configuration remains unchanged
2016-03-07T11:33:44.545580418Z INFO:haproxy:===========END===========


(Tifayuki) #6

@yourcursus
Your VIRTUAL_HOST setting look OK, and the configuration generated by Haproxy is as expected.

I simplified your stack, replacing the underlying service with a hello-word image, and everything works as expected.

Found an article about the 502 error here(http://www.checkupdown.com/status/E502.html). Maybe you can clear the browser cache and try again.


(Yourcursus) #7

Awesome @tifayuki ! Where can I see your simplified stack file ?

I will give it a try but as I said with docker-compose alone (before the switch to Docker cloud) it was working perfectly. But anyway, who knows. I’ll tell you.


(Yourcursus) #8

So I tried to clean my browser’s cache. It doesn’t help. (Actually I had tried already with another web browser before posting here).

Then I checked my DNS zone and found that the mailcatcher.staging.yourcursus.com was a CNAME to my prod server (it made sense before as I had only one server) so first thing, I have update it so that it point to the right place.
Anywayt I still have the issue (once 502 error, then page loaded without CSS nor JS, then 502 and so on).

I have then tried to access mailcatcher bypassing the HAProxy using the endpoint tab and here it’s working like a charm ! (http://mailcatcher-1.staging.4c221980.cont.dockerapp.io:32798/)

So it’s like the HAProxy is doing something wrong with mailcatcher or vise versa.


(Tifayuki) #9

@yourcursus I tested with the following stack:
cursus: image: 'dockercloud/hello-world:latest' environment: - VIRTUAL_HOST=staging.yourcursus.com target_num_containers: 2 lb: image: 'dockercloud/haproxy:latest' environment: - 'HTTP_BASIC_AUTH=admin:admin' links: - cursus - mailcatcher ports: - '80:80' roles: - global mailcatcher: image: 'dockercloud/hello-world:latest' environment: - VIRTUAL_HOST=mailcatcher.staging.yourcursus.com

you can verify the result by:
curl <lb_service_sendpoint> -H "HOST:mailcatcher.staging.yourcursus.com" -u admin:admin
curl <lb_service_sendpoint> -H "HOST:staging.yourcursus.com" -u admin:admin

The haproxy container dispatches the requests to the correct container.


(Yourcursus) #10

Well you’re not using mailcatcher in combined with HAProxy which is I guess the important point here.

I guess the issue is about WebSocket or something like that.


(Yourcursus) #11

Any update on this issue? I’m still facing this wired behaviour.


(Yourcursus) #12

Obviously there is problem with applications using Javascript/websockets and the DockerCloud platform.

I have replaced Mailcatcher with Mailhog, and it’s working the same as mailcatcher. When downloading the javascript files, the web browser got some 502 errors.


(Yourcursus) #13

I found on the internet people stating that accessing a Nodejs app through NGiNX wasn’t working while accessing directly the app works well.

I tried the same and I have the same result. When I use the dynamic URL from DockerCloud, it’s working while using the URL through the HAProxy still brake the application.