Hi,
I develop django app and I only know the basics of docker and docker-compose.
I have a standard django app structure with Postgresql database that works fine and know I try to implement asynchronous tasks using celery and celery-beat for automated backup for examples.
stack:
- windows 10
- django (2.2.5)/postgresql (10)
- redis/ celery
- celery-beat
- docker desktop (3.0.0)
- docker-compose (1.27.4)
So I have 4 containers running :
- redis
- django web app
- celery
- celery-beat
But there is something wrong with my settings or somethings do not really understand.
When I build and run my containers, I have errors with celery and celery-beat when migrate django database Looks like migrations are applyed in each containers and make unicity errors…
How should I resolve this issue?
I first tought using depends_on parameters but do not resolve my issue.
I have read about healthycheck but do not really understand how to implement …
I would appreciate some help
D:\Users\jl3\DOCKER\Apps\cafe_tropical>docker-compose -f docker-compose.dev.yml up
Creating network "cafe_tropical_default" with the default driver
Creating redis ... done Creating web ... done Creating celery ... done Creating celery-beat ... done Attaching to redis, celery, web, celery-beat
celery | Waiting for postgres...
celery | PostgreSQL started
celery-beat | Waiting for postgres...
celery-beat | PostgreSQL started
redis | 1:C 30 Dec 2020 15:21:18.345 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis | 1:C 30 Dec 2020 15:21:18.345 # Redis version=6.0.9, bits=64, commit=00000000, modified=0, pid=1, just started
redis | 1:C 30 Dec 2020 15:21:18.345 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis | 1:M 30 Dec 2020 15:21:18.346 * Running mode=standalone, port=6379.
redis | 1:M 30 Dec 2020 15:21:18.346 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis | 1:M 30 Dec 2020 15:21:18.346 # Server initialized
redis | 1:M 30 Dec 2020 15:21:18.346 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis | 1:M 30 Dec 2020 15:21:18.346 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled (set to 'madvise' or 'never').
redis | 1:M 30 Dec 2020 15:21:18.348 * Ready to accept connections
web | Waiting for postgres...
web | PostgreSQL started
celery | No changes detected
celery-beat | No changes detected
web | No changes detected
celery | Operations to perform:
celery | Apply all migrations: admin, api, auth, authtoken, cafe, contenttypes, django_celery_beat, parameters, sessions, sites
celery | Running migrations:
celery | Applying contenttypes.0001_initial... OK
web | Operations to perform:
web | Apply all migrations: admin, api, auth, authtoken, cafe, contenttypes, django_celery_beat, parameters, sessions, sites
web | Running migrations:
celery-beat | Operations to perform:
celery-beat | Apply all migrations: admin, api, auth, authtoken, cafe, contenttypes, django_celery_beat, parameters, sessions, sites
celery-beat | Running migrations:
celery | Applying auth.0001_initial... OK
celery-beat | Applying auth.0001_initial...Traceback (most recent call last):
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 82, in _execute
celery-beat | return self.cursor.execute(sql)
celery-beat | psycopg2.errors.UniqueViolation: ERREUR: la valeur d'une clé dupliquée rompt la contrainte unique « pg_type_typname_nsp_index »
celery-beat | DETAIL: La clé « (typname, typnamespace)=(auth_permission_id_seq, 2200) » existe déjà.
celery-beat |
celery-beat |
celery-beat | The above exception was the direct cause of the following exception:
celery-beat |
celery-beat | Traceback (most recent call last):
celery-beat | File "manage.py", line 22, in <module>
web | Applying auth.0001_initial...Traceback (most recent call last):
web | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 82, in _execute
web | return self.cursor.execute(sql)
web | psycopg2.errors.UniqueViolation: ERREUR: la valeur d'une clé dupliquée rompt la contrainte unique « pg_type_typname_nsp_index »
web | DETAIL: La clé « (typname, typnamespace)=(auth_permission_id_seq, 2200) » existe déjà.
web |
web |
web | The above exception was the direct cause of the following exception:
web |
web | Traceback (most recent call last):
web | File "manage.py", line 22, in <module>
web | main()
web | File "manage.py", line 18, in main
celery-beat | main()
celery-beat | File "manage.py", line 18, in main
web | execute_from_command_line(sys.argv)
web | File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
web | utility.execute()
web | File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 375, in execute
web | self.fetch_command(subcommand).run_from_argv(self.argv)
web | File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 323, in run_from_argv
web | self.execute(*args, **cmd_options)
web | File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 364, in execute
web | output = self.handle(*args, **options)
web | File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 83, in wrapped
web | res = handle_func(*args, **kwargs)
web | File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 232, in handle
web | post_migrate_state = executor.migrate(
web | File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
web | state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
web | File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
web | state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
web | File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 245, in apply_migration
web | state = migration.apply(state, schema_editor)
web | File "/usr/local/lib/python3.8/site-packages/django/db/migrations/migration.py", line 124, in apply
web | operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
web | File "/usr/local/lib/python3.8/site-packages/django/db/migrations/operations/models.py", line 92, in database_forwards
web | schema_editor.create_model(model)
web | File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 307, in create_model
web | self.execute(sql, params or None)
web | File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 137, in execute
web | cursor.execute(sql, params)
web | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 99, in execute
web | return super().execute(sql, params)
web | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
web | return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
web | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
web | return executor(sql, params, many, context)
web | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
web | return self.cursor.execute(sql, params)
web | File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
web | raise dj_exc_value.with_traceback(traceback) from exc_value
web | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 82, in _execute
web | return self.cursor.execute(sql)
web | django.db.utils.IntegrityError: ERREUR: la valeur d'une clé dupliquée rompt la contrainte unique « pg_type_typname_nsp_index »
web | DETAIL: La clé « (typname, typnamespace)=(auth_permission_id_seq, 2200) » existe déjà.
web |
celery-beat | execute_from_command_line(sys.argv)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
celery-beat | utility.execute()
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 375, in execute
celery-beat | self.fetch_command(subcommand).run_from_argv(self.argv)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 323, in run_from_argv
celery-beat | self.execute(*args, **cmd_options)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 364, in execute
celery-beat | output = self.handle(*args, **options)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 83, in wrapped
celery-beat | res = handle_func(*args, **kwargs)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 232, in handle
celery-beat | post_migrate_state = executor.migrate(
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
celery-beat | state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
celery-beat | state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/db/migrations/executor.py", line 245, in apply_migration
celery-beat | state = migration.apply(state, schema_editor)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/db/migrations/migration.py", line 124, in apply
celery-beat | operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/db/migrations/operations/models.py", line 92, in database_forwards
celery-beat | schema_editor.create_model(model)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 307, in create_model
celery-beat | self.execute(sql, params or None)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 137, in execute
celery-beat | cursor.execute(sql, params)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 99, in execute
celery-beat | return super().execute(sql, params)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
celery-beat | return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
celery-beat | return executor(sql, params, many, context)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
celery-beat | return self.cursor.execute(sql, params)
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
celery-beat | raise dj_exc_value.with_traceback(traceback) from exc_value
celery-beat | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 82, in _execute
celery-beat | return self.cursor.execute(sql)
celery-beat | django.db.utils.IntegrityError: ERREUR: la valeur d'une clé dupliquée rompt la contrainte unique « pg_type_typname_nsp_index »
celery-beat | DETAIL: La clé « (typname, typnamespace)=(auth_permission_id_seq, 2200) » existe déjà.
celery-beat |
celery | Applying admin.0001_initial... OK
celery | Applying admin.0002_logentry_remove_auto_add... OK
celery | Applying admin.0003_logentry_add_action_flag_choices... OK
celery | Applying api.0001_initial... OK
celery | Applying contenttypes.0002_remove_content_type_name... OK
celery | Applying auth.0002_alter_permission_name_max_length... OK
celery | Applying auth.0003_alter_user_email_max_length... OK
celery | Applying auth.0004_alter_user_username_opts... OK
celery | Applying auth.0005_alter_user_last_login_null... OK
celery | Applying auth.0006_require_contenttypes_0002... OK
celery | Applying auth.0007_alter_validators_add_error_messages... OK
celery | Applying auth.0008_alter_user_username_max_length... OK
celery | Applying auth.0009_alter_user_last_name_max_length... OK
celery | Applying auth.0010_alter_group_name_max_length... OK
celery | Applying auth.0011_update_proxy_permissions... OK
celery | Applying authtoken.0001_initial... OK
celery | Applying authtoken.0002_auto_20160226_1747... OK
celery | Applying cafe.0001_initial... OK
celery | Applying django_celery_beat.0001_initial... OK
celery | Applying django_celery_beat.0002_auto_20161118_0346... OK
celery | Applying django_celery_beat.0003_auto_20161209_0049... OK
celery | Applying django_celery_beat.0004_auto_20170221_0000... OK
celery | Applying django_celery_beat.0005_add_solarschedule_events_choices... OK
celery | Applying django_celery_beat.0006_auto_20180322_0932... OK
celery | Applying django_celery_beat.0007_auto_20180521_0826... OK
celery | Applying django_celery_beat.0008_auto_20180914_1922... OK
celery | Applying django_celery_beat.0006_auto_20180210_1226... OK
celery | Applying django_celery_beat.0006_periodictask_priority... OK
celery | Applying django_celery_beat.0009_periodictask_headers... OK
celery | Applying django_celery_beat.0010_auto_20190429_0326... OK
celery | Applying django_celery_beat.0011_auto_20190508_0153... OK
celery | Applying django_celery_beat.0012_periodictask_expire_seconds... OK
celery | Applying django_celery_beat.0013_auto_20200609_0727... OK
celery | Applying django_celery_beat.0014_remove_clockedschedule_enabled... OK
celery | Applying parameters.0001_initial... OK
celery | Applying sessions.0001_initial... OK
celery | Applying sites.0001_initial... OK
celery | Applying sites.0002_alter_domain_unique... OK
celery-beat | celery beat v5.0.5 (singularity) is starting.
celery-beat | __ - ... __ - _
celery-beat | LocalTime -> 2020-12-30 15:21:35
celery-beat | Configuration ->
celery-beat | . broker -> redis://redis:6379//
celery-beat | . loader -> celery.loaders.app.AppLoader
celery-beat | . scheduler -> celery.beat.PersistentScheduler
celery-beat | . db -> celerybeat-schedule
celery-beat | . logfile -> [stderr]@%INFO
celery-beat | . maxinterval -> 5.00 minutes (300s)
celery-beat | [2020-12-30 15:21:35,216: INFO/MainProcess] beat: Starting...
celery-beat | [2020-12-30 15:21:35,678: INFO/MainProcess] Scheduler: Sending due task hello (cafe.tasks.hello)
celery-beat | [2020-12-30 15:21:35,724: INFO/MainProcess] Scheduler: Sending due task backup (cafe.tasks.backup)
web | Watching for file changes with StatReloader
web | Performing system checks...
web |
web | System check identified no issues (0 silenced).
web | December 30, 2020 - 15:21:37
web | Django version 2.2.5, using settings 'core.settings.dev'
web | Starting development server at http://0.0.0.0:8000/
web | Quit the server with CONTROL-C.
celery | /usr/local/lib/python3.8/site-packages/celery/platforms.py:796: RuntimeWarning: You're running the worker with superuser privileges: this is
celery | absolutely not recommended!
celery |
celery | Please specify a different user using the --uid option.
celery |
celery | User information: uid=0 euid=0 gid=0 egid=0
celery |
celery | warnings.warn(RuntimeWarning(ROOT_DISCOURAGED.format(
celery |
celery | -------------- celery@e53033835ccb v5.0.5 (singularity)
celery | --- ***** -----
celery | -- ******* ---- Linux-4.19.128-microsoft-standard-x86_64-with 2020-12-30 15:21:37
celery | - *** --- * ---
celery | - ** ---------- [config]
celery | - ** ---------- .> app: core:0x7fe23b6ba220
celery | - ** ---------- .> transport: redis://redis:6379//
celery | - ** ---------- .> results: redis://redis:6379/
celery | - *** --- * --- .> concurrency: 4 (prefork)
celery | -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
celery | --- ***** -----
celery | -------------- [queues]
celery | .> celery exchange=celery(direct) key=celery
celery |
celery |
celery | [tasks]
celery | . cafe.tasks.backup
celery | . cafe.tasks.hello
celery | . core.celery.debug_task
celery |
celery | [2020-12-30 15:21:37,940: INFO/MainProcess] Connected to redis://redis:6379//
celery | [2020-12-30 15:21:37,951: INFO/MainProcess] mingle: searching for neighbors
celery | [2020-12-30 15:21:38,979: INFO/MainProcess] mingle: all alone
celery | [2020-12-30 15:21:39,009: WARNING/MainProcess] /usr/local/lib/python3.8/site-packages/celery/fixups/django.py:203: UserWarning: Using settings.DEBUG leads to a memory
version: '3.7'
services:
web:
restart: always
container_name: web
build:
context: ./app
dockerfile: Dockerfile.dev
restart: always
command: python manage.py runserver 0.0.0.0:8000
volumes:
- ./app:/usr/src/app
ports:
- 8000:8000
env_file:
- ./.env.dev
depends_on:
- redis
redis:
container_name: redis
image: "redis:alpine"
celery:
container_name: celery
build:
context: ./app
dockerfile: Dockerfile.dev
command: celery -A core worker -l info
volumes:
- ./app:/usr/src/app
env_file:
- ./.env.dev
depends_on:
- web
- redis
celery-beat:
container_name: celery-beat
build:
context: ./app
dockerfile: Dockerfile.dev
command: celery -A core beat -l info
volumes:
- ./app:/usr/src/app
env_file:
- ./.env.dev
depends_on:
- celery
- redis