Django GraphQL subscriptions using websockets within the dockerized API works fine locally but fails in production

Question

I have a Django GraphQL API and an Angular 12 UI that uses Apollo to interface with GraphQL.

The Django app is dockerized and uses NGINX. These are my files:-

settings.py (only relevant sections pasted below)

INSTALLED_APPS = [
    'channels',
    'corsheaders',
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'django.contrib.sites', # This is for altering the domain name in the migration
    'app',
    'graphene_django',
    'graphql_jwt.refresh_token.apps.RefreshTokenConfig',
    'graphql_auth',
    'rest_framework',
    'django_filters',
]

GRAPHENE = {
    'SCHEMA': 'project.schema.schema',
    'MIDDLEWARE': [
        'graphql_jwt.middleware.JSONWebTokenMiddleware',
    ],
    "SUBSCRIPTION_PATH": "/ws/graphql"
}

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'whitenoise.middleware.WhiteNoiseMiddleware',    
    'corsheaders.middleware.CorsMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
    'common.utils.UpdateLastActivityMiddleware'
]

AUTHENTICATION_BACKENDS = [
    'graphql_auth.backends.GraphQLAuthBackend',
    'django.contrib.auth.backends.ModelBackend',
]

GRAPHQL_AUTH = {
    "ALLOW_LOGIN_NOT_VERIFIED": False
}

GRAPHQL_JWT = {
    "JWT_ALLOW_ANY_CLASSES": [
        "graphql_auth.mutations.Register",
        "graphql_auth.mutations.VerifyAccount",
        "graphql_auth.mutations.ResendActivationEmail",
        "graphql_auth.mutations.SendPasswordResetEmail",
        "graphql_auth.mutations.PasswordReset",
        "graphql_auth.mutations.ObtainJSONWebToken",
        "graphql_auth.mutations.VerifyToken",
        "graphql_auth.mutations.RefreshToken",
        "graphql_auth.mutations.RevokeToken",
    ],
    'JWT_PAYLOAD_HANDLER': 'common.utils.jwt_payload',
    "JWT_VERIFY_EXPIRATION": True,
    "JWT_LONG_RUNNING_REFRESH_TOKEN": True,
    'JWT_REUSE_REFRESH_TOKENS': True, # Eliminates creation of new db records every time refreshtoken is requested.
    'JWT_EXPIRATION_DELTA': timedelta(minutes=60), # Expiry time of token
    'JWT_REFRESH_EXPIRATION_DELTA': timedelta(days=7), # Expiry time of refreshToken
}

ROOT_URLCONF = 'project.urls'

WSGI_APPLICATION = 'project.wsgi.application'

ASGI_APPLICATION = 'project.router.application'


REDIS_URL = env('REDIS_URL')
hosts = [REDIS_URL]

if DEBUG:
    hosts = [('redis', 6379)]

CHANNEL_LAYERS = {
    "default": {
        "BACKEND": "channels_redis.core.RedisChannelLayer",
        "CONFIG": {
            "hosts": hosts,
        },
    },
}

router.py

@database_sync_to_async
def get_user(token_key):
    try:
        decodedPayload = jwt.decode(
            token_key, key=SECRET_KEY, algorithms=['HS256'])
        user_id = decodedPayload.get('sub')
        User = get_user_model()
        user = User.objects.get(pk=user_id)
        return user
    except Exception as e:
        return AnonymousUser()

# This is to enable authentication via websockets
# Source - https://stackoverflow.com/a/65437244/7981162

class TokenAuthMiddleware(BaseMiddleware):
    # We get the auth token from the websocket call where token is passed as a URL param
    def __init__(self, inner):
        self.inner = inner

    async def __call__(self, scope, receive, send):
        query = dict((x.split("=")
                     for x in scope["query_string"].decode().split("&")))
        token_key = query.get("token")
        scope["user"] = await get_user(token_key)
        scope["session"] = scope["user"] if scope["user"] else None
        return await super().__call__(scope, receive, send)


application = ProtocolTypeRouter(
    {
        "http": get_asgi_application(),
        "websocket": AllowedHostsOriginValidator(TokenAuthMiddleware(
            URLRouter(
                [path("ws/graphql/", MyGraphqlWsConsumer.as_asgi())]
            )
        )),
    }
)

docker-compose.yml

version: "3.9"

services:
  nginx:
    build: ./nginx
    ports:
      - ${PORT}:80
    volumes:
      - static-data:/vol/static
    depends_on:
      - web
    restart: "on-failure"
  redis:
    image: redis:latest
    ports:
      - 6379:6379
    volumes:
      - ./config/redis.conf:/redis.conf
    command: ["redis-server", "/redis.conf"]
    restart: "on-failure"
  db:
    image: postgres:13
    volumes:
      - ./data/db:/var/lib/postgresql/data
    env_file:
      - database.env
    restart: always
  pg_admin:
    image: dpage/pgadmin4:latest
    container_name: app_io_pgadmin4
    ports:
      - "5000:80"
    logging:
      driver: none
    environment:
      - GUNICORN_THREADS=1
      - PGADMIN_DEFAULT_EMAIL=admin@email.com
      - PGADMIN_DEFAULT_PASSWORD=admin
    depends_on:
      - db
    restart: "on-failure"

  web:
    build: .
    command: bash -c "python manage.py makemigrations && python manage.py migrate && python manage.py runserver 0.0.0.0:8000"
    container_name: app_io_api
    volumes:
      - .:/project
    ports:
      - 8000:8000
    expose:
      - 8000
    depends_on:
      - db
      - redis
    restart: "on-failure"

volumes:
  database-data: # named volumes can be managed easier using docker-compose
  static-data:

Dockerfile

FROM python:3.8.3
LABEL maintainer="https://github.com/ryarasi"
# ENV MICRO_SERVICE=/app
# RUN addgroup -S $APP_USER && adduser -S $APP_USER -G $APP_USER
# set work directory


# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

COPY ./requirements.txt /requirements.txt
# create root directory for our project in the container
RUN mkdir /project
# COPY ./scripts /scripts
WORKDIR /project

# Copy the current directory contents into the container at /project
ADD . /project/
# Install any needed packages specified in requirements.txt

# This is to create the collectstatic folder for whitenoise
RUN pip install --upgrade pip && \
    pip install --no-cache-dir -r /requirements.txt && \
    mkdir -p /vol/web/static && \
    mkdir -p /vol/web/media

CMD python manage.py wait_for_db && python manage.py collectstatic --noinput && python manage.py migrate && gunicorn project.wsgi:application --bind 0.0.0.0:$PORT

nginx.conf (the $PORT variable here is assigned by Heroku and available as an env variable. Heroku is where I'm deploying all this.

upstream app {
    server web:$PORT;
}

server {

    listen 80;

    location /static {
        alias /vol/static;
    }
    
    location / {
        proxy_pass              http://app;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header        Host web;
        proxy_redirect          off;
        client_max_body_size    10M;
    }

    location /ws/ {
        proxy_set_header Host               $http_host;
        proxy_set_header X-Real-IP          $remote_addr;
        proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host   $server_name;
        proxy_set_header X-Forwarded-Proto  $scheme;
        proxy_set_header X-Url-Scheme       $scheme;
        proxy_redirect off;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;

        proxy_pass http://app;
    }
}

Schema.py that handles subscriptions middleware

schema = graphene.Schema(query=Query, mutation=Mutation,
                         subscription=Subscription)


def subscription_middleware(next_middleware, root, info, *args, **kwds):
    if(info.operation.name is not None and info.operation.name.value != "IntrospectionQuery"):
        print("Subscription Middleware report")
        print(" user :", info.context.user)
        print(" operation :", info.operation.operation)
        print(" resource :", info.operation.name.value)

    return next_middleware(root, info, *args, **kwds)


class MyGraphqlWsConsumer(channels_graphql_ws.GraphqlWsConsumer):
    async def on_connect(self, payload):
        self.scope["user"] = self.scope["session"]
        self.user = self.scope["user"]

    schema = schema
    middleware = [subscription_middleware]

Everything is working for me in local. In production all the regular http functions are working properly. But the websockets are failing.

One thing to note here is that in local I am able to use ws:// connections but in Heroku I can only use wss://, i.e. secure websocket connections. So I modify the request from the UI for this.

This is the error I keep seeing:-

WebSocket connection to 'wss://<link>.herokuapp.com/ws/graphql/?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1c2VybmFtZSI6InRlc3RhZG1pbiIsInN1YiI6IjIiLCJleHAiOjE2NDA1MTA0OTMsIm9yaWdJYXQiOjE2NDA1MDY4OTN9.tGrHphndoCBtE73qHwe2un49fk8qJHej-6QfpYtztrs' failed: 
SubscriptionClient.connect @ client.js:446
(anonymous) @ client.js:414
timer @ zone.js:2561
invokeTask @ zone.js:406
TaskTrackingZoneSpec.onInvokeTask @ task-tracking.js:73
invokeTask @ zone.js:405
onInvokeTask @ core.js:28645
invokeTask @ zone.js:405
runTask @ zone.js:178
invokeTask @ zone.js:487
ZoneTask.invoke @ zone.js:476
data.args.<computed> @ zone.js:2541

From the API logs, this is what I see:-

2021-12-26T09:14:30.059249+00:00 heroku[router]: at=info method=GET path="/ws/graphql/?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1c2VybmFtZSI6ImxlYXJuZXIiLCJzdWIiOiIzIiwiZXhwIjoxNjQwNTExMzUyLCJvcmlnSWF0IjoxNjQwNTA3NzUyfQ.DF16VMpJwIrlhRrnvkua6_-Mo-4CAJrgCPG7wjVSwfo" host=<link>.herokuapp.com request_id=07e05b16-f96a-4655-bf15-3e6f3a70667a fwd="157.51.140.85" dyno=web.1 connect=0ms service=2ms status=404 bytes=465 protocol=https

I did all of this following the documentation and various blogs online and I will admit that I don't know all the ins and outs of what I did. I simply followed instructions from various sources, primarily from documentation of the packages I'm using - django-channels and django-channels-graphql-ws.

I'm tired of trying to figure out where the issues exists. I've been wondering what to do about this for months now.