Deploy serverless, event-driven Python applications using Zappa

INTRODUCTION

Zappa is a  very powerful open source python project which lets you build, deploy and update your WSGI app hosted on AWS Lambda + API Gateway easily.This blog is a detailed step-by-step focusing on challenges faced while deploying Django application on AWS Lambda using Zappa as a deployment tool.

BUILDING YOUR APPLICATION

If you do not have a Django application already you can build one by cloning this GitHub repository.

$ git clone https://github.com/rishabh-velotio/django-zappa-sample.git

Once you have cloned the repository you will need a virtual environment which provides an isolated Python environment for your application. I prefer virtualenvwrapper to create one.

Command :

$ mkvirtualenv django_zappa_sample

Install dependencies from requirements.txt.

$ pip install -r requirements.txt

Now if you run the server directly it will log a warning as the database is not set up yet.

$ python manage.py runserver

Also trying to access admin page (http://localhost:8000/admin/) will throw an “OperationalError” exception with below log at server end.

In order to fix this you need to run the migration into your database so that essential tables like auth_user, sessions, etc are created before any request is made to the server.

$ python manage.py migrate

NOTE: Use DATABASES from project settings file to configure your database that you would want your Django application to use once hosted on AWS Lambda. By default, its configured to create a local SQLite database file as backend.

You can run the server again and it should now load the admin panel of your website.

Do verify if you have the zappa python package into your virtual environment before moving forward.

CONFIGURING ZAPPA SETTINGS

Deploying with Zappa is simple as it only needs a configuration file to run and rest will be managed by Zappa. To create this configuration file run from your project root directory -

$ zappa init

You can verify zappa_settings.json generated at your project root directory.

TIP: The virtual environment name should not be the same as the Zappa project name, as this may cause errors.

Additionally, you could specify other settings in  zappa_settings.json file as per requirement using Advanced Settings.

Now, you're ready to deploy!

IAM PERMISSIONS:

In order to deploy the Django Application to Lambda/Gateway, setup an IAM role (eg. ZappaLambdaExecutionRole) with the following permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:AttachRolePolicy",
                "iam:CreateRole",
                "iam:GetRole",
                "iam:PutRolePolicy"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "iam:PassRole"
            ],
            "Resource": [
                "arn:aws:iam::<account_id>:role/*-ZappaLambdaExecutionRole"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "apigateway:DELETE",
                "apigateway:GET",
                "apigateway:PATCH",
                "apigateway:POST",
                "apigateway:PUT",
                "events:DeleteRule",
                "events:DescribeRule",
                "events:ListRules",
                "events:ListTargetsByRule",
                "events:ListRuleNamesByTarget",
                "events:PutRule",
                "events:PutTargets",
                "events:RemoveTargets",
                "lambda:AddPermission",
                "lambda:CreateFunction",
                "lambda:DeleteFunction",
                "lambda:GetFunction",
                "lambda:GetPolicy",
                "lambda:ListVersionsByFunction",
                "lambda:RemovePermission",
                "lambda:UpdateFunctionCode",
                "lambda:UpdateFunctionConfiguration",
                "cloudformation:CreateStack",
                "cloudformation:DeleteStack",
                "cloudformation:DescribeStackResource",
                "cloudformation:DescribeStacks",
                "cloudformation:ListStackResources",
                "cloudformation:UpdateStack",
                "logs:DescribeLogStreams",
                "logs:FilterLogEvents",
                "route53:ListHostedZones",
                "route53:ChangeResourceRecordSets",
                "route53:GetHostedZone",
                "s3:CreateBucket",
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket name from zappa_settings.json>"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:PutObject",
                "s3:CreateMultipartUpload",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts",
                "s3:ListBucketMultipartUploads"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket name from zappa_settings.json>/*"
            ]
        }
    ]
}

DEPLOYING DJANGO APPLICATION

Before deploying the application, ensure that the IAM role is set in the config JSON as follows:

{
    "dev": {
        ...
        "manage_roles": false, // Disable Zappa client managing roles.
        "role_name": "MyLambdaRole", // Name of your Zappa execution role. Optional, default: <project_name>-<env>-ZappaExecutionRole.
        "role_arn": "arn:aws:iam::12345:role/app-ZappaLambdaExecutionRole", // ARN of your Zappa execution role. Optional.
        ...
    },
    ...
}

Once your settings are configured, you can package and deploy your application to a stage called "dev" with a single command:

$ zappa deploy dev

You should see that your Zappa deployment completed successfully with URL to API gateway created for your application.

 

TROUBLESHOOTING:

1. If you are seeing the following error while deployment, it’s probably because you do not have sufficient privileges to run deployment on AWS Lambda. Ensure your IAM role has all the permissions as described above or set “manage_roles” to true so that Zappa can create and manage the IAM role for you.

2. The below error will be caused as you have not listed “events.amazonaws.com” as Trusted Entity for your IAM Role. You can add the same or set “keep_warm” parameter to false in your Zappa settings file. Your Zappa deployment was partially deployed as it got terminated abnormally.

3. Adding the parameter and running zappa update will cause above error. As you can see it says “Stack django-zappa-sa-dev does not exists” as the previous deployment was unsuccessful. To fix this, delete the Lambda function from console and rerun the deployment.

4.  If you run into any distribution error, please try down-grading your pip version to 9.0.1.

$ pip install pip==9.0.1

or,

If you run into NotFoundException(Invalid REST API Identifier issue) please try undeploying the Zappa stage and retry again.

TIP: To understand how your application works on serverless environment please visit this link

POST DEPLOYMENT SETUP

Migrate database

At this point, you should have an empty database for your Django application to fill up with a schema.

$ zappa manage.py migrate dev

Once you run above command the database migrations will be applied on the database as specified in your Django settings.

Creating Superuser of Django application

You also might need to create a new superuser on the database. You could use the following command on your project directory.

$ zappa invoke --raw dev "from django.contrib.auth.models import User; User.objects.create_superuser('username', 'username@yourdomain.com', 'password')"

Alternatively,

$ python manage createsuperuser

Note that your application must be connected to the same database as this is run as standard Django administration command (not a Zappa command).

Managing static files

Your Django application will be having a dependency on static files, Django admin panel uses a combination of JS, CSS and image files.

NOTE: Zappa is for running your application code, not for serving static web assets. If you plan on serving custom static assets in your web application (CSS/JavaScript/images/etc.), you'll likely want to use a combination of AWS S3 and AWS CloudFront.

You will need to add following packages to your virtual environment required for management of files to and from S3 django-storages and boto.

$ pip install django-storages boto

Add Django-Storage to your INSTALLED_APPS in settings.py

INSTALLED_APPS = (

         ...,

         'storages',

    )

Configure Django-storage in settings.py as

AWS_STORAGE_BUCKET_NAME = 'django-zappa-sample-bucket'

AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME

STATIC_URL = "https://%s/" % AWS_S3_CUSTOM_DOMAIN

STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'

Once you have setup the Django application to serve your static files from AWS S3, run following command to upload the static file from your project to S3.

$ python manage.py collectstatic --noinput

or

$ zappa update dev

$ zappa manage dev "collectstatic --noinput"

Check that at least 61 static files are moved to S3 bucket. Admin panel is built over  61 static files.

NOTE: STATICFILES_DIR must be configured properly to collect your files from the appropriate location.

Tip: You need to render static files in your templates by loading static path and using the same.  Example, {% static %}

SETTING UP API GATEWAY

To connect to your Django application you also need to ensure you have API gateway setup for your AWS Lambda Function.  You need to have GET methods set up for all the URL resources used in your Django application. Alternatively, you can setup a proxy method to allow all subresources to be processed through one API method.

Go to AWS Lambda function console and add API Gateway from 'Add triggers'.

1. Configure API, Deployment Stage, and Security for API Gateway. Click Save once it is done.

2. Go to API Gateway console and,

a. Recreate ANY method for / resource.

i. Check `Use Lambda Proxy integration`

ii. Set `Lambda Region` and `Lambda Function` and `Save` it.

a. Recreate ANY method for /{proxy+} resource.

i. Select `Lambda Function Proxy`

ii. Set `Lambda Region` and `Lambda Function` and `Save` it.

3. Click on Action and select Deploy API. Set Deployment Stage and click Deploy

4. Ensure that GET and POST method for / and Proxy are set as Override for this method

SETTING UP CUSTOM SSL ENDPOINT

Optionally, you could also set up your own custom defined SSL endpoint with Zappa and install your certificate with your domain by running certify with Zappa. 

$ zappa certify dev

...

    "certificate_arn": "arn:aws:acm:us-east-1:xxxxxxxxxxxx:certificate/xxxxxxxxxxxx-xxxxxx-xxxx-xxxx-xxxxxxxxxxxxxx",

    "domain": "django-zappa-sample.com"

Now you are ready to launch your Django Application hosted on AWS Lambda.

Additional Notes:

  •  Once deployed, you must run “zappa update <STAGE-NAME>” for updating your already hosted AWS Lambda function.

  • You can check server logs for investigation by running “zappa tail” command.

  • To un-deploy your application, simply run: `zappa undeploy <STAGE-NAME>`

You’ve seen how to deploy Django application on AWS Lambda using Zappa. If you are creating your Django application for first time you might also want to read Edgar Roman’s Django Zappa Guide.

Start building your Django application and let us know in the comments if you need any help during your application deployment over AWS Lambda.

About the Author

IMG_4212.JPG

Rishabh is a Python specialist at Velotio Technologies. He has a strong understanding of Python web frameworks and Amazon Web Services. Lately, he has been working extensively in the world of web scraping and crawling.  Rishabh is numismatic and loves solving Sudoku.