Automate your Flask Deployments on AWS
In an ideal world, data scientists would focus their time on analysis and modeling, but there is often seemingly unavoidable overhead in visualizing and presenting their results. Because data scientists generally don’t have an infrastructure background, they often struggle to build out servers and use tools to visualize their modeling and analysis. At Insight, Fellows have to manually set up a web server on an Amazon cloud EC2 instance to deploy a Flask app to showcase their projects, which can take 6–8 hours or more, especially for those without prior experience. But, what if they could shrink that time 100-fold?
In this post, I will share the work I did in the Insight DevOps Engineering program, where I launched an open source project called One-click to deploy web apps in minutes. Check it out on GitHub.
From Local Web Development to the Internet
Local Flask development is so frictionless it conceals the hurdles required to host your application anywhere other than localhost. When you inevitably want your app hosted on the internet, your options, for the most part, fall into two buckets:
- Managed services (e.g. Heroku, PythonAnywhere, WebFaction).
- DIY! Manually configure and maintain a web server yourself.
Anyone who has sifted through hosting options will attest to weaknesses on both sides.
Managed services are great because they hide the details you don’t care about (software updates, firewalls, and ports). Unfortunately, they also hide the details you do care about (SSH access, hardware, and complex dependencies). In contrast, the DIY option offers that missing granular control, but it’s at the expense of a lengthy, and error-prone, setup.
I was inspired to offer an alternative to both of these options at Insight, when I saw how long data science Fellows were taking to deploy their Flask apps. They wanted features that were more advanced than standard managed services, but didn’t have the technical know-how to quickly set up their cloud servers.
How can we get the best of both worlds?
One-click
The goal of my project, One-click, was to solve this deployment riddle for our Data Science Fellows. It is a command line utility that automates the DIY method (so it is as easy as deploying to a managed service, like Heroku) while still exposing access to the underlying resources that it provisions on AWS.
Here’s how you use it:
- Give Terraform your credentials to create control AWS by setting the AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY environment variables.
- Run the shell command:
one-click deploy_github https://github.com/path-to-your-app.gitTwo minutes later, the Flask app in the Github repo is hosted on a freshly configured EC2 instance. The new workflow is local development -> One-click -> hosted on the internet.
All resources are accessible through the AWS console.
One command brings everything down.
one-click destroyArchitecture
One-click is a thin Python CLI wrapper for Terraform configurations that provision an EC2 instance to build and run a custom Docker image for the target Flask application.
Terraform is a cloud-agnostic Infrastructure as Code (IaC) tool. It is a domain-specific language to provision cloud infrastructure on a variety of providers. With Terraform, you write code instead of clicking through a console or using the AWS CLI. The result is cloud resources that are generated as explicitly and automatically as the programs that will run on them.
Creating an EC2 instance with Terraform looks something like this:
resource "aws_instance" "flask_server" {ami = "ami-70e90210",instance_type = "t2.medium"}
The backbone functionality of One-click is to programmatically control the cloud. Terraform was the ideal choice for the following reasons:
- Manages state: Terraform knows which resources are actively running and how to kill them.
- Configuration management: Execute shell commands on remote servers, which is a great way to configure fresh instances.
- Flow control: If statements, for loops, simple data structures.
During an automated One-click deployment, Terraform first creates the instance that will be the web server, then configures it by remotely executing shell commands.
These server-side steps are:
- Clone the Flask app repo from GitHub onto the server
- Install Docker
- Build a custom Docker image with uWSGI (application server), Nginx (web server), and all the Flask app’s custom Python requirements
- Run a container from the freshly built image
I chose to containerize the custom Flask application server-side as a convenient form of configuration management. Properly hosting a web application requires a web server and an application server. Both need to be installed and configured.
One option was to set up these components with remote shell commands via Terraform (or another CM tool like Ansible), but that involves too many moving parts.
Instead, I opted to use a pre-built image that already includes most of what I needed installed. Why reinvent the wheel? The custom Docker image, built on the instance, inherits from this well-maintained image, then installs the Python dependencies specific to the Flask app (these must be specified in a requirements.txt file in the root of the project).
Finally, Terraform remotely spawns a container from the custom image to complete the deployment.
Testing Infrastructure as Code (IaC)
I wouldn’t be worth my salt as a DevOps engineer if I wasn’t concerned with how to write automated tests for my code base.
The Python component of One-click was trivial to automatically test with Pytest. The hefty Terraform component was more challenging.
Testing Terraform code demands more thought because IaC can’t deploy simulated cloud resources locally for tests. The solution is to automatically deploy and validate real infrastructure during testing. I used Terratest, a Go library packed with utilities for automated infrastructure testing.
The One-click IaC tests deploy a sample Flask application, check if it’s responding to requests with 200, then spins it down.
I wired these Python and IaC tests into CircleCI to ensure that tests pass before any pull request gets merged into master. Continuous integration is a safety net so that I, or another contributor, won’t inadvertently break the project.
Safe development is a dependency for fast development.
Conclusions
One-click was my first effort to propagate DevOps within an organization. It shrunk deployment times for Fellows who used it on a pilot basis, and Fellows in later sessions can continue to contribute to my project so that it will evolve along with the needs of future Fellows.
Although I designed One-click for Insight Fellows, its use case stretches beyond our organization to anyone who needs to quickly share lightweight Flask applications. I would have leaned heavily on a utility like One-click in my past life as a data science consultant. Hastily hosting a front-end on AWS was the best way to share results or prototypes with clients. Automation would have saved me time better spent on data science.
A popular conception positions the data scientist as a hacker type, someone who is just as comfortable tackling complex SQL queries as they are with front-end development, Bayesian inference, AWS, or standing up a Spark cluster. That’s not how it should be. While I acknowledge the merit of Swiss Army knife-type workers in an emerging team, I don’t think such a person should be considered the paragon at most organizations. That’s a skills bottleneck. Data science is a mature enough field to become a specialty.
It’s up to engineers to design user-centric tooling to handle the rest.
Interested in transitioning to a career in DevOps engineering? Learn more about the Insight Fellows program and start your application today.
