Deployment to Google Cloud

This document describes the deployment process to a new Google Cloud environment. If you just want to learn how to get started or what tools you need to install, read the contribution guidelines.

Create Google Cloud project

Log in to Google Cloud console and create a new project. We’ll refer to the project ID you chose as $PROJECT.

You may have to activate billing and upgrade your account to access all features required for this project.

Run gcloud init to initialise your project in your terminal, then authenticate via gcloud auth login. In case you’re having trouble with Python 2 vs 3, try:

CLOUDSDK_PYTHON=python3 gcloud auth login

Create App Engine app

Open the App Engine dashboard and create a new app in the region of your choice with the flexible environment.

Create Storage buckets

Open the Storage dashboard and create the buckets $PROJECT-data, $PROJECT-logs and $PROJECT-responses in the same region as the App Engine app above. Leave the default options otherwise.

Create PubSub topic and subscription

Open the PubSub dashboard and create the topic users, then two subscriptions attached to that topic:

• crawl with “Pull” delivery type, “Never expire”, 600 seconds acknowledgement deadline, and 1 day retention duration,
• logs with “Pull” delivery type, “Never expire”, 600 seconds acknowledgement deadline, and 7 day retention duration.

Also create another topic responses and one subscription attached to that topic:

• response_logs with “Pull” delivery type, “Never expire”, 600 seconds acknowledgement deadline, and 7 day retention duration.

Then make sure to update the PubSub project, topic, and subscription:

• crawl in the scraper,
• logs and response_logs in .env and docker-compose.yaml.

Enable Google Container Registry API

Go to the APIs & Services dashboard, find the Google Container Registry API, and enable it.

Create credentials for default service account

Go to the IAM & admin dashboard, section Service accounts, and find the App Engine default service account. Select “Create key” from the actions, and download the key in JSON format. Move that file to the root of this project as gs.json. This is a private key, do not check it into version control!

Make sure gcloud uses these credentials by editing the following lines in your local ~/.boto:

# Google OAuth2 service account credentials (for "gs://" URIs):
gs_service_key_file = /path/to/gs.json

Now you should be able to log in to Container Registry:

cat gs.json | docker login -u _json_key --password-stdin https://gcr.io

Read more about using JSON credentials to access GCR.

Update settings

Edit your .env file to use the correct project:

GC_PROJECT=$PROJECT

Similarly, edit app.yaml to use the correct environment variables:

env_variables:
    GC_PROJECT: $PROJECT
    GC_DATA_BUCKET: $PROJECT-data
    PUBSUB_QUEUE_PROJECT: $PROJECT
    PUBSUB_QUEUE_TOPIC_USERS: users
    PUBSUB_QUEUE_TOPIC_RESPONSES: responses

The App Engine domain should be automatically added to ALLOWED_HOSTS in settings.py if $GC_PROJECT is configured correctly. Should you experience problems with your domain not being whitelisted, check there first.

Deploy

You should now be able to deploy the service. For a full release, simply run

./release.sh

If you don’t need to build a new recommender and datebase version, it should suffice to run

pipenv run pynt syncdata releaseserver

Either way, after successful deployment the service should be available at https://$PROJECT.appspot.com/.

Configure domains

If you’re using custom domains, navigate to the corresponding settings and follow the instructions. If the domain was previously used in a different project, you will need to unassign it first.

Enjoy!

Everything should be done! Sit back and relax…