Serverless Web application with Datastore

Last Updated: 2020-09-23

Overview

Google Cloud Platform (GCP) is a suite of cloud computing services that runs on the same infrastructure that Google uses internally for its end-user products, such as Google Search, Gmail, Drive and YouTube. Alongside a set of management tools, it provides a series of modular cloud services including computing, data storage, data analytics and machine learning.

Cloud Functions is a service on Google Cloud Platform that supports serverless deployments of functions. Much like serverless platforms with App Engine and serverless containers with Cloud Run, the infrastructure is hidden and is autoscaled to meet demand. Unlike App Engine and Cloud Run which typically run full applications, Cloud Functions are typically used to perform a single task. Cloud Functions are implemented using a set of container environments provided by Google. As a result, much like App Engine, only a restricted set of environments is supported.

What you'll build

In this codelab, you will create a basic serverless web application:

Front end will be served from a Storage Bucket.
Back end will be implemented as a set of Cloud Functions.
Data is going to be stored in Cloud Datastore serverless database.

What you'll learn

How to create Cloud Functions
How to use Cloud Datastore database
How to trigger Cloud Functions using HTTP requests
How to configure Cloud Storage buckets using command line

What you'll need

A recent version of your favourite Web Browser
Basics of BASH
GCP Trial account

Create an Account

In this step, you register for the Google Cloud Platform free trial and create a project. The free trial provides you:

$300 Credit for Free
Access to Google Cloud Platform Products
You Won't be Billed (though you need to enter your credit card)
Build with the Power, Speed, Security, Reliability, and Scalability of Google

To register for the free trial open the free trial Registration page.

If you do not have a Gmail account, follow the steps to create one. Otherwise, login and complete the registration form.

Read and agree to the terms of service. Click Accept and start a free trial.

Create a Project

Next, create your first project using the Google Cloud Platform Console. The project is used to complete the rest of the lab.

To create a project in the Google Cloud Platform Console, click Select a project > Create a project.

In the New Project dialog: for Project name, type whatever you like. Make a note of the Project ID in the text below the project name box; you need it later. Then click Create.

Upgrade Account (Optional)

In the upper-right corner of the console, a button will appear asking you to upgrade your account. Click Create a Project when you see it. If the Upgrade button does not appear, you may skip this step. If the button appears later, click it when it does.

When you upgrade your account, you immediately have access to standard service quotas, which are higher than those available on the free trial.

Finalize

On the GCP Console, use the left-hand side menu to navigate to Compute Engine and ensure that there are no errors.

At the end of this lab, you may delete this project and close your billing account if desired.

Setup Dev Environment

Enable APIs for Cloud Storage and Cloud Functions: click the Link.

Check if your cloud shell has Gradle build tool installed:

$ gradle --version

The output of the command should be similar to this:

------------------------------------------------------------
Gradle 6.0
------------------------------------------------------------

Build time:   2019-11-08 18:12:12 UTC
Revision:     0a5b531749138f2f983f7c888fa7790bfc52d88a

Kotlin:       1.3.50
Groovy:       2.5.8
Ant:          Apache Ant(TM) version 1.10.7 compiled on September 1 2019
JVM:          11.0.9 (Debian 11.0.9+11-post-Debian-1deb10u1)
OS:           Linux 5.4.49+ amd64

If it is not, check the documentation to install Gradle.

Create the working directory for the application and change to it:

$ mkdir -p serverless/api
$ cd serverless/api

Then, initialize a Gradle project by running:

$ gradle init \
  --type kotlin-application \
  --dsl kotlin \
  --package com.example.api \
  --project-name coolapi

Next, remove tests :) and upgrade the Gradle Wrapper to a more recent version :

$ rm -rf src/test/
$ ./gradlew wrapper --gradle-version 6.6.1

Create the Cloud ignore file so that unnecessary resources will not be uploaded to Google Cloud Platform when you use the gcloud command:

$ cat <<EOF >>.gcloudignore
.git
target
build
.idea
.gradle
EOF

Create build configuration

Edit the build.gradle.kts:

$ cloudshell edit build.gradle.kts

And replace the contents of the file with the following:

import org.jetbrains.kotlin.gradle.tasks.KotlinCompile

val invoker by configurations.creating

plugins {
    kotlin("jvm") version "1.4.20"
    kotlin("plugin.serialization") version "1.4.20"
    id("com.github.johnrengelman.shadow") version "6.1.0"
    application
}

repositories {
    jcenter()
    mavenCentral()
}

dependencies {
    // Align versions of all Kotlin components
    implementation(platform("org.jetbrains.kotlin:kotlin-bom"))

    // Use the Kotlin JDK 8 standard and serialization.
    implementation("org.jetbrains.kotlin:kotlin-stdlib-jdk8")
    implementation("org.jetbrains.kotlinx:kotlinx-serialization-json:1.0.1")

    // Use the Functions API library
    compileOnly("com.google.cloud.functions:functions-framework-api:1.0.1")

    // Use the Datastore API
    implementation("com.google.cloud:google-cloud-datastore:1.105.2")

    // Use minimal logging
    implementation("io.github.microutils:kotlin-logging:2.0.3")

    // We won't be running any tests (don't do this in prod :))
    // But we will use a local invoker to test the functions
    invoker("com.google.cloud.functions.invoker:java-function-invoker:1.0.0")
}

tasks.withType<KotlinCompile> {
    kotlinOptions {
        freeCompilerArgs = listOf("-Xjsr305=strict")
        jvmTarget = "11"
    }
}
task<JavaExec>("runFunction") {
    main = "com.google.cloud.functions.invoker.runner.Invoker"
    classpath(invoker)
    inputs.files(configurations.runtimeClasspath, sourceSets["main"].output)
    args(
        "--target", project.findProperty("runFunction.target") ?: "com.example.api.GetKt",
        "--port", project.findProperty("runFunction.port") ?: 8080
    )
    doFirst {
        args("--classpath", files(configurations.runtimeClasspath, sourceSets["main"].output).asPath)
    }
}

application {
    mainClassName = "com.example.api.GetKt"
}

tasks.named("build") {
    dependsOn(":shadowJar")
}

task("buildFunction") {
    dependsOn("build")
    copy {
        from("build/libs/" + rootProject.name + "-all.jar")
        into("build/deploy")
    }
}

To read more about this configuration you can read the documentation.

Create Datastore helper functions

Now, create the DataStoreConverter.kt file that will have our helper functions to save and load data from Datastore:

$ touch ${HOME}/serverless/api/src/main/kotlin/com/example/api/DataStoreConverter.kt
$ cloudshell edit ${HOME}/serverless/api/src/main/kotlin/com/example/api/DataStoreConverter.kt

And paste the following:

package com.example.api

import com.google.cloud.Timestamp
import com.google.cloud.datastore.*

private fun convertToEntity(
        message: Message,
        keyFactory: (String) -> KeyFactory
): FullEntity<IncompleteKey> {

    val rowKeyFactory: KeyFactory = keyFactory("Messages")

    return FullEntity.newBuilder(rowKeyFactory.newKey())
            .set("datetime", Timestamp.now())
            .set("message", message.text)
            .build()
}

fun saveToDataStore(
        msg: Message
) {
    val datastore: Datastore = DatastoreOptions.getDefaultInstance().service

    val keyFactoryBuilder = { s: String ->
        datastore.newKeyFactory().setKind(s)
    }

    val entity = convertToEntity(msg, keyFactoryBuilder)

    datastore.add(entity)
}

fun getFromDataStore(): List<Message> {
    val datastore: Datastore = DatastoreOptions.getDefaultInstance().service
    val query: Query<Entity> = Query.newEntityQueryBuilder()
            .setKind("Messages")
            .setOrderBy(StructuredQuery.OrderBy.desc("datetime"))
            .build()

    val messages: QueryResults<Entity> = datastore.run(query)

    val mutableList: MutableList<Message> = mutableListOf()

    messages.forEach { entity ->
        mutableList.add(convertToMessage(entity))
    }
    return mutableList
}

fun convertToMessage(
        entity: Entity
): Message {
    val text = entity.getString("message")
    return Message(text)
}

Let's break down the code.

Objects in Datastore are known as entities. Entities are grouped by "kind" and have keys for easy access. The convertToEntity() function takes our data and creates a new Datastore Entity. It uses a provided KeyFactory generator to generate a new key for the entity's Kind (Messages in our case). We use the newKey() method without arguments so the actual key is going to be auto generated. We could supply our own key as an argument if we wanted to.

private fun convertToEntity(
        message: Message,
        keyFactory: (String) -> KeyFactory
): FullEntity<IncompleteKey> {

    val rowKeyFactory: KeyFactory = keyFactory("Messages")

    return FullEntity.newBuilder(rowKeyFactory.newKey())
            .set("datetime", Timestamp.now())
            .set("message", message.text)
            .build()
}

The next function is saveToDataStore(). First, to make authenticated requests to Google Cloud Datastore, we must create a service object with credentials. You can then make API calls by calling methods on the Datastore service object. The simplest way to authenticate is to use Application Default Credentials. These credentials are automatically inferred from your environment, so you only need the following code to create your service object DatastoreOptions.getDefaultInstance().service. Then it creates a new KeyFactory anonymous function that is going to be used in convertToEntity() function. Finally, it calls the add() method to add the entity to Datastore.

fun saveToDataStore(
        msg: Message
) {
    val datastore: Datastore = DatastoreOptions.getDefaultInstance().service

    val keyFactoryBuilder = { s: String ->
        datastore.newKeyFactory().setKind(s)
    }

    val entity = convertToEntity(msg, keyFactoryBuilder)

    datastore.add(entity)
}

Finally, we have the getFromDataStore() function. In Datastore you can get entities by their keys, but also you can perform queries to retrieve entities by the values of their properties. A typical query includes an entity kind, filters to select entities with matching values, and sort orders to sequence the results. Java Datastore API supports two types of queries: StructuredQuery (that allows you to construct query elements) and GqlQuery (which operates using GQL syntax) in string format. In our case we use StructuredQuery, in which we only specify "kind" (as we want to get all messages) and set the sorting to use datetime property. Then we run() the query and transform the entity to the Message DTO to return the list to the user using a convertToMessage() helper function.

fun getFromDataStore(): List<Message> {
    val datastore: Datastore = DatastoreOptions.getDefaultInstance().service
    val query: Query<Entity> = Query.newEntityQueryBuilder()
        .setKind("Messages")
        .setOrderBy(StructuredQuery.OrderBy.desc("datetime"))
        .build()

    val messages: QueryResults<Entity> = datastore.run(query)

    val mutableList: MutableList<Message> = mutableListOf()

    messages.forEach { entity ->
        mutableList.add(convertToMessage(entity))
    }
    return mutableList
}

fun convertToMessage(
    entity: Entity
): Message {
    val text = entity.getString("message")
    return Message(text)
}

Create Cloud Functions

Now, edit the App.kt file that will host our functions:

$ cloudshell edit ${HOME}/serverless/api/src/main/kotlin/com/example/api/App.kt

Replace the content of the file with the following code containing basic imports and a DTO class:

package com.example.api

import com.google.cloud.functions.HttpFunction
import com.google.cloud.functions.HttpRequest
import com.google.cloud.functions.HttpResponse
import kotlinx.serialization.*
import kotlinx.serialization.json.*
import mu.KotlinLogging
import java.io.IOException
import java.net.HttpURLConnection

@Serializable
data class Message(var text: String = "")

The first endpoint is named Get and supports code for handling GET requests. It calls getFromDataStore() function to retrieve all of the entries in the Datastore. Then the list is converted into JSON and sent back to the client with an appropriate response type (application/json).

Add the following code in App.kt:

class Get : HttpFunction {

    private val logger = KotlinLogging.logger {}

    @Throws(IOException::class)
    override fun service(request: HttpRequest, response: HttpResponse) {
        response.appendHeader("Access-Control-Allow-Origin", "https://storage.googleapis.com")
        when(request.method) {
            "OPTIONS" -> {
                logger.info { "Handling CORS request" }
                handleCors("GET", response)
                return
            }
            "GET" -> {
                val messages = getFromDataStore()
                logger.info { "Retrieved ${messages.size} messages" }
                response.appendHeader("Content-Type", "application/json")
                val res = Json { encodeDefaults = true }.encodeToString(messages)
                response.writer.write(res)
            }
            else -> response.setStatusCode(HttpURLConnection.HTTP_BAD_REQUEST)
        }

    }
}

The other endpoint implemented within App.kt is named Post. The endpoint only supports HTTP POST requests and checks to ensure that a valid content type (application/json) is supplied. It then converts the JSON to a Message class and saves it to Datastore.

Add the following code in App.kt:

class Post : HttpFunction {
    private val logger = KotlinLogging.logger {}

    @Throws(IOException::class)
    override fun service(request: HttpRequest, response: HttpResponse) {
        response.appendHeader("Access-Control-Allow-Origin", "https://storage.googleapis.com")
        when(request.method) {
            "OPTIONS" -> {
                logger.info { "Handling CORS request" }
                handleCors("POST", response)
                return
            }
            "POST" -> {
                if (request.contentType.orElse("") == "application/json") {
                    val message = try {
                        Json { encodeDefaults = true }.decodeFromString<Message>(request.reader.readText())
                    } catch (e: Exception) {
                        null
                    }
                    logger.info { "Saving $message" }
                    message?.let {
                        saveToDataStore(it)
                        response.setStatusCode(HttpURLConnection.HTTP_OK)
                    } ?: kotlin.run {
                        response.setStatusCode(HttpURLConnection.HTTP_BAD_REQUEST)
                    }
                } else {
                    logger.error { "Wrong content type" }
                    response.setStatusCode(HttpURLConnection.HTTP_BAD_REQUEST)
                }
            }
            else -> response.setStatusCode(HttpURLConnection.HTTP_BAD_REQUEST)
        }
    }
}

The both functions also contain code to handle HTTP OPTIONS request methods.

This is to support CORS. It simply sets the HTTP response headers to allow all origins 'https://storage.googleapis.com' to perform the specified method request.

Add the following code to the end of App.kt:

private fun handleCors(method: String, response: HttpResponse) {
    response.appendHeader("Access-Control-Allow-Methods", method)
    response.appendHeader("Access-Control-Allow-Headers", "Content-Type")
    response.appendHeader("Access-Control-Max-Age", "3600")
    response.setStatusCode(HttpURLConnection.HTTP_NO_CONTENT)
}

The full file should look like this:

package com.example.api

import com.google.cloud.functions.HttpFunction
import com.google.cloud.functions.HttpRequest
import com.google.cloud.functions.HttpResponse
import kotlinx.serialization.*
import kotlinx.serialization.json.*
import mu.KotlinLogging
import java.io.IOException
import java.net.HttpURLConnection

@Serializable
data class Message(var text: String = "")

class Get : HttpFunction {

    private val logger = KotlinLogging.logger {}

    @Throws(IOException::class)
    override fun service(request: HttpRequest, response: HttpResponse) {
        response.appendHeader("Access-Control-Allow-Origin", "https://storage.googleapis.com")
        when(request.method) {
            "OPTIONS" -> {
                logger.info { "Handling CORS request" }
                handleCors("GET", response)
                return
            }
            "GET" -> {
                val messages = getFromDataStore()
                logger.info { "Retrieved ${messages.size} messages" }
                response.appendHeader("Content-Type", "application/json")
                val res = Json { encodeDefaults = true }.encodeToString(messages)
                response.writer.write(res)
            }
            else -> response.setStatusCode(HttpURLConnection.HTTP_BAD_REQUEST)
        }

    }
}

class Post : HttpFunction {
    private val logger = KotlinLogging.logger {}

    @Throws(IOException::class)
    override fun service(request: HttpRequest, response: HttpResponse) {
        response.appendHeader("Access-Control-Allow-Origin", "https://storage.googleapis.com")
        when(request.method) {
            "OPTIONS" -> {
                logger.info { "Handling CORS request" }
                handleCors("POST", response)
                return
            }
            "POST" -> {
                if (request.contentType.orElse("") == "application/json") {
                    val message = try {
                        Json { encodeDefaults = true }.decodeFromString<Message>(request.reader.readText())
                    } catch (e: Exception) {
                        null
                    }
                    logger.info { "Saving $message" }
                    message?.let {
                        saveToDataStore(it)
                        response.setStatusCode(HttpURLConnection.HTTP_OK)
                    } ?: kotlin.run {
                        response.setStatusCode(HttpURLConnection.HTTP_BAD_REQUEST)
                    }
                } else {
                    logger.error { "Wrong content type" }
                    response.setStatusCode(HttpURLConnection.HTTP_BAD_REQUEST)
                }
            }
            else -> response.setStatusCode(HttpURLConnection.HTTP_BAD_REQUEST)
        }
    }
}

private fun handleCors(method: String, response: HttpResponse) {
    response.appendHeader("Access-Control-Allow-Methods", method)
    response.appendHeader("Access-Control-Allow-Headers", "Content-Type")
    response.appendHeader("Access-Control-Max-Age", "3600")
    response.setStatusCode(HttpURLConnection.HTTP_NO_CONTENT)
}

Testing the function locally

Cloud Functions API has an ability to execute the function locally and test if it works OK.

The configuration is already present in build.gradle.kts.

Let's test the Get function. To run the functions, open the Cloud Shell and run the following:

$ cd ~/serverless/api
$ ./gradlew runFunction -PrunFunction.target=com.example.api.Get -PrunFunction.port=8080

You should get similar response:

INFO: Serving function...
Aug 28, 2020 10:22:12 PM com.google.cloud.functions.invoker.runner.Invoker logServerInfo
INFO: Function: com.example.api.Get
Aug 28, 2020 10:22:12 PM com.google.cloud.functions.invoker.runner.Invoker logServerInfo
INFO: URL: http://localhost:8080/

Open another Cloud Shell tab

And query the function. You should see an empty array:

$ curl http://localhost:8080/

[]

Go back to the first Cloud Shell tab and Ctrl + C to stop the execution of the function.

Create a Service Account

In order to ensure that only authenticated access is allowed onto our Datastore backend, GCP requires us to provide credentials to an authorized account. By default, Cloud Functions use a "default" service account that has full access to our project.

To follow the principle of least privilege, we will create a service account for the backend application and then add a role to it that authorizes the account to make changes to our Datastore backend. Using this account, we will be able to authenticate our web application to the Datastore service at run-time. All of this is managed via GCP's Identity and Access Management service (IAM).

To begin with, visit the console, go to IAM & Admin → Service accounts.

Then, create a new service account:

Specify name of account (backend-api) and find the Cloud Datastore User role that will give the account read/write access to the project's Datastore.

Skip the process for assigning per-user access to the service account as we will be assigning this account to a Cloud Function.

Finally, click Done.

Configure Datastore

If it's the first time you are going to use Datastore - go to Datastore configuration.

Select "Datastore Mode".

Store the data in europe-west3 or europe-west1 and create the database.

Deploying the API

We will now deploy our API endpoints. The commands below build and deploy the functions specifying a Java 11 environment. Since Cloud Functions can be triggered by many different kinds of events, not just REST API requests, we must specify an HTTP trigger. In addition, our two functions only require access to Cloud Datastore so we will deploy both using the service account created previously in order to practice least-privileges. We will also allow unauthenticated access so we don't bother with authentication at the moment.

$ ./gradlew shadowJar
$ ./gradlew buildFunction
$ gcloud functions deploy api-get \
  --entry-point=com.example.api.Get \
  --source=build/deploy \
  --runtime=java11 --trigger-http \
  --allow-unauthenticated \
  --region europe-west1 \
  --service-account backend-api@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com
$ gcloud functions deploy api-post \
  --entry-point=com.example.api.Post \
  --source=build/deploy \
  --runtime=java11 --trigger-http \
  --allow-unauthenticated \
  --region europe-west1 \
  --service-account backend-api@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com

Then, use the commands below to show the functions' settings along with the URL to access each (httpsTrigger):

$ gcloud functions describe api-get --region europe-west1
$ gcloud functions describe api-post --region europe-west1

Create a SPA

With a single page application (SPA), you download the entire site as a static bundle and then you interact with it seamlessly. As the application needs to send and retrieve data to the backend server, it does so asynchronously using Javascript and HTTP requests to and from the APIs it is programmed to access.

To mock a SPA, create a index.html file with the frontend:

$ mkdir -p ~/serverless/spa
$ touch ~/serverless/spa/index.html
$ cloudshell edit ~/serverless/spa/index.html

And paste the following:

<!doctype html>

<html>
  <head>
    <title>Cool SPA</title>
    <script src="./static/spa.js"></script>
  </head>
  <body>
    <div class=page>
      <h2>New Message</h2>
      <div>
          <div>
            <label for="message">Message: </label><br>
            <textarea id="message" rows=5 cols=50 name="message"></textarea>
          </div>
          <button onclick="send()">Send</button>
      </div>

      <h2>Messages</h2>
      <div id="entries"></div>
      
    </div>
  </body>
</html>

Then create a spa.js file to access the API:

$ mkdir -p ~/serverless/spa/static
$ touch ~/serverless/spa/static/spa.js
$ cloudshell edit ~/serverless/spa/static/spa.js

And paste the following:

"use strict";

const PROJECT_ID="REPLACE_ME"
const baseApiUrl = `https://europe-west1-${PROJECT_ID}.cloudfunctions.net/`;

const viewEntries = entries => {
  const entriesNode = document.getElementById("entries");

  while (entriesNode.firstChild) {
    entriesNode.firstChild.remove();
  }

  entries.map(entry => {
    const message = document.createTextNode(entry.text);
    const br = document.createElement("br");
    const p = document.createElement("p");
    p.appendChild(message);
    p.appendChild(br);
    entriesNode.appendChild(p);
  });
};


const send = async () => {
  const message = document.getElementById("message").value;

  const response = await fetch(baseApiUrl + "api-post", {
    headers: {
      Accept: "application/json",
      "Content-Type": "application/json"
    },
    method: "POST",
    body: JSON.stringify({ text: message })
  });
  if (response.status === 200) {
      getEntries();
  }
};

const getEntries = async () => {
  const response = await fetch(baseApiUrl + "api-get", {
    headers: {
      Accept: "application/json",
      "Content-Type": "application/json"
    },
    method: "GET"
  });
  const entries = await response.json();
  viewEntries(entries);
};

getEntries();

Deploy the frontend SPA

Storage buckets are often used to serve static web sites. When configured as multi-region buckets, the content within them can be automatically forward deployed to locations near to where clients are requesting the content from. Distributing our client application can be done via Cloud Shell.

$ cd ~/serverless/spa/
$ gsutil mb gs://restapi-${GOOGLE_CLOUD_PROJECT}

Since this will be a publicly accessible web site, we will assign an access policy allowing all users to access its content via IAM. In this case, the special identifier allUsers specifies everyone and objectViewer assigns read-access permissions.

$ gsutil iam ch allUsers:objectViewer gs://restapi-${GOOGLE_CLOUD_PROJECT}

Finally, copy the entire contents of the directory over to the bucket.

$ gsutil cp -r . gs://restapi-${GOOGLE_CLOUD_PROJECT}

Storage buckets by default are web accessible via the following URL:

https://storage.googleapis.com/<BucketName>

Visit the index.html file in this bucket:

https://storage.googleapis.com/<BucketName>/index.html

Enter a message to check if the Web Application and backend API are working.

Delete project folder in your home directory:

$ rm -r ~/serverless
$ rm -rf ~/.gradle

Delete the bucket:

$ gsutil rm -r gs://restapi-${GOOGLE_CLOUD_PROJECT}

Delete the Cloud Functions:

$ gcloud functions delete api-get --region europe-west1
$ gcloud functions delete api-post --region europe-west1

Delete the service account

$ export PROJECT_ID=$(gcloud config list --format 'value(core.project)')
$ gcloud iam service-accounts delete backend-api@${PROJECT_ID}.iam.gserviceaccount.com

Go to Datastore:

Select Kind "Messages". Check the box selecting all the entities and click delete.

Exit the cloud shell :)