Build a BigQuery Processing Pipeline with Events for Cloud Run for Anthos




Create a GKE cluster and set up cloud run events

Create a storage bucket

Query Runner

Chart creator

Test the pipeline

Build a BigQuery Processing Pipeline with Events for Cloud Run for Anthos

1 hour 15 minutes 7 积分


Google Cloud Self-Paced Labs


As serverless architecture is adopted, "events" become an integral part of how individual micro-services are connected yet de-coupled. Events for Cloud Run for Anthos makes "events" a first-class citizen of the Cloud Run for Anthos offering, so that it is easy to build event-driven serverless applications.

Events enable Cloud Run services to produce and/or react to events from a variety of on-cluster as well as off-cluster event sources. Events are delivered over HTTP in the portable CloudEvents format, and application developers do not have to worry about implementing a messaging solutions. Rely on the system for event ingestion, delivery, authentication, authorization, observability and error-handling, while you focus on developing an application code to solve a business problem.

In this lab you will build a BigQuery processing pipeline to query a public COVID-19 dataset on a schedule, create charts from that data, then notify users about the new charts via SendGrid with Events for Cloud Run for Anthos. To use SendGrid account, set up a trial account with your own email account if you don't already have one. Follow this doc for more details on how to set up SendGrid.


How does this work:

  1. Two CloudSchedulerSources are set up to call the QueryRunner service once a day for two countries.

  2. QueryRunner receives the scheduler event for both countries, queries Covid-19 cases for the country using BigQuery's public Covid-19 dataset, and saves the result in a separate BigQuery table. Once done, QueryRunner returns a custom CloudEvent of type dev.knative.samples.querycompleted.

  3. ChartCreator receives the querycompleted event, creates a chart from BigQuery data using mathplotlib and saves it to a Cloud Storage bucket.

  4. Notifier receives the event from the bucket via a CloudStorageSource and sends an email notification to users using SendGrid.

Join Qwiklabs to read the rest of this lab...and more!

  • Get temporary access to the Google Cloud Console.
  • Over 200 labs from beginner to advanced levels.
  • Bite-sized so you can learn at your own pace.
Join to Start This Lab