Access Google Cloud BigQuery from GO

In this post we will review how to access Google Cloud BigQuery from a GOlang Application.

Google Cloud BigQuery is a:

"Serverless, highly scalable, and cost-effective cloud data warehouse designed for business agility."

In simple words, BigQuery enables use to save huge amount of data in a relational DBMS, and access it using plain SQL language, enriched with some of BigQuery proprietary functions.

Access Key

Now that we have data in BigQuery, we'll probably want to process it.

To access the BigQuery, we first need to create an access key.

To create an access key, login to Google Cloud Platform console, and select IAM and Admin, Service Accounts. Then select the account, and using the menu, select Create key, and export to a JSON format.

Let's save the file in path key.json.

Notice:

The selected service account should be granted with permissions to access the BigQuery.

The GO Application

Now, we can use the key.json file to access BigQuery.

package main

import (
   "cloud.google.com/go/bigquery"
   "context"
   "fmt"
   "google.golang.org/api/iterator"
   "os"
)

func main() {
   projectId := "YOUR_GCP_PROJECT_NAME"

   _ = os.Setenv("GOOGLE_APPLICATION_CREDENTIALS", "key.json")
   bigQueryClient, err := bigquery.NewClient(context.Background(), projectId)
   if err != nil {
      panic(err)
   }

   sql := "select col1,col2 from YOUR_DATASET_NAME.YOUR_TABLE_NAME limit 100"
   query := bigQueryClient.Query(sql)
   result, err := query.Read(context.Background())
   if err != nil {
      panic(err)
   }

   for {
      var row []bigquery.Value
      err := result.Next(&row)
      if err == iterator.Done {
         return
      }
      if err != nil {
         panic(err)
      }

      stringColumn := row[2].(string)
      intColumn := row[3].(int64)

      fmt.Println(stringColumn, intColumn)
   }
}

Notice that the path to the key.json is supplied as an environment variable.

In addition, we need to specify our project ID (even though it is already specified in the key.json), and the SQL text that we want to run.

And that's it, very simple.