BigQuery

Get started with the BigQuery source!

Obtain BigQuery credentials

For the BigQuery target to work correctly, you need to create credentials in your Google Cloud Platform Web Console. The instructions to create these credentials are here.

Enable the source

First thing you need to do is head to the flow that you would like to add a BigQuery target to.

Now you should go ahead and click the Connect button for BigQuery.

Now you should fill in the credentials for all the BigQuery. If you don't know how to get these credentials, you can learn how to get them here. Once you are done filling in the credentials, click the Save & Continue button at the bottom of the page.

You have now linked the BigQuery source!

How to get your BigQuery credentials

Enable the BigQuery API

First and foremost, make sure you are logged in to the correct Google account that you would like to access BigQuery from. Once you're logged in to the correct Google account, head to the Google Cloud Platfrom Web Console.

Once you are on the home page of GCP's Web Console, head to the navigation bar on the left side of the screen. Once you open the Navigation bar, head to the APIs & Services tab, and select Library from the resulting drop down.

This will take you to a page where you should input BigQuery in the search box. Once it pops up, go ahead and click on it.

Now, go ahead and click the Enable button in order to enable the BigQuery API.

Authenticating with a service account

It is recomended that you use a service account with the BigQuery target. To create service account credentials, take the following steps.

Use the navigation bar on the left again to navigate to the APIs & Services tab and select Credentials from the resulting drop down menu. Once you are on the Credentials page, click the Create Credentials button at the top of the page, which will trigger a drop down menu. From that drop down menu, go ahead and select Service Account.

Under the Service account details, title the account target-bigquery and click the Create button.

Under Grant this service account access to project, make sure that you have two roles. The first role should allow the service account to be a BigQuery Data Editor. This allows the tap to edit the contents of the data sets (write permissions). Make sure that the second role is BigQuery Job User. This is a bit self explanatory, it allows the target to run jobs.

Now, using the navigation panel again, head back to the APIs & Services tab and go the Credentials tab within the tab. You should see the service account you just created near the bottom of your screen. Go and click on it.

Near the bottom of the page, you should go ahead and click the Add Key button which will prompt a dropdown. Once it does this, click Create new key.

Select JSON for your private key and click Create.

You should go ahead and open the file. Make sure you keep this somewhere safe. This file holds the credentials that you should use to connect your BigQuery account as a target in hotglue.

{
  "type": "service_account",
  "project_id": "acme",
  "private_key_id": "1**********************************f",
  "private_key": "-----BEGIN PRIVATE KEY-----\n*************\n*****************************\n************************n/*************************\n******************************\n*************************\n************************************\n*******************************\n**********************************\n**********************************\n**********************************\n**********************************\n**********************************\n**********************************\n**********************************\n**********************************\n**********************************\n**********************************\n**********************************\n**********************************\n**********************************\n**********************************\n**********************************\n**********************************\n**********************************\n-----END PRIVATE KEY-----\n",
  "client_email": "***********@acme.iam.gserviceaccount.com",
  "client_id": "1************7",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/**************image
    40acme.iam.gserviceaccount.com"
}

These are the credentials that are relevant for configuring your hotglue target. Paste them into their corresponding places in hotglue and you are all set!

Dataset ID output path

The output path can include the following parameters:

Parameter

Value

{tenant}

The user id this job is running for

{date}

The current date formatted as YYYYMM

{flow_id}

The flow id this job ran for

{env_id}

The environment this job ran for

{job_id}

The internal hotglue job id

For example, a valid dataset ID output path is data/invoices/{tenant}


Did this page help you?