# Email To BigQuery

# Introduction

The module allows you to send data files to a dedicated e-mail address. After the file is sent to your e-mail address, it will be automatically saved in the BigQuery table so that the data can be used for integration or analysis.

# An example of operation

Suppose a customer has transaction status in their CRM system and would like to send them systematically to BigQuery to be later linked to Google Analytics data. Unfortunately, automating this process would require developer involvement, which is not always possible. Fortunately, almost every CRM system allows you to set the sending of a CSV file with transactions to e-mail on a set schedule. The subscriber of such an e-mail can be a dedicated e-mail address provided by our module.

Every time a new file arrives at your dedicated email address, it will be automatically imported into your BigQuery WitCloud project.

At a later stage, it can be used for planned integration or data analysis.

image alt text

# Configuration

# Start creating the module

From the menu on the left, select the "Collect" tab, then click the "Add new pipeline" button.

image alt text

Select "Email to BigQuery" from the list of available modules

image alt text

# Initial settings

In the first step, we have 2 settings to choose from.

Name - the name of our process, e.g. "CRM Statuses"

Write Disposition:

  • Overwrite if partition exists - this option means that the data will be overwritten each time it is sent to the e-mail. It is worth selecting the options if there is duplication of data in the files sent. Such a situation may occur, for example, if we send data twice a day and the file contains the same transaction identifiers. To avoid duplicating the table, the data will be overwritten each time to keep only the most recent data.

  • Append to partitions - this option means that data will be added to an existing table. It is worth checking this option if the files being sent contain only new data.

  • Write only to empty partitions - this option allows you to write data only when the BigQuery table does not contain any data yet. For example, if we send 2 data files during the day, only the first one will be saved. The second one will be rejected because the BigQuery table will already have other data imported.

image alt text

# Receiving a dedicated email address

In the second step, we receive an e-mail address that should be set as the recipient of the e-mail with the data file. After sending the e-mail with the data as a CSV attachment, the data will be automatically imported into your BigQuery WitCloud project.

image alt text

Last updated: 2/3/2021, 10:28:38 AM