Export from Terraform to File

CloudQuery is an open-source data integration platform that allows you to export data from any source to any destination.

The CloudQuery Terraform plugin allows you to sync data from Terraform to any destination, including File. It's free, open source, requires no account, and takes only minutes to get started.

Ready? Let's dive right in!

Step 1. Install the CloudQuery CLI

The CloudQuery CLI is a command-line tool that runs the sync. It supports MacOS, Linux and Windows.

brew install cloudquery/tap/cloudquery

Step 2. Configure the Terraform source plugin

Create a configuration file for the Terraform plugin and set up authentication.

Configuration

Create a file called terraform.yaml and add the following contents:

kind: source
spec:
  # Source spec section
  name: terraform
  path: cloudquery/terraform
  version: "v1.5.2"
  tables: ["*"]
  destinations: ["file"]
  spec:
    backends:
      - name: local
        type: local
        config:
          path: /tmp/cloudquery
      - name: s3
        type: s3
        config:
          bucket: <bucket>
          key: <key>
          region: <region>
          role_arn: <role_arn>

You can have multiple backends at the same time, by describing them in the configuration. Every configuration block describes one backend to handle. CloudQuery currently supports local and s3 backends.

Fine-tune this configuration to match your needs. For more information, see the Terraform Plugin ↗ page in the docs.

Authentication

To authenticate CloudQuery with your Terraform state in S3 you can use any of the following options (see full documentation at AWS SDK V2 (opens in a new tab)):

Static Credentials: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN
Shared configuration files (via aws configure).
SDK defaults to credentials file under .aws folder that is placed in the home folder on your computer
SDK defaults to config file under .aws folder that is placed in the home folder on your computer
If your application uses an ECS task definition or RunTask API operation, IAM role for tasks
If your application is running on an Amazon EC2 instance, IAM role for Amazon EC2

Step 3. Configure the File destination plugin

Create a configuration file for the File plugin and set up authentication.

Configuration

Create a file called file.yaml and add the following contents:

This example configures the file destination, to create CSV files in ./cq_csv_output. You can also choose json or parquet as the output format. Note that the file plugin only supports append write-mode.

The (top level) spec section is described in the Destination Spec Reference.

kind: destination
spec:
  name: "file"
  path: "cloudquery/file"
  version: "v3.0.2"
  write_mode: "append" # file only supports 'append' mode

  spec:
    path: "path/to/files/{{TABLE}}/{{UUID}}.{{FORMAT}}"
    format: "csv" # supported values are 'csv', 'json' and 'parquet'

Fine-tune this configuration to match your needs. For more information, see the File Plugin ↗ page in the docs.

Step 4. Start the Sync

Run the following command in your terminal to start the sync

cloudquery sync terraform.yaml file.yaml

And away we go! 🚀 The sync will run until completion, fetching all selected tables from Terraform. Any errors will be logged to a file called cloudquery.log.

Export from Terraform to File

Step 1. Install the CloudQuery CLI

Step 2. Configure the Terraform source plugin

Configuration

Authentication

Step 3. Configure the File destination plugin

Configuration

Step 4. Start the Sync

Further Reading