Announcing CloudQuery History
December 5, 2021
- Name
- Ron Eliahu
- @p0werhead
This feature was deprecated, see blog post.
While TimescaleDB is no longer supported, CloudQuery v1 supports append write mode, which can be used to create a historical view of your data. See the write_mode documentation for more information.
Today we are excited to announce the release of CloudQuery History in alpha! CloudQuery History adds TimescaleDB (opens in a new tab) support to give users the ability to track their complete cloud asset inventory snapshots over time!
Achieving better visibility into your cloud infrastructure is key in maintaining security, compliance, cost and operational efficiency, and this is why we started CloudQuery in the first (opens in a new tab) place. Maintaining a historical record of your cloud infrastructure configuration is an integral part of your cloud environment lifecycle.
Why TimescaleDB?
TimescaleDB (opens in a new tab) is an open-source relational database with support for time-series data. It uses SQL and works as an extension to PostgreSQL. This ensures all current CloudQuery features such as policies and providers continue to work seamlessly and get out-of-the-box support for historical snapshots!
Audit logs vs complete snapshots
Current native solutions like AWS CloudTrail, GCP Cloud Audit Log , Azure Activity/Resource Logs and other SaaS services that have audit logs, records only API calls and changes. Obviously, it is advised to enable them (if not already enabled by default) as they can help with investigation & compliance.
Audit logs are great, although they only focus on what changed and not on what was the state of your whole cloud account at a certain point in time. CloudQuery History provides full historical snapshots of your cloud asset inventory, unlocking the following benefits:
- Visualize Historical State: Enhance your current visualization workflows such as Grafana and re-use the dashboards (opens in a new tab) to view current and historical state.
- Alert on change using standard SQL: Use TimescaleDB's hyperfunctions (opens in a new tab) and continuous aggregates (opens in a new tab) to aggregate at predefined intervals and materialize results and find changes that occurred between fetches.
- Compliance: To ensure you were compliant not only in point in time but also over time, you can re-use pre-made and custom CloudQuery Policies to prove compliance over-time.
- Visibility: Find resources that might already be deleted. Inspect what was created and understand what happened.
- Postmortems and incident response: Full historical snapshots of your cloud assets allow you to gain better insights into what happened in your environment and determine the blast radius. Re-use any standard analytics or BI tools.
Getting Started
History is currently in alpha version, so we welcome any feedback as it's not yet ready for production use.
Setting up History is fairly simple, you are required to either install the TimescaleDB extension on your existing PostgreSQL instance or setup a self hosted TimescaleDB instance. See here (opens in a new tab) for more details.
See CloudQuery quickstart guide and history configuration for more details on how to configure CloudQuery to run with History enabled.
What next?
CloudQuery history opens up endless possibilities for managing compliance, security, visualization and much more! We would love to hear your feedback, either on GitHub (opens in a new tab) or Discord (opens in a new tab).
