Tessell Availability Machine: Why, What, How?

Introduction

With more and more organizations seeing value in deploying their databases to the cloud, there has been an exponential demand for cloud-based database solutions. While most DBaaS services are limited to undifferentiated “heavy lifting” database tasks such as provisioning, backup and restore, and managing high availability, these DBaaS services do not address the various data management needs of organizations. As a result, many organizations are turning to siloed applications to manage their multiple data management needs. This also burdens database administrators as they handle daily operational tasks such as patching, backups, and provisioning databases, along with data management and data protection.

Availability Machine, an in-built application in Tessell, aims to eradicate these silos in data management by providing a unified approach for your data management needs, differentiated automation and heavy lifting operations, and data protection - all made possible in a few clicks at no extra cost. The Availability Machine, as the name suggests, lies at the core of Tessell’s data management architecture. When you provision a database service in Tessell, an Availability Machine (AM) gets associated with it which encompasses every possible data management-related aspect of the database service.

With the Availability machine, you can:

• Protect the associated Database Service (SLA)
• Capture data (snapshots, backups, logs)
• Transform data (data masking, data sanitization, DB exports)
• Manage the data for all environments using access-policies

Data Protection

The primary purpose of an Availability Machine is to protect the data that’s stored in your database service.

Assign SLA

A data protection SLA can be assigned to the database service during creation. The SLA can be changed at any later time. Tessell ships a few out-of-the-box SLA templates. Customers can choose either of them or create a custom SLA of their choice.

At a high level, the Availability Machine SLA helps to define either or both of the below requirements:

• Point-in-time recoverability
• ~ This is specified as the number of days
• ~ Example - if a customer specifies 7 days as the PIT recoverability, Tessell would make sure the Database Service is recoverable up to the last 7 days.
• Discrete recoverability
• ~ This is specified as the number of days
• ~ The snapshot capture time can also be specified
• ~ Example - if a customer specifies 15 days as the daily recoverability, Tessell would make sure the Database Service has snapshots available for recovery up to the last 15 days (one snapshot per day at the specified time).
• ~ In the future, advanced functionality would be added to support recoverability for weekly, monthly, quarterly, and yearly snapshots.

Automated Snapshots

As per the specified SLA, Tessell captures DB Service snapshots automatically. The snapshots also get deleted automatically, when they are no more required as per the SLA.

Optionally, the snapshots can be shared across other cloud regions or users, if required.

Automated Log Backups

To guarantee zero data loss, Tessell continuously takes backups of transaction logs.

PITR (Point-in-time recovery) or Discrete Recovery

With Tessell’s zero data loss guarantee, you can restore your database to the last committed transaction. Customers can also restore their database service using the discrete snapshots.

Data Management

The Availability Machine maintains a catalog of different kinds of data: snapshots, archive logs, database dumps, sanitized data, traditional backups, and more.

Manual Snapshots

Take snapshots at any time with a single click. Once captured, the manual snapshots would be maintained by the Availability Machine unless users explicitly request to delete them.

Optionally, the manual snapshots can be shared across other cloud regions or users.

The customers can use the manual snapshots to restore the database service or to clone the database service across regions.

Sanitized Snapshots

Tessell would provide a mechanism to sanitize/mask users’ database service data and make it available to specified target cloud regions and users.

Users are enabled to bring their own masking script and do the following

• Create sanitized snapshots on demand
• ~ The user can request sanitization for the system-created (SLA-based) automated snapshot or manually-created snapshots
• Create a schedule to produce sanitized snapshots regularly
• ~ A user can set a schedule to automatically sanitize new snapshots that get generated as per the SLA
• ~ The retention period for sanitized snapshots would also be specified as part of the schedule.

At a high level, data sanitization would work as below:

1. The sanitization would happen on top of a database service snapshot. The snapshot can be manual or automated.
2. The user specifies a Sanitization Script. The script has SQL instructions to update the data in the schem
3. The snapshot would be restored, the sanitization script would be applied against the restored DB, and a new snapshot would be created from the processed data. The new snapshot would now contain the state of data post-sanitization.
4. The sanitized snapshot is by default available in the source region of the DB Service
5. Optionally, the sanitized snapshot can be made available to different target cloud regions, as part of one or more Data Access Policy specifications and also be made available to other users

Data Access Policy (DAP)

The Availability Machine helps users define "what,” "where,” "when," “to whom,” and “how much” of their data to be made available based on policies. The core construct to configure it is Data Access Policy (DAP).

A DAP helps data-owners to manage the availability of database snapshots, sanitized snapshots for secondary environments, or other use cases like dev, QA, and analytics.

The data owner can define what view of data should be made available to the consumers. Below are the supported data views.

DAP for As-Is Data

Share the as-is production snapshots, without explicit processing or masking.

The consumer of these snapshots can either restore the snapshot, or create a secondary copy (clone) of the database service.

Use-cases: The UAT (User Acceptance Testing) use-case may need access to the as-is state of production data

Sharing types

• Automated
• ~ The users can configure a DAP to share automated contents
• ~ As the newer automated snapshots are captured, they will get made available as per the DAP
• ~ The snapshots replicated/shared using the DAP would also get deleted as per the configuration
• Manual
• ~ The users can pick and choose what snapshots to be made available as per the DAP
• ~ The manually shared snapshot would get deleted only when users decide to delete them

DAP for Sanitized Data

Due to the security or compliance requirements, users may not be comfortable making sensitive as-is data available for all users/user groups. In such cases, the data owner can define a policy and configure how the data should be processed (masked, sanitized) before making it accessible to other users.

Using the masked snapshots, the consumers can create a secondary copy (clone) of the database service.

Use-cases: As part of usual CI/CD flows, the QA or dev teams might request a copy of production data, not necessarily in an as-is view.

Sharing types

• Automated
• ~ The users can configure a DAP to share automated contents - snapshots generated from a given sanitization schedule
• ~ As the newer sanitized automated snapshots are created, they will get made available as per the DAP
• ~ The snapshots replicated/shared using the DAP would also get deleted as per the configuration
• Manual
• ~ The users can pick and choose what snapshots to be made available as per the DAP
• ~ The manually shared snapshot would get deleted only when users decide to delete them

Sharing and Access Controls

An Availability Machine can be shared across users, with different supported access levels.

The currently supported access levels are:

1. Co-owner: The users with co-owner access can:
2. ~ Share/revoke Availability Machine access with other users
3. ~ Create a Snapshot
4. ~ Create a Sanitized Snapshot
5. ~ Create a Sanitization Schedule
6. ~ Create a DAP
7. ~ Update SLA and snapshot time
8. Read-only: The users with read-only access can:
9. ~ View-only the SLA and snapshot time
10. ~ View-only Sanitization Schedules
11. ~ View-only DAPs

Dataflix

The Availability Machine is the producer’s view of the data, while the Dataflix is the consumer’s view of the data.

Anyone with access to the Availability Machine can actually consume the data to create clones, etc. The consumption will be exposed in the Dataflix app.

For a given user, Dataflix shows an intuitive and interactive view of the data available to him/her.

A Dataflix view provides complete insights about below aspects:

• Whether or not the continuous recovery regions are enabled
• What are the accessible as-is snapshots and their availability across clouds, and regions
• What are the accessible sanitized snapshots and their availability across clouds, and regions
• For every snapshot, show the insights:
• ~ Number of databases
• ~ Number of tables
• ~ The timestamp when the snapshot is captured

The snapshots and PIT recoverability which is available in Dataflix can be used to create a database service clone.

We'll cover Dataflix in more detail in a separate blog post.