1 Key Design

Redshift is a fast, fully managed, petabyte-scale data warehouse solution that makes it simple and cost-effective to efficiently analyze large volumes of data. It uses familiar data warehousing techniques, including columnar layout, per-column compression, co-locating compute and data, co-locating joins, compilation to machine code and scale-out MPP processing. It also had a number of additional design goals:

  1. Minimize cost of experimentation
  2. Minimize time to first report: from deciding to create a cluster to seeing the results of their first query
  3. Minimize administration: automate backup, restore, provisioning, patching, failure detection and repair
  4. Minimize scaling concerns
  5. Minimize tuning

The analysis gap between data being collected and data available for analysis is due to four major causes:

  • Cost: up front expense makes it hard to justify for large datasets with unclear value
  • Complexity: database provisioning, maintenance, backup and tuning are complex tasks requiring specialized skills
  • Performance: difficult to grow a data warehouse without negatively impacting query performance, IT teams sometimes discourage augmenting data or adding queries to protect current reporting SLAs
  • Rigidity: increasing number of data consists of machine-generated logs, audo and video not accessible to relational analysis

Redshift obtains a number of advantages by operating as a managed service in the AWS cloud:

  1. leverage the pricing and scale of EC2, the durability and throughput of S3 for backup and security of Amazon VPC
  2. making deployments and patching automatic and painless for customers, averaged addition of one feature per week
  3. automating capabilities that would not make economic sense for individual on-prem DBA

2 Architecture

2.1 Data Plane

Query processing begins with query plan generation and compilation to C++ and machine code at the leader node. At the compute nodes, the executable is run with the plan parameters and the intermediate results are sent back to the leader node for final aggregation.

A cluster is comprised of a leader node and one or more compute nodes (also support a single-node design where leader and compute work is shared on a single node).

Leader node: accepts connections from client programs, parses requests, generates & compiles query plans for execution on the compute nodes, performs final aggregation of results when required and coordinates serialization and state of transactions.

Data stored within each table is automatically distributed both across compute nodes, to enable scale out of large data sets and within a compute node to reduce contention across processing cores. A compute node is partitioned into slices; one slice for each core of the node’s multi-core processor. Each slice is allocated a portion of the node’s memory and disk space. Distribution can be round-robin, hashed by a distribution key or duplicated on all slices.

Each data block is synchronously written to both its primary slice as well as to at least one secondary separate node.This limits the impact of individual disk or node failure. They are also asynchronously and automatically backed up to S3. Customer can also choose to have backups occur to S3 in a second region.

2.2 Control Plane

Host manager software that helps with deploying new database engine bits, aggregating events with metrics, generating instance-level events, archiving and rotating logs, and monitoring the host, database and log files for errors.

2.3 Dependent AWS Services

EC2 - instances

S3 - backup

Amazon Simple Workflow - control plane actions

CloudWatch - customer instance metrics

Simple Notification Service (SNS) - customer alarms

VPC - network isolation

Route53 - DNS lookup

CloudTrail - audit logging

KMS and CloudHSM - key management

The use of these services has significantly accelerated Amazon Redshift development: avoided the need to build robust, fault-tolerant and secure components available in other AWS services; access to an ongoing innovation stream by sister development teams.

3 Highlights

3.1 The Case for Simplicity

  1. Simplifying the Purchase Decision Process
    1. Time to first report
    2. standard PostgresSQL ODBC/JDBC driver
    3. limited the information required to number and type of nodes, basic network configuration and administrative account information
    4. reducing the cost of an error
  2. Simplifying Database Administration
    1. do not have designated DBA to administer their database
    2. backup and ecnryption is just a checkbox
  3. Simplifying Database Tuning

3.2 Customer Use Cases

Enterprise data warehousing

Semi-structured “Big Data” analysis

Data Transformation

Small Data

3.3 Learnings

Design escalators, not elevators: design systems that degrade on failures rather than losing outright availability

Continuous delivery should be to the customer

Use Pareto analysis to schedule work

4 Related Work

Redshift is the first widely available data warehouse-as-a-service. It forgoes traditional indexes and instead focuses on sequential scan speed through compiled code execution and column-block skipping based on value-ranges stored in memory.

It can be provisioned in minutes, enabling customers to get started with no commitments and scale up to a petabyte scale cluster.