Scan 1 PB at 300MB/s (SATA r2)

Communication Models

Shared Memory

Non-Uniform Memory Access

Shared Nothing / Message Passing

Shared Memory

HDFS, S3, RAM+Modern OSes

NUMA

AMD CPUs

Shared Nothing

MPP, Actor Model

Today we want to clearly see the communications.

Putting Workers Together

"Pipeline" Parallelism

"Pipeline" Parallelism

Together

Distributing Data

Replication		Partitioning (Sharding)

How to Partition Data?

• Arbitrarily
• Range
• Hash

look familiar?

Can we run each worker on one partition?

Filter

Yes

$N$ partitions in, $N$ partitions out

Project

Yes

$N$ partitions in, $N$ partitions out

Union

Yes

Trick question, just combines partitions!

(Single-row) Aggregate

No

$N$ partitions in, $1$ partition out

Partial Aggregation

Algebraic Aggregates (Count, Sum, Avg, Min, Max)

Bounded-size intermediate state

Holistic Aggregates (Median, Mode, Count-Distinct)

Unbounded-size intermediate state

Aggregate needs to process $N$ partitions.

Final aggregate only needs to process $N$ tuples.

↓↓↓↓↓

Basic Aggregate Pattern

Init

Define a starting value for the accumulator

Fold(Accum, New)

Merge a new value into the accumulator

Finalize(Accum)

Extract the aggregate from the accumulator.

Merge(Accum, Accum)

Merge two accumulators together.

↓↓↓↓↓↓

$N$ workers gets us $\sqrt{N}$ scaling

How to Partition Data?

• Arbitrarily
• Range
• Hash

Back to $N$ scaling for $N$ workers

What if the partitions aren't aligned so nicely?

Data Transfer

• Limited IO/Network bandwidth
• Compute needed to receive data

Idea 1: Put the worker on the node that has the data!

Problem: What if the data is on 2 different nodes?

Idea 1.b: Put the worker on one of the nodes with data.

Can we reduce network use more?

Lossy Compression

(not all errors are equal)

False Positives

($b \in \pi_B(S_1)$ when it isn't)

not ideal, but ok

False Negatives

($b \not\in \pi_B(S_1)$ when it is)

bad, wrong answer!

Bloom Filters

Bloom Filter

Test always returns Yes if the element is in the set.

Test usually returns No if the element is not in the set.

Bloom Filters

A bloom filter is an array of bits.

$M$: Number of bits in the array.

$K$: Number of hash functions.

For one record/key

1. $\forall i \in [M] : filter[i] = 0$
2. $\forall j \in [K] : filter[\texttt{hash}_j(key)] = 1$

Each bit vector has $\sim K$ bits set.

(False positive)