Guide to Cassandra Thread Pools

Concepts

Cassandra is based off of a Staged Event Driven Architecture (SEDA). This separates different tasks in stages that are connected by a messaging service. Each like task is grouped into a stage having a queue and thread pool (ScheduledThreadPoolExecutor more specifically for the Java folks). Some stages skip the messaging service and queue tasks immediately on a different stage if it exists on the same node. Each of these queues can be backed up if execution at a stage is being over run. This is a common indication of an issue or performance bottleneck.To demonstrate take for example a read request: 

Note: For the sake of this example the data exists on a different node then the initial request, but in some scenarios this could all be on a single node. There are also multiple paths a read may take. This is just for demonstration purposes.

1. NODE1> The request is queued from node1 on the ReadStage of node2 with the task holding a reference to a callback
2. NODE2> The ReadStage will pull the task off the queue once one of its 32 threads become available
3. NODE2> The task will execute to and compute the resulting Row, which will be sent off to the RequestResponse stage of node1
4. NODE1> One of the four threads in the RequestResponseStage will process the task, and pass the resulting Row to the callback referenced from step 1 which will return to the client.
5. NODE1> It will also possibly kick off an async read repair task on node1 to be processed in the ReadRepairStage once one of its four threads become available

Accessing metrics

TPStats

For manual debugging this is the output given by

nodetool tpstats
 
 Pool Name Active Pending Completed Blocked All time blocked
 ReadStage 0 0 113702 0 0
 RequestResponseStage 0 0 0 0 0
 MutationStage 0 0 164503 0 0
 ReadRepairStage 0 0 0 0 0
 ReplicateOnWriteStage 0 0 0 0 0
 GossipStage 0 0 0 0 0
 AntiEntropyStage 0 0 0 0 0
 MigrationStage 0 0 0 0 0
 MemoryMeter 0 0 35 0 0
 MemtablePostFlusher 0 0 1427 0 0
 FlushWriter 0 0 44 0 0
 MiscStage 0 0 0 0 0
 PendingRangeCalculator 0 0 1 0 0
 commitlog_archiver 0 0 0 0 0
 InternalResponseStage 0 0 0 0 0
 HintedHandoff 0 0 0 0 0
 
 Message type Dropped
 RANGE_SLICE 0
 READ_REPAIR 0
 PAGED_RANGE 0
 BINARY 0
 READ 0
 MUTATION 0
 _TRACE 0
 REQUEST_RESPONSE 0
 COUNTER_MUTATION 0

JMX

These are all available via JMX as well in:

org.apache.cassandra.request:type=*

org.apache.cassandra.internal:type=*

with the attributes (relative to tpstats) being:

Metrics Reporters

Droppable Messages

Stage Details

-Dcassandra.available_processors

Index

ReadStage

Performing a local read. Also includes deserializing data from row cache. If there are pending values this can cause increased read latency. This can spike due to disk problems, poor tuning, or over loading your cluster. In many cases (not disk failure) this is resolved by adding nodes or tuning the system.

RequestResponseStage

When a response to a request is received this is the stage used to execute any callbacks that were created with the original request

MutationStage

Performing a local including:

• insert/updates
• Schema merges
• commit log replays
• hints in progress

ReadRepairStage

Performing read repairs. Chance of them occurring is configurable per column family with read_repair_chance. More likely to back up if using CL.ONE (and to lesser possibly other non-CL.ALL queries) for reads and using multiple data centers. It will then be kicked off asynchronously outside of the queries feedback loop, demonstrated in the diagram above. Note that this is not very likely to be a problem since does not happen on all queries and is fast providing good connectivity between replicas. The repair being droppable also means that after write_request_timeout_in_ms it will be thrown away which further mitigates this. If pending grows attempt to lower the rate for high read CFs:

ALTER TABLE column_family WITH read_repair_chance = 0.01;

ReplicateOnWriteStage

CounterMutation in 2.1, also counters changing dramatically so post 2.0 should consider this obsolete. Performs counter writes on non-coordinator nodes and replicates after a local write. This includes a read so can be pretty expensive. Will back up if the rate of writes exceed the rate that the mutations can occur. Particularly possible with CL.ONE and high counter increment workloads.

GossipStage

Post 2.0.3 there should no longer be issue with pending tasks. Instead monitor logs for a message:

Gossip stage has {} pending tasks; skipping status check ...

AntiEntropyStage

Repairing consistency. Handle repair messages like merkle tree transfer (from Validation compaction) and streaming.

MigrationStage

Making schema changes

MemtablePostFlusher

Operations after flushing the memtable. Discard commit log files that have had all data in them in sstables. Flushing non-cf backed secondary indexes.

FlushWriter

Sort and write memtables to disk. A vast majority of time this backing up is from over running disk capability. The sorting can cause issues as well however. In the case of sorting being a problem, it is usually accompanied with high load but a small amount of actual flushes (seen in cfstats). Can be from huge rows with large column names. i.e. something inserting many large values into a cql collection. If overrunning disk capabilities, it is recommended to add nodes or tune the configuration.

MiscStage

Snapshotting, replicating data after node remove completed.

InternalResponseStage

Responding to non-client initiated messages, including bootstrapping and schema checking

HintedHandoff

Sending missed mutations to other nodes. Usually a symptom of a problem elsewhere so dont treat as root issue. Can use nodetool disablehandoff to prevent further saving. Can also use JMX (nodetool truncatehints) to clear the handoffs for specific endpoints or all of them with org.apache.cassandra.db:HintedHandoffManager operations. This must be followed up with repairs!

MemoryMeter

Measures memory usage and live ratio of a memtable. The pending should not grow beyond size of number of memtables (front ended by set to prevent duplicates). This can take minutes for large memtables.

PendingRangeCalculator

Calculates the token ranges based on bootstrapping and leaving nodes. Instead of blocking this stage discards tasks when one already in progress so no value in monitoring

commitlog_archiver

(Renamed CommitLogArchiver in 2.1) Executes a command to copy (or user defined command) commit log files for recovery. Read more at DataStax Dev Blog

AntiEntropySessions

Number of active repairs that are in progress. Will not show up until after a repair has been run so any monitoring utilities should be able to handle it not existing.

Cassandra Database Consulting

Ready to handle massive data volumes with zero downtime?