Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update backup requests docs #89

Merged
merged 3 commits into from
Jan 27, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 15 additions & 14 deletions _docs/en/administration/backup-request.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,30 +3,31 @@ permalink: administration/backup-request
---

# Background
In the current implementation of Pegasus, reading from secondary replicas can lead to inconsistencies, so only primary replicas are used for reads. However, in certain scenarios, such as load balancing or hotspot writes can cause instability on the primary replica. To address this, we aim to enable reads from secondary replicas when the primary is unstable. While this approach sacrifices some strong consistency, it helps to alleviate long-tail latency in read requests and improves overall system availability. The **backup request** mechanism is designed to facilitate this functionality.
In the current implementation of Pegasus, reading from secondary replicas can cause inconsistencies, so Pegasus defaults reading from primary replicas. However, in certain situations (such as load balancing, hotspot writes, etc.), the primary can often become unstable. Therefore, we hope to read from the secondary when the primary is unstable, sacrificing some strong consistency to reduce the tail latency of read requests and improve system availability. Backup request is designed to achieve this functionality.

# Design and Implementation
Implementing of backup requests is relatively straightforward. For read operations (write operations currently do not support backup requests), when a client sends a request to the primary, it will wait for a specified delay period (typically p999). If no response is received within this time, the client will randomly select a secondary replica and send a backup request. The first response received will be used.

We recommend using p999 as the delay for sending secondary requests, as the purpose of the backup request operation is to eliminate long-tail latency rather than to improve cluster performance. Setting this value too low can result in an overwhelming number of backup requests, thereby significantly increasing the overall system load. For example, if the delay is set to p50, 50% of the requests would be sent to secondary replicas, causing a 50% increase in system load.
The implementation principle of the backup request is relatively simple: For read operations (currently, write operations do not support backup requests), when the client sends a request to the primary, if the response has not been returned after a certain delay (usually p999), a secondary is randomly selected and a backup request is sent to it. Finally, the fastest returned response is processed.

# How to Use
In Pegasus Java Client v2.0.0, we introduced an interface that allows users to enable the backup request feature for a specific table. The method is defined as follows:
We recommend choosing p999 as the delay time for sending secondary requests because the backup request operation is intended to eliminate tail latency, not to improve cluster performance. If the value is set too low, the large number of backup requests will increase the cluster pressure (assuming p50 is chosen as the delay, then 50% of the requests will send requests to the secondary, and the system load will increase by 50%).

# How to Use
In Pegasus Java client v2.0.0, we have added an interface through which the backup request function of a specific table can be enabled. The implementation is as follows:
```java
public PegasusTableInterface openTable(String tableName, int backupRequestDelayMs) throws PException;
```

Compared to the previous version of the `openTable` interface, we’ve added the `backupRequestDelayMs` parameter. This parameter defines the delay time in milliseconds: if a request sent to the primary replica does not receive a response within `backupRequestDelayMs`, a backup request will be sent to a secondary replica. Notice that setting `backupRequestDelayMs <= 0` disables the backup request feature.
Compared to the old version of the `openTable` interface, we have added a `backupRequestDelayMs` parameter. This parameter is the delay mentioned above, i.e.: sending a request to the primary, if the response has not returned after `backupRequestDelayMs` milliseconds, then send a backup request to the secondary. Note that `backupRequestDelayMs <= 0` means disabling the backup request feature.

In addition, in the old version of the `openTable` interface, the backup request feature is disabled by default.

In previous versions of the openTable interface, the backup request mechanism was disabled by default.
# Benchmark

# Performance Testing
The table below compares the performance between enabling and disabling backup requests. We used the p999 latency of read without backup requests (138 ms) as the delay for triggering backup requests. The data shows that enabling backup requests has **no significant impact** on the p999 latency for `get` requests, but the p9999 latency is **reduced by several times**.
The following table shows the performance comparison of whether the backup request is enabled. Here we selected the p999 time of read requests when the backup request is not enabled as the delay time for the backup request (138ms). The data shows that after enabling the backup request, the p999 latency of get requests **remains almost unchanged**, while the p9999 latency is **reduced by several times**.

Additionally, since the delay is set to the p999 value, approximately one out of every thousand requests triggers a backup request. This results in an additional request load (i.e., the overhead of enabling backup requests) of approximately 0.1%. Similarly, setting the `backupRequestDelayMs` to p99 can further reduce the p999 latency, which may increase the additional read request load by around 1%.
In addition, since the delay time is set to p999 time, about 1 out of 1000 requests will send a backup request, so the proportion of additional request volume (i.e., the additional overhead of enabling the backup request) is about 0.1%. By analogy, if you want to reduce P999 latency, you can set `backupRequestDelayMs` to P99 latency, which will increase the additional read traffic by 1%.

| Test Case | Backup Request Enabled | Read p9999 |
|-------------------------|------------------------|------------|
| 3 Clients, 15 Threads | No | 988,671 |
| 3 Clients, 15 Threads | Yes | 153,599 |
| test case | enable backup request | read p9999 latency in microsecond |
|----------------------|-----------------------|-----------------------------------|
| 3-clients 15-threads | no | 988,671 |
| 3-clients 15-threads | yes | 153,599 |
27 changes: 13 additions & 14 deletions _docs/zh/administration/backup-request.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,32 +3,31 @@ permalink: administration/backup-request
---

# 背景
在当前的Pegasus实现中,由于向secondary读取会导致不一致的情况发生,所以目前Pegasus仅仅支持对primary副本的读取。但是在某些情况下(例如:负载均衡、热点写入等)经常会导致primary不稳定。所以我们希望在primary不稳定时能够读取secondary,通过牺牲部分强一致性来降低读请求的长尾并提高系统的可用性。backup request便是用来实现此功能的
在当前的 Pegasus 实现中,由于向 secondary 读取会导致不一致的情况发生,所以 Pegasus 默认只对 primary 副本的读取。但是在某些情况下(例如:负载均衡、热点写入等)经常会导致 primary 不稳定。因此,我们希望在 primary 不稳定时能够读取 secondary,通过牺牲部分强一致性来降低读请求的长尾并提高系统的可用性。backup request 便是用来实现此功能的

# 设计实现

backup reqeust的实现原理比较简单:对于读操作(目前写操作不支持backup request),当client向primary发送请求后,如果经过一段时间延时(通常是p999)其response仍然没有返回,则随机选择一台secondary并向其发送backup request。最后获取最快返回来的response进行处理
backup request 的实现原理比较简单:对于读操作(目前写操作不支持 backup request),当 client 向 primary 发送请求后,如果经过一段时间延迟(通常是 p999)其 response 仍然没有返回,则随机选择一台 secondary 并向其发送 backup request。最后获取最快返回的 response 进行处理

这里发送secondary请求的延时我们建议选择p999,因为backup request操作是用来实现消除长尾的,并不是提升集群性能的。如果将该值设置过低,则会由于backup request的请求量过大而导致集群压力增大(假设选择p50作为其延时,这样便会有50%的请求向secondary发送请求,系统负载便会增大50%)。
这里发送 secondary 请求的延迟我们建议选择 p999,因为 backup request 操作是用来消除长尾的,并不是提升集群性能的。如果将该值设置过低,则会由于 backup request 的请求量过大而导致集群压力增大(假设选择 p50 作为其延迟,这样便会有 50% 的请求向 secondary 发送请求,系统负载便会增大 50%)。

# 如何使用
在Pegasus java client v2.0.0中,我们增加了一个接口,通过该接口可以打开某个表的backup reqeust功能。其实现如下:
在 Pegasus Java client v2.0.0 中,我们增加了一个接口,通过该接口可以打开某个表的 backup request 功能。其实现如下:
```java
public PegasusTableInterface openTable(String tableName, int backupRequestDelayMs) throws PException;
```

相比于老版本的`openTable`接口,我们增加了一个`backupRequestDelayMs`参数。这个参数便是上文所指的时延,即:向primary发送请求,如果过了`backupRequestDelayMs`毫秒response仍没有返回,则向secondary发送backup request。需要注意的是,`backupRequestDelayMs <= 0`代表禁用backup reqeust功能
相比于老版本的 `openTable` 接口,我们增加了一个 `backupRequestDelayMs` 参数。这个参数便是上文所指的时延,即:向 primary 发送请求,如果过了 `backupRequestDelayMs` 毫秒 response 仍没有返回,则向 secondary 发送 backup request。需要注意的是,`backupRequestDelayMs <= 0` 代表禁用 backup request 功能

另外在老版本的`openTable`接口中,backup request功能默认是关闭的
另外在老版本的 `openTable` 接口中,backup request 功能默认是关闭的

# 性能测试
# Benchmark

下面表格里展示了是否打开backup request的性能对比,这里我们选取了未打开backup request时读请求的p999时间作为backup request的delay时间(138ms)。数据显示,打开backup request之后get请求的p999时延**基本没有变化**,而p9999时延却有了**数倍的降低**。
下面表格里展示了是否打开 backup request 的性能对比,这里我们选取了未打开 backup request 时读请求的 p999 时间作为 backup request 的 delay 时间(138ms。数据显示,打开 backup request 之后 get 请求的 p999 时延**基本没有变化**,而 p9999 时延却有了**数倍的降低**。

另外,由于delay时间设置的是p999时间,大约1000个请求里只有1个请求会发送backup request,因此额外请求量(也就是开启backup request的额外开销)比例在0.1%左右。依此类推,若想要降低P999时延,则可以将 `backupRequestDelayMs` 设置为P99延迟,由此会增加1%的额外读流量。

| test case | enable backup request | read p9999 |
| ---- | ---- | ---- |
| 3-clients 15-threads | no | 988671 |
| 3-clients 15-threads | yes| 153599 |
另外,由于 delay 时间设置的是 p999 时间,大约 1000 个请求里只有 1 个请求会发送 backup request,因此额外请求量(也就是开启 backup request 的额外开销)比例在 0.1% 左右。依此类推,若想要降低 P999 时延,则可以将 `backupRequestDelayMs` 设置为 P99 延迟,由此会增加 1% 的额外读流量。

| test case | enable backup request | read p9999 latency in microsecond |
|----------------------|-----------------------|-----------------------------------|
| 3-clients 15-threads | no | 988,671 |
| 3-clients 15-threads | yes | 153,599 |
Loading