Bandwidth recommendations for direct copy #2884

drbasic · 2025-01-20T07:45:59Z

Продолжение #2368
Здесь делается ограничение по скорости со стороны ДискАгента.

Дискагент на старте зачитывает файл от инфры со скоростью интерфейса.
Дискагент считает со скольки девайсов за последнюю секунду производилось копирование.
Дискагент рассчитывает сколько полосы может занять копирование одного девайса, для этого он делит всю полосу, отведенную под копирование, на всех клиентов, ограничивая каждого клиента сверху величиной MaxDeviceBandwidthMiB
В ответе TEvDirectCopyBlocksResponse отправляется рекомендуемая полоса, которую может использовать клиент для этого девайса.

github-actions · 2025-01-20T08:53:25Z

Note

This is an automated comment that will be appended during run.

🔴 linux-x86_64-relwithdebinfo: some tests FAILED for commit 6d531ee.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
5826	5821	0	5	0	0

github-actions · 2025-01-20T13:26:29Z

Note

This is an automated comment that will be appended during run.

🔴 linux-x86_64-relwithdebinfo: some tests FAILED for commit 102b8ad.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
5827	5814	0	7	6	0

github-actions · 2025-01-21T06:04:11Z

Note

This is an automated comment that will be appended during run.

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit ad8643b.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
6332	6332	0	0	0	0

github-actions · 2025-01-21T10:08:10Z

Note

This is an automated comment that will be appended during run.

🔴 linux-x86_64-relwithdebinfo: some tests FAILED for commit 076c215.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
3583	3582	0	1	0	0

github-actions · 2025-01-21T11:16:24Z

Note

This is an automated comment that will be appended during run.

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit 26a408f.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
3584	3584	0	0	0	0

drbasic · 2025-01-21T10:57:33Z

cloud/blockstore/config/disk.proto

+message TDiskAgentThrottlingConfig
+{
+    // Host limits.
+    optional string InfraThrottlingConfigPath = 1;


Поле InfraThrottlingConfigPath уже есть в настройках тротлера NBS.
Если файл не получилось прочитать будет использован дефолт DefaultNetworkMbitThroughput

drbasic · 2025-01-21T10:59:37Z

cloud/blockstore/config/disk.proto

+    optional double DirectCopyBandwidthFraction = 3;
+
+    // Maximum bandwidth for one device in MiB/s.
+    optional uint64 MaxDeviceBandwidthMiB = 4;


Нужно ограничить нагрузку на один девайс, потому что если сеть быстрая, то просядут клиентские записи в девайс, куда проводится копирование.

drbasic · 2025-01-21T11:01:19Z

cloud/blockstore/libs/daemon/common/config_initializer.cpp

-        Rack
-    );
+        Rack,
+        HostPerformanceProfile.NetworkMbitThroughput);


если дискагент создается в процессе NBS то берем полосу сетевого интерфейса, которую зачитал тротлер NBSа. Чтобы не читать файл еще раз.

drbasic · 2025-01-21T11:02:51Z

cloud/blockstore/libs/disk_agent/config_initializer.cpp

 ////////////////////////////////////////////////////////////////////////////////

+std::optional<NJson::TJsonValue> ReadJsonFile(


копипаста по парсингу файла от инфры https://github.com/ydb-platform/nbs/blob/main/cloud/blockstore/libs/daemon/common/config_initializer.cpp#L168

drbasic · 2025-01-21T11:05:58Z

cloud/blockstore/libs/storage/disk_agent/actors/direct_copy_actor.h

@@ -20,6 +20,7 @@ class TDirectCopyActor final
    const NActors::TActorId Source;
    const TRequestInfoPtr RequestInfo;
    const NProto::TDirectCopyBlocksRequest Request;
+    const ui64 RecommendedBandwidth;


RecommendedBandwidth заполняется при создании актора. Он подмешивает это в ответ перед отправкой клиенту.

drbasic · 2025-01-21T11:07:55Z

cloud/blockstore/libs/storage/disk_agent/disk_agent_actor.h

@@ -71,6 +72,8 @@ class TDiskAgentActor final
    // Pending WaitReady requests
    TDeque<TPendingRequest> PendingRequests;

+    TBandwidthCalculator BandwidthCalculator {*AgentConfig};


BandwidthCalculator один на сервер.

drbasic · 2025-01-21T13:08:01Z

cloud/blockstore/libs/storage/disk_agent/model/bandwidth_calculator.cpp

+    TInstant now)
+{
+    DeviceLastRequest[deviceUUID] = now;
+    ClearHistory(now - WindowDuration);


забываем девайсы, которые не трогали больше секунды.

считаем сколько у нас девайсов к которым есть запросы и делим полосу поровну

drbasic · 2025-01-21T13:10:18Z

.../blockstore/libs/storage/partition_nonrepl/part_nonrepl_migration_common_actor_migration.cpp


    if (msg->AllZeroes) {
        ChangedRangesMap.MarkNotChanged(msg->Range);
    }
+    TimeoutCalculator->SetRecommendedBandwidth(msg->RecommendedBandwidth);


после получения рекомендации сохраняем ее в TimeoutCalculator и он будет использовать ее при расчетах таймаутов между запросами.

drbasic · 2025-01-21T13:11:09Z

example/0-setup.sh

@@ -77,6 +77,13 @@ FileDevices: {
    BlockSize: 4096
 }

+ThrottlingConfig: {


для упрощения локальной разработки

cloud/blockstore/libs/disk_agent/config_initializer.cpp

cloud/blockstore/libs/storage/disk_agent/model/bandwidth_calculator.cpp

github-actions · 2025-01-22T12:07:36Z

Note

This is an automated comment that will be appended during run.

🔴 linux-x86_64-relwithdebinfo: some tests FAILED for commit 6ae8204.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
3584	3583	0	1	0	0

Bandwidth recommendations for direct copy

6d531ee

drbasic force-pushed the users/drbasic/recommended-bandwidth branch from 20281ad to 1f10e5e Compare January 20, 2025 11:54

drbasic mentioned this pull request Jan 20, 2025

[NBS] Direct DiskAgent to DiskAgent devices replication #2368

Open

Configs for direct copy throttling

102b8ad

drbasic force-pushed the users/drbasic/recommended-bandwidth branch from 1f10e5e to 102b8ad Compare January 20, 2025 12:12

Fix tests build

ad8643b

drbasic force-pushed the users/drbasic/recommended-bandwidth branch from 7302424 to ad8643b Compare January 21, 2025 04:52

drbasic added the blockstore Add this label to run only cloud/blockstore build and tests on PR label Jan 21, 2025

Tests

26a408f

drbasic force-pushed the users/drbasic/recommended-bandwidth branch from 076c215 to 26a408f Compare January 21, 2025 10:47

drbasic marked this pull request as ready for review January 21, 2025 10:54

drbasic commented Jan 21, 2025

View reviewed changes

komarevtsev-d reviewed Jan 22, 2025

View reviewed changes

Fix review issues

6ae8204

komarevtsev-d approved these changes Jan 22, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bandwidth recommendations for direct copy #2884

Bandwidth recommendations for direct copy #2884

drbasic commented Jan 20, 2025 •

edited

Loading

github-actions bot commented Jan 20, 2025

github-actions bot commented Jan 20, 2025

github-actions bot commented Jan 21, 2025

github-actions bot commented Jan 21, 2025

github-actions bot commented Jan 21, 2025

drbasic Jan 21, 2025

drbasic Jan 21, 2025

drbasic Jan 21, 2025

drbasic Jan 21, 2025

drbasic Jan 21, 2025

drbasic Jan 21, 2025

drbasic Jan 21, 2025

drbasic Jan 21, 2025

drbasic Jan 21, 2025

github-actions bot commented Jan 22, 2025

		////////////////////////////////////////////////////////////////////////////////

		std::optional<NJson::TJsonValue> ReadJsonFile(

Bandwidth recommendations for direct copy #2884

Are you sure you want to change the base?

Bandwidth recommendations for direct copy #2884

Conversation

drbasic commented Jan 20, 2025 • edited Loading

github-actions bot commented Jan 20, 2025

github-actions bot commented Jan 20, 2025

github-actions bot commented Jan 21, 2025

github-actions bot commented Jan 21, 2025

github-actions bot commented Jan 21, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jan 22, 2025

drbasic commented Jan 20, 2025 •

edited

Loading