Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: 阮老师小分队 - 基于异步HTTP实现p2p分布式协议加速S3 Object存取实现 #2

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 58 additions & 0 deletions ruan_lao_shi_xiao_fen_dui/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# 基于异步HTTP实现p2p分布式协议加速S3 Object存取实现

### 其他文档

```bash
.
├── README.md
├── baseline # 基准测试,直接读取
├── deploy # 部署Minio S3存储
├── docs # API文档
├── genfile # 生成测试服务
├── p2p-with-tracker # 基于tracker的p2p实现
└── p2p-with-tracker-client # 基于tracker的p2p实现的测试客户端
```

### **其他文档在各自目录下**

##### 大致框架,使用Tracker存储部分节点信息和seed信息,Node主要负责下载和上传

![](https://github.com/datenlord/Hackathon-2023/assets/47499836/07cd5d99-f1a3-49cf-b4bf-e4caf6e225cc)

![](https://github.com/datenlord/Hackathon-2023/assets/47499836/0686f79e-9bb1-4f0a-a2d3-8b588ac79fba)

##### 流程拆分

1. 初始化构建阶段

初步思路:

N0将文件S(大小为S)切分n片,然后传输到N1-Nn中,W为带宽,耗费时间:
```math
T = \frac{S}{n * W}
```
第一步传输完成之后,P2P成员之间会继续交换,接下来的T时间,N1将Fs1传递给其他N-1个节点,以此类推,等于是再次花费一遍传输的时间,整个文件就同步了,就是上面的2T。(所有节点都在相互传输)

```math
T_{total}=2*\frac{S}{n*W}
```
这里就节省很多时间了!!!

==这里也可以进行优化,就是在传输的过程中,第一步N0上获取数据,其他p2p都在等待这个结果,然后第二步才开始进行p2p传输,这两个步骤也可以进行并行化处理==

> Tips: 减小分片长度进行初始化

就是尽量减小分片的长度,然后下载一点点之后就可以开始p2p传输了,粒度更小的话约等于同时开始发送,这样子启动速度就会变快。

就是针对上面的分片,再次切分成为m段,然后变成:
T0: m+1 2m+1 ...
T1: m+2 2m+2 ...

交换上一个由Tx周由N0发送的文件片段。
```math
t(m) \rarr \frac{1}{n}*\frac{S}{W}(m \rarr +\infty)
```
我们将总文件分割的片断越小,下载加速比就越大,理论极限为N倍速下载,N为P2P网络成员数量减1。

所有的N+1个节点都有了完整的文件F,现在假设新加入一个节点,它也需要这个文件,就不用重复刚才的过程了。
现在节点总数为N+2,文件可以分为N+1份片断,由N0到Nn分别传输给Nn+1。**同时使用心跳对节点进行维护**还要处理节点退出以及新节点加入等动态问题。
1 change: 1 addition & 0 deletions ruan_lao_shi_xiao_fen_dui/baseline/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
/target
Loading