Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

opt: downloader #1019

Merged
merged 9 commits into from
Sep 15, 2024
Merged

opt: downloader #1019

merged 9 commits into from
Sep 15, 2024

Conversation

Eric-Joker
Copy link
Contributor

@Eric-Joker Eric-Joker commented Aug 5, 2024

#1003 反复重做,重开个pr。

  • 使用平滑窗口平均 平滑速度计算、剩余时间估计。
  • 在继续任务前,确保之前的任务线程均结束,避免文件被占用。
  • 任务发生任何崩溃均将任务添加回_installItemQueue。
  • 优化下载限速器

与当前限速器的区别:

  • 精细化令牌补充间隔,避免流量全堆积在单位时间的开始、减少实际即时速度的误差。
  • 不限速时不使用限流器,以减少性能消耗。
  • 使用opt: downloader #1019 (comment) 的方式实现精确控制速度,数据流恢复为16KB避免I/O过于频繁。
  • 将限速设置的NumberBox上限设为2097151(int.MaxValue / 1024向下取整)

close #996 , close #997

@Eric-Joker Eric-Joker changed the title refactor: download limiter opt: downloader Aug 5, 2024
@Eric-Joker
Copy link
Contributor Author

Eric-Joker commented Aug 5, 2024

@Scighost 打扰,但这个有时间看看可以merge了?(

@Lightczx
Copy link
Contributor

Lightczx commented Aug 7, 2024

可以试试这个

internal static class TokenBucketRateLimiterExtension
{
    public static int Acquire(this TokenBucketRateLimiter rateLimiter, int permits, out TimeSpan retryAfter)
    {
        int count = Math.Min(permits, (int)Volatile.Read(ref PrivateGetTokenCount(rateLimiter)));
        return rateLimiter.AttemptAcquire(count).TryGetMetadata(MetadataName.RetryAfter, out retryAfter) ? 0 : count;
    }

    // private double _tokenCount;
    [UnsafeAccessor(UnsafeAccessorKind.Field, Name = "_tokenCount")]
    private static extern ref double PrivateGetTokenCount(TokenBucketRateLimiter rateLimiter);
}

@Eric-Joker Eric-Joker force-pushed the better-limiter branch 3 times, most recently from e2c3552 to af1414f Compare August 7, 2024 08:52
@Eric-Joker
Copy link
Contributor Author

可以试试这个

谢谢,好用,用了(

@Lightczx
Copy link
Contributor

Lightczx commented Aug 7, 2024

@Eric-Joker 我最终实际体验了一下还是改成了下面的样子
image
带count的Replenish方便返还 Read Span buffer 实际上没填充的字节数,这样会更加的准确

@Lightczx
Copy link
Contributor

Lightczx commented Aug 7, 2024

Update:
image

@Eric-Joker
Copy link
Contributor Author

Eric-Joker commented Aug 8, 2024

返还 Read Span buffer 实际上没填充的字节数

emm?没太明白,现在的实现就是根据ReadAsync返回的值来租用令牌的

@Lightczx
Copy link
Contributor

Lightczx commented Aug 8, 2024

返还 Read Span buffer 实际上没填充的字节数

emm?没太明白,现在的实现就是根据ReadAsync返回的值来租用令牌的

我的实现是尝试获取buffer长度的令牌,然后再读的,由于读取的长度可能会比buffer小所以我加了返还
如果你是在write时限速就不一样了

@Lightczx
Copy link
Contributor

其实我突然发现,都已经lock了,理论上是不需要Volatile.Read的

@Scighost Scighost merged commit 93c028d into Scighost:main Sep 15, 2024
4 checks passed
@Scighost
Copy link
Owner

谢谢你的贡献

@Eric-Joker Eric-Joker deleted the better-limiter branch September 16, 2024 05:20
Eric-Joker added a commit to Eric-Joker/Starward that referenced this pull request Sep 27, 2024
* refactor: download limiter

* opt: Decrease the time interval for replenishing tokens

* fix: crash by NumberBox possibly being -21e

* Improve more precise speed limits

Co-authored-by: DismissedLight <[email protected]>

* opt: reduce downloader writes

* refactor: simplify rate limiting

---------

Co-authored-by: DismissedLight <[email protected]>
Co-authored-by: Scighost <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature] 提高下载出错的包容性 [Bug] Limit download speed not working?
3 participants