support FP8 or FP4 in UAI_NV_TRT

[animeojisan]

Thank you for the great app!
Are there any plans to support FP8 or FP4 in UAI_NV_TRT?

[hooke007]

It's just the wrapper of vs-mlrt.
If vs-mlrt would support fp4/int8, it could follow up easily.

[animeojisan]

Thanks for the reply!

[animeojisan]

May I ask a question here since it is similar? Why only FP32 in UAI_DML, is there any possibility of FP16 support? Is this also a problem on the vs-mlrt side?

[hooke007]

Not entirely.
fp16 model can be used now in mlrt.(I have to update here later).

I'm not sure how much it would help. Because i already used the built-in convertion (to fp16 model) in my wrapper.

But fp16 I/O is not supported by mlrt.

[animeojisan]

Am I correct in understanding that UAI_DML is already operating at a processing speed equivalent to fp16? Is there a possibility that DirectML processing speed will improve in the future?

[hooke007]

I believe fp16 I/O would accelerate a lot.

[animeojisan]

I am very thankful!