feat: collect shared logic for blob storage and minio

[chuang8511]

Because

• blob storage will be used in different service

This commit

• collect get and upload data function

Note

• For minio pkg, the current pkg is not fit with the current design of blob storage. We will refactor that part with run logging.
• We will need to add the test code

[linear]

INS-6780 Integrate: Blob Storage to VDP

[jvallesm]

A little comment on this but not a blocker. Did you extract this code from any service (because another one needs to use it) or is it something several services will use in the future?

[jvallesm]

nit: I'd add the body (even the status code) as a separate log. You want to keep errors that come from the same path as consistent as possible. That allows you, e.g., to group errors to see how frequent an incident is.

[chuang8511]

I am not sure if I got it correctly.
Do you mean like this?

[jvallesm]

Sort of, I'd align with the logger package we use in other repos:

	err := fmt.Errorf("failed to upload file")
	logger.Error("Failed to upload file to MinIO",
		zap.Binary("body", body),
		zap.Int("status", resp.StatusCode),
		zap.Error(err),
	)
	return err

[chuang8511]

updated commit.
update pipeline-backend.

[chuang8511]

Did you extract this code from any service (because another one needs to use it) or is it something several services will use in the future?

I think UploadFile will be used in model-backend & pipeline-backend. So, I extracted it.
Is this what you are asking?

[jvallesm]

Did you extract this code from any service (because another one needs to use it) or is it something several services will use in the future?

I think UploadFile will be used in model-backend & pipeline-backend. So, I extracted it. Is this what you are asking?

Yeah, my question was if this had been copied from somewhere (e.g. api-gateway plugin) or if it was new code. Basically I wanted to know which will be the current and future clients for this code. Thanks for the answer, it's clear 👍