Vector Data Type Support

[sindhuthirugnanam]

Any plan to extend the support of Vector Data Type?
https://docs.snowflake.com/en/sql-reference/data-types-vector

[littleK0i]

I am planning to add it soon. Hopefully this week.

But at the same time.. I have my doubts. This data type seems to be poorly implemented and has too many limitations. For example, it seems to be crashing non-Python clients if present in result set. It may cause some software to fail due to prescense of tables with VECTOR type.

Do you have a legitimate use case for using it? Is it for TABLE object type only or for something else?

[littleK0i]

Alright. Implementing this type for TABLE is relatively easy. The only "gotcha" is string in first argument instead of number.

However, implementing it for FUNCTION and PROCEDURE is quite difficult. All vector sub-types seems to be considered as different types for function polymophism.

For example, these functions are different:

TEST_FN(VECTOR(INT, 16), VECTOR(INT, 4096), VECTOR(FLOAT, 16), VECTOR(FLOAT, 4096)) RETURN TABLE (VEC_INT_16 VECTOR(INT, 16), VEC_INT_4096 VECTOR(INT, 4096), VEC_FLOAT_16 VECTOR(FLOAT, 16), VEC_FLOAT_4096 VECTOR(FLOAT, 4096))

TEST_FN(VECTOR(INT, 16), VECTOR(INT, 4096), VECTOR(FLOAT, 32), VECTOR(FLOAT, 4096)) RETURN TABLE (VEC_INT_16 VECTOR(INT, 16), VEC_INT_4096 VECTOR(INT, 4096), VEC_FLOAT_32 VECTOR(FLOAT, 32), VEC_FLOAT_4096 VECTOR(FLOAT, 4096))

Unlike other types, we need to include full type definition while using VECTOR.

So.. basically, Snowflake does not seems to be bothered with correct and consistent implementation of new data types. I am not sure if we as users should support this.

Let me know what you think. Is there any benefit to actually define and maintain VECTOR type instead of casting ARRAY to VECTOR right before sending it to relevant functions?

[littleK0i]

@sindhuthirugnanam , VECTOR type should be available for tables starting with 0.32.0.

Functions and procedures are not supported for now, since Snowflake decided to implement overloading for this type differently, and it would require introduction of proper parsing to deal with that. Probably not worth it for now.

I've heard Snowflake may implement generic VECTOR type for UDFs in future, which should solve the problem automatically. We'll see.