allow arrow::ListType, arrow::StringType until permanent solution in place

[danielmawhirter]

Possible temporary solution for problematic types

[witchel]

Thank you. That is what you meant by add them as cases. I didn't realize it would be so easy. But will it cause them to be dropped or passed along?

[danielmawhirter]

Thank you. That is what you meant by add them as cases. I didn't realize it would be so easy. But will it cause them to be dropped or passed along?

It shouldn't change the "drop or pass along" behavior, which is still dependent on shared-memory vs distributed. What it will change is, once it's passed along, it won't fail due to this particular reason

[danielmawhirter]

Quick FYI @aneeshdurg @roshandathathri This will introduce List and String back to our visitors, that way code running on shared memory can still use the list properties that we can't distribute for now. A more permanent solution is wip in #250 to get those properties matching the types we expect, after which this can be rolled back.

[aneeshdurg]

Lgtm

[witchel]

I would love to just port this fix into my local PR. But when I do, I get these build errors. Am I overlooking something?

In file included from /home/witchel/katana/libquery/src/Table.cpp:16:0: /home/witchel/katana/libquery/include/katana/query/ArrowArrays.h: In instantiation of ‘T katana::ArrowArrayGetValue(const ArrayType&, int64_t) [with T = std::__cxx11::basic_string<char>; ArrayType = arrow::StringArray; int64_t = long int]’: /home/witchel/katana/libquery/src/Table.cpp:36:47: required from ‘{anonymous}::GetJSONValue::ResultType {anonymous}::GetJSONValue::Call(const ArrayType&) [with CType = satana::Er11::basic_string<char>; ArrayType = arrow::StringArray; {anonymous}::GetJSONValue::ResultType = boost::outcome_v2::basic_result<nlohmann::basic_json<>, k[10/6504]rorInfo, katana::internal::abort_policy>]’ /home/witchel/katana/libquery/include/katana/query/ArrowBreakoutVisitor.h:212:58: required from ‘arrow::enable_if_string<ArrowType, boost::outcome_v2::basic_result<typen::Call(t:decay<_Arg>::type::ReturnType, katana::ErrorInfo, katana::internal::abort_policy> > katana::internal::VisitorBreakoutAdapter<VisitorBaseType, VisitorType>[7/6504]ous}::GeVisitorBaseType::ParamType<ArrowType>) [with ArrowType = arrow::StringType; VisitorBaseType = katana::internal::ArrayVisitorBaseType; VisitorType = {anonym[6/6504]bort_polue&; arrow::enable_if_string<ArrowType, boost::outcome_v2::basic_result<typename std::decay<_Arg>::type::ReturnType, katana::ErrorInfo, katana::internal::a[5/6504] = const boost::outcome_v2::basic_result<nlohmann::basic_json<>, katana::ErrorInfo, katana::internal::abort_policy>; typename VisitorBaseType::ParamType<ArrowType>[4/6504] arrow::StringArray&]’ /home/witchel/katana/external/katana/libsupport/include/katana/ArrowVisitor.h:154:5: required from ‘katana::Result<typename std::decay<_Arg>::type::ReturnType> katana::ia::inter:VisitArrowInternal(typename VisitorBaseType::ParamBase, VisitorType&&) [with VisitorBaseType = katana::internal::ArrayVisitorBaseType; VisitorType = katan[1/6504]st::outcitorBreakoutAdapter<katana::internal::ArrayVisitorBaseType, {anonymous}::GetJSONValue&>&; katana::Result<typename std::decay<_Arg>::type::ReturnType> = boo[0/6504]ome_v2::basic_result<nlohmann::basic_json<>, katana::ErrorInfo, katana::internal::abort_policy>; typename std::decay<_Arg>::type::ReturnType = nlohmann::basic_json<>; typename VisitorBaseType::ParamBase = const arrow::Array&]’ /home/witchel/katana/libquery/include/katana/query/ArrowBreakoutVisitor.h:294:70: required from ‘auto katana::VisitArrowBreakout(const arrow::Array&, VisitorType&&) [with VisitorType = {anonymous}::GetJSONValue&]’ /home/witchel/katana/libquery/include/katana/query/ArrowBreakoutVisitor.h:304:28: required from ‘auto katana::VisitArrowBreakout(const std::shared_ptr<arrow::Array>&, VisitorType&&) [with VisitorType = {anonymous}::GetJSONValue&]’ /home/witchel/katana/libquery/src/Table.cpp:179:59: required from here /home/witchel/katana/libquery/include/katana/query/ArrowArrays.h:17:16: error: ‘const class arrow::StringArray’ has no member named ‘Value’; did you mean ‘GetValue’? return array.Value(index); ~~~~~~^~~~~ GetValue

[witchel]

It starts here.

In file included from /home/witchel/katana/libquery/src/OrderByOperator.cpp:9:0: /home/witchel/katana/libquery/include/katana/query/ArrowArrays.h: In instantiation of ‘T katana::ArrowArrayGetValue(const ArrayType&, int64_t) [with T = std::__cxx11::basic_string<char>; ArrayType = arrow::StringArray; int64_t = long int]’: /home/witchel/katana/libquery/src/OrderByOperator.cpp:40:49: required from ‘{anonymous}::SortIndicesByColumn::Call(const ArrayType&)::<lambda(size_t, size_t)> [with CType = std::__cxx11::basic_string<char>; ArrayType = arrow::StringArray; size_t = long unsigned int]’ /home/witchel/katana/libquery/src/OrderByOperator.cpp:36:14: required from ‘struct {anonymous}::SortIndicesByColumn::Call(const ArrayType&) [with CType = std::__cxx11::b::internalg<char>; ArrayType = arrow::StringArray; {anonymous}::SortIndicesByColumn::ResultType = boost::outcome_v2::basic_result<void, katana::ErrorInfo, katana[313/6535]::abort_policy>]::<lambda(size_t, size_t)>’ /home/witchel/katana/libquery/src/OrderByOperator.cpp:33:12: required from ‘{anonymous}::SortIndicesByColumn::ResultType {anonymous}::SortIndicesByColumn::Call(const Arrresult<voiwith CType = std::__cxx11::basic_string<char>; ArrayType = arrow::StringArray; {anonymous}::SortIndicesByColumn::ResultType = boost::outcome_v2::basic_[310/6535]d, katana::ErrorInfo, katana::internal::abort_policy>]’

[danielmawhirter]

It starts here.

In file included from /home/witchel/katana/libquery/src/OrderByOperator.cpp:9:0: /home/witchel/katana/libquery/include/katana/query/ArrowArrays.h: In instantiation of ‘T katana::ArrowArrayGetValue(const ArrayType&, int64_t) [with T = std::__cxx11::basic_string<char>; ArrayType = arrow::StringArray; int64_t = long int]’: /home/witchel/katana/libquery/src/OrderByOperator.cpp:40:49: required from ‘{anonymous}::SortIndicesByColumn::Call(const ArrayType&)::<lambda(size_t, size_t)> [with CType = std::__cxx11::basic_string<char>; ArrayType = arrow::StringArray; size_t = long unsigned int]’ /home/witchel/katana/libquery/src/OrderByOperator.cpp:36:14: required from ‘struct {anonymous}::SortIndicesByColumn::Call(const ArrayType&) [with CType = std::__cxx11::b::internalg<char>; ArrayType = arrow::StringArray; {anonymous}::SortIndicesByColumn::ResultType = boost::outcome_v2::basic_result<void, katana::ErrorInfo, katana[313/6535]::abort_policy>]::<lambda(size_t, size_t)>’ /home/witchel/katana/libquery/src/OrderByOperator.cpp:33:12: required from ‘{anonymous}::SortIndicesByColumn::ResultType {anonymous}::SortIndicesByColumn::Call(const Arrresult<voiwith CType = std::__cxx11::basic_string<char>; ArrayType = arrow::StringArray; {anonymous}::SortIndicesByColumn::ResultType = boost::outcome_v2::basic_[310/6535]d, katana::ErrorInfo, katana::internal::abort_policy>]’

This should be resolvable by duplicating this function https://github.com/KatanaGraph/katana-enterprise/blob/ac28ce27d7cfdea537d2583362e295289c724037/libquery/include/katana/query/ArrowArrays.h#L20-L27 with a change from LargeStringArray to StringArray, and a deprecation notice

[witchel]

It starts here.
In file included from /home/witchel/katana/libquery/src/OrderByOperator.cpp:9:0: /home/witchel/katana/libquery/include/katana/query/ArrowArrays.h: In instantiation of ‘T katana::ArrowArrayGetValue(const ArrayType&, int64_t) [with T = std::__cxx11::basic_string<char>; ArrayType = arrow::StringArray; int64_t = long int]’: /home/witchel/katana/libquery/src/OrderByOperator.cpp:40:49: required from ‘{anonymous}::SortIndicesByColumn::Call(const ArrayType&)::<lambda(size_t, size_t)> [with CType = std::__cxx11::basic_string<char>; ArrayType = arrow::StringArray; size_t = long unsigned int]’ /home/witchel/katana/libquery/src/OrderByOperator.cpp:36:14: required from ‘struct {anonymous}::SortIndicesByColumn::Call(const ArrayType&) [with CType = std::__cxx11::b::internalg<char>; ArrayType = arrow::StringArray; {anonymous}::SortIndicesByColumn::ResultType = boost::outcome_v2::basic_result<void, katana::ErrorInfo, katana[313/6535]::abort_policy>]::<lambda(size_t, size_t)>’ /home/witchel/katana/libquery/src/OrderByOperator.cpp:33:12: required from ‘{anonymous}::SortIndicesByColumn::ResultType {anonymous}::SortIndicesByColumn::Call(const Arrresult<voiwith CType = std::__cxx11::basic_string<char>; ArrayType = arrow::StringArray; {anonymous}::SortIndicesByColumn::ResultType = boost::outcome_v2::basic_[310/6535]d, katana::ErrorInfo, katana::internal::abort_policy>]’

This should be resolvable by duplicating this function https://github.com/KatanaGraph/katana-enterprise/blob/ac28ce27d7cfdea537d2583362e295289c724037/libquery/include/katana/query/ArrowArrays.h#L20-L27 with a change from LargeStringArray to StringArray, and a deprecation notice

Boy, ain't it easy when you know how. Yep, that does it and of course that change isn't in this PR because it is in enterprise. While I have picked up these changes on my branch, I'm happy for you to merge them. Though I realize an open/enterprise merge is a bit of a pain.

[witchel]

Hmm, I'm not sure this is such a great workaround because when I enable it, I get the following error from ArrowVisitor.cpp:79.

KATANA_LOG_FATAL("List visitors not yet supported");

[danielmawhirter]

Hmm, I'm not sure this is such a great workaround because when I enable it, I get the following error from ArrowVisitor.cpp:79.

KATANA_LOG_FATAL("List visitors not yet supported");

Ah yes, this would trigger that issue. This one should be worth fixing instead of working around, shouldn't take too long

[danielmawhirter]

@witchel should be fixed now, let me know if there's any more whack-a-mole to play!

[witchel]

My preferred path at this point would be for you to commit the PR, is that possible?

[danielmawhirter]

Yes, my intent is to merge it simultaneous with #246, https://github.com/KatanaGraph/katana-enterprise/pull/1148, and https://github.com/KatanaGraph/query-verify-neo4j/pull/31, hopefully tonight