feat(schema-compiler): add meta property for cube definition

[mharrisb1]

Issue Reference this PR resolves

#1539 and #6792

Description of Changes Made

Adds support for meta property at the cube and view level.

Examples (added to reference docs):

cube(`orders`, {
  sql_table: `orders`,
  title: `Product Orders`,
  meta: {
    any: "value",
  }
});

cubes:
  - name: orders
    sql_table: orders
    title: Product Orders
    meta:
      any: value

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

8 Ignored Deployments

Name	Status	Preview	Comments	Updated (UTC)
examples-angular-dashboard	⬜️ Ignored (Inspect)	Visit Preview		Dec 12, 2023 7:36pm
examples-react-d3	⬜️ Ignored (Inspect)	Visit Preview		Dec 12, 2023 7:36pm
examples-react-dashboard	⬜️ Ignored (Inspect)	Visit Preview		Dec 12, 2023 7:36pm
examples-react-data-table	⬜️ Ignored (Inspect)	Visit Preview		Dec 12, 2023 7:36pm
examples-react-highcharts	⬜️ Ignored (Inspect)	Visit Preview		Dec 12, 2023 7:36pm
examples-react-material-ui	⬜️ Ignored (Inspect)	Visit Preview		Dec 12, 2023 7:36pm
examples-react-pivot-table	⬜️ Ignored (Inspect)	Visit Preview		Dec 12, 2023 7:36pm
examples-vue-query-builder	⬜️ Ignored (Inspect)	Visit Preview		Dec 12, 2023 7:36pm

[mharrisb1]

I allowed for arbitrary values in meta including for null types. Adding that to the logic in transpileYaml broke the expectation that null values will cause an error so I added an additional predicate that will allow for null values but only within the meta field.

I really didn't want to treat meta as a special field so I'm open to suggestions on what to do there. I can fully remove support for null (which will need to be added as a clause in the docs) but I could see scenarios where that would be useful.

[paveltiunov]

@mharrisb1 Overall, it looks good to me! Could you please elaborate on null use case for meta?

[mharrisb1]

@paveltiunov truthfully I don't have any in mind (at least not for my team), but I'm sure other teams might have a need.

Maybe something like:

cubes:
  - name: orders
    sql: `select * from public.orders`
    meta:
      icon_png: null

  - name: accounts
    sql: `select * from public.accounts`
    meta:
      icon_png: gs://path/to/some/blob.png

A user can just use an empty string instead of null but they'd have to run into the unexpected input error first before they would know to try that, and that just feels like throwing darts at the wall and seeing what sticks.

Maybe for now, instead of me anticipating an imaginary use case, we remove support for null values in meta and let someone post an issue if they have a concrete use case.

[mharrisb1]

@paveltiunov I went ahead and removed support for nulls

[igorlukanin]

I think there's also a use case when meta is generated programmatically or brought from an external system that has flexible rules for allowed values. For example, I can see dbt models' meta directly injected as cubes' meta as part of dbt integration; my understanding is that it can be anything in there, including nulls.

[igorlukanin]

@paveltiunov @mharrisb1 Do we need more tests here, e.g., for views + to check that meta values are exposed via /load and /meta endpoints of the REST API?

[mharrisb1]

I think there's also a use case when meta is generated programmatically or brought from an external system that has flexible rules for allowed values. For example, I can see dbt models' meta directly injected as cubes' meta as part of dbt integration; my understanding is that it can be anything in there, including nulls.

Good call. Developer flexibility is probably the way to go.

@paveltiunov I haven't dug in too deep on the compiler code for this PR, but what would happen if null is handled in transpileYaml?

For example, if we had this YAML config:

cubes:
  - name: orders
    sql: `select * from public.orders`
    title: null

Would an error be thrown for title having a null value after transpilation with this JS code?

cube(`orders`, {
  sql: `select * from public.orders`,
  title: null,
});

If so, then could we just transpile the YAML code as is and let the error come from JS?

[mharrisb1]

Do we need more tests here, e.g., for views + to check that meta values are exposed via /load and /meta endpoints of the REST API?

@igorlukanin another good call. I was manually testing this by building the dev Docker image and running locally but actual tests would be better.

Can you point me to the integration tests where this would be added? I'm still trying to get familiarized with the codebase

[benhickson]

i think that meta should be any valid JSON-serializable value, and nulls are meaningful values.

[paveltiunov]

@mharrisb1 Ah. Gotcha. I thought there was meta: null use case. But I agree we need to support nested nulls inside meta for sure. So ideally to have it back.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (0214472) 48.02% compared to head (5a96fd9) 48.03%.
Report is 1 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #7327   +/-   ##
=======================================
  Coverage   48.02%   48.03%           
=======================================
  Files         154      154           
  Lines       20893    20897    +4     
  Branches     5378     5380    +2     
=======================================
+ Hits        10033    10037    +4     
  Misses      10120    10120           
  Partials      740      740           

Flag	Coverage Δ
cube-backend	48.03% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mharrisb1]

I reverted the removal of null support in meta 👍

[jwilson232]

this would be super useful for my team!

[hannosgit]

@paveltiunov Do you have any timeline on when this could be integrated into the main branch?

[paveltiunov]

@mharrisb1 Sorry for delay. Could you please rebase one more time?

[mharrisb1]

Saw that the commit for docs was deleted in previous large merge. Added back now.