Conversation
proposals/0015-variant-type.md
Outdated
|
|
||
| In addition to a new canonical encoding, we'll need a few more pieces to make variant columns useful: | ||
|
|
||
| 1. A set of new expressions, which extract children of variant arrays with a combination of path (similarly to `GetExpr`) and a dtype. |
There was a problem hiding this comment.
Maybe also worth mentioning expressions that convert to/from other variant-like data, e.g. JSON as a DType::Utf8 can be parsed into a DType::Variant.
I wonder if our JSON extension type has storage type DType::UTf8? or storage type DType::Variant...?
There was a problem hiding this comment.
In my mind JSON type is “string verified as JSON”, like a PG column.
So far my impression is that there’s no consistent naming, and any choice we make will end up conflicting with something
There was a problem hiding this comment.
So we would just also implement the variant expressions over a JSON extension type array?
proposals/0015-variant-type.md
Outdated
|
|
||
| Different systems have different variations of this idea, but at its core its a type that can hold nested data with either a flexible or no schema. In addition to this "catch all" column, most system include the concept of "shredding", extracting a key with a specific type out of this column, and storing it in a dense way. This design can make commonly access subfields perform like first-class columns, while keeping the overall schema flexible. | ||
|
|
||
| I propose a new dtype - `DType::Variant`. The variant type is always nullable, and its canonical encoding is just an array with a single child array, which is encoded in some specialized variant type. |
There was a problem hiding this comment.
how should we do execute_arrow for these, using the Parquet Variant? Or union?
There was a problem hiding this comment.
added some thoughts on this point, it might require a pretty big change some changes/extending our arrow exporting logic
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
AdamGS
force-pushed
the
adamg/variant
branch
from
c5d2aff to
e444968
Compare
|
I think this all makes sense, but I think it should explicitly call out what changes you want to make to DType enum / Scalar enum / Canonical enum / etc. |
connortsui20
left a comment
connortsui20
left a comment
There was a problem hiding this comment.
For something as complicated as the design space of variant, I think it would be worth putting together a few diagrams (could literally just be some text trees) that show the different kinds of variant and shredding designs as well as some concrete examples.
I'm happy to add this myself as well!
| Vortex currently requires a strict schema, but real world data is often only semi-structured and deeply hierarchical. Logs, traces and user-generated data often take the form of many sparse fields. | ||
|
|
||
| This proposal introduces a new dtype - `Variant`, which can capture data with row-level schema, while storing it in a columnar form that can compress well while being available for efficient analysis in a columnar format. | ||
|
|
There was a problem hiding this comment.
I know this is being pedantic but I think it would be good to have a motivation section here. Do we care about supporting all possible variant types, or just JSON, for example? And it might be good to say that we want this because other formats support this and people like that
|
|
||
| Different systems have different variations of this idea, but at its core its a type that can hold nested data with either a flexible or no schema. | ||
|
|
||
| Variant types are usually stored in two ways - values that aren't accessed often in some system-specific binary encoding, and some number of "shredded" columns, where a specific key is extracted from the variant and stored in a dense format with a specific type, allowing for much more performant access. This design can make commonly accessed subfields perform like first-class columns, while keeping the overall schema flexible. Shredding policies differ by system, and can be pre-determined or inferred from the data itself or from usage patterns. |
There was a problem hiding this comment.
I think "accessed" is the wrong word here? You could motivate this by giving an example of json data where a majority has the same type (string) but sometimes there happens to be a different type (int)
4 participants
Still WIP, but starting to extract my local notes into a publicly share-able form