did a lot of thinking/back end work recently about something that depot never had that I knew I wanted: groupings/sortings/formula fields. currently, sorting was implemented as a client side thing. this meant that sort/grouping/etc. cleared on reload. however, I envisoned new depot having something more akin to server-synced "views" (airtable-syle) that would act as ways for people to be able to look at just certain slices of data consistently, and share those views with others.
this meant though that these views would need to be persisten in some way.
the obvious candidate here was the primary .dpo file that Depot loads, but I also knew I didn't necessarily want to muck up the context of that file. In my mind, a .dpo file maintains a promise that it can be directly deserialized in your game/engine, and contains only useful information for that context. you will be able to customize other exports from depot (and maybe also have them server-synced), but I still want the primary .dpo file to largely be all-signal.
this means that I needed a different construct for the view-related settings. this also rubs a bit agains the thing I brough up earlier, which is trying to lean into what new depot can do with its new architecture that wasn't so easily possible in the vscode days. sidecar files weren't really possible (iirc) when I was doing og depot (though now seem more possible), so now given that I can do more or less whatever I want opens up new possibilites.
to that end, I came up with the idea of a file called .depot that lives alongside your actual data.dpo file. the idea of the .depot file is that it holds all depot-specific state, but doesn't hold any actual data. you could delete the .depot file and your data wouldn't be affected at all. the .depot file is synced as part of the same mechanism that syncs the data file itself though which means that as you add/remove fields and such that config file knows how to manage all of that with regards to its own values.
but what are those values?
this is one of the things I've been REALLY interested in finally diving into with new depot. it starts unrelated to sorting, but we'll get there.
in old depot, I knew it would be really nice to be able to filter the data. ie, give me all things like X or where X.attack > 0. old depot couldn't do this, so I would actually write little scripts on the side to analyze the data with either jq or, one of my favorites, Visidata. this worked (and still works!), but I knew that having that functionality built into depot itself would not only save me time, but also empower others to do their own little analyses against the data.
the other part of this though was that I knew that Depot isn't meant to be a generic data viewer. It is specifically looking at rows in a database, so whatever sort of filtering/querying I allowed would need to be able to sort of lock to the idea of being able to return rows in some way.
to that end, I looked at a LOT of json "languages":
jsonpath / jless / jsonata / CEL / jmespath / jq / jnv / miller /yq
to talk about the plusses and minuses of all of these would take too long for the scope of this (increasingly long) blog post, but to just shape my own thinking I came in here with a few high level goals:
- expressions/language should be as terse as possible and human-authorable/readable
- language should be primarily a query language, aka largely centered around being able to return true/false based on data conditionals
- language shouldn't be too "weird". this is largely a taste thing but ideally the user should be able to intuitively construct queries without needing to learn too much syntax
now, as I said above, when I started this effort I was thinking about filtering, but then had a bit of an insight that I could actually extend whatever I chose for filtering into a general purpose solution for basically all of depot's filtering/querying/sorting/grouping/expression needs. all of a sudden, things like forumla fields made a lot more sense (it's just evaluating the expression!)
writing to the .depot file itself then became trivial, it just serializes the expression!
i'll abbeviate my own thought process here, but of all the options, I was really leaning on JSONPath for a while, as it is purely a query language, which was, ostensibly, what I wanted. however with the insight that I actually maybe wanted something slightly more capable such that I could use it also for formula fields I (JSONPath doesn't have arithmetic) , I started to look into JMESpath more. JMESpath doesn't have arithmetic specifically, but it does have macros like sum(), min(), etc.
I was about to pull the trigger here, until I looked more in CEL. If someone wanted to add two values in JMESPath, they would need to do something like sum(@.field 1, @.field2). This started to rub my "syntax not for normal people" senses — if someone wants to add two fields, they should be able to just add those two fields like field1 + field2. and so I quickly looked at CEL, which was mentioned as an option before. and lo and behold, it can do exactly that! CEL is a lot more powerful than just that though, but importantly doesn't do any sort of transformative writing back to the source data. So CEL gives you "simple" data expressions that just make sense for how you would expect to be able to talk about json data, but then also you get a suite of other options for forumulas (and the eventual evaluator context for ad-hoc analysis).
going back to sorting/grouping then, this also means the group can easily be expressed by the column name (well, guid) itself at a baseline, but also could easily make custom groupings that are the results of expressions (think something like "high-powered enemies" being a top level group that is the result of data.damage > 10 && data.isEnemy). but also you could instead just make that a filter and save it to a view. Or make it a forumla column for IsHighPoweredEnemy and return true/false, then sort/group on that.
what's nice is that this shared language means that all these paths share all the same underlying logic and serialization structure, which makes things easy to maintain and extend, but also means that you only need to learn one scripting paradigm for all of Depot!
As a little tease then, here's an image of the groupings (and sub-groupings) in action!
And here's the backing data in the .depot file:
pretty nice! views are saved per-parent sheet, and then given a unique guid for each view itself. that view then can express itself in terms of sortings, groupings, and (soon!) filters. the expr for everything is resolved as a CEL expression. I use guids here to be stable, as column names themselves can change.
look forward to getting to play with all of this soon(-ish)! thanks for reading!