Hello all, my name is tommy and I love clojure. You can find me on github or email me. This article was created with clerk and published on clerk garden.
I am working on a gameserver plugin for my favorite fighting game MGE, and want to be able able to run tournaments automatically on a weekly basis, similar to how many melee groups run low-stakes weekly tournaments.
Thankfully, the company challonge (owned now by logitech) hosts software that lets you create and run your own tournaments. It displays the bracket, handles tournament state, player signups, multistage tournaments (swiss/round-robin/group-stage), winners/losers brackets, and much more. It also lets you embed the (realtime updating) bracket as an iframe.
It also has an API. So lets get started. Below is an example tournament that I've created, to decide which color is the best color. A couple matches have already been played (apparently crimson is favorable over rust by 15 points...)
Below is the parsed json we get back from the challonge api after asking for all the matches of the pictured tournament. Challonge shapes its responses according to a standard called json-api. Json-api describes itself as something to stop bikeshedding questions about how json responses should be formatted.
Its uniformity somewhat enables the rest of what happens in this post.
I asked for matches, I got very complicated piece of data. The :data
key has a vector of 10 resources, each with a type, id, some attributes, and some relationships.
For a each match in the tournament, its relationships are the two opponents facing off in that match. However, the participant nested in the match (get-in matches [:data 0 :relationships :player1])
has very little information. If you want to see any attributes
of a player, you must look in the toplevel :included
vector and find the right one.
The :included
vector contains all 12 participant resources (colors) in this tournament.
All resources have a uniform shape.
In each participant, there is a :misc
key in the attributes map. This is a bit of data specific to my domain that challonge allows me to attach to the resource. In my case, it the id of each player in my gameserver.
At first glance this feels structure feels heavy, but it's well thought out, and handles many edge cases. I personally would prefer to expose my data using something like pathom3, but I prefer the standardized json-api approach over having arbitrary shapes that are different for every company.
Now that we have all this data, we have to answer questions about it to run the gameserver logic.
To run my tournament, I need to know which matches are active (not completed, but have two defined participants), so I can enable the correct arenas ingame. Each match has a "state" attribute that defines this, so we can just filter on that. Easy enough.
This filters my 10 matches down to four, so now I know to allocate four mach arenas in the game.
I also need to know which players to allow in which arenas, so I need the corresponding uuids stored in the :misc
attribute.
Lets start by getting the player embedded in each match's :relationship
map.
Nice! But we're still not done. We have the player ids, but its not the entire resource. The piece of data we want is the included resources. Lets try again, but build a join from challonge id to gameserver uuid first...
Great! now we have a list of uuids I can send to my gameserver. Slinging maps around like this is one of clojure's strenghts, so this felt fairly natural to write.
This code works, but what about other questions we could ask of our api response? We went from match to gameserver uuid. What if we had a gameserver uuid and wanted to know which matches they are a part of? (the player runs a chat command to display their upcoming matches)
We would have to write more or less the same amount of code (~30 lines), this time building the join in the other direction.
Every new question we ask of the data requires a new set of functions to traverse the resource graph from question to answer, with very little code reuse, producing code that is hard to read.
Enter xtdb, the graph database from JUXT. We are receiving a normalized graph of resources from the json-api endpoint, and are computing queries on it. Why not have xtdb run the queries for us.
Lets start an xtdb node with an empty configuration map: Instead of persisting the data in one of their pluggable backends, we store it in plain java datstructures by giving an empty map as configuration.
Xtdb is schemaless so we don't have to worry about defining any attributes ahead of time.
Although you can keep the documents nested and query them in datalog (you can run (get-in _ [:attributes :timestamps :startedAt])
directly in the query if you wanted), its cleaner to flatten the data before ingesting it.
Lets use this (12 year old!!) function to recursively flatten keys, joining nested keys with a "."
Lets see a before and after of just one resource.
Much better. We don't need to, but we could do the reverse of this operation because section 7.8.2 of the json-api spec forbids "." in key names.
Now lets deal with toplevel responses.
Because of json-api's uniformity, we can have just one function that handles every possible api response. In particular, we use these properties:
:data
and :included
keys. They contain either a single resource, or a list of them.:id
key.This function takes every resource from the response, flattens it, and puts it into our database node.
Notice how we tranform the challonge notion :id
to :xt/id
, which every xtdb document needs.
Now lets ask all the same questions as before, this time in a declarative style.
If you have never seen datalog before, don't fret. This query can be read as
find the id of every document that has the key
:type
with a value of"match"
, and the key:attributes.state
with a value of"open"
.
Each triple in the :where
clause of the query is defining a rule/restriction in the shape [entity attribute value]
that the returned documents must comply to.
The effect of declaring that an attribute must match a data literal like [match :attribute.state "open"]
is a filtering of documents. This line is like my above function matches->pending
which took my 10 matches down to only 4 open ones. This is a nice delcarative filtering, but nothing mind shattering.
The magic comes when you put a variable in the value position of the triple. Now any time you use this variable again (in entity or value position of another rule), the values must match up.
Here we extract the p1 and p2 values from the match relationships, then use those values as entity ids to extract information about the player.
Now the reverse operation, (find match ids from gameserver uuid), is trivial.
Or we can ask, given two player names, which matches they are in.
Datalog can also run arbitraty code inside queries. Here, the challonge api gives us the winner of a match as an integer, even though the primary keys are all strings. Easily rectified by calling str on the key before using it further.
We can also run arbitrary predicates on logic variables (not=
) and access nested documents (get-in
).
There is much more you can do with datalog, almost any question you could want to ask is answerable. See here for more examples.
Thanks to json-api, every response to further api calls moving the tournament forward can be run through ingest-jsonapi-resonpse
and be otherwise forgotten.
Before xtdb, I would have to write code to answer my questions. After xtdb, every question I can ask of my data is already answered, I just have to phrase the quesiton!