How to Write a Ranker: API Documentation is here!
Including sample code and data
There are now detailed API docs for how our experimental framework interfaces with your ranker. They are subject to change as the science requirements are finalized and we plan our engineering roadmap. But they should give you a very good idea of what a ranker needs to do, and what a first round submission needs to look like.
What’s in a Ranker?
The a ranker is a function that takes as input the list of posts from the participant’s personalized feed, or at least the text content of these posts, with poster user IDs anonymized. Then it returns these posts in a different order. It can also remove items, or add items.
It is only possible to add items that are already on the platform and already visible to that participant, otherwise things like engaging and sharing wouldn’t work. So if you wanted to make sure a particular type of content appeared in someone’s feed, you would need to be publicly publishing such content on the platform.
Sample Code
We now have sample code for a minimal ranker available. We are adding more complex rankers and test data. Subscribe for updates.
What data can a ranker use?
Aside from the text content of the posts in a participant’s feed, the ranker can see a limited history of what items the participant recently engaged with.
Rankers can call external APIs in first round submissions. That is, at that stage you could call GPT4 or Claude with a custom prompt to determine your ranking. But you can’t call any external services in final round submissions. This is to prevent any possibility of leaking participant data. What you can do is distill your commercial LLM responses into an open source model. We’ll have sample code for that in not too long.
What will change going forward?
The basic API structure for reordering, adding, and removing items won’t change. The technical submission requirements won’t change (live URL in first round, Docker container source in final round.)
But there are many really interesting ways we could extend the capabilities of rankers. For example:
We keep getting asked if we can support rankers that look at the structure of the participant’s local social network. Maybe! There are privacy and security concerns here, but several plausible ways to do this.
We’d love to support an explicit map-reduce API to make it easier to meet our 500ms p95 latency requirement. This works for rankers which are complex but highly parallel, including the classic “score individually and then sort” approach.
We currently trying to get engineering time commitments settled so we can figure out realistically what features we can commit to. We have hard deadlines because we need to run the experiment in the months before the election. If you are able to donate engineering time, the odds go up a lot.
If you want a particular API feature, let us know!
We want to support your science. Join our discord! Tell us what you would need.