Updates: Add Your Own DVs, Info Session FAQs
Also updated DVs. Lots of new information for Challenge teams.
DVs have been updated
After meetings with our scientific advisory team we have updated the list of dependent variables — the outcomes we will test. We may still make some changes but the final version will be quite similar to this, as we are zeroing in on the interesting things to measure.
Watch for a post discussing what variables are known to be moveable from previous experiments, in the next few days. One seed: the short-term surveys (in-feed surveys) are probably a lot easier to move than the long-term outcomes. But of course long-term changes are more interesting, if you can manage to do it!
Add your own DVs
By popular request: we will consider adding up to three survey questions for each team, if there is a particular thing you want to study. These may be long-term or in-feed questions. This is not guaranteed — please make the case in your submission as to why there is good science to be done.
Behavioral or exposure outcomes, by contrast, do not take a toll on the user. We are very likely to consider collecting additional outcomes that we can derive from data we are already collecting.
All rankers will be tested on all outcomes, including any that are added by other teams.
Q&A from last week’s Info Sessions
These questions have all been added to the FAQ.
How will news knowledge be measured?
By asking the user if they recognize a number of current news events, with both actual and made up events thrown in, as in this paper.
What data will the ranker get about each participant?
Some basic demographics from the intake survey, including political party self-identification and intensity of social media use. Let us know if you’d want age, race, gender, or SES — these are sensitive so we’d want to review your plans.
A history of what each participant has previously seen and what they have engaged with will be available in the database.
We are still discussing whether we can provide any data about the user’s social network, e.g. a list of who they follow on X. But even if we do, we won’t be able to provide any information about any of those users, because there’s no time to retrieve that in the 500ms window. Let us know if you think you could do something interesting with this limited information.
What data will the ranker get about each post?
Text and basic metadata, including comment threading, session ID, and an indication of whether there is an image, with the alt text if so. See the API docs.
We are discussing including classifier output for each post the estimates a) is the post political or civic content and b) what is its political ideology on a left-right scale. Of course you could compute this yourself, but let us know if you would want to.
How many posts will be included in one call to the ranker?
Up to a couple hundred, depending on what is retrievable from the platform within the 500ms window. These will all be posts that the platform has already selected for the user, so depending on your goal there may or may not be value in reordering them, but of course you can remove and add posts too.
Ads in the feed will not be sent to your ranker, and all advertising on the page will be preserved.
Can I scrape data from within my ranker?
For security reasons you cannot call external APIs or services, but you can run a background process that imports public data or scrapes public social media data. You could ingest all of Wikipedia or monitor Google News, if you want. You cannot scrape the participant’s social network. See this post and the repo.
Can we ask users for information and use that to personalize ranking?
Not currently planned. We love the idea of greater user control over ranking algorithms, but we also want to figure out good defaults because most people won’t use controls. And this simplifies the experiment.
However, if you wanted to add up to three questions to the intake survey, you could get that data as part of the user demographics.
Thanks to everyone who showed up to the Info Session. We are planning more!
Join us on our discord.