Datasets
Navigating the Open Measures datasets
The data that Open Measures offers, whether through the API or premium dashboard tools, is mostly left as it was found within each source. That means we share everything that is publicly available. Use this resource to map the fields used from network to network.
Data is delivered via JSON blobs that look like this when making an API request.
Data Fields
Elasticsearch Fields
To get to the actual data, you need to navigate into hits.hits and then each result will be under a number in the nested structure.
When we add a post to our database we generate a number of fields that can be found at the level above the actual data responses. If we are looking in response "0", the meta-fields continue until "_source" which is the beginning of the actual data. These are generated by Elasticsearch and are not part of the data from the site. Learn more about these fields in Elasticsearch's guide.
Data Fields Overview
Each dataset will be described in detail in the following sections with more fields, but this chart can function as a quick-start guide for a few "Key Fields".
Platform: This is the plain text readable name of the platforms not the name of the platform field in the api.
Platform Endpoint: This is the name of that platform when you're making an API request.
Username: This is the handle (especially user slug) for the account that is posting the message, though not necessarily the author of the post contained (i.e. if it's a forward). This is their @, not their full name.
Post content: This is the actual body of an individual post or message.
| Platform | API Site Parameter | Username Field | Content Field |
|---|---|---|---|
|
| htmlparsedcom | |
|
| htmlparsedcom | |
|
| text | |
|
| content | |
|
| N/A but look at meta.description or meta.title | |
|
| content_cleaned | |
|
| content | |
|
| txt | |
|
| post_text | |
|
| comment | |
|
| N/A but look at value.title and value.description | |
|
| content | |
|
| body | |
|
| content | |
|
| body | |
|
| content | |
|
| text | |
|
| N/A but look at full_description and metadata.name | |
|
| text | |
|
| N/A but look at value.title and value.description | |
|
| content | |
|
| message | |
|
| desc | |
|
| text | |
|
| content_cleaned | |
|
| text | |
|
| content |
Data Sources
4chan
4chan is an imageboard website where users can anonymously post. Users primarily participate in threaded discussions in response to an original post containing an image. Threads are categorized into “boards”, which are a many-to-one relationship between a thread and a forum room. One of the most popular boards, “/pol/” or “politically incorrect” is where the majority of internet attacks and threats of real world violence are posted. Content and users across the site, especially on “/pol/”, suggest a free-speech maximalist ideology.
Example API JSON
8kun
8kun, previously called 8chan, is an imageboard site where anonymous users respond in a threaded format to an original post. 8kun was created in 2013 by Fredrick Brennan as a free speech alternative to 4chan after 4chan began banning some objectionable topics. Like 4chan, 8kun threads are categorized into various “boards”. Brennan no longer controls the site and has since advocated for it to be shut down due to the real world violence it has spawned. The site is associated with several white supremacist, neo-Nazi, alt-right, and anti-Semitic hate groups globally. Activity on the site has been linked to several mass shootings and terrorist events, including three in 2019 alone (Christchurch, New Zealand, Poway, CA, and El Paso, TX). The site is also known as the home of Q, the user behind the notorious QAnon conspiracies.
Example API JSON
Bitchute
BitChute is a British alt-tech video sharing platform and an alternative to YouTube. Founded in 2017 by Ray Vahey, BitChute hosts content involving QAnon conspiracies, hate speech, and neo-Nazi propaganda. Its users often turn to the platform after being kicked off other video sharing platform like YouTube and even Rumble. As with other alt-tech sites which promote “free speech”, the site is full of videos and comments containing racist slurs, Nazi imagery, and calls for violence.
Example API JSON
Bluesky
Bluesky is an American text-based, decentralized social network created by a group of former Twitter employees. Also known as “Bluesky Social”, it is a microblogging social network that uses the “AT Protocol”. It officially branched out from Twitter in 2021, but maintains a similar feel and user experience. Moderation, however, works very differently at Bluesky compared to Twitter or other legacy platforms. Called ‘Composable Moderation’, Bluesky’s moderation begins with a ‘basic default’ level of moderation followed by additional layers that are left to individual users to determine.
Fediverse
Fediverse is comprised of decentralized platforms that position themselves as networks that give users more control and autonomy. These networks are often backed by servers running open-source code and maintained by pseudonymous administrators. The open-source libraries running on these servers, like Mastodon or Lemmy, implement communications protocols that allow the servers to “federate” and share information with others using a shared protocol. One of these protocols is called “ActivityPub”, which operates as a server-to-server federation communication network.
Gab
Gab is an American social media platform that was launched in 2016. It promotes itself as a platform that upholds free speech and claims to be committed to providing an open and decentralized alternative to mainstream social media platforms. Gab allows users to create accounts, share posts, and engage in discussions.
Generally speaking, Gab users are drawn to the platform as an alternative to traditional social networks like Twitter, but with much less content moderation. Gab has been a subject of controversy due to concerns about the presence of extremist and controversial content on the platform. Some critics argue that its commitment to free speech has resulted in the hosting of content that promotes hate speech or violence.
Example API JSON
GETTR
GETTR is an American social media platform that was launched in July 2021. GETTR was co-founded by Jason Miller, a former spokesperson for the Trump administration. GETTR, like other alt-tech platforms, positions itself as a platform for free speech. The platform was developed as an alternative to mainstream social media platforms and has gained attention for attracting users who share alt-right political views. GETTR also has ties to Chinese businessman and dissident, Guo Wengui and former Trump aide, Steve Bannon. Today, GETTR is rife with alt-right extremism, racism, antisemitism, child sexual abuse material, and terrorist propaganda.
Example API JSON
Kiwi Farms
Kiwi Farms, formerly known as CWCki Forums, is an American internet forum commonly associated with extreme doxxing, organized harassment, and real-life stalking. Founded in 2013 by Joshua "Null" Moon - a former 8chan administrator - Kiwi Farms continues to host targeted racist, antisemitic, and transphobic harassment campaigns. The volume and depth of these campaigns on Kiwi Farms is associated with at least 3 suicides. Because the data contains a high volume of personally identifiable information, we do not display the source in Open Measures' Public API / App.
Example API JSON
LBRY/Odysee
LBRY was a blockchain-based, peer-to-peer file-sharing and payment network. It was founded in 2015 by a group of libertarian activist and served as the foundation for decentralized platforms like social networks and video sharing platforms. It was shut down in 2023 following a lawsuit brought by the SEC for selling unregistered securities.
The creators also launched Odysee, a fringe decentralized alternative to YouTube and is run on the LBRY network. Odysee contains COVID-19 misinformation, neo-Nazi propaganda, and antisemitic conspiracy theories. While the site may delist some content that includes pornography and violence, the videos remain available on the Odysee blockchain data store.
Example API JSON
MeWe
MeWe is an American alt-tech social networking formally launched in 2015. It describes itself as the "anti-Facebook" as it does not moderate mis- or disinformation on its platform. MeWe's light content moderation results in the hosting of dangerous users like conspiracy theorists, white supremacists, and far-right militia groups.
Example API JSON
Minds
Minds is an American peer-to-peer blockchain-based social network. It was launched in 2015 as a free speech, minimally moderated alternative to Facebook. Users can earn crypto rewards for platform engagement. Its founders insist that they allow extremist content as part of an effort to deracidalize users through discourse. Naturally, Minds can be found to be full of white supremacist and far-right content due to said lack of moderation. Like many of the other datasets, Minds saw a large influx of users following the January 6th US Capitol attack and the removal of many thousands of users from Twitter and Facebook.
Example API JSON
OK
Odnoklassniki (OK) is a Russian was founded in 2006 by Albert Popkov. Since then, it has been sold to the VK conglomerate which also owns VKontakte. The merger has resulted in a monopoly of the Russian social media landscape. Supremacist and neo-Nazi activity on OK has been tied to real world violence, including a 2023 mass shooting in Allen, TX.
Example API JSON
Parler
Parler was one of the highest trafficked American alt-tech microblogging social network. Now shuttered, Parler had deep ties to the alt-right in the United States. Like other alt-tech microblogging platforms, Parler advertised itself on the premise of maximal free speech so they performed little content moderation. Parler data contains QAnon, white nationalist and neo-Nazi content, as well as organized calls to violence. Parler is known as one of the primary social networking sites used to coordinate the January 6th insurrection in the United States.
While Parler shut down following an acquisition by Starboard in April of 2023, Open Measures maintains an archive of Parler data and usage.
Example API JSON
Poal
Poal is an American alt-tech threaded forum site modeled after the mainstream forum network, Reddit. Poal insists they maintain a “free speech” approach with their community as they conduct very minimal content moderation. Like in other datasets, this leads to content containing harmful and harassing posts including violent, antisemitic, and white nationalist propaganda.
Example API JSON
Rumble
Rumble is a Canadian video sharing platform and web host founded in 2013 by Chris Pavlovski. Billed as a free speech alternative to YouTube, it has made a series of acquisitions in years' past to compete with the video giant. Objectionable content creators that violate terms of service (often for hate speech or extremism) with YouTube and other video hosting sites often turn to Rumble to share their videos. In May 2021, the platform received investment from Peter Thiel and J.D. Vance. A few months later, it entered into an agreement to serve as cloud services provider for Truth Social. In September 2022, Rumble went public via special-purpose acquisition company.
Example API JSON
RUTUBE
RUTUBE is a Russian video platform and alternative to YouTube founded in 2006. Now owned wholly by Gazprom Media, RUTUBE has been used to push Wagner and state-authored talking points and propaganda. State-sponsored material via a library of licensed content includes movies, series, cartoons, shows, and live broadcasts. It also hosts blogs, podcasts, video game streams, and educational content.
Example API JSON
GET https://api.smat-app.com/content
Get RUTUBE comment content from SMAT's public API.
Path Parameters
| Name | Type | Description |
|---|---|---|
term* | String | ukraine |
site* | String | rutube_comment |
Scored
Scored (also known as .win or Win Communities) is a collection of alt-tech threaded based conversation forums that operates very similarly to its traditional counterpart - Reddit. The sites first came into existence when Reddit banned the subreddit r/The_Donald and users responded by creating their own site (now called c/The_Donald). Scored claims to "unblur the lines between entertainment and politics". The Donald remains a very popular channel for users to discuss January 6th, conspiracy theories, and hateful rhetoric.
Example API JSON
Telegram
Telegram Messenger, commonly known as Telegram, is an encrypted, cross-platform, cloud-based messaging application. Telegram was founded in 2013 by Nikolai and Pavel Durov (founders of VK) and hosts its operational center in Dubai. Telegram data schema consists of channels which users can join to post messages, images, videos or other media. Open Measures users frequently search this source to identify activity from extremist and neo-Nazi groups in the United States and for news related to Russian influence throughout Eastern Europe and Africa. In the past, our researchers have also seen it leveraged by authoritarian regimes like the Myanmar Tatmadaw.
Example API JSON
TikTok
TikTok is a social media platform that allows users to create, share, and discover short-form videos. The app was developed by the Chinese tech company ByteDance and was launched in September 2016 under the name Douyin for the Chinese market. It was later released internationally as TikTok in September 2017.
Open Measures focuses its attention on TikTok content that ranges from harassment, white supremacy, conspiracy, and war crime related posts.
Example API JSON
Truth Social
Truth Social is an American microblogging social network alt-tech platform created by Trump Media & Technology Group. Originally only a mobile only site, the site now operates like other microblogging social media platforms on web browser and in app stores. The site bills itself as a "Big Tent" social media platform that encourages an open, free, and honest global conversation. It has been reported to shadowban users that counter the site's conservative and conspiratorial narrative. Open Measures has observed neo-Nazi, white nationalist, and QAnon content on the platform as well as calls to violence.
Example API JSON
VK
Founded by the Durov brothers in 2006, Vkontakte (VK) is a Russian social networking site. VK is based out of Saint Petersburg and considered the Russian Facebook. It is one of the most popular websites in Russia and is known to have loose enforcement on policy-violating content. State-run insurer Sogaz purchased a majority stake in VK in 2021. Research has shown the platform plays host to neo-Nazi groups in Russia and the United States.
Example API JSON
Wimkin
WimKin is an American alt-tech social network that promotes itself as a free speech alternative to Twitter and Facebook. It was found in 2017 by Jason Sheppard and the platform allows calls to violence including videos and recordings of violent acts and threats. WimKin was pulled from major app stores in January of 2021 following the storm on the Capitol but has since returned. Activity on the site is often linked to QAnon, far-right militia, and white nationalist groups.
Example API JSON
Last updated
