April 10, 2024

Product Release: Crawl Requests

With Crawl Requests, researchers can direct our crawlers at the keywords, profiles, or Telegram channels that they’d like to track.

TLDR

Open Measures is launching a new tool, Crawl Requests, to allow researchers to point crawlers at the information they care about most.
Authenticated users can now add keywords, profiles, or Telegram channels they’d like to back-crawl and continuously monitor.

Background

crawl requests dashboard — ***In the Crawl Requests UI, users can view all crawl requests, manage request statuses, add new requests, and monitor trends.***

The research community needs comprehensive coverage of unique keywords, user or group profiles, and Telegram channels for project-specific investigations. Historically, we met that need with request-based manual configuration and our API. Today, we’re happy to share the launch of the Crawl Request App, where users can take control of Open Measures’ scope of coverage themselves.

Here’s a high level overview of how this new tooling works:

crawl requests diagram — Crawl Requests works in five steps: Partners provide targets for the crawl (keywords, profiles, or channels) (1). Open Measures’ crawlers pick up jobs (2) and run collection for data using a source’s native search interface (3). That data is then collected and stored (4). Partners can then access the collected data on the Open Measures platform via API or Research Dashboard (5).

With this new feature, users will have better control over Open Measures’ collections and the ability to target thorough crawls of subsets comprised of sources we collect. Depending on the dataset, we will either enumerate through a list of targeted keywords, user profiles, or channels.

The * in the above diagram represents the other crawling processes that Open Measures maintains to collect data from our sources. Crawl Requests are a standalone collection stack that runs in parallel to these default data collection systems.

Keywords and Profiles

crawl requests keywords — ***Requests for profiles or keywords are made on a source-by-source basis***

Most social media sites have a native search bar. When a crawl request is made for a keyword, Open Measures’ crawlers run a search for that keyword using the dataset’s search bar, enumerate all the results, and index the data.

***Example search interface on Gab. In this example, the keyword request is for “trump 2024”***.

Crawl Requests allow Open Measures users to request keyword crawling on a per source basis.

Telegram Channels

add channels — ***The addition of new Telegram channels, now at the touch of a button***

Crawl Requests also enable users to add Telegram channels. When Open Measures receives a request to crawl Telegram data, we back-crawl the channel to the first message and then continuously monitor for new data moving forward.

Conclusion

Open Measures’ Crawl Requests App and API are now available to authenticated users and organizations. This innovative new application allows users to take control of our crawlers, pointing them at unique keywords, profiles, and channels that their researchers care about most.

Identify disinformation and extremism with the Open Measures platform.

Organizations use Open Measures’ tooling every day to track trends related to networks of influence, coordinated harassment campaigns, and state-backed info ops. Click here to book a demo.