Open Measures has introduced a few key changes to our Public API in order to counter abuse of our platform.
TLDR
Open Measures is introducing new rate limits to the Public API to counter abuse of our platform. These include limits on the number of free daily public queries that a single IP can make and the window of time for data that users can view. We are also restricting the Public API so that non-credentialed users may only see collections that are at least 6 months old.
If you’d like to have unlimited access to our API or the ability to run queries over all of our entire live collections, please get in touch with us at [email protected] or fill out this form.
Background
As Open Measures has continued to grow, we’ve wanted to get a better understanding of our use base and a sense for our daily active users. Since introducing mixpanel to our Public API, we’ve been able to monitor our tools’ anonymized usage. What we found was somewhat unsurprising: a handful of users are taking advantage of our open source services in a few ways, namely by:
- Using Open Measures in their own commercial products without citing, crediting or supporting us
- Continually running the same search terms and rotating IP addresses to avoid detection
- Spoofing headers to make it appear that origin traffic is coming from our own resources
Beyond the fact that resource intensive queries reduce load capacity for authentic users, misuse of our tools also presents Open Measures with a few concerns:
- We want to be appropriately credited for what we have built and continue to create. We are proud of our work!
- We’d like to be able to accurately measure usage and in order to do so, we need to filter out inauthentic traffic.
- We want to prioritize authentic user requests. The scale of abusive traffic we face inherently deprioritizes them.
- When we spend resources on covering the hosting costs of malicious traffic that means less funds for expanding our collections, innovating, and developing new tooling. Running Open Measures costs money!
To give readers a sense of scale, we are estimating that ~80% of our traffic is coming from the same group of users attempting to bypass our rate limits.
New Limits
To counter abusive traffic, Open Measures is introducing some new limits to our API:
- We are introducing a cap of 39 requests a day per IP address.
- We are limiting the visibility of our most recent collections of data. Any requests from unknown users will only be able to access data that is at least six months old.
Conclusion
These changes should be a net positive for everyone! They are a necessary reality that will help protect Open Measures from malicious abuse and to ensure that our tools are available for authentic users who need them.
If you’re a user who will be negatively affected by these changes or if your use case requires more recent data, please drop us a line at [email protected]. In alignment with our values, Open Measures’ mission is to get our collected data in as many hands as possible.
It is important to note that Open Measures is still collecting live data. Our team works hard to ensure that our crawlers stay live, and in the coming weeks, we hope to produce public visualizations and dashboards to reflect that. Stay tuned for more updates and robust solutions to your API needs!