I once briefly chatted with a man who ran a company that used satellite imagery to predict crop yields. I found it amusing how rather than providing this information to farmers ("It looks like you're going to produce only 60% of what you did last year, time to cut some expenses!"), there was so much more money in providing this information to hedge funds and trading firms. Morality aside, the potential for technology like this is exciting.
It's kind of cool how the combination of big data, scraping, statistics, etc. allows this small niche of analytics to thrive. I have a decent background in most of these subjects, and this seems like a very freelance-able job, so I'd love to dabble in this area and learn how it works.
It is context dependent, but I can give a basic idea. In the example you've given, high four figures per customer per quarter, assuming you just give them a deliverable report once a month and the data is turnkey enough that many customers (30 - 40) can "subscribe" to it. For a fund that wants something more boutique, like exclusivity or a specific company, five figures becomes the floor.
The nature of the business is significant upfront work (setup of data collection and analysis model) followed by infrequent tweaks going forward. Lucrative data on a single hot company could be sold for mid - high six figures per year with very high profit margin for as long as the data is available and the analysis model is both functional and interesting (i.e. delivers an insight not already priced into the market).
I'm also curious as to the level of "polish" that you are expected to provide. Let's say you know how many widgets Acme sold this last quarter. Is that by itself enough information, or would you need to calculate projected expenses / other stuff to actually be able to deliver an earnings estimate? What if you just had projected revenue?
Thank you!
That's great. Were tracking numbers vendor-specific in some way? So you could, let's say, order an iPhone once a week and get an idea for how many iPhones (or total Apple products) were sold in that time period, just by using the tracking numbers? Was it dependent on geography in some way?
What changed BTW? Did the format of tracking numbers change?
Just two per carrier would do: one at the start of the quarter, and one at the end. Earnings dates happen weeks after the quarter ends, so you'd have heaps of time to peddle your analysis around.
I mined data in the real estate, QSR, automotive and airline sectors (and a few peripherally related ones). We would identify a source of data that was a demonstrably strong proxy for a specific company's revenue (that is to say, if we broke out a naive timeseries of the data it would map nearly 1:1 to earnings results each quarter). Then we would collect this data en masse, writing software over a custom crawler and infrastructure, bespoke to each case. During ingestion we'd process and normalize the data and dump it in a database. Once we had a significant amount of data we'd build a forecasting model, which could be very straightforward ("how many products have they sold this quarter", for simpler companies) to very complex ("can we reverse engineer the amount of business which is now online" or "can we determine the undisclosed amount of business being done in this specific region"). This analysis (and not the data itself) constituted the product - we never even provided the data directly to the hedge funds.
Once the crawling and analysis was mature, we'd incubate it for a few quarters while building interest; then, after showing a very strong mapping over several quarters, offer it as a data product. It was not uncommon to achieve <5% or even <1% margin of error for forecasting earnings announcements on a quarterly basis.
The data was always public and legal, though challenging to identify and difficult to effectively crawl. It was typically derived from very uncommon sources, and would range in complexity from the extremely simple ("we can crawl sequentially incremented integers from this third party that appear to map to products sold by this company") to the very sophisticated and complex (e.g. "we've collected a zero sum distribution of ad partners and their spend on the network, are more partners churning off, and is this a better or worse outcome for the company?"). I've spoken about this in comments here before.
This is still very doable, and I still do this sort of work for personal research (and profit). However, the work I do is now much more quantitative - I'm currently taking a similar approach for baskets of equities with the goal of forecasting macroeconomic trends (e.g. subprime auto loans) instead of the earnings results of individual equities. To give a very specific throwaway example (because it doesn't violate an NDA and it no longer works): it used to be possible to pretty accurately forecast large tech retailers' product sales each quarter (like Apple) by reverse engineering FedEx and UPS tracking numbers.