News for hackers. Stuff that matters.
Re:Hedge-fund managers that do the most research will post t... by piker on Thursday March 22, 2018 @09:57AM (#16648661)
Wouldn't some sophisticated shops potentially use crawlers, etc. hosted from cloud providers (i.e., third-party IPs) that might significantly skew this data?
Re:Hedge-fund managers that do the most research will post t... by fwdpropaganda on Thursday March 22, 2018 @10:11AM (#16648774)
I assumed the sophisticated shops ran their own datacenters. Can you imagine RenTech running their code on other people's computers? I can't.
Re:Hedge-fund managers that do the most research will post t... by dsacco on Thursday March 22, 2018 @10:48AM (#16649110)
Yes, Renaissance runs everything on its own infrastructure (as do most firms of that caliber, and even a tier below).

That said, many of these firms also ingest data from aggregated sources, so their research would activity be very difficult to accurately track via IP presence.

Re:Hedge-fund managers that do the most research will post t... by sseveran on Thursday March 22, 2018 @12:06PM (#16649806)
Why? The reasons for having their own datacenters are either legacy or needing some sort of specialized hardware that is not available at a cloud provider. Mana which is a new pure quant fund uses AWS extensively.
Re:Hedge-fund managers that do the most research will post t... by MagnumOpus on Thursday March 22, 2018 @01:15PM (#16650629)
The reason for having your own computers is assured secrecy of your algorithms, and complete control over redundancy, latency and network linkages.

The reason for not having your own computers is because you can't afford them. A consideration for upstart funds, not a problem for RenTec.

Re:Hedge-fund managers that do the most research will post t... by sseveran on Monday March 26, 2018 @12:09PM (#16679917)
If it is discovered that Amazon is stealing data from AWS customers than AWS will go out of business fairly quickly. Also even though Mana is new they do have ~$1B AUM.
Re:Hedge-fund managers that do the most research will post t... by fwdpropaganda on Thursday March 22, 2018 @01:29PM (#16650743)
> Why?

Because you don't want anyone to have access to your code and data.

If you found out that Facebook is running on AWS, wouldn't you find that strange?

Re:Hedge-fund managers that do the most research will post t... by ikeboy on Thursday March 22, 2018 @06:36PM (#16653363)
Snapchat runs on Google. Dropbox ran on aws for a long time. Instagram was on aws before Facebook bought them. Netflix has basically everything but the actual video content on AWS.
Re:Hedge-fund managers that do the most research will post t... by gowld on Thursday March 22, 2018 @01:34PM (#16650786)
AWS, Google, Microsoft, all have lots of customers. Why are those customers not strange but FB would be?
Re:Hedge-fund managers that do the most research will post t... by fwdpropaganda on Thursday March 22, 2018 @01:44PM (#16650862)
In my mind, if technology is a big part of your business then running on other people's computers only has one upside: might be cheaper. Everything else is downsides.

Therefore, A) companies that operate at a scale where the only upside disappears, and B) companies where the cloud saving outweights their expected liability cost, both will have their own datacenters. Facebook is clearly in category A, RenTech is clearly in category B. This is my understanding anyway.

Re:Hedge-fund managers that do the most research will post t... by YawningAngel on Thursday March 22, 2018 @02:49PM (#16651477)
Public cloud is not cheap. Public cloud is really expensive. The value of public cloud is that it's flexible (my current employer, which is massive, uses AWS in part because it has poor governance of leased data centres) and has hosted offerings that save time and operational overhead - it's quicker and easier to use SQS and DynamoDB than to host Kafka and Cassandra. Cheap it is not, however.
Re:Hedge-fund managers that do the most research will post t... by jeffreyrogers on Thursday March 22, 2018 @10:20AM (#16648852)
Some do use AWS and other cloud providers for collecting and processing data.
Re:Hedge-fund managers that do the most research will post t... by scarletham on Thursday March 22, 2018 @10:10AM (#16648766)
Yes, exactly. Wouldn't surprise me if most of these firms were rescraping the data every night and overwriting.
Re:Hedge-fund managers that do the most research will post t... by fwdpropaganda on Thursday March 22, 2018 @10:13AM (#16648788)
Why would they do that?
Re:Hedge-fund managers that do the most research will post t... by scarletham on Thursday March 22, 2018 @10:27AM (#16648914)
Take this with a grain of salt as I don't work at any of the funds mentioned (but still in the industry), but why not?

Filings are subject to change due to errors, updates, etc. It's cheaper, easier, and less error prone to just scrape and overwrite data than search for any updates.

I may have been heavyhanded in saying that the firms are rescraping the entire universe of filings on a daily basis, but I guess my point here is that these numbers can very easily be skewed and are generally a pretty poor indicator of actual data access.

Re:Hedge-fund managers that do the most research will post t... by calabin on Thursday March 22, 2018 @11:21AM (#16649377)
SEC EDGAR is terrible for a variety of reasons. It breaks, few things are truly standardized/normalized, etc. If you have the time and an invested interest there's no reason not to.

Filings are subject to change but they are restated, not modified in place. That said, anything goes on the site.

The only real reason not to scrape everything on a continuous loop is that there is a rate limiter in place, so you may end up impeding your intake of new data by loading yourself down with old jobs to handle.