lkowalcz

At IEX, we use kdb.

Here’s a blog post from a former employee at a new startup comparing kdb against some alternative databases for common operations on financial data: https://medium.com/prooftrading/selecting-a-database-for-an-...

I think IEX has a free API with historical and real-time stock info: https://iextrading.com/developer/docs/

I think it's appropriate to restrict usage of the word "encryption" to methods that come with some justification for their security.

I don't think that doing this stifles innovation (In fact, I think requiring crypto innovations to have security justifications is probably better overall for innovation in the field)

I've never heard of neural encryption before, do you have a good source for me?

From the Wikipedia page for "Neural cryptography", it seems like there's some success in using NN's for cryptanalysis, but not for constructions...

Edit: Do you mean the Google GAN experiment? (https://arxiv.org/pdf/1610.06918v1.pdf) Ahh ok, well at least for this there looks like an attempt at defining a security model (security against some other NN). I don't really believe the security model is realistic (how do we know NN's are really that effective as adversaries?), but at least there is a model, so calling that "encryption" sits somewhat better with me. I'm pretty sure this is not what's being used by Numerai since it seems like it would not result in ciphertexts with the structure necessary to perform ML operations on.

Edit2: Maybe you're right and it is more advanced. In any case, as a crypto nerd I wish they would disclose what they are actually doing / the rationale instead of tantalizingly suggesting that they have made (what would be) a breakthrough in a practical use case of advanced encryption schemes, but not saying how.

Thanks for that link! I was never able to find any details from someone who works for Numerai (or claims to at least)

I still don't think it's fair to market their method as comparable to Aslett's scheme or "standard" notions of homormorphic/order preserving encryption, no matter how "chill" they are :)

I suspect they might just be anonymizing their features (removing any labels), then normalizing them to [0,1].

Worth pointing out that Numerai actually doesn't use encryption in any standard sense (including structure-preserving encryption), but instead seems to be using some heuristic method of obfuscating their data.

Their (closed-source) method of obfuscating their data apparently does have the property that it preserves the structure of the data, but calling it "structure-preserving encryption" is misleading imo since it risks confusing it with standard notions of encryption and structure-preserving encryption which have much stronger security guarantees.

(Their marketing seems to encourage this conflation by, for example, citing academic advances in standard notions of homomorphic encryption and SPE and implying that these advances have enabled Numerai's technology)

https://medium.com/numerai/encrypted-data-for-efficient-mark...

Most NSF grants and the like are for PhD students (the university still welcomes the funds they bring in, but they are a relatively small portion of income).

Universities make much more money off their Masters students, most of whom are international, and pay full tuition (or someone does for them).

There is a huge difference in how universities treat their Masters vs. PhD students. Indeed, universities are not making loads of money off international PhD students (most receive free tuition, a stipend, etc). However, international Masters students are cash cows for most universities (most pay full price -- or someone does for them).

Ah ok, sure. I wouldn't call something like a linear transform on anonymized features "encryption" (more like obfuscation?), but I guess it's good marketing in that it lets them associate with the "recent advances in [real] homomorphic encryption"

I'm almost positive it's not using homomorphic encryption (at least in any real sense). It's misleading how they seem to suggest that they are, though.

The encryption here is being done by "adding 20" / "multiplying by -0.5"?

Given this "encrypted" X , y dataset, I could easily find the unencrypted version... (even if I don't know 20 or -0.5, this still reveals so much of the structure that I don't believe it provides any real protection against anything except the most lazy attackers)

Anyone want to speculate about how the data is being "encrypted"? It seems like they don't want to say, which immediately sets off red flags in my head...

I am pretty sure homomorphic encryption is not being used. I think if they are doing anything rigorous, maybe they are using order-preserving encryption (http://www.cc.gatech.edu/~aboldyre/papers/bclo.pdf). This would mean that the only valid operations on the ciphertexts are comparison operations. I can't seem to find anywhere where numer.ai actually says how to interact with the "encrypted" data. I think it's a little strange that they would suggest that they are using homomorphic encryption yet have only comparison operations actually make sense on their "encrypted" data.

A second hypothesis would be that no encryption is being used at all, and this is just unlabeled features that have been renormalized within [0,1].

A third would be that order-preserving encryption is being used, but in an ineffective way which is basically just resulting in the second scenario. (understanding the security guarantees of order-preserving encryption is practice is very complicated)

I am pretty sure homomorphic encryption is not being used. In fact, I suspect that no real encryption is being used.

Isn't it the case that if I just removed the labels, and renormalized all my data to fall in [0, 1], then what I end up with looks a lot like what Numer.ai gives you?

I'm not aware of any homomorphic encryption / structure preserving schemes that have homomorphic evaluation on ciphertexts equivalent to literal multiplication and addition of ciphertexts, and this seems to be what they want you to do to train your model. (unless I'm misunderstanding how to interact with the "encrypted" dataset)

EDIT: seems like most people think they are using Order Preserving Encryption, which allows one to compare ciphertexts with the "less than" predicate. This makes more sense looking at what they give, but I never saw anything where they say "only do comparisons on the encrypted data."

Yeah, my first guess would be that the genetic solver is degenerating to basically a brute force search here (which is feasible in this weak instance). Will have to take a look to confirm...

What is this part? Isn't it just by assumption? (also, by secret key, I'm guessing you mean 'a', 'b')

Surprising it took this long. (Especially since, for example, RS and A have had Turing awards for more than a decade)

The "dummies" are the authors of "Fool's Gold," to which this article is a rebuttal.

Well I can audit the source code of my client and be assured that it will only rekey when it sees proof (posted via twitter) that the person the data was encrypted for has joined.

Keybase doesn't have my private key (only I do), so they can't re-encrypt the contents.

(sorry if I misunderstood your question)

Also note that most homomorphic encryption schemes proposed so far are based on LWE and are therefore also conjectured to be resistant to quantum attacks.

Apparently they reuse the primary wallet quite frequently:

http://www.coindesk.com/cryptowall-325-million-bitcoin-ranso...

For (1), this is the reason the ransom is small. Since "many" are actually trustworthy, it's a small risk to pay the relatively small ransom. (Also, you can verify via bitcoin address if you're dealing with a hacker who is known to give data back.)

For (2), could you also find a way to get the FBI to release a statement saying you are trustworthy?

The hacker actually has incentive to fulfill their end of the bargain. If they didn't, the victim might go public with this, and then no one would ever pay the ransom.

The hacker wants to be trustworthy here so that new victims will be more likely to pay the ransom because they believe they will actually get their data back.

Maybe try joining a Fantasy Football league with some of your friends? Doing this will greatly improve your enjoyment because you will be invested in the performance of the players / teams on your fantasy team.