phaefele

I found this book very helpful in several difficult conversations over the years - https://www.amazon.com/Difficult-Conversations-Discuss-What-...

It has quite practical advice as well as a framework helping one navigate.

My favourite cell documentary is this. Has more information about the various transport mechanisms in the cell itself. Simply awesome - have watched several times to understand better. See https://www.youtube.com/watch?v=FzcTgrxMzZk

In the qualia / physical world distinction, I wonder if there are good reasons we see things the way we do.

Take colors: colors map to wavelengths of light. I wonder if there is a good reason for our perceptions of color to have red be lower frequency and indigo be higher frequency. I guess what I'm wondering is if our brain mapped these differently, it would be suboptimal in some way - perhaps the 'mixing' of colors would work out less 'well' (e.g. red+yellow=orange wouldn't work in the new layout as well.) If so, then perhaps one could use evolutionary selection pressures as an explanation to lead us to the qualia we have.

Or have I perhaps just missed the whole point?

Thanks for doing this Peter. I am currently thinking of accepting a US computer programming job (part in US, part in Canada) and am thinking of using a TN visa to travel back and forth (1 week per month in US, 3 weeks in Canada.) Do I need to be concerned about what might occur if Donald Trump et al decide to drop NAFTA?

Did you try an inversion tableÉ I went from shuffling around with lower back pain to fixed in about 6 weeks.

Most of those floor traders is running a small business (2/3 guys/girls) based on past contacts and trader know-how (such as it is - some better, some worse.) It's really more of a 'bazaar' for small, likely well connected, trading firms rather than anything else. It has been 'hard' for them in recent years as order flow has moved to the largest, too-big-to-fail brokers (GS, JMP etc). But certainly the NYSE gets a lot out of the media coverage, but the brokers themselves are not owned by the NYSE.

I'd be happy to pay my $45.71 per person per year. The BBC makes amazing shows - I don't see that Canada can't do the same. As Canadians, if we believe that we have something to say to the world, we need to fund saying it.

Am reading his book called Breach of Trust. An easy read, but very compelling. I'd vote for him for sure.

If you haven't yet, check out Noam Chomsky - he holds the US up to their supposed 'good guy' status - fascinating. Andrew Bacevich, a retired Colonel, also has lots of interesting things to say on this. See https://www.youtube.com/watch?v=iFpXFFPEZ9w

'Between 1946 and 1958, the equivalent of more than two hundred million tons of TNT was detonated (on the inhabited Marshall Islands by the US Military) — like a Hiroshima every day for almost forty years.'

...

The local language grew full of horrible expressions for birth defects: “jellyfish” (babies born without bones), “grapes” (spontaneously aborted clumps of tissue), “turtles,” “octopuses,” “apples,” “devils.” The Crossroads tests were the beginning of one of the more disturbing American nuclear legacies—a trade of flesh for knowledge.

Has anyone looked at SciDB (http://www.paradigm4.com/) as a replacement for KDB. Seems like it would fit the bill and has a community edition. I do see some usage in the financial markets.

Most definitely - 'Additionally, DeepSpark, during its code scanning, is capable of generating code for new components of Spark'

This is great. Wasn't too clear on exactly how this is used from the documents (e.g. do I just run 'python') - didn't seem that there where any 'imports' either.

Found this example which seems a little clearer: https://pypi.python.org/pypi/quasiquotes/0.2.0

The one key part this seems to be missing is a backtesting framework. Did you look at https://www.quantopian.com/?

Gladiator, Baraka, My Neighborhood Totoro, Holy Motors, 12 Monkeys

Did you consider Summingbird? Seems like a lot of what you are doing might have been simplified by using Summingbird rather than building separate speed and batch layers in Hadoop / Spark.

Content free - there is really no information in this article outside of the fact they say Norway's Sovereign fund use "Block Trades" which is really what dark-pools are supposed to be all about. Dark pools and block networks have been around for at least the last 5 years, and introduce their own set of issues (not getting your orders filled, showing Goldman Sachs your order on their promise that they won't don't anything with that information :-)

After building a large Hadoop simulation environment and contemplating how to use the same codebase in a streaming fashion, Summingbird seems like an exciting answer. I am concerned that I don't see much discussion around its use online and wonder what its adoption is like. Are people using it? I wonder if the usage of new terms like "moniod" scares people away.

We have run into issues with VPC being tied to a specific zone and the availability in that zone.

Yes - this is what we are doing. With this 4x cost differential however, we calculated that it takes only 6-8 months to have the hardware pay for itself, so if you plan to be in business in that time frame you are better of buying.

Understood that Amazon needs to make a profit, but these are still dollars that I need to pay for, that I could be feasibly be using for something else (hiring great developers perhaps.)

I think the advantage of EC2 is exactly fast scaling, and an easy sell to management (no big upfront.) But as we started using it at size and consistently, it is just cheaper to run our own hardware as the blog attests to.

We do all the racking and cabling using "remote hands" directed from our SAs in India, and so far for us this has not been the complex part. Getting the Hadoop configured and our software running efficiently if several orders of magnitude harder, and EC2 doesn't I think help here. If anything it hinders as we are dealing with virtual hardware. There have been several posts about "lemon" EC2 instances and how you should test your instance before using it.

Some of the difference comes about due to the number of cores in the machines we purchased - they are 16 core machines (each E5-2650 has 8 cores.)

In terms of storage, we are utilizing HDFS with commodity 3TB drives, thus no specialized storage from EMC (or the like.)