jing

No it can’t. That’s d to d

In practice the functions just need to be piecewise differentiable. The RELU is the canonical example for deep learning. At kinks a subderivative is used.

Why not just get the 7702p?

Actually I'm pretty sure the Rome numbers are for double precision whereas most numbers quoted for GPUs are for single precision or less, making Rome's 3.4tf even more impressive.

Particularly since it would not be unreasonable to assume that the "mi" in mimalloc is incorrectly pronounced like the "mi" in Microsoft.

Yep. And Mellanox does it too fwiw -

https://community.mellanox.com/s/article/vma-improves-redis-...

I've got the 9550 and run Ubuntu on it. No issues whatsoever and I do very compute-intensive work on it.

One thing that I've found is very important is to clean out the fans often, otherwise dust builds up and prevents cooling. Just unscrew the plate on the underside of the laptop and blow / brush / pick out the dust that's built up (both the fan intake and the fan outlet). Doing this every few months has been a game changer for my 9550.

I think this resource could be helpful:

http://colah.github.io/posts/2015-08-Understanding-LSTMs/

Essentially, RNNs and feed forward networks are very similar - RNNs are just "unrolled through time" and every timestep shares the same weights. The activations are slightly different as well, but the core concept is the same as feed forward networks; it's not a completely different concept or idea.

Note that it's not just the loss function. It's the loss function combined with a very specific problem formulation - namely a neural network with only linear activations (equivalent to a 0-layer network). Once you go to non-linear layers or a different loss it's no longer solved analytically.

I do see a lot of people writing tutorials like OP's. See for example:

https://towardsdatascience.com/linear-regression-using-gradi...

The existence of these articles should not be taken as an indication of best practice. They often have the goal of teaching SGD in a simplified setting, not teaching best practice for LLS. I suppose only nice thing about using TF / SGD for such a simple problem is that you now have starting point for solving more complex problems (RELU activation, cross-entropy loss, more layers, etc.).

A few other points as to why you would never SGD for LLS:

1) it's always way slower than the closed form matrix solutions

2) if you're doing SGD instead of just GD, there's noise in which "rows" are in a given batch - as a result, repeated runs may not converge to exactly the same final weights. This never happens with the analytical solution which always gets exactly the same result.

3) if you're doing this as part of a data science pipeline which is likely the case in the real world, you'll likely want to do some cross-validation. In the SGD case you have to recompute the entire solution for each fold whereas in the LLS case you can immediately compute CVs once you've calculated the initial XTX / XTYs. This makes the process of using LLS even faster than SGD.

I find it hard to believe that SGD would be faster than the closed form solutions for linear regression (gels, gelsd etc.). The closed-form solutions give a lot of other benefits in practical settings as well which makes them more likely to be used if possible. SGD + related optimizers give benefits with non-convex or non-analytical loss functions or with non-linear layers / more than one layer.

The Xeons support much more RAM.

Are physics engines not yet accurate enough to enable "virtual" pre-training / full training of the networks, lighting conditions, etc? If they are, exclusively using physical robots seems somewhat inefficient.

DAC is an order of magnitude cheaper than optical and might be fine for their use based on their comments of cat6 v cat6a and "short runs in their lab".

Also, if they are using two or three NICs per box, you'd need to factor in the cost of two / three of the X540's vs. a single 40gbe connection and you'd need to double the cost-per-port of the switch vs the 40gbe option.

Some claim that this is part of a broader trend of "generational theft" from the young to the old and have some interesting slides to demonstrate their thesis:

http://online.wsj.com/news/articles/SB1000142412788732348570...

http://vimeo.com/65731171#t=11m55s

The pairing idea isn't new although it does seem to be very successful - there's a company called Thinkful (http://www.thinkful.com) that does online learning by pairing a mentor who gets paid with every student who signs up and pays for a course. They apparently have a very good amount of traction with thousands of paying students. Should be interesting to see how askadev's free model compares.

ZFS may have been, but not the 128 bit support. I mentioned that there are "admittedly much more relevant features" which are in fact what smugmug likes:

"ZFS is the most amazing filesystem I’ve ever come across. Integrated volume management. Copy-on-write. Transactional. End-to-end data integrity. On-the-fly corruption detection and repair. Robust checksums. No RAID-5 write hole. Snapshots. Clones (writable snapshots). Dynamic striping. Open source software."

http://bit.ly/bg2xlf

I never disputed ZFS being a good file system. I claimed that 128 bit support wasn't a key feature and was in fact a potential waste of time.

Yes, this is theoretically cool and all, but when I hear something like:

"Let's start with the easy one: how do we know it's necessary? Some customers already have datasets on the order of a petabyte, or 250 bytes. Thus the 64-bit capacity limit of 264 bytes is only 14 doublings away."

It raises a huge red flag from a business point of view. In 2004 "some customers" had datasets that would cause issues sometime around 2015 - a problem that ZFS purported to solve (among admittedly much more relavant features). In planning for a future that was possibly ten years away for "some customers" instead of focusing on what was relevant to the larger customer base, Sun managed to continue their slow profitability death march until Oracle finally snatched them up this year.

Something to keep in mind when you decide to add "cool theoretical feature X" or "unparalleled scalability" to your four month old startup...

A recent HN post which may be helpful for you:

http://news.ycombinator.com/item?id=1752139

Here's a few I've really liked that have a broader subject matter than only what you asked for. I figure that if you like to read you'd like these books.

--- NON-FICTION ---

Darkness Visible

A Mathemematician's Apology

Omnivore's Dilemma

Fermat's Enigma

The Code Book

Born to Run

Judgement of Paris

Siddhartha

Killing Pablo

The Victorian Internet

Lone Survivor

I Hope They Serve Beer in Hell

Surely You're Joking Mr. Feynman

Papillon

Into Thin Air

Born on a Blue Day

Touching the Void

Got Fight

The Conquest of Happiness

--- FICTION ---

Switch Bitch

Lost Horizon

The Cuckoo's Egg

The Simlarillion

The Time Traveler's Wife

The Alchemist

The Count of Monte Cristo

A Canticle for Leibowitz

Neverwhere

My Uncle Oswald

Stories of Your Life and Others

Invisible Cities

Life of Pi

Atlas Shrugged

--- WALL STREET / FINANCE ---

When Genius Failed

Liar's Poker

Buffett (Lowenstein)

Bull!

Black-Scholes and Beyond (textbook)

Fixed Income Securities (textbook)

The Road to Serfdom

The Smartest Guys in the Room

Fortune's Formula

The Fourth Turning

Inside the House of Money

--- ENTREPRENEURSHIP / COMPUTERS (non-textbook) ---

Masters of Doom

It's Not About the Bike

Dot. Bomb

Founders at Work

The PayPal Wars

Coders at Work

The Mousedriver Chronicles

Leaving Microsoft to Change the World

The Soul of a New Machine

Startup

The Hypomanic Edge

Small Giants

Small is the New Big

Four Hour Work Week

The Boy who Harnessed the Wind

Four Steps to the Epiphany

Pour your Heart into it

Beyond Software Architecture

Setting the Table

--- DESIGN / UX ---

On Writing

The Design of Everyday Things

Understanding Comics

The Elements of Typographic Style

Don't Make me Think!

A Pattern Language

--- PROGRAMMING ---

Programming Pearls

Effective Java

The Algorithm Design Manual

Advanced Programming in the UNIX Environment

The Pragmatic Programmer

Refactoring

Java Concurrency in Practice

Java Puzzlers

Expert C Programming

--- COOKING ---

The Secrets of Cooking Revealed

Cooking for Geeks

The Bread Baker's Apprentice

The Brewmaster's Table

--- MISC ---

Robot Building for Beginners

Mountaineering - The Freedom of the Hills

A First Course in Coding Theory

The Computational Beauty of Nature

Total Control

The Ashley Book of Knots

Make: Electronics

I'm curious - you say,

"A do-it-yourself wedding invitation kit costs $45, while professional wedding invitations are hundreds or thousands of dollars."

How much of that "hundreds or thousands" is in the actual printing vs. the design + margin?

If it was profitable to harvest / clean up the spill, wouldn't BP be doing it already? For that matter, wouldn't all of its competitors such as Exxon be doing the same as well?

Exactly. Unfortunately, since Palantir is in the enterprise space, few people on HN immediately know of the how 1) cool and 2) difficult the problems and solutions in the space are. And of course, this post hast immediately degenerated into Java vs. X as to be expected.

I don't mean to sound rude, but your comment shows how little you know understand about securities, investing, risk, and reward.

Of course nobody comes in asking for "crappy mortgages". Pension funds, endowments, etc. have certain investing goals. These are typically set based on future obligations to their investors. Let's say for example that you are a portfolio manager at one of these institutions whose job is to achieve a 7.5% annual return because teachers, firemen, and police officers actually expect to get paid in retirement. Well, the first thing to note is that this isn't easy. The investors diversify their investments among stocks, hedge funds, fixed income (bonds), and yes - mortgage backed securities, based on where they think the best relative risk-reward is.

Along the entire spectrum of assets, investors expect a much higher yield for what the market perceives to be a crappy asset as compared to a good asset. You can loan money to the US government at a 1% yield or you can load money to high grade corporate companies in the US for 5%. Why would you choose the "safe" US government vs. the relatively "crappy" high grade corporate securities? Well, it all depends on what yield (interest rate) you seek versus what risk you are willing to take.

The investors who bought these securities were looking for a specific risk (and for the interest rate that they'd get for taking it) which was detailed very clearly in the offering memorandum (http://bit.ly/9zwBqB). Some specific lines I'll cite are as follows:

-----Prior to making an investment decision, prospective investors should ensure that they have sufficient knowledge, experience and access to professional advisors to make their own legal, tax, accounting and financial evaluation of the merits and risks of investment in the Notes and should carefully consider the nature of the Notes, the matters set forth elsewhere in this Offering Circular and the extent of their exposure to the risks described in "Risk Factors". -----Concentration Risk. The concentration of the Reference Obligations in the Reference Portfolio in any one particular type of Structured Product Security subjects the Notes to a greater degree of risk with respect to credit defaults within such type of Structured Product Security. Investors should review the list of Reference Obligations set forth herein and conduct their own investigation and analysis with regard to each Reference Obligation. See "The Credit Default Swap—The Reference Portfolio". ----- The Collateral Securities may include Commercial Commercial Mortgage-Backed Securities. Mortgage-Backed Securities. CMBS bear various risks, including credit, market, interest rate, structural and legal risks. CMBS are securities backed by obligations (including certificates of participation in obligations) that are principally secured by mortgages on real property or interests therein having a multifamily or commercial use, such as regional malls, other retail space, office buildings, industrial or warehouse properties, hotels, rental apartments, self-storage, nursing homes and senior living centers. Risks affecting real estate investments include general economic conditions, the condition of financial markets, political events, developments or trends in any particular industry and changes in prevailing interest rates. The cyclicality and leverage associated with real estate-related investments have historically resulted in periods, including significant periods, of adverse performance, including performance that may be materially more adverse than the performance associated with other investments. In addition, commercial mortgage loans generally lack standardized terms, tend to have shorter maturities than residential mortgage loans and may provide for the payment of all or substantially all of the principal only at maturity. Additional risks may be presented by the type and use of a particular commercial property. For instance, commercial properties that operate as hospitals and nursing homes may present special risks to lenders due to the significant governmental regulation of the ownership, operation, maintenance and financing of health care institutions. Hotel and motel properties are often operated pursuant to franchise, management or operating agreements which may be terminable by the franchisor or operator; and the transferability of a hotel's operating, liquor and other licenses upon a transfer of the hotel, whether through purchase or foreclosure, is subject to local law requirements. All of these factors increase the risks involved with commercial real estate lending. Commercial lending is generally viewed as exposing a lender to a greater risk of loss than residential one-to-four family lending since it typically involves larger loans to a single borrower than residential one-to-four family lending. Commercial mortgage lenders typically look to the debt service coverage ratio of a loan secured by income-producing property as an important measure of the risk of default on such a loan. Commercial property values and net operating income are subject to volatility, and net operating income may be sufficient or insufficient to cover debt service on the related mortgage loan at any given time. The repayment of loans secured by income-producing properties is typically dependent upon the successful operation of the related real estate project rather than upon the liquidation value of the underlying real 26 estate. Furthermore, the net operating income from and value of any commercial property may be adversely affected by risks generally incident to interests in real property, including events which the borrower or manager of the property, or the issuer or servicer of the related issuance of commercial mortgage-backed securities, may be unable to predict or control, such as changes in general or local economic conditions and/or specific industry segments; declines in real estate values; declines in rental or occupancy rates; increases in interest rates, real estate tax rates and other operating expenses; changes in governmental rules, regulations and fiscal policies; acts of God; and social unrest and civil disturbances. The value of commercial real estate is also subject to a number of laws, such as laws regarding environmental clean-up and limitations on remedies imposed by bankruptcy laws and state laws regarding foreclosures and rights of redemption. Any decrease in income or value of the commercial real estate underlying an issue of CMBS could result in cash flow delays and losses on the related issue of CMBS. A commercial property may not readily be converted to an alternative use in the event that the operation of such commercial property for its original purpose becomes unprofitable. In such cases, the conversion of the commercial property to an alternative use would generally require substantial capital expenditures. Thus, if the borrower becomes unable to meet its obligations under the related commercial mortgage loan, the liquidation value of any such commercial property may be substantially less, relative to the amount outstanding on the related commercial mortgage loan, than would be the case if such commercial property were readily adaptable to other uses. The exercise of remedies and successful realization of liquidation proceeds may be highly dependent on the performance of CMBS servicers or special servicers, of which there may be a limited number and which may have conflicts of interest in any given situation. The failure of the performance of such CMBS servicers or special servicers could result in cash flow delays and losses on the related issue of CMBS. At any one time, a portfolio of CMBS may be backed by commercial mortgage loans with disproportionately large aggregate principal amounts secured by properties in only a few states or regions. As a result, the commercial mortgage loans may be more susceptible to geographic risks relating to such areas, such as adverse economic conditions, adverse events affecting industries located in such areas and natural hazards affecting such areas, than would be the case for a pool of mortgage loans having more diverse property locations. Mortgage loans underlying a CMBS issue may provide for no amortization of principal or may provide for amortization based on a schedule substantially longer than the maturity of the mortgage loan, resulting in a "balloon" payment due at maturity. If the underlying mortgage borrower experiences business problems, or other factors limit refinancing alternatives, such balloon payment mortgages are likely to experience payment delays or even default. As a result, the related issue of CMBS could experience delays in cash flow and losses. In addition, interest payments on CMBS may be subject to an available funds-cap and/or a weighted average coupon cap (which cap will, in each case, have the practical effect of deferring part or all of such interest payments) if interest rate rises substantially. Residential Mortgage-Backed Securities. The Reference Obligations will include and the Collateral Securities may include Residential Mortgage-Backed Securities. RMBS bear various risks, including credit, market, interest rate, structural and legal risks. RMBS represent interests in pools of residential mortgage loans secured by one- to four-family residential mortgage loans. Such loans may be prepaid at any time. Residential mortgage loans are obligations of the borrowers thereunder only and are not typically insured or guaranteed by any other person or entity, although such loans may be securitized by Agencies and the securities issued are guaranteed. The rate of defaults and losses on residential mortgage loans will be affected by a number of factors, including general economic conditions and those in the area where the related mortgaged property is located, the borrower's equity in the mortgaged property and the financial circumstances of the borrower. If a 27 residential mortgage loan is in default, foreclosure of such residential mortgage loan may be a lengthy and difficult process, and may involve significant expenses. Furthermore, the market for defaulted residential mortgage loans or foreclosed properties may be very limited. At any one time, a portfolio of RMBS may be backed by residential mortgage loans with disproportionately large aggregate principal amounts secured by properties in only a few states or regions. As a result, the residential mortgage loans may be more susceptible to geographic risks relating to such areas, such as adverse economic conditions, adverse events affecting industries located in such areas and natural hazards affecting such areas, than would be the case for a pool of mortgage loans having more diverse property locations. In addition, the residential mortgage loans may include so-called "jumbo" mortgage loans, having original principal balances that are higher than is generally the case for residential mortgage loans. As a result, such portfolio of RMBS may experience increased losses. Each underlying residential mortgage loan in an issue of RMBS may have a balloon payment due on its maturity date. Balloon residential mortgage loans involve a greater risk to a lender than self- amortizing loans, because the ability of a borrower to pay such amount will normally depend on its ability to obtain refinancing of the related mortgage loan or sell the related mortgaged property at a price sufficient to permit the borrower to make the balloon payment, which will depend on a number of factors prevailing at the time such refinancing or sale is required, including, without limitation, the strength of the residential real estate markets, tax laws, the financial situation and operating history of the underlying property, interest rates and general economic conditions. If the borrower is unable to make such balloon payment, the related issue of RMBS may experience losses. In addition, interest payments on RMBS may be subject to an available funds-cap and/or a weighted average coupon cap (which cap will, in each case, have the practical effect of deferring part or all of such interest payments) if interest rate rises substantially.

Note also Schedule A which LISTS EVERY SECURITY IN THE CDO. With a document like this (which is required), it is very difficult for someone to make a credible claim that GS did not forward appropriate info as to to what the buyer was actually investing in. These investors took a risk. They knew they were taking a risk and they knew the risk they were taking - these weren't mom and pop retail investors, but some of the most professional in the business. The simple fact is that they did a poor job of assessing risk versus reward and lost a ton of money as a result.

Again, when you go into a McDonalds, nobody says, "please give me unhealthy food", but instead they say, "give me a double quarter-pounder with cheese". This person is taking the risk of heart disease versus the "reward" of enjoying a calorific sandwich. They know the risk they're taking and have free will to take that risk. It was the same with these investors, only now it's as if they're suing McDonalds after having a heart attack.

Does that mean that we should throw the CEO of McDonald's in jail too because he knowingly sells junk food to customers? No, we shouldn't. And the reason is that people are free to buy whatever food they want. If they want shitty fast food, that's up to them. Similarly, the clients of Goldman were free to buy or sell whatever securities they want - Goldman's job is specifically not to make a claim as to whether something is "good" or "bad" especially when it's not very clear cut. Their job is instead to be ready to sell to or buy from clients who come in with requests. If these securities had worked out well, GS would be considered geniuses - hindsight is always 20/20, especially in the blogosphere.

This article is so incorrect that it's a joke. I guess there's a reason why Michael Lewis and Maria Bartiromo don't talk about the differences between splice() and sendfile()...

Charity isn't always about being as "effective" as possible. People there are literally dying from not having food, water, and medical care. I hate to say this on hacker news, but not everything is a numbers game. Haiti is in a humanitarian crisis right now. Yes, you could logically say to yourself that X dollars would be more "effective" elsewhere and that Haiti will never become a flourishing country, but that largely misses the point. It also presumes that you have the balls to say who should live and who should die. People there need help. Now.

A good charity I donated to is Partners in Health which was founded in Haiti (by an American) and has always been in Haiti. You can read more about them in "Mountains Beyond Mountains" which coincidentally and topically was written by Tracy Kidder who also wrote "Soul of a new Machine".