logancg

And, consequently, this may have serious implications for language models that use the Scots Wikipedia corpus. [0]

[0] https://twitter.com/r_speer/status/1298297872228786176

We do. If you’re interested in the architecture particulars (I.e. which layer types, layer shapes, layer parameters, activation functions, etc) you’ll commonly see these in deep learning papers as a directed acyclic graph with layers represented by a collection of vertices or rectangles. Other architecture specifics are sometimes noted with text alongside layers. We also have more general representations of deep models which look like, and sometimes are exactly, Probabilistic Graphical Models. PGMs have their own formal language which makes it quite easy to write complex models describing a joint distribution very simply.

However, this is different than interpreting how a deep model works. “Interpreting” is an overloaded and poorly-defined term of active research. The sense I get from my friends in the interpretability/explainability research world is that despite the buzz, there is no common definition that lends to a clear set of requirements for an acceptable interpretation of a neural network.

I'm econ undergrad -> DS -> Machine learning. Econ is very useful for data science if you focus on the right subjects: statistics, math, and experimental design. You get all the hard skills you need to interact with data that a statistician or computer scientist gets, with the (significant, unique) benefit of learning how to ask the right question or design the right experiment given what is likely a messy, weird, social scientific question.

On the other hand, if you don't do any quantitative, empirical, or experimental economics -- i.e. you only do theory or political econ -- then you won't pick up these skills (as much).

How do you come to the conclusion that ML is a buzzword here? It's natural to publicize interesting new research that grounds a field -- as statisticial learning theory does for machine learning. Historically, Ben-David and co-authors have produced foundational work on the implications of SLT in ML.

Deep Graph Library (DGL) [1] came out in November and I've heard good things. It looks easy and intuitive.

[1] https://github.com/dmlc/dgl

For more, I highly recommend looking at the accepted papers from the NeurIPS 2018 Relational Representation Learning workshop. [1] I really enjoyed the workshop and I hear workshops tend to represent a (rough) frontier of the subfield.

[1] https://r2learning.github.io/

There are a lot of talented ML researchers in China. This is a product of (a) the government and major companies (i.e. BAT) investing heavily in fundamental ML research (b) the population size (c) a long tradition of STEM-focused education in China. So, it's not surprising that would be the case.

The interesting questions are if China is uniquely focused on deep learning over other ML techniques, and Chinese research compares in terms of quality. Anecdotally (speaking as a researcher in the field) papers from Chinese institutions seem disproportionately focused on deep learning (whereas, for example, the UK does great work in Bayesian ML and the US does disproprotionately well in NLP). I'm not a deep learning researcher so I can't judge the technical merit, but I was just at NeurIPS in Montreal, and I saw about equal representation of Chinese institutions as South Korean ones. South Korea, with ~1/25 the population, punches way above its weight per capita.

I'm curious: if you think so much about climate change and its effects, and even plan your actions and location around it, have you considered directing your energy to combatting climate change with your skills on a systemic level? (For example, building sustainability companies, being part of climate activism, earning-to-donate?)

I'm in a similar spot but optimistic about systemic action.

Sounds interesting. The two big barriers to this system are:

1. Political buy-in to set a cost of carbon emission 2. Economic viability of removal (often referred to as carbon capture and storage, or CCS)

1 is non-trivial, but happening. Canada has a provincial carbon tax measure which requires setting a price. A price is also defined in the European cap-and-trade system.

2 is also non-trivial, as the economics are pretty bad right now. However, CCS costs are projected to decrease rapidly. [0]

Right now, integrating CCS is on average more expensive than purchasing carbon credits. Carbon offsetting companies are, to your point, an example of private enterprise filling a gap.

[0] http://www.ccsassociation.org/why-ccs/affordability/

This is a great idea Ben, and I appreciate the work you do. Do you see Kaggle datasets as a tool to encourage better data formatting, or are you also thinking about building tools for automatically visualizing, cleaning, and organising data?

The link at the bottom should be emphasized: https://github.com/awesomedata/awesome-public-datasets

It is a very expansive collection of datasets, some well-prepped for ML and most not (which is part of the fun of it, anyways).

Totally agreed. For the thought experiment, make the super liberal assumption that most value was produced and realized in the past year (or a single year), and it becomes comparable. Otherwise, the inefficiency is a hard lower limit. So in the extreme case, the lower bound is 5x inefficiency. If we assumed a value equivalent to 1-5% of market cap is realized every year, it becomes 100x to 500x inefficient.

A thought experiment: how much energy should Bitcoin use?

Bitcoin's mkt cap is ~$140B [0]. The world's GDP is ~$127T [1]. If we assume: 1) BTC market cap is representative of its economic output in accounting terms, or is at least an upper limit 2) Energy use by product should be proportional to its output

Then at 0.11% of world output, Bitcoin is at least ~5x more energy intensive per unit of value than the average product.

(This is obviously not wholly accurate. For one, market cap != annual value. If accounted for, that might make Bitcoin several orders of magnitude less efficient. And assumption 2 is probably a linear approximation to a highly nonlinear relationship. But I propose this as a fun thought experiment that questions the energy-value relationship.)

[0] https://coinmarketcap.com/currencies/bitcoin/ [1] https://www.cia.gov/library/publications/the-world-factbook/...

This was awesome. Thank you! It’s incredible that it only takes ~70 people’s lives, end to end, to get back to the invention of cuneiform.

Imagine all those people in a room. You could fit them in a small banquet hall. And between the oldest and youngest is the difference between the invention of writing and the utterly complex, global economic flow that we’ve networked ourselves into today.

I think Cassidy would agree with you. I think it's strategic wording to sell his mission and motivation better.

It also potentially engages people who continue to think it's a subject of belief. That may provide an opportunity to educate them.

Not in SF, but just wanted to pass on that I took a look through your personal site and really enjoy it. Best of luck with Impossible Labs!

This would be a fantastic idea.

SEWTHA made me see the world in a completely new way. When I finished it, I started seeing everything I use, see, or have as an energy process. I felt like I learned a first-principles toolkit I could use to break down anything in terms of energy.

I highly recommend this book to everyone – especially if you're serious about thinking about climate change. No BS, no platitudes – just energy from a practical physics perspective.

Updating it today could make it an even more important educational tool for thinking about climate change.

Fantastic, thank you. I didn't know Robinson had a (deliberate? unlike the Mars trilogy) cli-fi novel.

Aye – the list is great. I always admire personal recommendations! Will take a look at Bacigalupi.