meigetsu

Thank you - your comment is right and I conflated two things which are conceptually totally different.

For a given number of experiments and block of time (i.e. available samples over time), it's not useful to say that bandits have higher power / a worse FPR, bc the values are adjustable. F1 or AUC would probably be the right way to compare and it seems likely to me that bandits have better performing precision-recall curves. Basically, this is actually irrelevant to the point, and actually favors bandits.

I was totally thinking about the scenario you mentioned where the number of experiments are unconstrained and old experiments run long. Bandits will spend a lot of their bandwidth on very marginal improvements that are below the effect size cutoff that shorter fixed RCT will set. I think you can fix this with early stopping (or just stopping), so maybe it's not really an issue after all.

Thanks for helping clarify my thinking on this :)

huh, wierd - I saw this post in Aug and can't understand how I missed that. Thanks for pointing that out - it does indeed discuss it.

I have mixed feelings about using multi-armed bandit for product testing like this. Regret minimization makes sense 100% as a framework if you are testing a large inventory of things - i.e. the classic examples of showing ads or recommendations - since there might be some real opportunity cost in not showing some of the things in inventory (particularly if the inventory has a shelf life). (I'm also quite surprised they don't use thompson sampling...)

For testing product features though, I feel like there is often a high long term cost to the dev team and the regret from showing users a non-optimal treatment during the experiment is pretty minimal (the regret is usually to first order only the cost of experimental bandwidth).

The team cost comes in several subtle forms:

- in practice, bandits encourage lots of small experiments which leave behind a large surface area graveyard of code - you can mitigate this by having strict stopping points for bandit experiments

- bandits have higher statistical power, but also higher false-positive rate; false positives can be quite high cost since they cause thrash and require time to investigate if a feature that tested well does poorly in production

- you are introducing novelty effects over time as new sample groups get added in the dynamic allocation; probably nbd for most experiments, but it's complicated to correct for this if your experiment has novelty effects

- there are often cyclical time-dependent changes in the composition of users being exposed (daytime vs night time, week day vs weekend, geography bc of timezone differences); also, probably nbd for most experiments, but requires complex stratification to correct for if this is an issue

I would also say that the majority of product changes have small, but measurable effects on metrics, so I'm not sure that bandits help all that much in those cases. If there are runaway successes or failures, early stopping techniques seem like a better way to free up resources - early stopping policies can be tuned to address the experiment design problems above fairly simply.

Again, this is all for product testing. I think for recommendations and personalization, contextual bandits make lots of sense.

Any thoughts on the ultimate hacking keyboard as an option?

https://ultimatehackingkeyboard.com/

How do you convert kindle books into PDFs?

I generally agree with this, but I also think there is a legitimate use case for saying “Hello”, namely when you have a request that actually needs a fairly imminent response and if the person you are pinging isn’t responding right away, you need to move on to someone else.

If you type the full message to the first person and don’t get an immediate response, as you paste the same message to other people, the first person or few people who didn’t respond immediately might waste their time duplicating work that someone else you reached is already doing.

One question for any attorneys here - if the FTC were not investigating the 2014 hack, would there not be any charges for these alleged actions? The indictment doesn't seem to mention any statutes violated except for in connection to impeding the existing investigation.

If you could get this to run fast on GPUs you might get some converts from other APL family languages.

For anyone who has spent significant time with data and ML pipelines, read posts about Dagster. Even if you don't end up using it in your pipeline, it is full of well articulated and extremely valuable lessons about how to manage the problems and complexity of data applications. I can't recommend reading about dagster enough!

The software itself has matured significantly in the last year as well - highly recommend taking a look at this if you're building new pipelines or have a need to upgrade existing ones.

- Twitter

- Quora - remote first - https://news.ycombinator.com/item?id=23646164

- Atlassian - remote choice - https://news.ycombinator.com/item?id=24085546

Since the GPU uses the system memory, is there any advantage to using these APUs for machine learning?

GPU RAM is typically under 32GB (more commonly under 11GB) and quite expensive - for the price of a V100 you could buy 1TB of system DDR4 RAM.

I'm guessing there are disadvantages in memory bandwidth, number of GPU cores, overall FLOPS, but was curious if anyone knew how these pros/cons balance out.

I've never done pairing, but this seems like a good suggestion. I've also been looking for an excuse to try https://tuple.app