The name for the product category you're broadly describing is "smart beta." This exists in quantity. "SPY but without overvalued stocks in it", etc etc.

The reason the exact product you want doesn't exist is because it will have the same returns as SPY but cost 20X as much due to lower economies of scale, broadly because people don't want it.

> but cost 20X as much due to lower economies of scale, broadly because people don't want it.

For any existing fund manager who tracks an index (i.e. Vanguard), this shouldn't be too difficult or expensive. They're already buying/selling a TON of SP500 shares at the close of trading day. Now they just need to subtract some of those buys/sell for the overvalued companies that people do not want. Am I missing something?

EDIT: Oh, I guess the (SEC?) would require a different fund management and prospectus for every index "smart beta" permutation, and therefore wouldn't be realistic.

Whenever I feel the need to use the adverb "just" when talking about something I think someone else should do, it usually means my understanding of the situation is the tip of an iceberg I can't fully see.

Oh man, reading up on that, smart betas could exercise enormous power over the market if they became popular enough.

Imagine if someone was able to immediately exclude you from 20% of investment capital on the market.

Just buy SPY with 95% of your money and then allocate 5% to put options on the losers. if they don’t work out, it was just a hedge. if it did, you had your cake and ate it too.

> allocate 5% to put options on the losers

How do you do it withing the frame I described the problem (cost effectively and without increasing risk) for relatively small investor?

In terms of making it a fund, I don't see how it can done effectively. But if you're an individual investor that knows what you're doing (ie have experience with options, etc), it's fairly straightforward to just allocate 5% of your portfolio to liquid long-dated put options.

There's still some complication here. Do you carry options to expiry or trade them before if their value skyrockets? How often do you rebalance?

Picking losers is just as hard as picking winners. And you know this otherwise you would just short those 20 picks of yours.

So what would happen is that investors who don't pick outperform those who do.

Is picking the rotten apples at the store just as hard as picking the very best?

If you made an index of penny stocks, do you think markets are efficient enough this would be basically as good as an S&P 500 fund? What if you made two indexes, one of penny stocks that pay dividends, and one of those that do not?

I've done this experiment years ago with a stock market simulator utilizing real market data, and it suggested that, no, there is not a price at which total shit is a good investment, or if there is, it's not the market price. Therefore it follows that excluding that shit from your index is potentially worthwhile.

Penny stocks are usually traded OTC and aren't in any index fund I've seen anyway.

Picking losers is just as hard as picking winners. If you think you know of a loser, just short it. You'll be hard-pressed though, because other people, likely with more info than you, have already done this, and the price has already come down.

Why do you think shorting a stock is (uniformly) just as easy as going long? I don't know where you people get the idea it's symmetric. And the difficulty of shorting a stock varies massively from one company to another, which is a not irrelevant aspect of it. You can't (legally) print unlimited shares yourself to sell short.

If you could you make money shorting rotten apples at the store, visibly rotten apples would no longer exist and you would not be able pick them out at the store.

The EMH applies to stocks, not apples. Your analogy is nonsensical.

Wait, I thought the efficient market position was that the "rotten" companies were all priced correctly. But you seem to be saying there aren't any.

Some recent research showed that the market doesn't even take into account the contents of SEC filings which seems like the minimum for a semblance of efficiency. I have to say, if you are going to take the position that there are no worthless public companies out there, you ought to take a sizeable sample of 10-Ks and just glance at the business description of each.

Also...I'm not an economist or expert, but what I think you are saying is that the EMH only applies even in principle to things you can sell short and that sounds wrong to me.

What does picking apples have to do with trading financial assets?

Analogies are useful to illustrate an idea, but they lack any value when establishing a chain of reasoning (i.e. providing proof that a statement is true).

Analogies always match up in a limited number of ways, so saying that the things in an analogy are different in some ways is not a specific criticism.

Apples are sold in markets. Most of us are not experts on apples any more than stocks. Yet we use what we do know to select from what is available, rather than having faith in market efficiency. Because there is information that the non-expert can easily perceive.

How can that be wrong with stocks? I'm getting the impression people are attributing the difference to the possibility of shorting stocks, but I don't think that is necessary in theory for efficient markets, and selling short is clearly not just as easy as going long, so the symmetry isn't there anyway.

It's easy to identify patterns that let you consistently (consistently enough to make more than you lose) pick companies that will win or lose a little. That's why that task is now automated. Picking the companies that will win and lose by orders of magnitude is hard to do consistently.

You couldn't implement this in a fund structure. It would need to be a separately managed account or something customized to each individual.

That what I was thinking too.

I think it could be implemented as SP520 index fund with a low cost swap sold separately against the fund.

You’re not going to be able to trade swaps with a party that doesn’t have an ISDA.

I think with a sufficient number of investors, this fund would correlate very strongly with the S&P 500 itself. So it should be relatively cheap for the issuer to simply buy the S&P 500 and enter into a swap agreement to cover any difference.

Until it's not and the fund manager goes bankrupt...

No, just like with synthetic ETFs, there would be a number of swap counterparties that would have to cover the difference.

If some of them fail (which is very unlikely), the damage would be limited to the difference between the actual portfolio and the S&P 500.

Wouldn't the effect of, say 15 of your picks underperforming(good for you) be dwarfed by the other 480 ones performing the same? Couldn't you simulate this fund by buying s&p index fund and buying puts on your 15 picks?

It's actually the other way around, at least historically speaking. Typically a very small number of names account for the lions share of the index return.

Just to add an important note, this is true but it’s different names for different people depending on when they enter & exit the index.