Government departments notoriously have serious technical debt, so while they might entitled to demand this kind of data, I sincerely doubt they are doing it with anything resembling modern data analysis. These orgs aren't going to have working data lakes, spark clusters or data warehouses on cloud infrastructure. They maybe have legacy SQL databases being pulled into spreadsheets.
So, while their data access might be better theoretically, I doubt they truly are able to even remotely analyze the data effectively. Data science in govt is notoriously poor outside of Statscan or other tech heavy departments.
The issue here has nothing to do with the size of the data. A grocery store will have 100,000 products at most, probably sets prices about weekly and maybe you have ten years of data. The market is an oligopoly so you have 10-15 store chains. That fits in memory.
The issue is not at all the mechanics of running the analysis but what analysis you run. Or don’t run bc as I said there is no easy way to just “identify collusion” in data like this or every agency would already be doing it
The appropriate incentive is lacking.
Just like how the SEC whistleblower laws will awarded part of the fines to those who they reported.
2. Collusion is a strong word. It hints at breaking the law, willfull action, etc. Looking at the price of a competitor and acting upon that, and internal information, is not collusion. It is completely legal and economically rational behaviour. There are hints and information everywhere on which to act. Groceries are an oligopoly, and they are ogipolistic buyers as well. Lots of information for grocers in the price bids of their suppliers, that all groceries get in tandem.
3. I've worked as an economist at one of those government agencies. They are filled with keen PhD / MSc economists looking to bust kartels and find collusion. Geographical price dispersion models are so '00s. There is a lot of positive energy and interaction between academia and government. Actually _proving_ collusion is terribly hard. Prices that move in parallel is not collusion. Proving collusion is finding documents, emails, phone calls, tracking cars with staff members of various firms to motels , paper trails of inside information in another firm. That's collusion. And I fear that's not what happens in the grocery sector.
The motel one comes to mind as real life example of how DoJ in the US prosecuted a proper cartel. The talk was by more a broad-shouldered officer type, than an economist. Certain set of transport firms always had a interesting way of responding to auctions. Always the second best offer was quite far behind and there was a pattern to which firms did or did not reply to a bid request. End of the investigation, they could trace employees of the firms to meeting in a motel before bidding. And somebody left a paper trail.
There is the skillset of the employees, and then there is what management allows employees to work on. A decade or so ago, a Statistics Canada senior manager was interviewed and claimed as fact that they were unable to reproduce an Excel model that analyzed the % of real estate sales to recent immigrants to Canada, one that an individual CRA team (one person I suspect, it's not complicated) in Richmond BC had done a few years before.
At the same time, they were given additional budget of many millions of dollars to accomplish this task and other things. They did not accomplish the task.
> Proving collusion...
The government has tipped their hand already as far as I'm concerned. I don't expect anything would ever be proven, I just want to see as much evidence as possible to help me decide what I am going to believe - I'm certainly not going to take epistemic advice from anyone in the Canadian government.
Surrounding collusion. I can’t tell you what to believe, but keep in mind that there are a lot of reasons why correlated moves in prices can occur without communication. Lots literature in industrial organization (the field of why markets have particular ordening and how actors behave on markets) on signaling. A though experiment might be the games of poker versus bridge on explicit vs implicit signaling / coordination.
https://en.m.wikipedia.org/wiki/Closed-world_assumption
It would be fun if this fellow's initiative actually got some traction, and attention, and then out of the blue they launched a new educational section of their website to assist the public (and journalists...or not) in their thinking on such matters.
I suspect these giant retailers have a risk department, and at least some people in that department would be able to recognize that which the general public and participants in this thread lack the ability to recognize.
That article covers the initial report, not the subsequent claim that they do not have the ability to reproduce it.
> But the report was not made public until a five-year access-to-information battle concluded recently.
This demonstrates that Canadian democracy does not rule according to the will of the people - there was never a referendum on whether the government should hide the truth from citizens.
> Of the 46 houses bought in Burnaby, staff found 72 per cent were purchased by new arrivals to Vancouver who reported an average total family income of just $16,000.
Immigrate to Canada, enjoy all of its world class (at least formerly) social services, but pay no income taxes. Also not the will of the people.
Hanlon's Razor seems to be the guiding principle of journalists, and the general public in general. Gosh, I wonder if our educational curriculum (or lack of specific content) has anything to do with this state of affairs. Note also who it is that determines curriculum.
Not that prices aren't consistent across a chain, individual stores in one chain will have different prices, calibrated to local supply and demand. In many cases they'll also have local products that aren't in most of the stores in that chain. I think you have to track prices in each and every store, not just per chain.
Still, I think you're right about this fitting in memory. This isn't "big data".
This paper:
https://www.aeaweb.org/articles?id=10.1257/aer.20170116
Is the best case scenario (=easiest) for the exercise you want to do and will illustrate what is hard about the grocery case. In the paper there is:
- One product
- one clearly isolated market.
- four or five firms.
- A clear mechanism for collusion (a government price sharing website).
Even then there is no statistical or econometric test out of the box which “shows collusion”. It’s a great paper (both in my opinion and that’s why it got into AER) but there is nothing they do that says “hey here’s collusion based on this t-stat.”
The grocery case (overlapping markets, thousands of products, many firms) is vastly, vastly harder.
“ A Study of Cartel Stability: The Joint Executive Committee, 1880-1886” By Rob Porter (northwestern)
That paper is a classic and I hope porter gets the Nobel prize for the work he has done.
In that paper, we actually know for sure that there was collusion. Porter is trying to assess how hard it was to sustain collusion using the data. (One thing you are not crediting in your website is that collusion is hard to sustain bc all of the colluders have the incentive to cheat!) It takes a lot of assumptions and serious econometric chops to make the analysis credible.
That paper is 40 years old but what you’re really trying to do with your project is called “structural industrial organization.” It’s been around in the economics literature for a long time. It’s my field. It’s hard.
If you want a recent overview which still won’t answer the question the way you wanted to, get “the handbook of industrial organization” edited by hortacsu et al, or the book draft “empirical methods in industrial organization” by aguirregabiria (who is at U Toronto!) or “quantitative techniques for competition and antitrust” by Davis and Garcés.
Good luck.
Also imagine the false discovery rate of <whatever analysis you think you can do> with the thousands of products which exist in a grocery store!
“Ah look we found collusion in salty snacks between weeks 38 and 45 of 2022 in stores 1, 5 and 13!”
Even if you knew what analysis to do (and you don’t) you are going to end up crying wolf a million times and solving no problem at all.
Again - a worthy effort but the methods just don’t exist.
Investigators work on hunches just fine if you pay them enough.
Mocking the epistemic skills of others while engaging in literal soothsaying (stating facts about future events) shows how culturally conditioned North Americans have become regarding government wrong doing.
I don't expect these people to be successful in their primary goal, but an enticing second order effect is it could accidentally catalyze public interest in epistemology, logic, rhetoric, the nature of Human culture and cognition, and a variety of the other things that keep commoners locked in a sophisticated virtual reality without their awareness (in any sophisticated manner at least). Now that would make this whole game a lot more interesting, because it applies to everything.
You can publish some correlations between prices. (Remember there is no known method where you can look at a time series of prices across stores and say “that’s collusion!”). And hey if you have 10,000 - 100,000 products per store you probably have correlations as high as you want. None of that will actually show collusion or more importantly suggest a policy response.
Even very high correlation _within a category_ across stores isn’t going to do that. Say you find price movements in tomato paste across all N chains are 0.95… ok now what? There are 10,000 or 2,000 product categories so you found a high correlation in one of them what do you want to do about it?
What I want to happen is for those people to realize that they're not alone, and that the whole world is not working to put them at a disadvantage. Really, it's a rather small group of people who are setting the rest of us up to be rats in a maze with no exit. Then I want us to find those people, and go after their bottom line. Nothing violent, just like... welding the doors shut on opening day, that sort of thing.
It might be less rational than something more restrained like getting a law degree and working in the system to bring them to actual justice. But it has... a glimmer of rationality. You're no longer an insane lone wolf lashing out against an impossible situation... now you're a shitty activist, and even if you're not acting with a cohort, you know that cohort is out there. You're not stuck, you're not alone, there's a path forward--you can aspire towards being a less shitty activist.
But it requires a justifiable target, a representative of the untenable situation. A list of potential enemies that's far less than everybody. Once that list exists, I want companies to compete to not appear on that list.
Markets work without being rational, as does evolution (some predation required). This would be the same.
What an unbelievably stupid thing to advocate for, much less under the guise of non violence.
That’s all this metric will show.
It also won’t show supply chain issues that make the Canadian grocery industry look the way it does. As an example, buy 100 head of cattle in western Canada and get the beef into a store. Are you having any luck finding a place to process it?
The fact that they keep approving mergers and acquisitions in telecom, banking, and groceries despite those being some of the most comically oligopoly-esque industries within a country on earth shows just how tragically wrong you are.
If there was 2 telecom companies left standing in Canada, our competition bureau would pretend to study their desire to merge for a year before approving it, under the condition that their collection of subsidiary brands pretend to be different companies while advertising.
So, unfirable? Essentially useless and beholden to little if any performance metrics?
The agency is likely underfunded, understaffed, and disincentivized from actually fixing the issues which are so pervasive across the country that it's just business as usual for most people..
Would this be the sort of analysis that measured transfer entropy or granger causality?
You can do economic analysis of pricing data, and especially the interaction/correlation of multiple streams of prices across time ‘’’
The item in that quote is significantly, significantly harder than the author is giving it credit for. Canada has a competition policy agency. They are (almost surely) entitled to demand data from firms as part of an investigation. Their data will be better than the data here.
You can rarely if ever prove these cases on the basis of data analysis alone. (Indeed if you could the world’s antitrust agencies would just monitor such data! And anyone engaging in collusion would attack the monitoring adversarially - not that hard to predict!)
In grocery data you will be looking at thousands and thousands of prices of different goods from different suppliers with different costs who are exposed to different shocks due to variation in the costs of their inputs or god knows what else.
Not to be negative bc the idea is nice, but this is a complete waste of time.
~ an actual antitrust guy.