The issue here has nothing to do with the size of the data. A grocery store will have 100,000 products at most, probably sets prices about weekly and maybe you have ten years of data. The market is an oligopoly so you have 10-15 store chains. That fits in memory.
The issue is not at all the mechanics of running the analysis but what analysis you run. Or don’t run bc as I said there is no easy way to just “identify collusion” in data like this or every agency would already be doing it
The appropriate incentive is lacking.
Just like how the SEC whistleblower laws will awarded part of the fines to those who they reported.
2. Collusion is a strong word. It hints at breaking the law, willfull action, etc. Looking at the price of a competitor and acting upon that, and internal information, is not collusion. It is completely legal and economically rational behaviour. There are hints and information everywhere on which to act. Groceries are an oligopoly, and they are ogipolistic buyers as well. Lots of information for grocers in the price bids of their suppliers, that all groceries get in tandem.
3. I've worked as an economist at one of those government agencies. They are filled with keen PhD / MSc economists looking to bust kartels and find collusion. Geographical price dispersion models are so '00s. There is a lot of positive energy and interaction between academia and government. Actually _proving_ collusion is terribly hard. Prices that move in parallel is not collusion. Proving collusion is finding documents, emails, phone calls, tracking cars with staff members of various firms to motels , paper trails of inside information in another firm. That's collusion. And I fear that's not what happens in the grocery sector.
The motel one comes to mind as real life example of how DoJ in the US prosecuted a proper cartel. The talk was by more a broad-shouldered officer type, than an economist. Certain set of transport firms always had a interesting way of responding to auctions. Always the second best offer was quite far behind and there was a pattern to which firms did or did not reply to a bid request. End of the investigation, they could trace employees of the firms to meeting in a motel before bidding. And somebody left a paper trail.
There is the skillset of the employees, and then there is what management allows employees to work on. A decade or so ago, a Statistics Canada senior manager was interviewed and claimed as fact that they were unable to reproduce an Excel model that analyzed the % of real estate sales to recent immigrants to Canada, one that an individual CRA team (one person I suspect, it's not complicated) in Richmond BC had done a few years before.
At the same time, they were given additional budget of many millions of dollars to accomplish this task and other things. They did not accomplish the task.
> Proving collusion...
The government has tipped their hand already as far as I'm concerned. I don't expect anything would ever be proven, I just want to see as much evidence as possible to help me decide what I am going to believe - I'm certainly not going to take epistemic advice from anyone in the Canadian government.
Surrounding collusion. I can’t tell you what to believe, but keep in mind that there are a lot of reasons why correlated moves in prices can occur without communication. Lots literature in industrial organization (the field of why markets have particular ordening and how actors behave on markets) on signaling. A though experiment might be the games of poker versus bridge on explicit vs implicit signaling / coordination.
https://en.m.wikipedia.org/wiki/Closed-world_assumption
It would be fun if this fellow's initiative actually got some traction, and attention, and then out of the blue they launched a new educational section of their website to assist the public (and journalists...or not) in their thinking on such matters.
I suspect these giant retailers have a risk department, and at least some people in that department would be able to recognize that which the general public and participants in this thread lack the ability to recognize.
That article covers the initial report, not the subsequent claim that they do not have the ability to reproduce it.
> But the report was not made public until a five-year access-to-information battle concluded recently.
This demonstrates that Canadian democracy does not rule according to the will of the people - there was never a referendum on whether the government should hide the truth from citizens.
> Of the 46 houses bought in Burnaby, staff found 72 per cent were purchased by new arrivals to Vancouver who reported an average total family income of just $16,000.
Immigrate to Canada, enjoy all of its world class (at least formerly) social services, but pay no income taxes. Also not the will of the people.
Hanlon's Razor seems to be the guiding principle of journalists, and the general public in general. Gosh, I wonder if our educational curriculum (or lack of specific content) has anything to do with this state of affairs. Note also who it is that determines curriculum.
Not that prices aren't consistent across a chain, individual stores in one chain will have different prices, calibrated to local supply and demand. In many cases they'll also have local products that aren't in most of the stores in that chain. I think you have to track prices in each and every store, not just per chain.
Still, I think you're right about this fitting in memory. This isn't "big data".
Government departments notoriously have serious technical debt, so while they might entitled to demand this kind of data, I sincerely doubt they are doing it with anything resembling modern data analysis. These orgs aren't going to have working data lakes, spark clusters or data warehouses on cloud infrastructure. They maybe have legacy SQL databases being pulled into spreadsheets.
So, while their data access might be better theoretically, I doubt they truly are able to even remotely analyze the data effectively. Data science in govt is notoriously poor outside of Statscan or other tech heavy departments.