A major problem was the high breakage rate for the DVDs they were renting. This created issues with the supply chain of new DVDs to replace broken ones since DVDs are not considered continuously consumed, disposable products. Additionally, it made it easier to manage the bursty nature of demand for particular DVDs since you no longer had to optimize the economics around the number of rentals per physical DVD.
It was Redbox that got the studios on the ropes. Not only were home entertainment revenues dropping because of over-saturation, a bad economy, and burgeoning digital rental options -- and these were revenues the studios had started to rely on and build into their budget forecasting for films that would often only become profitable after a successful home release -- being able to get a new release for $1 from a kiosk (a kiosk that can store 500 copies and might be loaded up with just the top X new releases) and then returned and re-rented within 24 hours, that was a problem. Because now I don't have to pay $15 for that movie I'm not sure I'll ever watch again and that will clutter up the living room, I can just drive up to the grocery store or the gas station or whatever and grab a disc and go. And a website will tell me where the disc is available if it's out at the place up the street. And it's $1 and I can just return it right after I watch it. $1 rentals was a threat, but Netflix never had that pricing. They were like $7.99 for one out at a time and then it would go up from there.
But the studios didn't have a problem with Netflix and in the early years, were probably able to get better terms (for the studio) with them on bulk purchases than they had with Blockbuster.
Studios didn't start to have to grapple with Netflix until Netflix signed its landmark deal with Starz, allowing it to stream Starz's catalog of content (there was no provision in the Starz contract with studios that disallowed offering a digital streaming option -- future contracts obviously added those clauses but that would take several years to expire), thus truly making good on Reed Hasting's vision of having the internet deliver movies.
And even then, the relationship was adversarial but also somewhat necessary from both sides, because Netflix needed to license content for its library, and studios needed that sweet licensing revenue (at least before Netflix started to create its own content that competed directly against the studios).