I also got similar advice to shard my users or something.

So right now, we have crossed that limit and are unable to know how many users are on our system. Their server just fails and takes down the DB for 10 or so mins if I do that query.

Firebase is good for MVPs and prototypes but not at all scalable.

I don't know if you are aware, but their DB can't handle more than 1000 requests/sec, so if you are iterating through a list of nodes and requesting data for each one you can hit that limit (not a good practice, but sometimes you have to). Additionally, once you hit that limit, the DB slows down but keeps accepting requests meaning, if you keep hitting it, even at a slower rate you make the backlog worse. Seriously, be very very careful not to go over that limit, we found out the hard way.

I am not even sure how I should get keys for all my user nodes anymore.

I tried doing it from an offline JSON backup they generate. But that one giant 100GB+ JSON is impossible to parse with any available tools.

This sounds like a classic use case for a streaming parser. The data is a mile wide and an inch deep, so at any point the memory requirements should not be too high.

What do you want to do when you've parsed it? Insert it into a real database? Iterate over it? Would simply turning it into a list of user IDs one per line suffice?

  {
    users: {
      "userid1": {...},
      ...
    },

    ...
  }

But jq fails silently after 12-24 hours of processing. I am still researching this in free time.

If I can just get the keys (like "userid1") I can do the rest from firebase itself.

  cat input.json | jq -c --stream '. as $in | select(length == 2 and $in[0][0] == "users") | {}|setpath($in[0][1:]; $in[1])' > users.jsonlines

  {"userid1":{"name":"user1"}}
  {"userid2":{"name":"user2"}}

You could to the splitting yourself (it's just plain text) and create multiple files whose contents is just an array in the format:

```

[

{},

...

]

```

Then you can use JSONStream to load each of those files individually and map/reduce on the contents.

Running a query takes down the db?

That sounds like a major problem. How can that happen?

Maybe returning incorrect or incomplete results due to sharding... But taking the db down? That's very... Unexpected.

Oh come on, it was a startup. The same people who made the tool told the lies (if that's what they were). Now under Google, this is still the case - the original founders now lead the Firebase department if I'm not mistaken. I'm pretty sure they have influence over the marketing messages.

And then people are surprised when I advise them to avoid startup products.

It's the classic tragedy of the commons; lying your way into an acquihire is a win for the successful founders, and a loss for the everyone else in the ecosystem, as more and more people grow sick of being played that way.

Also, Google's Firebase acquisition was many things, but not an acquihire.

You basically have 2 weeks to implement user management functionality? An acquisition so hot they basically say fuck you to all customers. Amazing.

--

[0] - https://stormpath.com/blog/stormpaths-new-path

I can roll my own using Rails but I'm trying to avoid that.

That is a huge number of concurrent users though.

Our app has multiple millions of users and we wanted to add a small feature which we were going to use Firebase to back. We ended up having to go back to the drawing board when we started a gradual roll out and hit that limit pretty quickly.