One thing I would like to add after reading some of the comments here. The "1 billion requests/day" is actually an understatement so the title has a nicer ring. Last time I checked we were around 2B -and that's according the most conservative approximation-. Those are requests that hit the application, excluding the CDN caching layer.
BTW, happy to answer any questions!
For Zendesk, what are the business consequences of using a monolith? Is this a mono- or multi-repos? Are tests simpler? Does the system require lots of special-purpose tooling?
Thanks!
I experienced this first hand when I first started doing timed coding interviews and didn't prepare my DRF environment. Rookie mistake for sure, but it really emphasized to me that things can be difficult to find in there unless you know exactly what you're looking for.
In the long term, with a lot of complexity and a large organization, a more mature architecture that relies on components that wrap tables, composed together to form components that wrap business logic, which are composed into endpoints. Something like that is much easier to test and more scalable for disparate teams to step into and understand.
Plus, I'm not sure if I ever met someone who used ZenDesk and actually liked it.
There’s most likely is better but donno what is.
My use case is pretty low volume so I could probably have succeeded with anything but it's been sophisticated and fast and easy to use so far.
However RT is really just a request tracker and doesn’t do some of the fancier pseudo PM stuff zen desk does. That’s a plus in my book since when I’m tracking tickets I don’t want a JIRA clone but worse.
Also it’s worth noting I haven’t used it in 10 years so it might have changed substantially since then.
I found that after I went through their free training programs, I could use it powerfully, the way it's designed, and it works really well when you know how to use it really well.
Granted, we haven't "met," so your statement stays true, but I really like ZD.
> It is currently in maintenance mode (bugfix, but no new features) because we are migrating off it.
Also, we do handle more than a billion requests/day, that was just for giving the title a nice ring.
Do you know of any larger studies on that?
Profitability also may correlate differently; smaller places usually focus on growth instead, right?
I'm now at an elixir place. I'm not convinced it is the right tool for what we do but it was chosen to leverage the BEAM arch. Headcount growth is the only thing in the way of being profitable at this juncture.
In short, I disagree with your thesis.
... that supposed to be good ? it's 11600 requests per second... for reference Seastar (a C++ framework) reaches 7M requests per second (https://seastar.io/http-performance/)
1 day has 246060=86400 seconds. 1bn/86400 is more than 10000 reqs/second, so each request has to be served in less than 100 us. According to [1], random access on an SSD is about 150us. This suggests to me it's likely that most of these are being served cached from a CDN. Are we supposed to be surprised that this can be done by a rails monolith? We don't know how many of those requests are actually hitting the rails app.
An org might deploy a complete copy of the monolith for a single tenant, and it would still be a monolithic architecture.
Native Command Queuing (NCQ) [1] is a SATA extension that lets the drive optimize the order commands are performed in. For HDDs, this means it can do the commands in an order that minimizes overall seeking time. For SSDs, it can concurrently execute commands that are operating on different physical chips.
I can't find a good reference, but it seems like this is also true for RAID volumes, which are almost certainly being used on the server -- the RAID controller can perform parallel reads on each of the independent disks. How well this works is highly dependent on the RAID controller itself.
I'd suggest that the word means whatever the majority of people assume it means - that's also how we get the meaning of "agile" to be so inconsistent, depending on which companies/teams/people you talk to. Essentially, people who've only worked with projects that run as single instances might have a pretty different opinion on what a "monolith" is.
I've definitely met a lot of people who'd claim that a single application package across multiple servers is no longer a monolith. The reasoning would probably go along the lines of this: "If a monolith can live on multiple servers, what do you call an application that can only ever live on a single server, with a single instance being launched at the same time?"
So essentially, what would be the names best suited to describe:
- a single codebase that can only run as a single instance
- a single codebase that can be deployed with multiple concurrent instances
Oh, also, in regards to the definition of "monolith" that is centered around only how the code itself is deployed, personally I think that a modular monolith ("modulith"? heh) is a great architecture that doesn't get explored nearly enough! A single codebase that is easy to reason about, a single executable that is probably scalable but also simple to deploy however you need, with different parts of the functionality (think front end, different types of API, reporting functionality, file upload functionality etc.) being possible to enable/disable in each of the instances based on feature flags. Want everything on a single instance? Flip all of the flags to "on". Multiple instances? Configure whatever modules you want, where ever you need.
That exists? Are there examples of this, especially once where there is a good reason for this? I cannot even begin to list all the awful issues with this in my head.
Most certainly. I'd suggest that many systems out there that ever only needed to run on a single server are structured like this. Even though you could technically take plenty of these systems and launch two parallel instances, you'd get problems because they haven't adopted the "shared nothing" approach, or even just basic statelessness principles.
We tend to forget ourselves with all of our modern and scalable container developments, but there are untold amounts of PHP code out there that stores files and other uploaded data on the very same server, in any number of folders. Of course, you can technically set up a clustered file system, or at least a network based one, unless you are running in a shared hosting environment, in which case you are out of luck.
Oh, and speaking of shared hosting, in theory you should be able to get rid of environments that use cPanel and instead switch to containers, right? Well, no, because workflows are built around it and dozens of sites might be run on the same account with any given shared hosting provider.
You'll be lucky to even find such an environment that has an up to date version of PHP installed and running and resource contention issues will present themselves sooner or later: "Oh hey, this one slow SQL query in this site brings down this other dozen sites. Could you have a look at it?"
I actually helped an acquaintance with that exact problem, I dread myself for agreeing to help because it wasn't a good experience.
Looking at the enterprise space, I've also seen systems out there that store state (e.g. information about business rules) in the actual application memory liberally, as well as things like user session information, because someone didn't know how or couldn't be bothered to set up Redis.
So there an app restart would mean that everyone is logged out. Not only that, but if you have a system which allows users to make some sorts of requests, with business rules about what order they can be accepted in, that means that you can store the output of these states in the DB, but during the processing you have an in-memory queue, which means that you couldn't feasibly have multiple instances running in parallel, because then you'd have a split brain problem. It's like those people had never heard of RabbitMQ while designing it.
Apart from that, there are also issues with scheduled processes. If you've never heard of feature flags or don't see a good reason to use them, you'll run into the situation where you'll have your main application instance executing scheduled tasks in parallel to serving user requests. Worse yet if it's coupled tightly and the application will do "callbacks" for reacting to certain changes, instead of passing the message through the DB or something. Oh and in regards to performance, you better hope that the reporting process you wrote doesn't cause the service's GC to thrash to the point where everything slows down.
Oh, and in addition to that, there are hybrid rendering technologies like PrimeFaces/JSF out there, which store the user's UI state on the server (in memory), whilst sending diffs back and forth, as well as making the client execute JavaScript in the browser for additional interactivity. Think along the lines of GWT, but even more complicated and way worse. A while back some people talked about how the productivity can actually be pretty nice, but what I saw was 100% the opposite, but more importantly there's also no viable way to (easily) distribute this UI state across multiple instances, at least with the way the eldritch monolith is written. I've also seen Vaadin applications with the same problem.
Another factor that can cause situations like this to eventually develop is having a tightly coupled codebase, where you cannot reasonably extract a piece of code into a separate deployment, because it has 20+ dependencies on other services in the app and is called in about 40+ places (not even kidding). While you could try, before you know it you would be sending 20 MB of JSON for simple data fetching calls between applications (again, not kidding - once actually saw close to 100 MB of network traffic between back end services and DB calls for a page to load).
Those are just some of the issues. My suggestion would be to never build systems like that no matter how "simple" they seem and instead just stop being lazy and use Redis, RabbitMQ, or even just PostgreSQL/MySQL/MariaDB tables for ad-hoc queues, anything is better than writing such messes. And if you are ever asked to help someone with anything that starts looking like the above, tell them that your schedule is sadly full or at least very carefully consider your options.
In practice, many web applications are stateful. The load balancer would see to it that clients keep talking to the same frontend. For larger applications it is important for cache locality.
> untold amounts of PHP code out there that stores files and other uploaded data
This is quite normal when you have some type of blob, and normally what networked file systems are user for.
In regards to front end resources, it shouldn't matter which instance you're talking to, if all web servers are serving copies of the same bundle, given that the resource hashes would match, outside of A/B testing scenarios. It's also nice to explore stateless APIs where possible, and not have to worry about sticky sessions.
In many dynamically scalable setups if you tried talking to API instance #27, you might discover that it is no longer present because the group of instances has been scaled down due to decreased load. Alternatively, you could discover that the instance that you were talking to has crashed and now has been replaced by another one.
Hence, having something like Redis for caching data, or even a cluster of such services becomes pretty important! Of course, there are ways to do this differently, such as taking advantage of CDN capabilities, but for the most part sticky sessions are a dated approach in quite a few cases. It's easier for everyone not to care about ensuring such persistence.
An excellent exception for this: geographically distributed systems where even if you don't care about that exact instance, you still want stuff in this data center to be reached, instead stuff half way across the world.
> This is quite normal when you have some type of blob, and normally what networked file systems are user for.
Nowadays, I'd argue that S3 (or compatibles like MinIO or Zenko) is one of the very few ways to do this properly, or perhaps GridFS in MongoDB - an abstraction on top of the file system, that handles storing and accessing data as necessary. Then, using a distributed or networked file system, or block/object storage (depending on the setup) is a good idea.
However, in general cases, you should never use the file system directly for the storage of your blobs, regardless of whether those are stored locally or in a networked file system, as that is just asking for trouble. Things like maximum files per folder, inode limits, maximum folder nesting/file name length limits, maximum file sizes, writing bad code that allows browsing other directories than the intended ones, the risk of files that might be executed in the case of bad code/configuration, case sensitivity based on the file system, encoding issues, special characters in filenames or directories, need to escape certain characters as well, reserved names in certain file systems and frankly too many issues to list here.
So yes, it is "normal" but that doesn't make it okay, though one also has to understand that often in a shared hosting environment there aren't good options on offer, versus just spinning up a MinIO container and using the S3 library in your app.
"Monolith" is a term of art in software engineering. You're in a discussion about software engineering. Saying "it means whatever people want it to mean!" is like talking to a bunch of chemistry people and saying most people think car springs when they hear "suspension".
Look, you didn't know what "monolith" means in a software context. Everyone has stuff they don't know, even in their field. Learn something and move on.
The problem is that most of these terms are loosely defined and evolve over time. And I do mean in practice, as opposed to some thesaurus definition that gets lost in conversation.
REST? A set of architectural constraints, but nowadays most just selectively pick aspects to implement and forget about things like HATEOAS or hard problems like resource expansion and something like HAL isn't popular.
Microservices? Some think that those need to be "small" which isn't true, and yet somehow we see time and time again people ending with a pattern of service-per-developer instead of service-per-team, because people just aren't good at DDD and aren't experienced with what works and doesn't yet, if they haven't been building systems like that for a while.
Agile? I'm sure that you're aware of what the management and consultation industry did to the term, now we have something like SAFE which goes exactly against what Agile is and shouldn't be allowed to be named after it.
Cloud native apps? Who even knows, everyone keeps talking about them, there are attempts to codify knowledge like "12 Factor Apps" but I've seen plenty of projects where apps are treated just like they were 10 years ago, to the point of dependencies being installed into containers at runtime, logs being written to bind mounted files and configuration coming from bind mounted config files, as well as local filesystem being used for intermediate files, thus making them not scalable.
> Look, you didn't know what "monolith" means in a software context. Everyone has stuff they don't know, even in their field. Learn something and move on.
Another person suggested the distinction between "monolith" applications (referring to the codebase) and "singleton" applications (which describes the concept of only a single instance being workable at a time). That advice is workable and is useful, since now we have two precise concepts to use.
Your advice isn't useful, because while you're right about the "proper" definition, you brush aside the fact that people misuse what "monolith" is supposed to mean and don't engage with the argument that we need a proper set of terms to describe all sorts of applications to avoid this.
If we don't address this, then monolithic apps become some sort of a boogeyman due to claims that they aren't scalable when that isn't true for all monolithic apps (since that should only imply how the code is deployed), just the subset of arguably badly written ones, as explained in my other comment.
So essentially:
- a single codebase that can only run as a single instance --> could be called "singleton apps"
- a single codebase that can be deployed with multiple concurrent instances --> the proper definition of "monolith"
Hope that clears it up.
I've heard this referred to as a "singleton app".
It's like saying "this thing says it is built as a suspension bridge, but there are ten of them, so I wouldn't call it a bridge."
[0] https://docs.microsoft.com/en-us/dotnet/architecture/contain...
This would also mean that deploying a microservice architecture onto a single server, using docker for example, would be considered a monolith. Or even better, any application that leverages a database on a separate server is no longer a monolith! That's way easier than rearchitecting your whole app.
I've never come across a definition for monolith in the context of software that had anything to do with the actual infrastructure employed
Microservices on the other hand in theory have a number of very small services that at most share some libraries, but can be deployed independently and could each even be written in different languages if you want. This mostly is an advantage if you know that different portions of your app are gonna scale very differently from each other and rely on disparate data models as well and only share a small interface. In practice, things frequently don't end up as clear cut as you expected on the outset and you end up having to heavily coordinate many of your deploys across services.
I wonder why they're not brought up more when talking about large Rails apps. Is there an interesting (read: bad) history with them and Ruby?
[0]: https://www.youtube.com/watch?v=mJw3al4Ms2o