I would really like to try working in an organization that uses something simpler, like Trello (although now that this is also an Atlassian property, maybe not exactly Trello?).

Other ticketing systems do not work nearly as well for this purpose because they are designed mainly as external brains or communication platforms for workers, and they assume a level of worker autonomy in moving tasks through their lifecycle. In Trello you cannot make it so that a PM has to sign off before a card is moved to the in-progress column, or that only in-progress cards can have code reviews associated with them. JIRA eats these kinds of requirements for breakfast.

EDIT: This is not to say you can't use JIRA in a workflow-neutral way, or that everyone uses it for this reason, but I would submit that it's JIRA's differentiated advantage.

However, regarding ticketing systems, in team environments, it is very effective and helpful to have a system that manages the data about the work that has been completed, is being worked, and is planned to be worked on .

Part of that system might be defining restrictive workflows for some teams, not for control, but to ensure the agreed upon process is followed for quality or consistency.

One of the many problems Jira has is that if you don't have a Jira admin on your team, it's impossible to build an effective and efficient workflow for your team. Coupled with Jira making many things global by default (it takes a lot of care to make a change that only affects specific Jira projects) most configurations end up being a pile of garbage automatically inherited from changes an admin(that is not part of the team) made when intending to change something for another specific team.

> It sounds like you've been hurt by the some terrible management practices, I'm truly sorry that some managers think their job is to control their subordinates.

When we assume someone was hurt, and imply they hold an opinion only because they were hurt, we risk delegitimizing their position. The interpolated message we might be sending is "your experience is personal and not representative of the subject at hand, and so your thoughts are only applicable to your situation; so, after we express our sympathy, your thoughts can be dismissed." Or the message we might be sending can be patronizing: "you hold your opinion for emotional, rather than rational, reasons; I'm sorry that you are so unfortunate."

To be clear, though, I'm sure this wasn't your intent, and it makes me glad to see someone being compassionate (i.e. that you bothered to consider the experiences and feelings of the parent commenter).

A personal story: I was raised devoutly religious but left the church in my twenties. My family and friends assumed I left because I wanted to be free from guilt, had been hurt by a culture that belied the doctrine, and so on (and they said as much). My change of belief occurred after recovering from a few years of mental illness, and while it is true that I may not have left when I did were it not for the opportunity to reexamine my beliefs (while trying to piece back the fragments of my life into a sense of self), the reasons why I left were the result of a lot of research and thinking. It was mildly frustrating when people assumed my decision was made for emotional convenience, when in reality, the research was uncomfortable and contemplating an unfamiliar universe was scary.

I recognize the irony here – the issue I'm highlighting in this comment may be something that only I feel is an issue, born from a personal experience. But I think it's more common than that.

That is what I mean here by "assembly line" and "control." Making sure that processes lead and individuals follow.

Citing consistency as a terminal value in the same breath as quality is also exactly what I mean by the middle-manager aversion to local differences.

I work with a variety of different environments, and depending on the environment I can either solve my problem in minutes and get it deployed in another few minutes or solve the problem in minutes and spend hours figuring out how to safely deploy it without breaking everything. JIRA is terrible if you do anything that it offers by default, but when used properly it can absolutely help with this.

Obviously not all work is this way. Sometimes you need to drive a migration that touches every team, and then the technologies of bureaucracy and process become important. But most work should be done in human-scale groups that can be more towards the self-organizing and trust-based end of the spectrum.

However some middle managers take offense to the idea that their different sub-teams have different operating models internally, and lean on technologies like JIRA to try to make them all the same. Middle managers at my company have tried this, not very effectively , so it hasn't hurt me too bad. But I've seen their vision and recoiled in horror.

I think the point is that Jira is particularly granular in the way that it lets you do things with permissions, workflow rules, roles, metrics, etc. There's a fair number of places that use that granularity to create a weird digital sweatshop.

Meaning the complaint is more about really deep "micromanagement as a service" than what you might get with lighter tools.

And I wouldn’t assume you’re not one of them. The worst cases I’ve run into aren’t even the psychos that embrace micro management as part of their “management style”. It’s the ones that genuinely believe they aren’t engaging in the behavior. They’re not micro-ing, they’re “helping” their team because they are an awesome manager and their team is almost awesome, they just need to be monitored very carefully and given “suggestions” until they nail it. But they’ll never nail it. Because no one is as smart, experienced or does a task “just so”. They view themselves as a mentor to all. All decisions must be theirs to make. Jira becomes the perfect tool since the team effectively becomes little boxes that accept tickets or stories and return work both performed and delivered as specified.

For any managers reading this that don’t see a problem with this or see some of those behaviors in yourself please understand that you are sacrificing your team’s happiness and motivation at the altar of your own insecurities. No one can grow where they’re not trusted and no one can improve their skills when they’re never given latitude to make meaningful decisions. Your people will make mistakes. They will accomplish things in ways that are different from how you would do them. It might even be objectively worse. That’s ok. That’s how you grow into a strong team with confident members.

I worked on the line (Toyoda Iron Works) and used a real-life Kanban implemented by the plant engineers. It was used for quality control, to broadcast quality control and station output, and was checked regularly against their internal estimates and baselines and used also as a gauge for employee output.

Control is what it's designed to do. The very fact that Kanban is the tool of choice should support at least some of OP's points, objectively.

Could you elaborate? What kind of fear? “You’re fired”? I wonder how effective it actually is because of the current job market and also because I (and others) react very poorly to this kind of tactics: “you want me to fear getting fired? Joke’s on you, please DO fire me, I dare you”

Counterpoint: software developers aren't necessarily well paid or highly regarded everywhere, since remote working for companies abroad hasn't quite gotten mainstream enough.

So it might just be effective against some people, or in cases where the hiring process itself has become increasingly unreasonable - the job being working on boring CRUD apps but the hiring process being multiple stages of Leetcode and complex interviews.

That's probably not applicable to everyone since plenty of folk can grokk Leetcode and find jobs without too much trouble, but i still recall "The Unseen 99%" article: https://www.hanselman.com/blog/dark-matter-developers-the-un...

It probably applies to the industries and companies where devs are treated as a cost center and since those companies aren't all out of business, plenty of people must be working in such environments, with sometimes sub-optimal conditions.

Isn't that just a more corporate way of phrasing "control"?

And that is not a useful way of thinking when you have real engineers writing software that people depend on.

JIRA then becomes a tool for enforcing arbitrary rules, e.g. control

You can use Jira as a simple Scrum board, a Kanban board, or you can build enforced-process monstrosities. You can build customer-support / internal-helpdesk workflows, or even model internal work-item-oriented business processes, etc. Now, as you point out, just because you can doesn't mean you should, and many orgs fall into the trap of making issue workflows overly-restrictive. But most companies (I believe) choose Jira before they choose those hairy task workflows. Startups with zero process use Jira.

Also, you can integrate it all together to give good-enough dashboards/roadmaps, good-enough (for some, not me) docs integrations with Confluence, Git integration with Bitbucket etc. -- while there are big issues with these systems, I think it would be myopic to ignore the real benefits of working in one integrated stack where every design doc you write has dynamically-updated labels and auto-complete for each issue you type in.

For context, I use Jira for tasks and don't love it, found Confluence to be really annoying and so I don't use it, and prefer Gitlab to Bitbucket, but I think you have to recognize these unique selling points. If all Jira had to offer was the rule engine it would not be as widely used.

Each member can actually organize their sprint and create tasks.

Point assignment is not a big deal, it's just there so we avoid promising more than we can chew.

I've found Jira really pleasant to use for lightweight processes.

The same exists for levels, enemies, quests and tons of other elements.

I would not be surprised if a lot of studios had similar workflows.

However, that's just not what most people go through in companies using JIRA. Worse, they have to toggle between pages multiple times, each taking at least a few decent seconds to reload. I'd like to give JIRA the benefit of the doubt here, but it sounds like the tool is just very easy to misconfigure and abuse.

And you generally do them both at a lower level than tickets, certainly commits, so you don't want to have too much automation between them as that starts adding constraints.

One JIRA is the project that's used for development of the core product, where there are no constraints— anyone can add a comment, create links, change assignee, add new tags, push the tickets through whatever state transitions they want, and so on. It works, though it is a little chaotic sometimes as subgroups of people have different preferences for how things should go (eg, for tickets requiring test team validation, should the ticket assignee remain as the person who did the original work so it's clear who has more to do if it fails validation, or should the assignee change to the test team person, so that it's clear that that's the next person who has it as an action item?)

The second JIRA is the IT team's internal support project, which is completely locked down— no one except them can close tickets or move them around, or even edit the contents, closed tickets can't be commented on any more, and so on. This is the one that gives me the vibes you are talking about. Every time I have to interact with it, I loathe it because every inch of it is transparently a funnel, railroading me along a path toward one of either DONE or WONTFIX. This is absolutely efficient, in the sense of meeting the goal of closing all the tickets, but I feel it introduces friction for the larger business goal of actually helping people resolve their problems. To the point where eventually most of the IT support activity moved away from the JIRA project to an informal Slack channel, which is way more accessible, but worse in basically every other way: it's harder to effectively search, impossible to properly link, bad for async, bad for dealing with more than one thing at once, etc.

> Free software has zero acquisition cost, but non-zero TCO, which can measure in millions USD

Often a primary driver is exactly the opposite -- for-profit companies are accustomed to paying money for a good or service, with a billing pattern and legal obligations. The company financial deciders do not want a setup that does not have a billing pattern and clear legal obligations. Meanwhile, Open Source Software went from niche to mission-critical in the 2000s via the Internet. For-profit companies (and their publicists) scrambled to explain it, and came up with that exact line repeated again today. I do not blame any person for saying it, it was in print in some reliable place. It does not capture the reality in 2022 IMO.

> The company financial deciders do not want a setup that does not have a billing pattern and clear legal obligations.

I haven’t ever met a CTO or CIO, who would make budget decisions like that, neither I do it this way myself. The reality in 2022 is the same as it was in 2012 or in 2002: when you choose a solution, you consider all long term costs. In 2022 TCO for the server software includes everything that I mentioned in my comment and more. There’s a lot of use cases for OSS in corporate environment, for sure, but not every OSS solution is cheap or even affordable. Running on-premise open source collaboration tool is certainly not cheap if you do it right.

If JIRA didn't allow you to make it terrible, it wouldn't allow for some of the absurd things that people want it for and those companies might not buy it.

The saying is apocryphal and unlikely to be accurate, but the shape of the thing its describing applies to almost every piece of enterprise software whether installed on-prem or SaaS.

And as another comment points out, at Enterprise scale you can substitute "team" or "group" for customer. Every team might use a different 5%, and unless you standardize their processes, you have to buy the product that can accomodate all of their needs.

>The saying is apocryphal and unlikely to be accurate

Well its mathematically impossible to be accurate as soon as you have > 20 users.

However, if we assume there are, say, 100 features in Word (the real number is likely much higher), the number of combinations is orders of magnitude higher than 20.

It's probably in the semantics.

Text input and editing is clearly a part of functionality that's probably used by everyone (or at least most users), so it's not possible for "different 5%" to mean what you're alluding to, maybe the phrasing needs work.

In any given 5% there might be 1-4% of overlap with what others are using and the remainder of that is specific to the company.

If it's a uniform distribution of discrete features then each feature is equally "important" and worth equal resources and dev time. If 81/100 companies use the exact same 5% of features and the remaining 19 cover the remaining 95%, then all else equal you can probably drop 95% of your features and still do well.

Typically you do the most popular features first, but most Enterprise vendors end up working on a long tail of niche features that nevertheless are profitable.

There's a long conversation to be had about how this ends up being a trap where Enterprise software gets bloated and shitty and eventually gets disrupted by a small vendor that does "less," but in a powerful, transformative way that obsoletes the Enterprise "standard," which leads us back to discussing Atlassian :-)

They're a good example of this dynamic, because they have a "constellation" of products to sell. So if they build a niche feature that gets a new customer to buy Jira seats, having "landed" in the account, their salespeople can "expand" by selling OpsGenie and other related products very profitably.

    75,287,520
  +  3,921,225 
  +    161,700
  +      4,950
  +        100
  ------------
    79,375,495

But your statement doesn't make sense; there might be millions of features, and trillions of ways to combine them to make 5%.

As the lone dev on the team I've been continually astounded by my leadership's willingness to commit more and more to tech debt laden paths. The notion that all software requires maintenance is anathema to them, and it's led us to be 'cornered' into decisions re: what software we can use / where we can invest our discretionary funding.

Moreover, we're constrained by the parent mega-enterprise's software purchase policies; JIRA's already approved (and run elsewhere in the enterprise), whereas off-the-shelf or SaaSy alternatives are significantly harder to get buy-in for. (No using corporate cards for SaaS, all purchases need to go through the quote/purchase-order process, etc).

Clients add their notes to the card, I check the boxes as I hit the notes, and I move the card further right as we enter different stages of the post production process. We then have a column of every completed project, which is incredibly easy to sift through if we need to revisit something. It’s literally left to right in the workflow, it visually is telling me where we are at all times.

It’s incredibly simple and elegant. For fast turnaround, relatively stripped down content (like podcasts) there is nothing like it.

It’s like my taste in wine. I don’t want an overdeveloped sense of taste where only a $400 bottle will do. I’m fine with what we have because the work is what excites me and if people are documenting projects and managing workloads and committing code, we’re 90% of the way there.

Wine that costs 400$ is for fun.

You don't drink that professionally.

JIRA does try to be all things to all people…and mostly succeeds. For instance, we use the same workflow and mostly the same nomenclature across our development and helpdesk teams. Some of our software projects use Kanban-style workflows, while others use sprints, but we can keep track of a project across multiple teams using the same tools. I’m sure other products also offer this, but we liked the integration and overall capability for the price.

There are definitely issues: some feature requests and bugs have languished in their backlog for years. But you can get started very quickly and we’ve had great feedback from users.

The fact that it’s so feature packed and customizable is the point.

I think the complainers are not really investing the time in to change project settings to fit their needs.

My only complaint about the Atlassian suite is the performance of Jira and Confluence. The overall page load speed is too slow.

How do you eliminate all non-task ticket types in a Jira board and allow any ticket to be a child of any other ticket?

It’s hard to configure away complexity from a product if it’s designed to be complicated.

This here is the single most insane thing about Atlassian.

I have no idea why you would want this from a work management point of view, but you can just use issue linking to describe a parent <-> child relationship.

Re 2: Set the project's issue type scheme to one that only allows tasks and subtasks. That gets you one level of nesting. (And even though task and subtasks are different issue types, changing from one to the other is trivial since they have identical fields.) Allowing epics gets you another at the top level. That's a bit limited, but wouldn't arbitrary nesting be even more complex?

In the beginning, with me plus 2 engineers, I noticed it was slow but since I used it for 20 minutes a week, that didn't really matter. By the time I started using it for an hour a day, we had 10 engineers on 2 teams using it. I got to see a friend using linear, and I had some spare time that I was going to use to switch, but I couldn't get in the beta. By the time they let me in, the opportunity was over and I was too busy.

That means the tool is often the wrong one for the job, but instead of picking something that's a better match out of the box folks stick with the easy choice (extend what they have).

To this day I still don't know what JIRA does so much better that other products don't which big corps are willing to waste months worth of manhours over. It's biggest selling point is integration with the remainder of the Atlassian stack, not exactly known for being great either.

Personally, I like JIRA. I think it adds a ton of transparency in our org, and while I've used Trello for personal and home projects, I don't see how it's good enough for business. Trello doesn't even allow for time estimates (last I tried), which for us is part of planning. Search in JIRA is also really good, so no ticket is ever just lost to the ether.

Sure, it's not perfect, and waiting for a board to load is annoying, but for distributed work and visibility, I haven't seen something as professionally useful.

Open to exploring though.

How good is ticket search? I have to be honest, JQL is the superpower that makes or breaks for me.

The idea that the C++ committee are unthinking people pleasers it patently false.

C++ does have a lot of cruft, but mostly because it aims to: i) support new features ii) maintain pretty strong backward compatibility guarantees

In general the new features are actually pretty well liked, but in conjunction with (ii) it creates a big language. There's a reasonably decent subset that can be carved out, but it's also clear why newcomers without legacy baggage (e.g. rust) are making inroads.

EDIT: clarified wording a bit.

Then, you pay for JIRA, and that expert customizes it the way they like. It still doesn't work very well for most people. Nobody likes it except one stakeholder, and the engineering lead who acts as a admin on it. A while later, those people have left the company, and everyone else is out of luck.

Seen this exact scenario play out at two different companies now. Am witnessing it play out in real time at a third.

That being said, because it can do anything, it doesn't take much effort to make a workflow as painful as possible. Somebody with the "right" mind might make all kinds of checkpoints in a workflow, which makes a lot of operations a pain in the ass because you wind up hopping through a bunch of steps. Pretty sure in our org we just make our workflow "you can hop from any state to any other state"--basically a free-for-all.

Dunno my point, but there you go!

IDK if this was to cheap out on the licensing with a minimal number of users, or if it was to insulate the developers from the experience of using Jira. Perhaps some of both.

Clearly that usage pattern would only scale so far.

Maybe Asana or Monday would work for you.

I've eventually gotten a tenant exporter to work. Practically, this requires some deep and nasty digging through the information_schema to build a graph of tables and foreign key constraints. Once it had that, it generates selects with a simple where clause for tables with the tenant_id, and selects with weird joins all over the place for other tables to dump the tenant data.

All of that sounds complex, but that part took a day or two to hammer together to 90% completion, since it's just some graph handling. The other 10% were getting some weird date formatting questions right to produce a properly importable sql dump. And interestingly enough, it's working for more than just that one product.

But that's just where the journey started. After that, it took a weeks and months to sort out legacy tables, old tables, tables without indexes, tables no one knew about, tables that were important (but not), tables with inconsistent data, .... And it's just handling a single relational database. And compared to \copy in psql, it's slow. And at times, weird things happen if you import huge chunks of sql into a postgres with deferred foreign keys (because our schema has cyclical references).

Point is, I know how painful it can be to handle that kind of database schema, at a ridiculously smaller scale. I'm kind of happy to not work there.

Would still be some maintenance, don't get me wrong. But far from impossible.

source: every single place I've worked at that poo-poos referential integrity has a database that is full of bullshit that "the application code" never cleaned up

Always use referential integrity. The people who are against it almost always are against it for superstitious reasons (eg: "it makes things slow" or "only one codebase calls it so the code can enforce the integrity"). All it takes is exactly one bug in the application code to corrupt the whole damn thing. And that bug will happen over the lifetime of the product regardless of how "good" or "awesome" the programmers think they are....

... I'll get off my soapbox now!

You're lecturing about table design. I'm talking about more general transactionality over any errors.

Oh, and just forget about allowing your customers to share their data with each other, which most enterprises want in one way or another.

Don't do any of the above unless you understand the implications.

Giving every customer their own table means you're going to need database administrators. For these folks their dedicated job was maintaining, operating, and changing their fleet of databases, but they where very technical and were amazing to work with.

Does this extend to services as well? We have a suite of (micro) services. Are they all segregated?

I (and many others) assumed they had to graft in data from backups since a full restore would clobber newer changes from unaffected customers.

If they're all isolated in their own logical per-tenant DBs, I'm really at a loss for what is making restoration take 3 weeks for 400 tenants.

I understand if you'd rather not venture into it, but care to offer any speculation?

For example, they have the main PostgreSQL data store. Surely that's easy to restore. But the users in that DB have a "foreign key" (in a logical sense, not physical) to the Identity service. This is a real life example that occurred while I was there. So now we have a mixture of multi and single tenancy. So perhaps the identity records are also tied to this app ID and deletes were propagated to that service. And perhaps there is an SQS queue and a serverless function to handle, say, outgoing mail from Jira. Where does this data go? I dunno maybe some Go-powered microservice with its own DocumentDB store. Do deletes propagate here too? Who knows. You can see how this gets complicated and how issues multiply with more services.

Again, this is only speculation. But "decomposing the monolith" was a big deal and it was coming from the top.

Really interested if you can share any details.

Edit: I know each wiki is on a subdomain. Does each wiki also have it's own server?

Some of this was open source before we unified all of our wiki products, which has a lot of the selection / db logic, at https://github.com/Wikia/app.

Is it not a good idea to spin up separate db instances for each client/company?

1. An account/tenant_id field for each table

2. A schema for each tenant wrapping all of the tables

Option 2 gives you cleaner separation but complicates your deployment process because now you have to run every database change across every schema every time you deploy. This gets more complicated as your code is deploying in case the code itself gets out of sync, there's a rollback or an error mid deploy due to an issue with some specific data.

The benefit of the approach is the option to do different backup policies for different customers, makes moving specific customers to specific instances easier and you avoid the extra index on tenant_id in every table.

Option 1 is significantly easier to shard out horizontally and simplifies the database change process, but you lose space on the extra indexes. Plus in many databases you can partition on the tenant_id.

Most people typically end up with option 1 after dealing with or reading horror stories about the operational complexity of option 2.

Business wants to run a query across customers? In most DBs you need either custom code or to create a stored procedure to iterate across schemas.

Every table that you create is multiplied by the number of customers. This has implications for some database systems (like PG's vacuum).

Your migrations will take _forever_ to run.

Etc.

It depends, really. There is a trade-off in terms of software and operational complexity vs scalability/perf and isolation. And probably a bunch of other factors.

If you have separate databases for each customer, schema migrations can be staged over time. But that means your software backend needs to be able to work with different schemas concurrently. You can also benefit from resilience and isolation guarantees provided by the dbms. On the other hand, having a dbms manage lots of databases can affect perf. Linking between databases can be a minefield, especially w/r/t foreign keys and distributed transactions.

https://docs.microsoft.com/en-us/azure/azure-sql/database/sa...

Not if you're truly multi-tenant and each customer has their own app servers. Then your code and schema version are always in lock-step.

I've also had to restore partial data from backups on a few occasions when customers fat-fingered some data and asked pretty-please to undo. If someone on staff understands the system well, it's not hard. I suspect Atlassian suffers from a complicated schema and a post-IPO brain drain.

At least it would not be the first time in history that a company has lost the engineering spirit. And instead the business people have taken over, so that details like disaster plans become less of a priority.

A business person and an engineer will always view risk differently, better disaster plans is a kind of insurance that is a lot harder to sell when too many business people run the company.

Just wait until a migration doesn't run on 2 of your 400+ customer databases. Or multi-hour migrations.

The best would be multiple separate database instances, which is not even hard to manage specially for qualified engineers like Atlassian surely has plenty of. The problem are business decisions of ignoring the tech debt, usually...

Automation wasn’t the issue here. It’s the symptom not the cause.

Source: worked at Atlassian, on Jira, 4 years ago.

I wonder if they're not doing this due to some tech limitations, to avoid taking the financial cost of running two systems, or to avoid having to reconcile the systems.

For just this sort of problem, we actually had three DB servers running all the time: active, passive, and hour behind with the ability to break hour behind's copying of the write-ahead log of active as the DBA's secret weapon for just this problem. If all customers had accidentally lost an hours worth of data it would have been embarrassing, but much less than completely shutting out hundreds of paying customers for two weeks, I think?

It's true that nothing is simple at scale, but it's important to note that simple concepts are the only concepts that work at scale.

I don’t really understand what this has to do with “monolithic” or not.

Atlassian’s software is probably very complex and convoluted but from my experience it’s almost impossible to keep a clean architecture in a software system that has grown over many years and is used and customized by many customers so you have to avoid breaking backwards compatibility.

Perhaps they don’t have the right people on hand to do hard things like this.

They also apparently lack an incident response plan since a critical component of that is coms to affected customers.

They also lack good practices around preventing human error. It should not have even been possible to make the initial mistake. It certainly should have involved multiple steps of “are you sure” and potentially even review.

Sounds like an operations shit show. Glad it’s not my circus.

How would they lose committed data? Even after restoring the backups can't they run the logs so that everyone is caught up?

You wouldn't be able to do that without forcing downtime for all customers, for the duration it takes to restore the snapshot and then replay the logs. Not to mention the risks of the process failing somehow

You could narrow the window to just the "replay" portion, if you were able to stand up an extra database/infra, to switch over to when it was ready. But at some point you'd probably still have to go read only to checkpoint the logs and begin the replay.

It's of course possible to do something more complicated here and stream the changes then eventually enact a failover, but this would all be too complex and error prone to introduce in their current crisis mode. It's something I'd suggest considering when architecting their DR/BCP, but it's too late for that kind of elegance (and complexity) now.

This seems to be exactly what they are doing, as described in the article. They don't have automated tools to do this.

You don't think that's exactly what they are doing?