Now we have Airtable and Smartsheet, where you can have a proliferation of undocumented shadow IT systems core to your business processes AND pay by the user every month for the privilege! That's progress, right?

My general theory of shadow systems is that they're usually the product of either:

1) Poorly trained users that don't know how to use the system for their needs

2) A poorly chosen system (or system implementation) that doesn't fill the needs of the users.

2-b) This includes systems that technically do the thing users need it to do, but in such a bad/slow/sloppy way that no one wants to use it.

I'm not sure which is more prevalent... I've had to say "Your system will do that for you" plenty of times, but I've also been on the other side creating shadow systems myself. One such was an ad-hoc data integration using python's splinter library to automate data download from a system w/o an API to a very kludgy "select .... from dual union" series of SQL statements to populate some of the data needed for a daily analysis. The ETL folks were already in integration hell on over-scoped projects, and also not accustomed or tooled up to run arbitrary python code-- especially any code they didn't write-- to power a process.

Or 3 - Users have a new requirement outside of any existing solution and perceive building their own solution in Airtable as being quicker, easier and cheaper than dealing with the people in IT.

I’ve been in a position before where talking to IT to get something small made means absolutely tonnes of work producing business cases, requirements documents, asking for funding, only to be told the huge bill to produce something internally isn’t worth paying, for something that takes a few hours to build and implement yourself sitting next to the team that’s going to use it. Ok it’s not supportable, but it’s airtable!

In my current role as a ops/logistics consultant, when I talk about IT change to clients a lot of the time the question comes back as “can we find a way to do this without talking to our IT team? You can’t get anything done through them without it costing at least X”.

Shadow systems are prevalent in large MNC's where systems are decided centrally without being able to take into consideration local needs.

Local parts of the MNC's end up using the centrally required system, as well as their own local shadow system.

I've been involved in the selection & implementation of multiple systems like this, and there was a reasonable effort to include stakeholders from functional areas. After the RFP responses and selection of potential vendors, they each come in we had basically a week-long conference with sessions for functional areas, technical areas, implementation, etc before selecting the vendor. Even then, the problem wasn't lack of consideration for local needs: The problem was that pretty much every question asked of the vendor's demo team was "Sure we can do that, it's just a configuration!" or "That's a customization that a few of our customers have implemented, we can piggyback on their work to do the same for you!"

Then implementation comes along and what were claimed to be configurations are more like "You just rewrite this COBOL module, but updates will break it so you'll need to rewrite it for every update." And those shared customizations just never materialized. In one case the actual "built in" functionality shown via screenshot slides literally didn't exist in the actual product. ::ahem Oracle:: and resulted in a lawsuit for it, among other things like like deadlines they didn't meet and & refused to complete without being paid an additional exorbitant fee, even though it was a fixed-price contract.

Also yes, there are IT departments that take a request that's basically "Please create this database trigger" and run it through multiple layers of review & approval & business case justification, ROI, etc over the course of months for something that would take about 10 minutes to implement in a test environment, another hour to sufficiently test, and another hour to pass to UAT so they can confirm it's working as desired, and push to production.

So you're absolutely right: I missed a causal category for shadow systems. Honestly I don't even think shadow systems are inherently bad in some cases: I just think there should be formal documentation lodged with IT so there's institutional awareness of it, and the departure of a single employee who created it doesn't cripple a department. This crippling of a department is something I've actually seen.

> Them: "Oh we just share an excel file each week and they add their own stuff to it. Here's 300 weekly files for the last 5 years. We really need the analysis in a few days."

> Me: ::tries to concat 300 excel files, discovers 50 variations on column arrangements, names, data types etc., tells department

Had to laugh seeing as how familiar this is in my company.