What I have heard of - and directly witnessed - is MS Access and MS Excel being used by non-developers to build surprisingly powerful tools to solve their most acute problems.
They often don't even realize that they are programming.
It’s an immensely valuable digitization too in enterprise setups where you operate s lot of closed and/or legacy systems. Like we operate around 600 it systems, and very few of them talk to each other.
So if you want to transfer data from one system to another, you’d need to have someone manually do it. Or you could automate it by using RPA software.
Basically it’s what will do to the office space what the assembly lines did to factories.
It’s also really terrible, because it breaks when the UI changes. But when companies refuse to sell open apis, it’s what you have to work with. The most ironic part is the IT companies that now sell RPA solutions for their own shitty software, but that’s the world we live in.
The tricky part is that there is a finesse involved to make it reliable and the exercise becomes more process improvement than programming. Most people struggle with this and you end up with a 2000 line-of-code Rube Goldberg machine that always breaks.
Part of it is because our IT department is underfund and dysfunctional and RPA is the only “controllable” way to improve processes (in execution it’s not) and part of it is because our management have drank the cool aide that automation anywhere puts out about the impending singularity.
I’m all for automation, more for process improvement; but at 2.5MM for a bunch of brittle macros running critical functions; it’s way oversold and too immature
As you have learned, RPA is mostly screen scraping, macros, and as another person referred to it below "Rube Goldberg machines" that break when UIs change. For all of these reasons above (the need for automation and the limitations of RPA), we started Soroco. Like you said (and we agree) you can't build serious automation without real development.
We take a fundamentally different approach which is to build a platform/SDK for automation that provides a significant amount of reliability on top of a number of unreliable automation layers. There is no other way to automate Windows applications or Java applications than through either A) Screen scraping, or B) Their accessibility layers (e.g., Windows UIA). The problem is that BOTH are extremely unreliable. For web, that equivalent is Selenium (at least as the primary layer).
As another commenter mentioned below, a "watered down version of Selenium" is exactly what is NOT needed -- nor are Windows UIA or other accessibility layers great. They are all extremely unreliable. When something like Selenium fails to find an element, what do you do? Halt your business critical process?
All of these frameworks were never built to handle business critical processes, they were mainly built for application testing. The RPA industry has basically tried to leverage them and pretend they are enterprise ready by hiding them behind flow chart builders. They are NOT enterprise ready.
To draw an analogy, it's like having the Internet and IP, which do NOT provide or guarantee reliability. You need the equivalent of TCP and reliability layers.
At Soroco, we have dealt with all of this and have put a tremendous amount of effort into building reliability layers on these unreliable systems, along with security, scalability, and machine learning. We construct automation systems using a full programming language (Python) and have built flow control layers and reliability on top of it. Sort of like an automation SDK for Python with a full IDE. We have lots of supporting microservices for storing credentials securely, storing information scalably, and deploying. We've even open sourced how we encrypt Python (https://blog.soroco.com/)
What does "cognitive" mean in this field? At Soroco, we don't pretend automation systems can "self heal" or "self construct" -- that's just silly. For Soroco, cognitive means that if you had your automation system do the same thing over and over (e.g., during User Acceptance Testing)... what can it learn about the distribution of inputs or outputs to flag anomalous behavior? We also apply Machine Learning and Computer Vision to specific use cases to make them achieve what is not possible through a laundry list of conditionals. For example, ML/NLP to classify intents of text and to do things like binary or multi-class classification.
There's a lot of noise in the RPA space. I'd be happy to share more with you if you want to reach out directly to me: [email protected] or [email protected]
I was hoping that your company offerred a product similar to Automa, but it looks like your business model is different to that.
Anyway I ended up having to re-implement my Automa-using Python scripts using a different technology (TestStack.White and C#) and I 100% concur with your analogy about TCP over IP and building on unreliable foundation. It's also analogous to the sensor fusion problem of AI: picking out what to do when different methods of interacting with the automation layer report conflicting results. (And you have multiple ways because you need the redundancy to detect and deal with problem #1 - automation interfaces are incredibly unreliable.)
For example between IT-mandated invasive antivirus software injecting shims in the OS and/or screwy win32 code in the program-under-test, or bugs in TestStack.White, (I've found indications of all three causes) there is some kind of randomness or ambiguity in the Window handles you get when you ask for certain controls on a dialog. You might ask for the OK button and get the button 50% of the time and 50% of the time get a handle which turns out to be the textbox above it.
So I've taken a belt-and-suspenders approach and built a database of all of the controls I expect to find and their screen positions, a folder full of screenshot snippets which I intend to make my test automation program compare, and use that to augment what Windows UIA is telling me is on-screen. Due to the problem of receiving the wrong window handle (on this app) when I ask TestStack.White to find a control at a certain position or by name, I resort to just cross-collating the entire list of controls on the window with the best match from the database. However even this is 4x harder than it should be, because for some reason the positions and sizes in the database (obtained by a previous attempt with a different automation tool) differ by small pixel amounts from what TestStack.White reports. So I had to implement a fuzzy match.
And then the entire object-oriented architecture of my automation utility program collapsed on itself like a house of cards, because now I have 2 or 3 different derived classes to represent "the same" conceptual object: if you code "button.Click()" should it click the button with the handle found by TestStack.White? Or click the coordinate from the database? Or click where the screenshot matched? Or do nothing because this is a mock object? Worse, I now have a constellation of 1..N OOP instances associated with each conceptual object: maybe the database says I should have a button, but I can't find a handle for it thus I cannot create a ButtonFromHandle but no matter I can still click on the center coord of ButtonFromData. But if I have both I really want to default to one (which?) that is most reliable. Object-oriented programming design methodologies are completely inadequate to deal with the case of "some number between 1 and N instances actually represent one conceptual object and should be treated like a sheaf of overlaid transparencies". Yes, it can be done, but the design turns into something that makes sense from that aspect but makes no sense from the API-user's point of view: you just want to click "the" button you don't want to do "button.Implementations.First().Click()" or some such. Then add in multiple layers of mocks for unit testing the automation tool and you get N*M combinations...
Not sure where I was going with this, I guess I'm just excited to find someone who's worked on the same (or similar) problem.
But even though RPA is mostly screengrapping and macros, we’ve found it impossible to teach to people who aren’t developers.
Similar with better BI tools it’s become possible for sql savvy business process people to do BI, but without s fundamental understanding of data efficiency, which we can’t seem to instill in these highly educated people, what they make is often useless because it slows the systems.
Programming is completely trivial, but understanding what you’re doing, and why. Is much harder to teach than programming. Especially when you’re working toward productiveness where you can’t just write best practices for everything because best practices are often unnecessary for some systems and necessary for others.
I know this wasn’t really that related to the article, it’s just a general observation, and believe me, we tried to decentralize our RPA processes because maintaining them is hell.