Instead, create software that allows 1 person to do the job of 5. You create massive business value without getting sucked down the hole of edge cases.
A simple probabilistic programming solution can work really well.
It grew out of consulting projects and is almost ready for beta testing if anyone is interested in playing with it and giving me some feedback - my email is in my profile :)
"Some people profess difficulty at finding applications to write. I have never understood this: talk to people. People have problems — lots of problems, more than you could enumerate in a hundred lifetimes. Talk to a carpenter, ask him what about carpentry sucks. Talk to the receptionist at your dentist’s office — ask her what about her job sucks. Talk to a teacher — ask her what she spends time that she thinks adds the least value to her day."
https://www.kalzumeus.com/2010/03/20/running-a-software-busi...
That's your job. Develop hypotheses about what the underlying problem is and ask them questions to try to falsify them. Develop hypotheses about solutions and build mockups or proofs-of-concept and have them try them out to falsify them.
No one will hand you a business idea on a platter. But problems to solve are the easiest thing to find in the world.
I meant to say 'Just Ask' and 'Problems to solve are easiest to find in the world' are very misleading things for a beginner.
Just asking won't lead you to a solution and identifying solvable problems are incredibly hard.
I don't think "just ask" is misleading at all. The fact that most people will happily complain about their business processes but won't have ideas to solve them doesn't make it "tricky", it makes it an opportunity.
There might be fields of endeavor where identifying solvable problems is incredibly hard, maybe in academia or politics, but business processes? Execution and, if you want to get rich, scaling up are hard. But identifying solvable problems with people's business processes is totally one of the "easiest [things] in the world".
See also: http://www.paulgraham.com/schlep.html
We had a whole slew of standard letters that we'd send out, most of the time requests would go to the typing pool (yes, this was a thing) that were basically "Do letter ABC for client XYZ", where letter ABC was a standard thing that always went to the same recipient.
I built a screen in our system that let you tell it to request a letter (only the simple letters, obviously), it would pull the relevant data out of the client file and write the request out as an XML file. On the other side of things, a VB6 program would watch a folder for those XML files to appear and go through OLE Automation (as it was called back then) to make Word fill in the letter, print it, print an envelope, and save the result in the client's folder.
This kind of thing is an absolutely incredible time saver.
Another "domain-specific mail merge" that I built was a tool to pull summary data out of the client financial records and populate specific fields in some off-the-shelf income tax software.
That was a terrible tool to write and maintain. The tax software's import feature didn't actually work correctly. I reached out to the vendor and the response was basically "Works for us", so doing it the right way wasn't an option.
Instead, I used VBA to blindly enter data into the program using SendKeys[0]. Maintaining the program was a case of tabbing through and counting the number of times I had to tab. Enter first name, tab, enter last name, tab three times, enter address line 1, etc. Next year when the forms in the tax software changed I'd have to add/remove tabs in the appropriate places, sometimes input stuff in a different order, etc.
As a program, it was horrible, horrible code. As a useful tool for the business, it was incredible. It saved a ton of time and the staff that used to do this by hand, in addition to doing actual work on the more complex files, were incredibly happy.
Overall, it was completely worth it.
[0] https://docs.microsoft.com/en-us/office/vba/language/referen...
Holy cow why didn't I know about this 6 months ago =|
VBA is something I have had to reluctantly learn. It's a horrid/ancient language. Sorry if someone gets offended at that but to someone who uses a lot of modern development languages/tools it's just very very difficult with near-zero "nice to haves"... you just end up taking this huge lib of code around with you with functions like "inArray" etc.
> As a program, it was horrible, horrible code. As a useful tool for the business, it was incredible.
That's something that's rough for us engineers but can be hedged with great documentation, both inline and and otherwise =)
You have data in a database, you flatten it into a non structured document, send it, and then the other end uses data entry to add it to a database. What should be the realm of EDI/CrystalReports/Blockchain take your pick, instead everyone wants to interchange data in the pretty human readable format, as opposed to just rendering it that way for humans while also communicating the raw structured data.
It's finicky technology that is hard to make sustainable, but a lot of companies are really interested in it because of the amount of work that could be cut out like you said.
This 100 times over. The only way I've been able to make it work is being one heck of a generalist. I've found that training on what I do is near-impossible and I don't even know if I could scale a real company in this beyond myself.
Maintaining & expanding the automation is a heck of a challenge though. Like, once you got the stuff deployed, make sure that it works without headaches, in the face of other software/ OS upgrades. The robots/automation services are essentially a "fleet of microservices" that you have to maintain for a customer whose strength is likely not IT, and who will change the infrastructure from underneath your "microservices". It's a hard problem, but I feel we're well positioned to tackle that, and if we do, the sky is the limit really. (disclaimer: I work for UiPath, though that should be obvious from the message)
Everything you're saying is 100% something I deal with on a day-to-day basis, and ongoing support is a huge struggle to workout with the companies I've implemented software at.
Looking at your product offerings made me chuckle - I definitely have versions of a lot of what you do! My robots are called workers, my orchestrator is called a broker, etc etc ha. Lemme know if you guys are ever looking for a remote automation engineer w/tons of cloud experience ;)
- Never use an RPA tool that doesn't generate plaintext-serialized scripts. You're going to have a bad time if they're binary locked
- Never choose an RPA tool that's been around fewer than 3 years. It's probably just a shim on top of MS automation libs, and can't handle the really gnarly stuff
- Never promise anyone you'll automate 100% of their workload. Never try to automate 100% of their workload. Never hesitate to tell a VP you're not going to automate 100% of their workload. There's value in 50%+, get the easy win and move on. Only come back when you've gotten all the easy wins
- RPA is fundamentally about target selectors (or match rules, or whatever else your tool calls them). Their robustness is the only real feature of an RPA platform, and a smaller toolset is going to result in some fragile, quick-to-break stuff
Ultimately, RPA is about one thing: creating a more tactically malleable layer on top of your existing software. Development and change speed is the biggest advantage.
It shouldn't ever be a core system, but it should be where you prototype functionality.
Taught me a big lesson about realistic messaging and never up-negotiating expectations.
(A) Selenium's UX isn't nearly where it needs to be to upskill an analyst to create their own automations.
(B) Selenium's Windows app compatibility is haphazard.
(C) Selenium doesn't have the kind of corporate support it would take to expand compatibility quickly enough to catch up with its competition.
The RPA space is the Linux desktop problem in a nutshell. Polish and niche compatibility are the final 10%.
Nobody on the open source side has the interest in making a VB6 app work. And nobody on the corporate side really wants to use it for more than what they're currently using it for.
I can't overstate the sheer number of bizarre situations a tool needs to be able to handle to be effective here.
Because being able to automate something 95% of the way to completion is usually a lot less valuable than 100% (note: talking about percentage of happy-path process, not of total incoming workload here).
At this point, I've been doing it for... about 7 years?
It's been an interesting ride.
Email in profile
Tl;dr - different target user & no corporate sponsor seriously incentivized to improve legacy compatibility
Would love to get the Selenium viewpoint though! Have always been curious, and I'm not as well versed in that side of the house.
Maybe - but how would one go about contacting you? (I have my email in my profile, should you wish to contact me).
This is a problem with almost any type of automation. I wouldn't say our software (I am only familiar with UiPath) is finicky. GUIs are finicky, but there are ways to deal with them. That's the stuff a good RPA dev can handle.
I was automating before I started at UiPath, and GUI interaction adds a new layer to automation, of course. But it is still maintainable when you implement them using best practices and CI/CD. I didn't have UiPath at my previous job but it would have made a lot of our automations more reliable and more maintainable. We're also making strides to address these types of problems easier.
I strongly believe that UiPath should a tool in any automation developer's toolbox, as well as GUI testing. UiPath is also pretty easy to use, so business users can automate simpler tasks on their own after going through the academy.
The industry is exploding, and good devs are in high demand. Salaries are high, and you can download Community edition and get certified for free.
Check out this video: https://www.youtube.com/watch?v=3B4Y_aUBWTM
It's a very simple automation. It just gets text from a browser and inserts it into notepad. But it's a 3 minute video that does something. Check out Selenium tutorials and compare how much you would learn from a 3 minute video. You don't have to dive into the HTML, you just click on what you want to click/type into/scrape and it knows what you want.
But it still has all the power a developer would want. You can create custom activities using C# and VB. The product itself is not open source, but it is very extensible and flexible. The workflows it generates are text files, which work well in source control. It has source control integration built in, which a custom diff tool. It enables code sharing and encourages code reuse.
The only downside I would point out is that it only runs on Windows for now, which might be a problem for linux only shops.
First, it was a massive waste of those 2 human's intellect, entering data over and over into a generic form from a proprietary business' website, in order to spam scientists. I started a ruby project to automate out all those processes, just using selenium to enter the data onto the web form for us. Before I could get very far, I got a huge promotion elsewhere and left that under-compensating journal.
It all gave me a depressing vibe though, both the ethics of what we were doing and the waste of time/life that those employees had to endure to do it.
This is usually the reality and it's nothing to be ashamed about.
At work, do what you're paid to do. In your free time, pursue interests (literally, things that are interesting to you).
I work in a small, physical goods business in the Midwest (~30 employees), and the amount of time spent doing data entry is incredible.
Documents (often PDFs, sometimes excel) flow between buyers, shippers, freight forwarders and internally. These documents are often updated.
I'm looking into EDI for a solution, but there's so much value to be created from cutting out all this data entry. At least 1-1.5 full time employee's worth at our size.
Software solutions require ongoing work: ongoing maintenance costs of a customised solution could easily be higher per year than 1 cheap FTE.
More than the # of hours involved, it's the error-prone nature of the work and the lag between when the data should be entered vs when it's actually entered.
These three factors -- hours saved, errors prevented, time lag avoided -- make me think that it's worth automating.
You're right, however, it's not clear cut it's the right way to go.
However, the cost of training staff and implementation is often out of reach for SMBs, still.
Facebook and Co. get a bad rap for many things, which I agree with. But I credit these social platforms in speeding up tech adoption rates for the average person, who can now use this knowledge at their job.
> Programming as a profession is only moderately interesting. It can be a good job, but you could make about the same money and be happier running a fast food joint. You're much better off using code as your secret weapon in another profession. People who can code in the world of technology companies are a dime a dozen and get no respect. People who can code in biology, medicine, government, sociology, physics, history, and mathematics are respected and can do amazing things to advance those disciplines.
But it's like math, if you know it well enough, you'll find ways to use it everywhere. If you don't, you won't. You have to be the type of person that likes to innovate. Its hard to sell to prospective employers, but its great for demonstrating value once you are with an organization. All of my previous employers fight over trying to get me back when I've found myself looking for work. ...Now I work for myself and make my own work and I've priced myself out of their offers, but that's not so bad.
I would be interested in thoughts from anyone with ML experience who has reviewed said course's materials?
Formally from school, I've only had 3 semesters of scientific programming in Fortran and a shitload of math. That and years and years of building models and massaging data in Excel.
Mostly I'm just really used to learning a new API/tool and applying it to new things.
A lot of the ML stuff hasn't been fancy ML, just basic things but applied in really clever and novel ways.
In my limited experience, it's a mixed bag.
Good: you get special treatment/opportunities because of a unique skill set and increased visibility on the end results of what you do.
Bad: management doesn't really know what you do between software releases, you're paid the going rate for your industry while SWEs make far more, and in-house software quality standards might not be established/followed.
As a tangential bonus, I've accidentally converted my PhD research coworkers to strict git/markdown thanks mostly to typora (windows application). I showed one person what my work flow and version history look like for some internal documentation and now they do the same and convert to word/pdf as a last step. As far as I can tell, no one outside of the math/cs intersection has the patience for latex.
Source: in that boat.
Edit: In regards to market rates, that can be alleviated somewhat in follow up negotiations (6-12 months in or so). It's hard to convince someone what you're worth / what your value proposition is when they're not used to hiring software people. You need to demonstrate your business effect first, since they typically don't have a clear picture of it.
It's already a huge pain dealing with know-nothings inside of tech firms. Now imagine having to cater to know-less-than-nothings somewhere else.
1. you _won't_ be happier running a fast food joint - the work is gruelling, and it's so easy to go bust. And you won't bring home six figures
2a. people who _actually_ can code are scarce, even in the tech industry. Source: conducted over 300 interviews
2b. quite logically, your coding skill will be most appreciated and compensated at a Big Tech Co, not at a government department where they will be simply unable to see the difference
As a young person with an interest in software programming (currently studying chemical engineering but still write C code now and then), what do you look for when trying to find out people who can _actually_ code?
examples of simple things: DFS/BFS walks; simple Project Euler problems; or "write a simple game in terminal, maybe with some form of minimax search" (a bit harder), or maybe a parser for simple arithmetic expressions
Granted I don't know anything about that business, a quick google search comes up with an article from 2015 saying running a McDonald's provides an average annual profit of $150k so it sounds about in the right ballpark range: https://www.mymoneyblog.com/mcdonalds-franchise-cost-vs-prof...
My source is from the McDonald’s franchise disclosure documents. The money blog mentioned in sibling comments claim it’s less.
Very possible I skimmed and may have read it incorrectly, accounting is not my thing. As I have now reached the maximum amount of effort I'm willing to put into a forum comment, I'm not going to dig any further. But if you can tease out better information, I'd be curious to know.
Average profit margin 26-28% Average gross sales 2.2 - 2.6M Average operating income before rent/tax: 570k - 716k
"The rent paid to McDonald’s will vary based upon sales and McDonald’s investment in land, site improvements, and building costs."
It looks like that rent paid to McD's home planet is somewhere in the neighborhood of 10% of that investment (yearly? I guess?) but it seems to average about 100k-150k.
There are tons of software "engineers" who try to escape fintech for FAANGs or even startups, coming from top tier firms like Goldman and McKinsey.
Likewise, bioinformatics generally pays a lot less than traditional SWE roles.
So what I have seen is the exact opposite in practice: people with minimum coding skills in other fields trying to jump into tech firms.
Any time, a piece of paper is passed around or when people enter data manually you can either automate or improve the process. People are prone to errors.
Right now, I am rewriting a mission critical application, some trivial changes will save hours a week and improve the integrity of the information. It is not as exciting as writing algorithms but it is nice to see an application written by you used by 200+ people.
Off the top of my mind:
https://sellercentral.amazon.com/forums/c/selling-on-amazon/... https://community.ebay.com/t5/Tools-Apps/bd-p/tools-apps-db
I know similar situations have to be all around us. The problem, as you say, is finding out about them.
- more than 10 documents per day on year average, e.g a bank will receive new annual reports only in some calendar months but in massive scale
- average number of pages or lines of text per document, the longer the document the more mistakes will be made by humans, as they don't have the time to read everything in detail
- average pay of the FTE who is able to understand and process the document manually should be higher than the average pay of all employees in the company, to make sure the documents encapsulate business value
It's not a fixed set of KPI but helps us to sort out too narrow use cases.
By we I refer to the team behind my startup Konfuzio:
Small payback : on your English home page under "Operation of the software" > "Information security", there's one too many sentences:
> We set the highest standards both when creating the software and when processing your data. Both when creating the software and when processing your data we set the highest standards.
(First one is better imho).
What is the job title I should search for more information? Automator?
The Big Four consultancies and Accenture spring to mind. I think the likes of Fidelity and Citi have few roles too.
Note however that you'd be using off the shelf tools to do the automating - you wouldn't be developing the automation tools yourself.
Hit me up if you ever want to chat! ryan luma.im
That's the industry I'm currently working on... they're generally regressive and have shot themselves in the foot in regards to tech and ease-of-use for a lot of their solutions.
Those systems are easy to sell with "This system will pay itself back in 2,5 years, taking into account your current labour cost".
Given all the AI/deep learning possibilities, lots can be automated still.
I often am not selling to middle management, I ideally sell at the exec and/or product level. Middle managers are a conflict of interest when it comes to what I do.
Any advice about how to even get started doing this kind of work? What tools to study or learn? How to find work?
Don't move around all the dang time. I've lived in two places and have an incredibly strong network in both. I find people who are bouncing around 50-times-over usually lack the deep business relationships that it takes to establish trust to solve "big" problems.
As far as getting started - once you start, just never stop. Solve every dang problem you can get your hands on big or small, and continue to publish code every day.
Tools to study and learn - web web web. Also Microsoft Excel automation is an incredible need for lots of businesses. Puppeteer, and learning how to mimic what a browser can do without using a browser would be the two most useful skills I have personally. If you don't have a VERY solid background in webdev you need to get that going first, so much of what I do is reverse-engineering someone's website so I have to be very versed in the way that sites are built (all the way from ancient ones to the most modern of frameworks).
How to find work is network. Once you have the skills, you need to know people that have potential needs. I've already reached out to two people who found me through this thread and regardless of "making money", I genuinely want to help them, if even to just point them in the right direction (like yourself).
People are (*generally) awesome if you put aside your own ego, and I would be absolutely nowhere if I wasn't a sociable person. I love to learn about the world by making connections with people, and genuinely want to help them with what I can do!
Charging for value can sometimes feel exorbitant... i.e. saving a $100+/hr person ~20 hours a month. Sure I can calculate out it on an excel sheet the customer ROI, but still feels crazy.
If you think about it - $35-40k for a 2-3 month dev timeline (which is what I'd say is my average) is worth it if you're cutting the salaries of 5+ people out of the equation, no need to pull out an Excel spreadsheet for the ROI that. All-in-all - you're not crazy, they're the ones who are crazy if they continue to sink money into human robots.
> Charging for value can sometimes feel exorbitant
Just get confident (this is hard advice I know). I'm incredibly confident in the problems I can solve and will put in the hours to get back on a timeline/budget if needed. Businesses aren't buying my "products" per-say, they're buying me.
Usually it's better for the business I'm supporting to document their own workflows before it's handed over to me for a lot of common sense reasons.
I've never really done the job board thing so no idea on that.
What I’d like to do is fixed price for initial dev and then subscription based contract for support and minor updates, but this seems wholly unpalatable for buyers.
Personally this didn't match what I found online (they said it was not supported i.e. if it breaks tough luck) but what would I know. Money must be spent retrofitting other solutions now.
I joked we should pay a minimum wage high school kid to sit in a room and 'watch' the servers but I guess that's too low tech ;)
Total Microsoft play on that for sure - I've never ran into the same issue though.
These things are really a big deal within large organisations.
You should trace the executable and analyze the function calls. Get a million samples then form a statistical model. The production model replays the function calls given some input. Doesn't need graphical memory anymore and it also has higher resolution inputs (function symbols, addresses) than coordinates on a screen.
Also, at large orgs, software updates can take _forever_. I know of some examples where updating to the newer version of a B2B software taking 7+ years.
And a lot of those automations will need things like OCR or NLP. Even just out of the box Excel integration is a huge time saver. UiPath has activities built in for that. It's more than just GUI interaction. There's a ton of partner technology with out of the box integrations, and you get frameworks to build your automations on, as well as all of the infrastructure and orchestration.
Disclaimer: I work at UiPath
When I first saw it, I thought: this looks like a glorified Autohotkey or Automator. But companies like UiPath are doing very well selling this type of automation software.
It sounds mundane but for a company with lots of entry level staff, the paybacks are appreciable.
Coupled with cloud based OCR as a service offerings etc. a lot of data entry tasks can be automated.
Yep I've had to employ this software to get things done before. Also OCR doesn't need a cloud vs. a beefy computer - I do a lot of that too.
Can you elaborate on this?
Most of my day is literally just getting one step further and one step further into a "process" of some sort, and then handling a metric boatload of edge-cases as they come up. Then the last portion of the contract is the actual integration into a business, and handling the "ninety-ninety" rule that always ends up happening when the rubber meets the road.
I take a complex, convoluted mess and turn it into a coherent "something", usually an API or web experience.
I understand that, like regular application programming, a lot of this kind of automation comes down to requirements-gathering and formal specification of workflow.
But, in theory, could there be some kind of sufficiently-advanced software wizard that does most of your job? Maybe by monitoring entry-level employees' interactions with the software-to-be-automated over some period, and using that as a training input to a workflow guesser?
I'm not imagining that you could be cut out of the loop entirely, mind you, but rather that you'd reduce the human-interaction man-weeks part of your job down to a few hours of pairing with a domain expert to clean up the output of workflow-guesser into a final model.
Nah - I'm going to be cocky for a second so please take this with a grain of salt... Automation is the only job that's safe from automation ;)
That being said, it's prime for development of tools... I've been working on taking a lot of my workflow out of Charles Proxy and moving it onto a more "man-in-the middle" model that can replay certain web traffic... PARTS of what I do, but automating the automating is probably the most difficult problem that I've had to solve (ie: building my own tools).
https://techcrunch.com/2019/04/30/uipath-rpa-series-d/?rende...
My GitLab is my toolbox in so many ways and I've got a wealth of clean/documented code that I can re-purpose depending on the task at hand.
Lots of times I have to measure certain things before I begin to fix a problem, and the above stated tooling (typically ran in Docker containers) has really really helped with that.
I have dashboards, warnings, circuit breakers, etc etc. that all help me monitor complex processes, and that help me react to ongoing changes (ex: when a 3rd-party website updates their DOM).
These are the things I need at my current role, and I'm woefully overwhelmed. Tying together processes spread out over multiple systems is a large portion of my role.
I'm having trouble tracking all the various scripts running on different versions of my own packages, and reacting to breaking changes on 3rd party sites. I definitely need a dashboard to tie all these automations together.
I'd love to chat if you're up for it.
This isn't as "up and coming" as all of the other items people are mentioning, but I'd put it on a "always increasing in popularity" trajectory due to an ever-increasing need. It's not really sexy or interesting, but there will always be a HUGE market for the things that I can do =)
I will warn people that "up and coming" tech is often fad-based and has boom and bust cycles, and personally I'd rather be working for a paycheck then waiting to win the lottery in this regard.