Now of course we don't know what the architecture of this system is and what the deltas in cost would have been to allow this to scale-out more - but I do know that all too often the more robust solution giving you much greater protection and lower cost down the road is often discarded if it costs even just 5%-10% more. Then the day comes when the people making these decisions get caught flat-footed and they try to blame everyone but themselves. It doesn't always happen like this - but it happens a lot.
He decides to do it again, this time with accounts payable, and is promptly fired.
So, probably like most software of this nature, the reason it's not scaling is simply because the people who made it probably weren't the greatest engineers on the block.
Consider: single auth across all the state's services, external APIs, identity verification, address verification, employer ID verification, federal/military ID verification, income/tax verification, phone verification, bank account information, translation into multiple languages, accessibility features, etc. Also, there's probably a lot of legacy infrastructure and process.
Also, if "ability to burst to 10x normal filings per week that might happen once every 40 years" wasn't in the spec, I think they were right not to engineer for it.
It sometimes borders on sealioning.
>IMPORTANT NOTICE: Because of the high volume of claims, we are asking that you help us help you and our greater community.
>If you need to file an unemployment claim and your last name begins with the letter A - M, file a claim on Sunday, Tuesday, Thursday, or after 12 noon on Saturday.
>If you need to file an unemployment claim and your last name begins with the letter N - Z, file a claim on Monday, Wednesday, Friday or before 12 noon on Saturday.
> Ooooh, like gas rationing in the 70s.
I wasn't born back then, but I heard about that being based on License plates at a few car meetups by the older guys in the group and I had the same thought when I heard that on CPR.
Odd, but it could work if you have total compliance; lets see how that pans out.
It doesn't take many resources to show the user a form, validate it, and save to a DB.
You are talking about comparing a basic web form with an application for unemployment benefits which must go into a federal tax database and be processed using a what I assume is a garbage mainframe system.
It not only needs to be validated, it needs to securely store records, be able to compare them, and hook up to the system that handles payments, etc.
They can't just circumvent it and dump it into some silly Amazon or MySQL database and call it a day. That would require the employees to basically copy and paste that data into the actual warehouse and considering they have 3+ million to go through as it is making it easy for them to process is just as important as allowing people to submit.
For the time being the correct response is a queue gate.
Stop being silly.
The reality of government paperwork systems on the backend is much, much closer to this hell and is part of why so many like myself ran screaming from public sector because when you see so many peers doing so well at FAANGS, why would you subject yourself to something that resists change and wants to keep it the same way? https://www.washingtonpost.com/news/federal-eye/wp/2014/03/2...
There is the option of going to some area with a much lower cost of living and trying to hire there, but the problem might be getting enough people together to form a team. If you can easily get enough people with skill and experience, the area probably has jobs for them that pay better, and if those jobs don't exist, it might be hard to find the people.
California is a huge state and the Bay Area is going to have drastically different stats for even the same industry comparing San Diego, Los Angeles, Sacramento, and San Luis Obispo (yep, there's software jobs there too).
Now, not so much, but if my circumstances change, then who knows?
What if the backend rejects the form? The user's already moved on before their form made it through the queue. So then you're stuck re-implementing all the validations the backend needs in order to give the user feedback (which you may not even be able to do) or trying to get the user to come back later to try again.
> Making the problem of getting the application through backend systems the states' to deal with, not the applicants'.
Reducing permanent staff involved in processing applications is probably one of the main reasons the automated system was built in the first place. If they still have to do that, then you might as well just replace the frontend with a printable PDF.
There is already processes (a workforce and/or outbound written letters) to reach out to applicants in the case of eg a dispute (terminated for cause vs laid off).
The point is that it's easy to say things should be easy when you don't know anything except the very surface details of the problem, and it's not your job to actually solve it.
Maybe the team that built the system in question were a bunch of dumb-dumbs who just needed a rockstar developer to show them how easy it is to scale, or maybe the problem is actually more complicated than it seems due some hidden complexities or constraints none of us actually know anything about (either technical or business).
Hence mainframe maintainers should really move to charging $1 million/year in a decade or two.
Once you have more people who can understand that there are scientific and moral issues with manifest destiny, and religion isn't going to solve global warming, there will be some shifts in the public discourse and public policies.
https://www.canada.ca/en/services/benefits/ei.html
In the US, though, it's by-state.
If the form has to go into a mainframe well just set up an asynconous Queue
...okay, whatever you say.
I bet that's what the previous developers thought.
What happens if you need to validate the form data against an external service that's coming and going due to the traffic spike?
What if your database is rejecting transactions occasionally?
What happens when your backup process locks all the database tables?
How do you reject duplicate form submissions from people hammering the submit button? Do you query the database to find previous submissions?
What happens when a scriptkiddie decides it'd be fun to DDOS the site? How do you differentiate good traffic from bad traffic?
What do you do when the cloud provider runs out of space and you can't scale up any more (https://news.ycombinator.com/item?id=22691926)?
You need to think of all of these things and many, many more to run a robust online service that can handle spikes hundreds of times bigger than the usual level. It's really not straightforward or simple.
When "hundreds of times the usual level" is still only 50 page loads per second, and 10 milliseconds of CPU per page would be extreme overkill for anything written in a reasonable way, it actually is straightforward.
(That is not to say it's necessarily the devs' fault.)
Aside from potentially having to interface with dozens of unreliable, painfully slow SOAP-based web services, everything is often hosted on creaking, over-subscribed VMWare hosts, in VMs that would be under-specced regardless.
There is also often a "governing body" that severely restricts your tech stack choices.
Want to use Postgres? Nope, our standard is SQL Server - 2008 edition, actually!
Want to use Python/Ruby/Elixir/Clojure/Kotlin? None of that hipster nonsense here, we use good ole Java/VB.NET here!
Message queue, you say? It's Windows Message Queue with distributed COM all the way down here!
"Containers"? What's a one of those? You'll get a crappy VM with 1 vCPU and 1GB of RAM, and you'll thank me for it! etc...
As a dev, it's horrible and soul-destroying to work under such limitations, but if you have no choice...
I have had to build python distribution completely in home/user-space in some cases, working on conservatively managed servers.
Queuing access to the form itself and telling someone to wake up at 4:52 AM so they can then merely access the static assets is a less-than-desirable user experience.
And now you've given a private company access to market-moving unemployment data. And a million other issues, especially legal ones.
The technology part in and of itself isn't that difficult, it's all of the constraints (and, often, mountains of laws) that are the bigger issue.
You’re assuming that the people who built it in the first place (or the people that may or may not be contracted to fix it later) know or care. Remember, this is government contracting we’re talking about - lowest bidder wins. How do you win the lowest bid? By doing it as cheap and quick as you can. That means hiring inexperienced/cheap developers who can build something that looks like it will work for far less money than you can build something that actually will.
I briefly interned with a state judiciary's IT department around 2015 and got to get lunch with the CIO. He described to me how most court filings in the state had been manual prior to 2008 when the mortgage crisis hit and judges in the tax courts got _slammed_ with cases surrounding foreclosures. This , in turn, drove a need to develop a platform to automate the process of filing a case. It started with the tax court and gradually expanded to automate filings for other court divisions as well (e.g. Family, Civil).
I wouldn't be shocked if the revelation of "holy shit no one can file for unemployment" drove such an investment. I honestly think the next generation of politicians should take a page from product owners by isolate some shitty process that they'd have jurisdiction over, and finding some way to automate it. Bonus points if it's right before a watershed moment- imagine if someone had considered the problem you described prior to the coronavirus epidemic.
Pennsylvania reported 378k claims.
California reported... 186k claims.
Yesterday, California's governor said they've received more than 1 million claims since March 13th (so, over a 12 day period from the 13th to the 25th). This DOL report covers March 14th through the 21st.
Are we to believe that the remaining 800k+ people all filed on March 13th, or March 22nd through the 25th?
But there's more. Utah reported an increase of only 9 claims compared to the week before. They went from 1,305 to 1,314.
Then, New York, where more than half of Covid-19 cases in the US are, reported only 80k?
I could believe it over the 22-25 stretch.
I assume some less computer-literate higher up thought that someone needed to be around to actually accept the form, same as in-person submissions.
My wife managed to get her filing completed a little after 1am this morning. She was the only one of her 20 coworkers to successfully file, the rest are continuing to attempt to get the state web site to work today, while more people pile in.
These numbers are going to get much, much worse.