Within the last couple years, I've overheard senior, so-called technical government employees 1) complain about Git and wonder aloud why we weren't using Visual Source Safe; 2) insist that rotating through a list of 12 hard-coded passwords, in code, checked into Git, was totally fine; 3) refuse to believe that automated deployments were possible (not hard, or against norms, but physically impossible); 4) try to explain to another so-called technical gov't employee the difference between CSS and JavaScript, and get it wrong; 5) stand up in the middle of a conversation and walk out the door because it's 2:30 PM and their day is over; 6) even more nonsense you wouldn't believe if I showed you a video of it.
I would hope Federal is a little better than State, but I'm not convinced.
I worked with Federal briefly. I was at a startup at the time, and we had this integration with the Dept. Homeland Security. I was young and stoked. I was the one who was going to wire everything together. It seemed really big and important. I thought it'd be a gold star on my resume.
Reality was super disappointing. So much "we're working on a document for when we can give a date for a date on another document" kind of bureaucracy. I'm sure down in the deep cores of these agencies there are super bright and talented people. The ones at the edges, though.. the problem seems to be that you have to be more of a politician than an engineer to actually get anything done.
In Red Mars, they try to colonize Mars. At first, the challenges are completely technical, and the engineers prevail. But the success is chased by increasing bureaucracy and involvement of stakeholders, and it quickly becomes all politics. c:
And then quickly becomes colonial revolution.
You practically have to assault someone in the break room to get fired from a government job.
Organizational inertia is definitely a thing, and technology trends do not move as quickly in all sectors.
My guess is, in most circumstances, probably not
"New tech" lately feels more and more like whatever the next breed of inexperienced geeks decided to reinvent
I'm told that at this point sales will quietly try to talk you out of buying it.
If you want people in a company to be able to produce good ideas, they must have reasonable leeway to disagree or be contrarian.
My dad would do the same stuff. He worked at Samsung. When he got into his 50s, it was like a switch turned on and he just wouldn't stay beyond a standard work day. Got a meeting you scheduled late in the day? Too bad, he was there at 6am and was going home. He was always home at the same time.
In comparison, during this period of time my mom would go into the office at 8am and stay as late as 1am. She would even pick up 2nd dinner for me on her way home.
When I was in training with the RCAF, I was in lecture where the Information Systems Security Officer (i.e. the senior most infosec person at this base of 6-8000 people) told us about a time where "two guys were emailing back and forth and just picked up a virus".
I made the mistake of pointing out that that's not really how computers work, i.e. an email can't just pick up a virus in transit like a dog picks up a tick.
I learned two lessons that day:
1. Militaries don't like it when people stand out
2. Governments are fundamentally incompetent
Medicare spent months just digging into the laws and talking to their lawyers just to prove they actually had the jurisdiction to change the policy. That really blew me away. One of their bigger concerns was "this might not be our job".
You make it sound like they spent months trying to finagle a way out of working.
> Don't underestimate the absolutely jaw-dropping incompetence from even senior "tech" folks employed by governments.
100% agree. Except each agency can significantly change on this. Some agencies (DEA, USCIS) can and are completely competent. I've seen then turn around on certain moderately complex tasks within hours.
Then you have the idiots at HISN and VA that takes a year for even basic one-way integration. Or, they will argue with you how SAML works (or doesnt) for weeks at a time.
> 1) complain about Git and wonder aloud why we weren't using Visual Source Safe;
Ah so you were talking with their developers. Sounds DoD'ish as that's what they use, along with Firebird.
> 2) insist that rotating through a list of 12 hard-coded passwords, in code, checked into Git, was totally fine;
Ive dealt with network infrastructure entities that did similar for the commercial side of things. I wish it were the "OMG" exception. There's more terrible out there than this, admittedly.
> 3) refuse to believe that automated deployments were possible (not hard, or against norms, but physically impossible);
Well, they ARE impossible by policy. FedRAMP and FISMA requires manual and intentional deployments. Of course we all know that this is the norm for a software SaaS company - but its not the norm when you have to go through a VPN with a CAC or PIV card, and then another VPN to either the testnet or prodnet... And the only software support for both are Windows. (And the VPNs will also kick you out if you're not running windows.)
> 4) try to explain to another so-called technical gov't employee the difference between CSS and JavaScript, and get it wrong;
I've had screaming fights break out in calls when technical issues came up like this. And usually its the single real employee screaming at the legions of 3rd party contractors. And also, Snowden was correct in that 95% of the "government employees" aren't. They are contractors working for Accenture or the host of other Gino (govt in name only) employees. They don't get govt benefits, nor do they have whistleblower protections. Snowden in Hawaii was such a 3rd party govt employee.
I have only ran across 1 fed employee (contractor or real) that was in any way and shape competent.
> 5) stand up in the middle of a conversation and walk out the door because it's 2:30 PM and their day is over;
I get your point. But it's usually 4p. Ive also seen government employees (employed by the local/state/fed) get in trouble for working non-critical issues outside their hours.
> 6) even more nonsense you wouldn't believe if I showed you a video of it.
One thing I've learned is that there is almost always a reason (and a decent one too) of why there is a rule. It's because someone previously abused something, or did something that broke stuff, and the new rule is the usual overcompensation so that anybody in power can claim "They didnt follow our rules, which protect against that." And remember, it's not far up the foodchain to an elected official.
But yeah, there's absolutely crazy shit out there, especially in the fed side of things. And much of it is for complete show. One such tempest in a teacup issue is the new (as of 2017) NIST password guidelines: not a single fed has implemented them. They still enforce the old and terrible ruleset.
It probably happened like this: In 1995 people started planning for this Y2K bug. Some developer said "Nah it's not an issue I'll just change this field" so they changed it like they change everything else, and then 6 months later people got paid the wrong amount because a date calculation was wrong.
So they went looking for someone to blame. They fire the developer of course, but they want someone responsible, who signed off on the release. There was no one, so and with the biggest software fixing exercise in their history coming up they figured they should make sure this problem didn't happen again.
So they introduced sign-off procedures, where someone above a certain level had to sign off on things.
And that is how manual deployments are "banned". They probably aren't exactly, but some poor person will have to put their job on the line to sign off on automating them.
Nobody who gives a shit about their product/service is deploying from a developer directly into production without any kind of, you know, code review, pipeline testing, etc.
It's 'automated' in that you git push. The pipeline handles code reviews, sign-offs, etc., and then automatically deploys the approved code change.
This isn't banned at all.
Just randomly pushing shit to production is banned (separation of concerns, amongst other controls). You can still have everything be fully automated, just not fully autonomous. Someone reviewing a change and then clicking 'LGTM' is still an automated, continuously integrated and deployed system.
I know 'why' people might think that, and make assumptions. The actual controls are pretty level headed though.
How do we replace them with folks that know what they're doing? It almost sounds like it would be better for the government to outsource to a tech firm.
These people have huge salaries and pensions, too, right?
Huge salaries? No. The agencies I contracted for topped out for senior folks in the $85-90k range while the same title in the private sector is going to be about 40-50% higher than that. They have pensions but unless you're already 15 years in you really need to work the system to get anything out of it. New hires (rightly) have to put a lot of money into the system so it's not much better than a private retirement plan in terms of returns, however it is guaranteed by the taxpayers.
There are exceptions, though. If you're retiring from any of the agencies I contracted for, and you worked there your entire career, you're probably getting a lump sum $80-100k vacation time payout and anywhere from $60-100k/yr for the rest of your life depending on your exact job. This is in addition to whatever personal retirement accounts you might have. Obviously no 401(k) with the pension in place, but it was not unheard to have some of the higher ups cashing in just enough vacation every year to fully fund their Roth.
So yeah you can set up a nice retirement nest egg if you're willing to commit to the state (at least where I live). But for short term stuff it's terrible. You don't even vest any pension benefits until I think 5 years now, and then you're getting something like 5-10% from age 70 on or something.
State employees with 25 years get 54 days of leave a year[1] and 10 federal holidays. Additionally many agencies work 9 hour days, adding every other friday off, for an extra 26 days. All told, that is 90 days off a year, or more than 1 in 3 work days.
https://www.calhr.ca.gov/employees/Pages/vacation-vs-annual-...
$80k/yr with pension benefits and a third of your work days off is better than a lot of private sector comp. I'd happily take that last benefit.
The agencies I worked at routinely had employees' shifts ending at various times, anywhere from 2:30 until 5.
Here's an example from my own experience:
I can recall a FOIA request I made for the results of searches in an internal government database in spreadsheet format. To my knowledge this can be exported using existing functionality in the web interface for the internal database. The request was denied, and I appealed. They claimed that the information was publicly available, which is absurd. In the appeal I showed that actually almost none of what I wanted was publicly available and the independent appeals office agreed (after a long delay). Then it became clear that the FOIA office thought the request would be burdensome and they tried to deny it on those grounds. I insisted that no, it wouldn't be burdensome.
When it finally went to the person who would actually do the work, apparently it took only a few hours to do, and they declined to charge me anything for it. I assume the majority of that time was writing some useful comments in the headers of the spreadsheets. They didn't need to write the comments, so clearly they weren't pressed for time.
In the end, the FOIA office spent far more time trying to deny this request than it would have taken to fulfill it.
That's exactly what I was hinting at here. The agency has no interest in the public finding out how much of that feedback was bullshit.
> In my experience it's likely that no technical people were consulted on it.
From the sounds of it, they did at some point consult someone technical and tried to parse their answer as part of their justification for not executing the FOIA request. The fact that they knew it would take a script to extract the data as opposed to just dropping it into a search tool suggests as much anyhow. Though more likely it would just be a couple `sed` commands, a perl/ Python script, or a few command line tools.
They are using the "defense" that it may be difficult to do, but their aim (avoiding the more sinister possibilities) is to stop precedent for other unwieldy requests.
You’ve never dealt with a bad log setup. Things I’ve seen:
- inconsistent format depending on which part of the software was logging (think of some modules using logging lib and others using print())
- multiline logs without any indication they are multiline
- logs in some dumb binary format that requires the original software to reverse into legible text
There are more, but the point is that logs are not often designed to be parsed and searched. They are normally added by developers to be human readable means of reversing internal state to debug issues. Anything above that is (shockingly) bonus material that doesn’t make it into home-grown enterprise/govt software.
I'm not sure how this is relevant. I've seen miles of incompetence in my career, that someone can fudge something up royally doesn't change the fundamental purpose of the thing. The point of log files is to store and make retrievable data about server state and actions over time. That someone somewhere can screw that up doesn't change that.
In this particular case, I find it unlikely they rolled some custom bizarro log file format regardless because of the nature of the subject matter: Proxy server log files. It's unlikely they rolled their own proxy server (and they certainly don't imply any particular difficulties like that in court statements) so we're most likely talking about reading standard Apache logs or something similar.
That’s a point of log files. However, it’s not the point tons of developers have in mind when they add log statements. Developers are primarily concerned with state required to reproduce bugs, not long term standardized storage containing enough information for useful attribution or even all important state transitions. You would be surprised how many developers don’t even bother logging item deletions because it’s such a trivial code path.
You’re idea of “the fundamental purpose” is from an operators view. It rarely lines up with the developers’ views and you’re in for a bad time if you think people who don’t see your perspective are “fudging something up royally” with “miles of incompetence”.
https://en.wikipedia.org/wiki/MAI_Systems_Corp._v._Peak_Comp....
P.S. Please correct me if I got any of this wrong.
What it really stands/stood for is the notion that a "copy" sufficient for the operation of copyright law occurs when a program is loaded from disk to RAM, and therefore it's possible to infringe copyright merely by using software that you aren't personally licensed for.
MAI basically used this theory to make third-party maintenance of their systems illegal, so Congress responded by amending the Copyright Act, but in an extremely narrow way: there's a carve out specifically for maintenance and repair.
[1] https://www.vice.com/en_us/article/4xaqz9/hillary-clinton-te...
If we replied to a lawful order with "This demands 'research' instead of simply a 'search'," they would tell us tough-shit.
Funny how a government agency considers it overly burdensome to write what is likely a 4 line script, so goes through a lawsuit that costs taxpayers likely millions of dollars to avoid it. Unless their infrastructure is well and truly F*ed, this is a 30-60 minute task for a junior server admin. Most likely they already ran it and didn't like what it revealed so it's even less effort.