News for hackers. Stuff that matters.
Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by raverbashing on Monday April 30, 2018 @02:12PM (#16960876)
The question for the industry remains: how do we deal with systems that should be up for 10 years? 25 yrs? 50 yrs? 100 yrs? We didn't even have computers as they exist today 50 years ago.

How do we separate business logic in a self-contained way that can be independent of hardware and (from a certain level) software?

Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by romaniv on Monday April 30, 2018 @06:13PM (#16962903)
>The question for the industry remains: how do we deal with systems that should be up for 10 years? 25 yrs? 50 yrs? 100 yrs? We didn't even have computers as they exist today 50 years ago.

Is this a question the industry actually cares about? If anything, I'd say it got much worse at creating long-living software. It sucks to run a mainframe-based system from 50 years ago, but running a 10-year-old Web Forms application can be borderline impossible. I'm scared to think of what will happen with all the cloud-based stuff in the next 10 years.

>How do we separate business logic in a self-contained way that can be independent of hardware and (from a certain level) software?

I often say that there is no such thing as "business" logic. There is just logic. The least maintainable systems I've worked with were the ones with the most sophisticated attempts at abstraction.

In my opinion, the key to software longevity is making systems that allow for local reasoning, i.e. analyzing parts without knowing the whole. This can be achieved in a lot of different ways, but it needs to be a deliberate goal in system design.

Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by raverbashing on Tuesday May 01, 2018 @07:51AM (#16966238)
I agree it's a tough cookie. I actually don't think layers and layers of abstraction is a good idea.

But something were you can describe something like "Read this value from this field, do some math based on fields in another table, return the result"

That's why I raised the question, how do we do that over long periods of time without rewriting it in several different ways? (ASM to flat files, COBOL and JAVA to different SQL flavours, what else is next?)

And if you migrate, let's say from Oracle to MSSQL it will work for 90% of the things, for the last 10% it will be a PITA.

Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by bartread on Tuesday May 01, 2018 @05:53AM (#16965699)
> The least maintainable systems I've worked with were the ones with the most sophisticated attempts at abstraction.

Seconded. Enterprise patterns mandating layers and layers of abstraction, making it incredibly hard to reason about the far flung pieces of code that actually do anything.

Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by dx034 on Tuesday May 01, 2018 @09:19AM (#16966779)
> Is this a question the industry actually cares about? If anything, I'd say it got much worse at creating long-living software. It sucks to run a mainframe-based system from 50 years ago, but running a 10-year-old Web Forms application can be borderline impossible. I'm scared to think of what will happen with all the cloud-based stuff in the next 10 years.

Cloud deployments, especially modular systems (and of course microservices) are often sold to companies as the best way to reduce technical debt. I don't think they've been around for long enough to truly test it but would be curious if a 30yr old system of microservices running in containers is easier to update than COBOL software running on a mainframe.

Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by ChuckMcM on Monday April 30, 2018 @04:04PM (#16961755)
We have shown pretty clearly that any computer we build at day 1 can be emulated perfectly by a computer built day 1 + 50 years. However, building systems that can evolve is a larger issue.

The nut here is that these systems evolve in a very haphazard way do to funding availability, scaling stresses, and changing requirements.

For the IRS, I think it would be useful to engage the US Digital Service to see if they could design a system based on a cluster of non-specific hardware with only network interfaces, that could host a 'tax computation engine' and a growable data store. While it is a 'big' problem, it is not as big as some of the problems I have seen hosted on 25,000 machines as a 'small' cluster. That is 800,000 cores (assuming 32 core machines) so for 140M tax payers that is 175 payers per machine. Something that could no doubt process every return in a single day, and run fraud analysis as well.

Clearly the government needs warehouse scale computers that their vendor (IBM) is not in a position (yet?) to deliver.

Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by jcranmer on Monday April 30, 2018 @04:13PM (#16961822)
> We have shown pretty clearly that any computer we build at day 1 can be emulated perfectly by a computer built day 1 + 50 years.

We can emulate pretty much any old system, but emulated "perfectly" is at best not demonstrated and at worst already proven wrong. Perfect emulation implies emulating the undocumented bugs that end up working--and there are a lot of those. Getting the correct behavior in the case of parallel components (note that this applies even to single-core systems, since dedicated hardware constructs can race with the CPU) is particularly tricky. Even without parallel issues, you can have cases where software blatantly violates the API but it works reliably and consistently, necessitating complex emulation to make it work correctly (there are some games that don't work on WINE for this reason).

Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by DrScump on Monday April 30, 2018 @04:39PM (#16962096)
"Getting the correct behavior in the case of parallel components (note that this applies even to single-core systems, since dedicated hardware constructs can race with the CPU) is particularly tricky."

Heck, even if you emulate the documented performance perfectly, you can still suffer regression behavior if the existing program just happens to rely on misbehavior of the original hardware due to less tolerant exception handling, stricter memory alignment protections, etc. in the later deployment environment.

I've seen this a lot in memory misuse / misalignment when porting between various UNIX implementations. If your development environment just happens to forgive a lot of sloppiness (cough Solaris cough), misbehavior may only show up later on other platforms.

Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by dfox on Monday April 30, 2018 @08:03PM (#16963619)
As for the solaris it is probably only UNIX where ignoring the ISO C rule that anything ending with "_t" is reserved for implementation will bite you hard as solaris' libc headers are probably the main reason for that and typedef stuff like lock_t.

As for the rest doing cycle exact emulation of 50 year old system is and probably always will be doable and from some PoV trivial: you can just simulate the hardware logic of the original implementation on RTL or even gate level and it still will be faster than realtime.

[Edit: "reserved for implementation" was missing from the first paragraph]

Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by fouc on Monday April 30, 2018 @10:19PM (#16964159)
It occurred to me just now that the US Digital Service will become increasingly powerful as it helps digitizes the government, and 30 years from now it could be synonymous with the government.
Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by Retric on Monday April 30, 2018 @02:18PM (#16960915)
> How do we separate business logic in a self-contained way that can be independent of hardware and (from a certain level) software?

As strange as this may sound, documentation. Software is at it's core rather brittle it's designed to operate inside of a very specific system and trying to make it portable is only reasonable when you know the target ahead of time.

Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by arcticbull on Monday April 30, 2018 @09:04PM (#16963885)
Better yet, test cases. That way you can throw your new implementations against them.
Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by dman on Monday April 30, 2018 @02:15PM (#16960895)
How do you create incentive structures for new employees who have to spend months (years?) coming upto speed with giant state machines before they feel remotely confident of the the effect of their changes? For better or for worse, currently the glory is in writing new systems rather than improving existing ones.
Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by PopePompous on Monday April 30, 2018 @07:18PM (#16963363)
"state machines" - nice pun in this context.
Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by s2g on Monday April 30, 2018 @05:02PM (#16962309)
You get them doing interesting work on the system as often as you can.

Fixing bugs and babysitting all day isn't going to convince people to stick around. Best you can do is get them something they can try to work with.

Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by DrScump on Monday April 30, 2018 @04:41PM (#16962129)
Unfortunately, it's not just about "glory". If you work in "obsolete" languages/environments, you likely will be perceived as less valuable to your next employer.
Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by JustSomeNobody on Monday April 30, 2018 @04:57PM (#16962255)
This right here is a huge problem with our industry. If you're not jumping from employer to employer every 1.7 years, you're labeled as out of date. It takes some really talented developers to truly understand some of these old systems. They should be given the proper respect they deserve.
Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by ror6ax on Monday April 30, 2018 @06:38PM (#16963103)
In reality it often means these systems are built in bad, ineffective way. Most likely in bad, ineffective languages/technologies. You don't need 17 years to be able to contribute to Apache Spark. You would not call that a small project, would you?
Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by s2g on Monday April 30, 2018 @05:04PM (#16962322)
No, you can stick around.

I wouldn't consider someone who spent 5 years on AWS, especially if they moved internally, out of date.

Whereas I've done nothing remotely close to where I want to be in about 3 years. I feel very scared about my skills, especially since I'm not even 10 years into my career.

Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by hermitdev on Monday April 30, 2018 @09:18PM (#16963949)
I spent 9 years at my first FT job out of college at a hedge fund known at the time for culling the bottom 10% of performers annually at the time. That long of tenure at a place with that reputation has openned more doors than it has closed, especially within the finance industry. Lately, I've been getting 4-6 emails from local recruiters per week.

An aside, even when you spend that long in a single position, you do need to keep learning on new and relevant technology; I attended every on house training seminar I could, which helped a lot, and spent a good portion every day reading. I spent 7 years at that company in the same effective role, and never changed job titles, but saw a >200% increase in take home pay in that same period.

Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by gowld on Monday April 30, 2018 @07:53PM (#16963557)
I heard that COBOL programmers do pretty well because they are a rare commodity serving expensive industries.
Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by raverbashing on Tuesday May 01, 2018 @02:33AM (#16964966)
If you're happy with working with something that destroys your brain and makes you useless to any other software technology developed in the last 20 years be my guest.
Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by DrScump on Tuesday May 01, 2018 @07:46PM (#16972803)
COBOL doesn't destroy your brain, that I recall. In fact, if it bores you, it leaves a lot of brain capacity available for funner stuff, like MVS exploits via Resolve.

To me, it's the cool-app-of-the-month-that-harvests-private-data-off-peoples-devices that is brain-killing.

Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by colechristensen on Monday April 30, 2018 @10:54PM (#16964273)
>How do we separate business logic in a self-contained way that can be independent of hardware and (from a certain level) software?

Come up with programming languages with limited domain specific functionality that don't allow you to do funny machine specific things. (or even ordinary things like pointers)

Use the restricted languages for business logic and have a second language to do the plumbing.

Or just target a generic x86_64 virtual machine.

Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by dx034 on Tuesday May 01, 2018 @09:21AM (#16966794)
So all critical software in Python or JS? Not so sure about that. I get the idea but think having strict code guidelines might be more effective than choosing a more restrictive language.
Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by colechristensen on Wednesday May 02, 2018 @12:57AM (#16974165)
No no, restricted, a programming language that can't do a lot of things. A language that can only do business logic kinds of things.
Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by js8 on Monday April 30, 2018 @02:42PM (#16961119)
IBM mainframe assembler is actually somewhat abstracted from the hardware. The original assembler is very CISC-like, but today it's running on RISC-like cores.

As for the hardware, I/O is done through channel programs, which can have any device as a backend - it's a generic way to do DMA.

There is a lot of legacy stuff (like CCKD, VTAM) but more or less it's all emulated today by the hardware/software.

Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by pishpash on Monday April 30, 2018 @02:18PM (#16960917)
I'm sure academics have thought about such information representation problems. As for industry, why would they be interested in that? It's like killing the goose that keeps laying golden rewriting eggs.
Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by Shivetya on Monday April 30, 2018 @05:49PM (#16962744)
IBM has already shown that the machine layer and the OS layer can be separate and that progression in one does not necessarily require a full change in the other.

the method by which it works is that code compiled into a new form which can be translated on the fly to match the hardware level below it.

now what is probably trapping the IRS besides a base that not everyone full understands but the sheer complexity of what they have to deal with, the tax code and whims of those in Congress.

Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by yazaddaruvala on Monday April 30, 2018 @07:36PM (#16963463)
This is a relatively solved problem. Abstractions.

50 years ago the IRS had to build a webserver, the servlet application code, it had to build in fault tolerance without SQS or Kafka.

If built today, the IRS' entire system would probably run on Rails with JRuby, deployed with Docker and K8s, and be 1000x more maintainable.

Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by DonaldFisk on Monday April 30, 2018 @08:07PM (#16963633)
50 years ago the WWW didn't exist and there was no such thing as a servlet.

The system ran on a mainframe not 50 but 60 years ago. The only high level language at the time was FORTRAN, but this system was apparently written in assembly language.

Does anyone know the type of computer it ran on? The article it linked to says IBM mainframe but which model?

Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by cat199 on Monday April 30, 2018 @09:14PM (#16963929)
> The only high level language at the time was FORTRAN, but this system was apparently written in assembly language.

No direct insight to the actual system used, but FWIW IBM mainframe assembly IMHO seems 'surprisingly' user friendly due to the large amount of standard macros/routines provided, clear/direct integration with JCL, and the minimalist ethos of the OS. Take a look at some youtube videos for some examples, it actually looks like a fairly interesting/productive, if arcane, environment. Writing clean/well architected applications in it doesn't actually look like it would necessarily be a ridiculous notion.

Of course, how well any given production system was constructed and what cruft has accumulated in it over 50y is probably another story..

Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by saratogacx on Monday April 30, 2018 @10:04PM (#16964101)
They use IBM/360[1] mainframes. Bloomberg had a good story [2] some time ago that was re-run last April 17th on some of the history.

[1] https://www.washingtonexaminer.com/tech-timebomb-the-irs-is-...

[2] https://www.bloomberg.com/view/articles/2018-04-17/the-irs-c...

Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by beezle on Monday April 30, 2018 @10:13PM (#16964136)
COBOL, ALGOL, PL/1
Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by DonaldFisk on Monday April 30, 2018 @11:34PM (#16964419)
Not in 1958.

The first COBOL compiler was available 2 years later in 1960. Algol 58 wasn't available until 1960 either.

The ALGO (Algol 58) manual for the Bendix: http://www.piercefuller.com/collect/bendix/algo6008.pdf

The first compiler for PL/1 was delivered much later, in 1966.

You could have suggested LISP 1.0 but that appeared in 1960 too (as an interpreter, the first compiler was written in 1962).

LISP, COBOL, ALGOL: 1960 was a remarkable year in the history of programming languages.

Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by justinator on Monday April 30, 2018 @07:49PM (#16963533)
Oh yes, 1000x maintainable with 1000x more lines in the codebase.

Speak softly into my ear, these little lies you like to tell me.

Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by chiph on Monday April 30, 2018 @08:10PM (#16963649)
Make sure you're including all the lines, including those in libraries/frameworks you reference. Because they all count towards your MTBF.
Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by samlents on Monday April 30, 2018 @08:09PM (#16963647)
Having not worked on large scale systems like what I imagine they must be working with, is this really true? I would have thought that a modern framework would make things easier. I've certainly had life made easier by modern web frameworks like React and Angular, although the support timeline is more on the order of 2-3 years, rather than 20-30.
Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by dx034 on Tuesday May 01, 2018 @09:23AM (#16966814)
Not sure, what if you have an enterprise system that uses a few hundred docker containers and at some point in 10-20 years there's a major vulnerability that can only be fixed by breaking compatibility for one function? Securing that system could be significantly harder than fixing a monolithic system.
Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by empath75 on Monday April 30, 2018 @11:13PM (#16964345)
From my experience the government is excited about kubernetes and openshift and are slowly starting to use it for prod (security approval is tricky)
Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by DonaldFisk on Monday April 30, 2018 @08:13PM (#16963659)
If the hardware is unsupported and no longer maintainable you could run the software on an emulator. That way you don't need to know how the software works. The hardware's instruction set would be a lot less complex and almost certainly better documented.
Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by tonyarkles on Monday April 30, 2018 @08:22PM (#16963698)
Whenever this comes up, I’m reminded of two of my favourite calculators: the HP48 and the HP49g+. HP used a processor called the Saturn for a long time, and when they finally could no longer source it, they switched to ARM running a Saturn emulator for the ‘49. The ‘48 had battle-hardened firmware, and this was probably a great decision.

https://en.m.wikipedia.org/wiki/HP_Saturn

Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by matchagaucho on Monday April 30, 2018 @04:45PM (#16962160)
At least this article acknowledges a period of "parallel validation" prior to retiring the old system.

Too many legacy migration projects fail to account for the active-active nature of migration.

Re:IRS’ 60-Year-Old IT System Failed on Tax Day Due to New H... by OrganicMSG on Monday April 30, 2018 @10:44PM (#16964241)
Code all long term business logic in magnetic core memory running on hardened valves with dixie tube displays.