News for hackers. Stuff that matters.
Re:How ACH works: A developer perspective (2014) by joshribakoff on Thursday August 24, 2017 @09:55PM (#15095554)
I had an integrator request this so I stood up a nodeJS server that only implements upload, not download. This way if they leaked their own password, a malicious actor is limited to forging data, and no real data can be leaked. Because it didn't work in FileZilla, they didn't want to use it. Worked at another company that shuffled data between big name gyms & health insurance companies, it also used CSV files sent over FTP in all directions, to my dismay. CSV isn't even a well defined format, and you get all kinds of impedance mismatches with different delimiter & escaping mechanisms, character encodings, BOM, etc. Other companies will just give you a SQL user & let you go to town mining their database directly. I don't understand whats so difficult about making an API, but sometimes it seems like no one wants to do it. You can't push back too much or they will just see you as a problem & decide not to integrate with you.
Re:How ACH works: A developer perspective (2014) by marktangotango on Thursday August 24, 2017 @10:24PM (#15095670)
Funnily, you're complaining about csv formatting, guess you never worked with ACH formatting. It's cobol fixed length files. I implemented parsing/creation for this file format at two different companies. Hired on at one company, first day, my manager said, "hey, you have experience with ACH right? we have this project...". That's when you learn to start leaving stuff off your resume.
Re:How ACH works: A developer perspective (2014) by bct on Thursday August 24, 2017 @11:24PM (#15095879)
You can do a lot worse than ACH. It's hard to read, but it's simple and pretty well-defined.

What _really_ sucks is one-off fixed-width formats that aren't well defined, or that change suddenly (oh, you thought that field would always be populated? lol no.)

Re:How ACH works: A developer perspective (2014) by HillRat on Friday August 25, 2017 @12:03AM (#15096022)
Yeah, ACH is surprisingly not-unpleasant, at least relative to nightmares like X12 EDI with its implicit looping constructs and billions of companion guides that supercede random parts of the base spec.
Re:How ACH works: A developer perspective (2014) by sixdimensional on Friday August 25, 2017 @12:29AM (#15096127)
X12 EDI, been there earned the badge, both on the generating and receiving sides. 837, 834, 835 and others in health care - fun! Positional format, with situational meaning... really it is a fascinating format.
Re:How ACH works: A developer perspective (2014) by 20after4 on Friday August 25, 2017 @01:36AM (#15096350)
Don't you mean nightmare format?
Re:How ACH works: A developer perspective (2014) by eric_h on Friday August 25, 2017 @12:37PM (#15099835)
Nightmares can be fascinating
Re:How ACH works: A developer perspective (2014) by pg_is_a_butt on Friday August 25, 2017 @01:07AM (#15096256)
i wrote the X12 EDI stuff for a major clearwater distributor... also wrote the ACH stuff that dumped someone a file and their job was to upload it via SFTP to the bank. so many badges.
Re:How ACH works: A developer perspective (2014) by oddsignals on Friday August 25, 2017 @04:01AM (#15096919)
What _really really_ sucks is well-defined, reasonable, meticulously documented formats where all in-the-wild implementations stray from the spec in different ways :-/
Re:How ACH works: A developer perspective (2014) by tonyedgecombe on Friday August 25, 2017 @03:39AM (#15096850)
Years ago I worked with a product that parsed line printer data, it was a nightmare trying to get every piece of optional data into the reports.
Re:How ACH works: A developer perspective (2014) by paulie_a on Friday August 25, 2017 @12:05AM (#15096037)
Previously I had no experience with ACH but my current project is essentially reverse engineering a very proprietary version of NACHA and another file that is 400 fields long. Honestly the current parser does work but I can only assume due to some black magic or other voodoo.
Re:How ACH works: A developer perspective (2014) by dmlittle on Friday August 25, 2017 @01:05AM (#15096247)
Every time someone complains about file formats I bring up a similar story. It's hard to believe fixed width files are still being used today. Parsing these kind of files with 45 million records, each 600+ bytes is a huge pain. I chuckle when I think about the first time I dealt with these files and thought I could open them in an editor...
Re:How ACH works: A developer perspective (2014) by DaiPlusPlus on Friday August 25, 2017 @02:28AM (#15096562)
> It's hard to believe fixed width files are still being used today. Parsing these kind of files with 45 million records, each 600+ bytes is a huge pain. I chuckle when I think about the first time I dealt with these files and thought I could open them in an editor...

Depending on the business-application, fixed-width formats are fantastic because they allow for genuine O(1) random-access to records without needing to precompute an index first. Writing parsers for fixed-width formats is also much simpler, for example, because it eliminates ambiguity around null-terminators vs length-prefix in text/string/array data - for resource-constrained environments it's great because you can guarantee you won't need to dynamically-allocate buffers. I feel the only real arguments against fixed-width records are concerned with wasted space - which are mitigated if your system lets you compress them somehow - because they'll typically compress very, very well.

Re:How ACH works: A developer perspective (2014) by krylon on Friday August 25, 2017 @04:11AM (#15096954)
> I feel the only real arguments against fixed-width records are concerned with wasted space

Sometimes, fields can turn out to be too short. I have received a few letters from business with my last name truncated, because somebody way back when decided eight or nine characters are definitely enough for a surname.

Re:How ACH works: A developer perspective (2014) by marktangotango on Friday August 25, 2017 @06:01AM (#15097300)
Ha! I wrote an editor for fixed length files, it lets you map to a cobol copybook, or create the layout manually. I never released it. Anyone interested?
Re:How ACH works: A developer perspective (2014) by wolfgang42 on Friday August 25, 2017 @12:02PM (#15099452)
If it works on Windows, I'd love to give it a try! Our ERP system is written in COBOL and I have to deal with fixed-width files all the time. Does it also handle REDEFINES and level 88?
Re:How ACH works: A developer perspective (2014) by marktangotango on Friday August 25, 2017 @02:06PM (#15100754)
Yes it handles redefines and 88s, my emails in my profile, I have a Windows build.
Re:How ACH works: A developer perspective (2014) by nvahalik on Thursday August 24, 2017 @10:51PM (#15095773)
NACHA cha cha cha!!
Re:How ACH works: A developer perspective (2014) by labster on Thursday August 24, 2017 @11:00PM (#15095803)
You can't spell Atlach-Nacha without ACH.
Re:How ACH works: A developer perspective (2014) by pryelluw on Thursday August 24, 2017 @10:10PM (#15095619)
Ugh, reminds me of the time I had to integrate with a bunch of school software. They used csv over ftp (no auth). You just had to connect to the server and get the personal information of a district's students. Worse is you could google the URLs... I built a system that solved the problem but left shortly after it began to be integrated nationwide.
Re:How ACH works: A developer perspective (2014) by matt_wulfeck on Friday August 25, 2017 @12:11AM (#15096057)
wait.. are you saying that a district had public ftp with personal student information? Pretty sure that breaks the law.
Re:How ACH works: A developer perspective (2014) by joshribakoff on Friday August 25, 2017 @12:27AM (#15096116)
Email me at josh.ribakoff at Gmail I'd be interested in chatting. I think we worked at the same company. Or direct competitors
Re:How ACH works: A developer perspective (2014) by pryelluw on Friday August 25, 2017 @02:16AM (#15096510)
Sure, I'll do.
Re:How ACH works: A developer perspective (2014) by posguy on Thursday August 24, 2017 @10:23PM (#15095668)
ACH needs to die. Its a very old system, and you have no clue if a transfer ever actually goes through. Only ACH rejects ever get sent back, and there isn't a definitive window in which you'll get that reject.
Re:How ACH works: A developer perspective (2014) by colejohnson66 on Thursday August 24, 2017 @11:27PM (#15095893)
Wait. ACH doesn’t even tell the sender if it succeeded??!
Re:How ACH works: A developer perspective (2014) by dragonwriter on Friday August 25, 2017 @12:02AM (#15096021)
NAK-but-no-ACK is apparently a super common pattern in enterprise (both internal and between parties.)

I have no idea why it is and why people are so resistant to adopting ACKs.

Re:How ACH works: A developer perspective (2014) by kqr on Friday August 25, 2017 @02:55AM (#15096682)
Why it's there in the first place is fairly excusable. It (at least) halves the number of datagrams. Why people still resist it I'm not so sure about.

But on the other hand, it's a complicated architectural decision. US military wants absolutely reliable communications? They invent TCP which relies a lot on acks. Swedish telecom company wants fast and absolutely reliable communicating systems? Cue throw hands in air with a "there's no guarantees even with socks so better to simply never assume your message arrived and instead focus on detecting anomalous behaviour." They invent Erlang where there are no acks for messages.

Re:How ACH works: A developer perspective (2014) by ves on Thursday August 24, 2017 @11:37PM (#15095924)
Yeah there are no acks. It sucks.
Re:How ACH works: A developer perspective (2014) by saarons on Friday August 25, 2017 @12:04AM (#15096032)
The lack of acks is why most companies receiving money through ACH will "season" the transactions for a set amount of time. Most of the returns occur in the first 3 business days so most seasoning periods are just around that long. The reality is you can receive a reject up until (I believe) 60 days after the transaction date. I've seen a lot of companies be a little smarter here and reduce the seasoning times for repeat users using the same source bank account. It is a fascinating system to work with every day. Wire transfers are their own bundles of fun too.
Re:How ACH works: A developer perspective (2014) by DaiPlusPlus on Friday August 25, 2017 @02:30AM (#15096573)
Now I finally know why my credit-union (which serves the high-technology industry exclusively) told me that they couldn't put a hold or otherwise segregate any cheques I deposit until they've cleared... because they have no way of knowing when/if they've cleared!
Re:How ACH works: A developer perspective (2014) by wiredfool on Friday August 25, 2017 @05:12AM (#15097140)
It's 60 days, + some possibility of up to 2 extra days at the end for Fed Reserve holidays, and then in cases of outright fraud, occasionally banks will ignore the rules and return stuff later than that.
Re:How ACH works: A developer perspective (2014) by tomc1985 on Thursday August 24, 2017 @10:20PM (#15095655)
They already have an API, it's called FTP or SFTP or whatever.
Re:How ACH works: A developer perspective (2014) by dcow on Thursday August 24, 2017 @10:49PM (#15095765)
I don't know why everyone thinks they need an API to reimplement (poorly) basic unix functionality that's been around be for years.
Re:How ACH works: A developer perspective (2014) by Spivak on Friday August 25, 2017 @12:29AM (#15096124)
Especially when FTP and HTTP performance is basically comparable and will depend more on the network and storage than the protocol.

Have we gotten to the point where anything not HTTP is considered old?

Re:How ACH works: A developer perspective (2014) by digi_owl on Friday August 25, 2017 @05:34AM (#15097204)
never mind that most "API" are JSON (or some such) delivered over (S)HTTP anyways.

But yes, anything but HTTP is "old". In large part because of corporate firewall rules that block anything but HTTP traffic (and more often than not filter even that traffic).

Re:How ACH works: A developer perspective (2014) by joshribakoff on Friday August 25, 2017 @12:56AM (#15096213)
No. I'm referring to an API to query data in realtime instead of batching the entire freaking database and then having out dated snapshot of the data
Re:How ACH works: A developer perspective (2014) by EpicEng on Friday August 25, 2017 @01:05AM (#15096246)
>Because it didn't work in FileZilla, they didn't want to use it

Sounds like they didn't need much if any of what you did. They wanted X and you gave them Y. Y would require them to change the way they worked. Did they need query capability? Seems like they didn't and SFTP would have suited them better.

Honestly sounds to me like you went for an overly complex solution where something simple would have done just fine.

Re:How ACH works: A developer perspective (2014) by joshribakoff on Friday August 25, 2017 @02:00AM (#15096444)
Our company and their company have a common customer. Neither of us need each other we want to make our customer happy. Customer would be happiest with realtime data but company B wants to send snapshots via sftp. So I setup an endpoint that speaks sftp. They even told me to "do something" to make it secure. Then they complained I used a port other than 22 and that it only worked on the command line. It sounds like the problem is the other company doesn't want to do any work / wants to send the file manually. I don't see how I went for an overly complex solution. I attempted to do exactly what they asked and they moved the target so of course I missed. In fact it's this company that is just trying to avoid doing any programming. But that comes with the territory. I respect your opinion but disagree with you
Re:How ACH works: A developer perspective (2014) by EpicEng on Friday August 25, 2017 @09:27AM (#15098223)
Fair enough, I didn't have the backstory.
Re:How ACH works: A developer perspective (2014) by gaius on Friday August 25, 2017 @01:37AM (#15096353)
But but rewrite all the things in node.js!! Don't worry about understanding the file format, which is well established, or what the users of the file are actually doing with it, also well established, the important thing is node.js!!!

Frankly once they hear it's node.js the users should be bowing down to worship!!!!

Re:How ACH works: A developer perspective (2014) by joshribakoff on Friday August 25, 2017 @02:20AM (#15096527)
Actually I wanted him to make an https API for us to query real time data. I would have used PHP to consume his API because it's easier to find developers on our end and synchronous code is easier to maintain when performance is not a concern
Re:How ACH works: A developer perspective (2014) by corndoge on Thursday August 24, 2017 @10:39PM (#15095726)
why use a huge codebase like node for such a simple security critical task
Re:How ACH works: A developer perspective (2014) by jlgaddis on Friday August 25, 2017 @12:04AM (#15096033)
When all you have is a hammer...
Re:How ACH works: A developer perspective (2014) by joshribakoff on Friday August 25, 2017 @12:29AM (#15096126)
How else do you suggest you standup a sftp that cannot leak data even if the password is compromised? I'm all ears. Node is a runtime by the way and huge in what regard? The Linux codebase is also large.
Re:How ACH works: A developer perspective (2014) by notyourday on Friday August 25, 2017 @08:36AM (#15097954)
It is a system administration problem, not write a new software problem.

You:

a) think about the scope of the problem and what you are trying to achive - you are blocking reads of the files by the uid/gid of the server. It is a known and solved problem. The tool is called chmod

b) think about the surface area of the attack - it should only be the SFTP server. We have ones that are very well known. I recommend the one that comes with OpenSSH but there are others.

c) think about business requirements - user's provisioning etc. That has been a solved problem for years - for the super-complex cases LDAP is used. For the simpler ones you can use PAM.

After thinking about that you just write the needed glue.

Re:How ACH works: A developer perspective (2014) by Karrot_Kream on Friday August 25, 2017 @01:49AM (#15096396)
Seconding what kalleboo said, setting directory permissions has been a staple of file-sharing for decades. Setting up a node server to block read-access seems incredibly byzantine.
Re:How ACH works: A developer perspective (2014) by joshribakoff on Friday August 25, 2017 @02:06AM (#15096470)
Good point but we also have to load balance this / have failover and provision users on the fly. I could have used messaging queues to provision PAM users on the whole cluster and this guy would have still complained because I used a non standard port. Not sure why that was a problem either but again we can't press back or question him. he would cancel the whole integration. Bottom line is our end user wanted real time data (presumably via an API)
Re:How ACH works: A developer perspective (2014) by Karrot_Kream on Friday August 25, 2017 @02:55AM (#15096681)
A lot of FTP servers have provisions to essentially "ignore" the user and proxy into the directory. As far as load balancing is concerned, most FTP servers can handle very high throughput because of the simplicity of the protocol. If you really did want to load balance it, haproxy can do that just fine.
Re:How ACH works: A developer perspective (2014) by le-mark on Friday August 25, 2017 @09:09AM (#15098126)
> we can't press back or question him. he would cancel the whole integration

This attitude will not serve you well. There's saying no, and then there's getting more information (requirements!?!) to then guide people to better decisions.

Re:How ACH works: A developer perspective (2014) by corndoge on Friday August 25, 2017 @02:14AM (#15096498)
what's wrong with nonstandard ports?
Re:How ACH works: A developer perspective (2014) by kalleboo on Friday August 25, 2017 @12:57AM (#15096221)
Can't you just host a directory with -w- permissions? "Drop box" directories have been a staple of file sharing software for decades.
Re:How ACH works: A developer perspective (2014) by patmcc on Friday August 25, 2017 @02:18AM (#15096514)
You use chmod on the directory...writing any code for this kind of task would be crazy overkill.
Re:How ACH works: A developer perspective (2014) by Karunamon on Friday August 25, 2017 @12:45AM (#15096178)
First thing that comes to my mind is a PAM plugin that generates a new root directory for the user when they log in, named something sane under a fixed hierarchy. A recursive walk can later pick them all up and clean up.
Re:How ACH works: A developer perspective (2014) by microcolonel on Thursday August 24, 2017 @10:23PM (#15095665)
I did a similar thing, but decided to have LIST/MLSD/NLST, but stub out GET/RETR/MFCT/MFF/MFMT to return nothing or static data.
Re:How ACH works: A developer perspective (2014) by joshribakoff on Friday August 25, 2017 @12:31AM (#15096135)
What did you use to implement?
Re:How ACH works: A developer perspective (2014) by microcolonel on Friday August 25, 2017 @02:37AM (#15096603)
Wrote it myself because it was Elixir so why not. I'd be more careful doing this in C.
Re:How ACH works: A developer perspective (2014) by kccqzy on Friday August 25, 2017 @12:24AM (#15096104)
Reminds me of a time when I had to implement a parser for a certain kind of file to transfer money using an "API" provided by a financial institution in Asia. Yes we are using CSV too, also sent over SFTP. The file format is, unsurprisingly, ill-defined.
Re:How ACH works: A developer perspective (2014) by voltagex_ on Thursday August 24, 2017 @11:13PM (#15095855)
Wouldn't FileZilla have been happy with a fake blank directory response?
Re:How ACH works: A developer perspective (2014) by joshribakoff on Friday August 25, 2017 @12:31AM (#15096133)
Yeah I added that but the particular node sftp implementation I used from github just simply didn't work with FileZilla. Only the command line. Don't know why.
Re:How ACH works: A developer perspective (2014) by teddyh on Friday August 25, 2017 @01:55AM (#15096419)
Whenever someone says “from github”, I mentally replace it with “from some random guy on the street” to give it its appropriate intuitive feeling.
Re:How ACH works: A developer perspective (2014) by joshribakoff on Friday August 25, 2017 @02:12AM (#15096493)
Sure I don't care who wrote it if it works and passes our code review and the license matches. Plenty of crap software comes out of big name companies and plenty of random people on the street can write good code.
Re:How ACH works: A developer perspective (2014) by teddyh on Friday August 25, 2017 @02:47AM (#15096640)
If you have no other alternative and can’t take the time to write it for yourself, and need something now, sure, go nuts and take something random off Github, PyPI, npm, or whatever.

But code that does not have upstream development and support is dead code. A piece of code in production is like an Internet connection or electrical or water hookup – it must be, so to speak, “connected” upstream to whoever is providing continuous upgrades and security/bug fixes. Otherwise, it is (to continue stretching the analogy) like a mystery battery or water barrel – it could go bad in many ways at any time and you wouldn’t know it until your equipment starts to fail because of low voltage or voltage spikes, or you start to die of legionnaires’ disease. And unlike a battery or water, there are only cursory, no good and thorough, ways to test their quality, and no way to test their age as measured by the way they interact with the ever-changing outside world.

Re:How ACH works: A developer perspective (2014) by kqr on Friday August 25, 2017 @03:11AM (#15096751)
I've always been overly eager to add dependencies to projects and while I know why it may not be good I was missing a good intuitive sense of why. This is such a good way to view it. Thanks!