News for hackers. Stuff that matters.
Re:State found $117k in double payments through data analytics by DoofusOfDeath on Wednesday December 11, 2019 @12:45PM (#21764832)
All things being equal, catching $117k in double payments sounds great.

I'm curious of it's a net win, though, once all the costs (and benefits) are considered.

Re:State found $117k in double payments through data analytics by DoofusOfDeath on Wednesday December 11, 2019 @01:50PM (#21765515)
> I'm curious of it's a net win, though, once all the costs (and benefits) are considered.

To be fair, I wouldn't be surprised if there are additional benefits beyond the $117k. For example:

- Reducing the attack surface of fraudulent invoicing.

- Retaining staff/skill that are useful in other projects not discussed in the article.

- Seeding awareness/interest for similar projects in other departments of the state government.

Re:State found $117k in double payments through data analytics by dangerboysteve on Wednesday December 11, 2019 @01:01PM (#21765018)
after I read the article I thought was this a situation of stepping over a dollar to pick up a dime?
Re:State found $117k in double payments through data analytics by chance_state on Wednesday December 11, 2019 @12:51PM (#21764900)
I'm wondering the same thing. $117K out of what total?
Re:State found $117k in double payments through data analytics by gunshai on Wednesday December 11, 2019 @01:43PM (#21765435)
So the article is a bit confusing, as it says in a year the state processes over 18 million dollars in transactions.

However the pilot program only analyzed transactions from Jan to Sept. For all state agencies (which seems impressive from an aggregation and governance stand point). With in that time period it found 117k in duplicates, so assuming that dollar/transactions are at a constant rate through out the year (they aren't).

117,000/(18,000,000 * (8/12)) ~ .975 % loss in state to vendor(?) transactions.

Put another way, with in the time period they studied assuming 12 million dollars in transactions were made, they can potentially recuperate roughly 1% of the total value transacted.

Re:State found $117k in double payments through data analytics by Mathnerd314 on Wednesday December 11, 2019 @02:27PM (#21765841)
They process 18 million transactions per year, total, not 18 million dollars of transactions. And they only found 56 duplicates, i.e. a 0.0004% rate. (56 / (18e6 * 8/12))

For comparison there's a Kaggle dataset of 2 days of European credit card transactions from 2013. It contains 284k transactions of which 0.147% are fraudulent.

This 1000x difference can either be good since it means the state is mostly paying appropriately or bad since the detection is not as good as the credit card company's.

Re:State found $117k in double payments through data analytics by gunshai on Wednesday December 11, 2019 @07:57PM (#21768406)
>18 million payment transactions

god damnit you're right. whoops

Re:State found $117k in double payments through data analytics by mushufasa on Wednesday December 11, 2019 @01:10PM (#21765112)
the article said all across the state of ohio. the overall ohio state gov't budget was $67.5 billion in 2016.
Re:State found $117k in double payments through data analytics by daeken on Wednesday December 11, 2019 @01:17PM (#21765179)
Jesus, so this is all of 0.000173% of the budget. Significantly less interesting in that context.
Re:State found $117k in double payments through data analytics by JMTQp8lwXL on Wednesday December 11, 2019 @01:27PM (#21765277)
If you instead considered this an audit of the state's expenses, rather than a money-recouping initiative, then it's an impressive feat the state's money goes where it should be.
Re:State found $117k in double payments through data analytics by mushufasa on Wednesday December 11, 2019 @01:33PM (#21765341)
I'd be surprised if they audited 100% of state expenses. This seems to focus on invoices and not, for example, payroll. Even so, if it was $1b and only 117k of mistakes, that's just .0117%, which seems pretty good to me.
Re:State found $117k in double payments through data analytics by gunshai on Wednesday December 11, 2019 @01:48PM (#21765495)
The closest comparison I can make is as follows.

This was 117K was found in transactions during 8 out of 12 months in the year. The article doesn't specify the amount of money transacted during that time. It doesn't give us the total amount of transactions either.

It gives an estimate of 18 million in the total year so the closest estimate is 117k out of 12 million. Which is ~1% and seems like a win win. State employees and folks should feel good that they aren't fucking up very often and the analysts should feel good because functionally this number should go to 0. Operationally no idea what it would cost to maintain though. cough

Re:State found $117k in double payments through data analytics by geogra4 on Wednesday December 11, 2019 @01:22PM (#21765229)
Yeah, thats actually remarkably good
Re:State found $117k in double payments through data analytics by mistrial9 on Wednesday December 11, 2019 @12:44PM (#21764823)
A recent Masters graduate trained in data science with Pandas, and also accounts payable work experience, got hired very very fast on good terms when the hiring department found out how facile and fast the Pandas python approach was to finding duplicate or other problematic payments. Maybe the novelty will wear off, but right now, this is absolutely practical.
Re:State found $117k in double payments through data analytics by toomuchtodo on Wednesday December 11, 2019 @12:51PM (#21764893)
Would love to see this skill set put to work automating away a lot of accounting functions. I know too many accountants (Controllers, specifically) doing mind numbing work.
Re:State found $117k in double payments through data analytics by user5994461 on Wednesday December 11, 2019 @01:16PM (#21765164)
The mind numbing work is what give them a salary to provide for their families. They might not like unemployment as an alternative.

I've met certified accountants whose daily job was to take an excel spread sheet, add 1% to every cell with a number, then return to the client. It's not automated but not because it can't be automated.

Re:State found $117k in double payments through data analytics by toomuchtodo on Wednesday December 11, 2019 @01:19PM (#21765200)
Automation is coming regardless. Political and economic systems must adapt to soften the blow. People will need to find purpose and meaning elsewhere than a job that can be automated. I suggest family, community, and self development, but to each their own.

As a disclaimer, I have recently become enchanted with automating accounting work due to hearing about the problems accounting functions face. I'm a tad biased.

Re:State found $117k in double payments through data analytics by user5994461 on Wednesday December 11, 2019 @01:35PM (#21765359)
You'd think that but nope. Well, not as quickly or as strongly as one would anticipate.

Large banks, automated trading shops and assimilated have automation, because it's their jobs. Other things don't.

Recruiting chartered accountant is easy and cheap. Recruiting developers that can deliver is not as easy or as cheap. Bear in mind that it's not a tech environment so no source control in place, no servers to deploy to, won't be easy to execute.

Re:State found $117k in double payments through data analytics by toomuchtodo on Wednesday December 11, 2019 @01:35PM (#21765366)
Okay. I'm still going to take a crack at it, working in tech within finance. Worst case scenario, I fail and have a story to tell.
Re:State found $117k in double payments through data analytics by user5994461 on Wednesday December 11, 2019 @01:40PM (#21765408)
I worked in finance too, trading platforms in investment banks and prop trading shops. There hasn't been accountants there for a long time. Places that still rely on manual labor are a very different kind.
Re:State found $117k in double payments through data analytics by toomuchtodo on Wednesday December 11, 2019 @01:43PM (#21765446)
Agreed. In those cases, where the business is stubborn, you might have to consume the entire business [1] [2] to be able to enact automation in the underlying functions. That's also the question, right? What tools can you develop (Similar to Brex and Stripe) that make it exceedingly more simple for upstarts to compete against the entrenched? If I can't sell to your internal divisions, maybe I go get some VC and compete against your whole business. Lots of margin hiding as inefficiencies in the economy still, just have to go get it.

Give me a lever long enough.

[1] https://a16z.com/2011/08/20/why-software-is-eating-the-world...

[2] https://techcrunch.com/2016/06/07/software-is-eating-the-wor...

Re:State found $117k in double payments through data analytics by user5994461 on Wednesday December 11, 2019 @02:48PM (#21766040)
Remind me, saw a fintech company with almost a billion in VC funding earlier this year. Basically a bank doing loans.

The whole business was based low cost labor taking whole days to evaluate stuff manually and write risk reports. Crazy thing is they were hiring loads of developers and data scientists, but the core of the business is all manual.

Re:State found $117k in double payments through data analytics by kirubakaran on Wednesday December 11, 2019 @01:42PM (#21765432)
I think, just because of this attitude, you won't fail. All the best!
Re:State found $117k in double payments through data analytics by throwawayhhakdl on Wednesday December 11, 2019 @02:00PM (#21765606)
As someone who passed the cpa exams, worked at a large bank, and now does data science elsewhere, I strongly disagree. Many large banks are stagnant with legacy tech and processes. Many people do pointless excel work and will defend pointless excel work to the death.

There were multiple people, of varying skill, willing and able to automate significant chunks of this with introductory R/Python/VBA skills. It’s not a cool shiny front end app, but a few scripts here and there could save an FTE easily. But politics prevents these things from being adopted.

Cost and scarcity of tech talent is not a real issue for light officework. Servers, Source Control, etc. are missing the picture. You just need people with light programming capability and a little bit more agency and voila they’ll be 10x as productive as everyone else.

Re:State found $117k in double payments through data analytics by mcherm on Wednesday December 11, 2019 @03:41PM (#21766585)
> I've met certified accountants whose daily job was to take an excel spread sheet, add 1% to every cell with a number, then return to the client. It's not automated but not because it can't be automated.

Well, that's just dumb. We can do better. Those accountants you met can do better!

Re:State found $117k in double payments through data analytics by eitland on Wednesday December 11, 2019 @01:57PM (#21765578)
At one company I worked with our external auditor knew SQL. My colleagues loved him because he would actually do useful queries that proved something instead of just verifying random bus tickets from Asia as certain others would do.
Re:State found $117k in double payments through data analytics by toomuchtodo on Wednesday December 11, 2019 @02:08PM (#21765690)
Folks like that are worth their weight in gold.
Re:State found $117k in double payments through data analytics by SamuelAdams on Wednesday December 11, 2019 @02:30PM (#21765865)
I keep hearing this but no one wants to pay me 3M USD / year.
Re:State found $117k in double payments through data analytics by rahimnathwani on Wednesday December 11, 2019 @02:47PM (#21766034)
Renting your weight in gold would cost much less than 3M USD per year.
Re:State found $117k in double payments through data analytics by dangerboysteve on Wednesday December 11, 2019 @01:32PM (#21765339)
I know the comments are all "how much money did they spend to find $117K in duplicate payments" and I am one of them. But if this system helps in identifying process/audit flaws then its a good thing.
Re:State found $117k in double payments through data analytics by kylek on Wednesday December 11, 2019 @01:41PM (#21765422)
Does this mean the state isn't SOX compliant? I thought reconciliation was a requirement

https://en.wikipedia.org/wiki/Reconciliation_(accounting)

Re:State found $117k in double payments through data analytics by xxpor on Wednesday December 11, 2019 @01:43PM (#21765444)
Are states required to be in the first place?
Re:State found $117k in double payments through data analytics by ConcernedCoder on Wednesday December 11, 2019 @01:29PM (#21765299)
which begs the question: "how much did the data analytics cost?"
Re:State found $117k in double payments through data analytics by madenine on Wednesday December 11, 2019 @12:42PM (#21764816)
SELECT TransactionID, Amount, COUNT(*) FROM Transactions GROUP BY TransactionID, Amount

I'll take 10% of savings

Re:State found $117k in double payments through data analytics by JMTQp8lwXL on Wednesday December 11, 2019 @01:04PM (#21765047)
This presumes all of the transaction data is centralized one place. 99% of the battle was probably getting that data aggregated, then performing the analytics, which is chalked up to this simple SELECT statement.
Re:State found $117k in double payments through data analytics by slap_shot on Wednesday December 11, 2019 @01:57PM (#21765573)
This is exactly the issue here. I worked as a contractor for the State Department for a couple of years on software that basically held the record of every dollar spent over the last 20 years.

What seemed like a very small and simple problem initially revealed itself to be massive problem that even a team of 35 struggled to maintain. The purchase history for any given item spanned multiple systems with completely different topologies of data glued together by, literally, tens of thousands of lines of SQL.

The project had been around for ~17 years when I worked on it and while data was landed in a final format where a query like the above could have been done, I wouldn't bet any serious money that the calculations were correct.

Some five or six multi-million dollar rewrites had been attempted, but could never be done.

Not to say Ohio's system is that this level, but I doubt there's a giant table sitting somewhere that such a simple query could be applied to.

I like to think that somewhere in all that SQL was an Office Space-esque line of SQL syphoning off a few cents per transaction, but was never able to find one :)

Re:State found $117k in double payments through data analytics by dang on Wednesday December 11, 2019 @01:45PM (#21765462)
"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something."

"Don't be snarky."

https://news.ycombinator.com/newsguidelines.html

Re:State found $117k in double payments through data analytics by ellisv on Wednesday December 11, 2019 @12:47PM (#21764854)
HAVING COUNT(*) > 2;

I'll take 9% );

Re:State found $117k in double payments through data analytics by madenine on Wednesday December 11, 2019 @12:59PM (#21764992)
HAVING COUNT(*) >= 2;

Back to 10%

Re:State found $117k in double payments through data analytics by elhudy on Wednesday December 11, 2019 @01:26PM (#21765266)
Add a date field or you might get legitimate duplicates if the payment amounts are the same across months.
Re:State found $117k in double payments through data analytics by TheSoftwareGuy on Wednesday December 11, 2019 @02:25PM (#21765823)
They shouldn't have the same transactionID then, no?
Re:State found $117k in double payments through data analytics by elhudy on Wednesday December 11, 2019 @02:40PM (#21765962)
Wouldn't think so. Each transaction ID would represent a unique transaction?

>Other double payments made by mistake included times that the state received multiple invoices.

I would assume each paid invoice was a new transaction ID. The real problem seems like there are two invoices being paid. Not that there are two transactions (which seems like just a symptom). It's possible that each invoice even has it's own ID.

Re:State found $117k in double payments through data analytics by mgraczyk on Wednesday December 11, 2019 @12:59PM (#21764989)
`>=`

8%?

Re:State found $117k in double payments through data analytics by edwhitesell on Wednesday December 11, 2019 @12:50PM (#21764885)
And when there are two payments with different transaction ids from the gateway?
Re:State found $117k in double payments through data analytics by madenine on Wednesday December 11, 2019 @12:54PM (#21764930)
most likely? my joke fails to pan out in reality
Re:State found $117k in double payments through data analytics by DoofusOfDeath on Wednesday December 11, 2019 @12:46PM (#21764845)
I would guess that "TransactionID" is where a lot of the challenge lies.