News for hackers. Stuff that matters.
Re:Ask HN: What are the things that you have automated in yo... by A_Person on Sunday June 24, 2018 @12:35PM (#17387147)
Ok, I'll bite!

About 20 years ago I inherited enough money to retire on for the rest of my life. But the only evidence of those assets was, a dot matrix printout that anyone could have done in five minutes. So over the next few weeks, I created accounts in the various share registries and online banking websites, and manually cross-checked the printed list against those accounts. All was good, but I wanted to do that on a regular basis, and it was clear that doing it manually, on a regular basis, was not gonna happen.

I am personally -- NSFW SPOILER ALERT!! -- a long time Windows/IE user, and I knew that IE has a comprehensive automation interface. In literally a few lines of VBScript (say), you can programatically start IE, navigate to URLs, parse the DOM, create new local documents, and so on. But although those interfaces are very simple, they're not robust against failures, and have various weird and wonderful corner cases that will sometimes trip you up.

So over the next 10 years or so, I created a robust, general-purpose IE automation wrapper, and wrote some applications on top thereof. The result is that at any time, I can click a few buttons on a handsome user interface, and enter a single master password, at which point my application will automatically log into each share registry and online banking website in turn; download all new and amended data therefrom; transform all data into common formats; store it in a local cache, so it doesn't matter if it later disappears from the website; then creates a single share holdings document, showing all share holdings from all share registries, and a single online banking document, showing all transactions from all accounts at all banks since the start of time. Both documents have dynamic, user-defined sorting and filtering. They also hilite significant changes; check all share holdings against banking credits to confirm that all expected dividends were received; and so on.

In other words, I've completely automated various important financial checks that I simply couldn't do by hand on a regular basis. The downside is, I'm tied to IE - and - several hundred thousand lines of VBScript! If anyone would like to re-write all that code for Chrome or whatever - for free - you're more than welcome to contact me!!!

Re:Ask HN: What are the things that you have automated in yo... by Cyphase on Sunday June 24, 2018 @04:01PM (#17388122)
> If anyone would like to re-write all that code for Chrome or whatever - for free...

For free? I thought you inherited enough money to retire on for the rest of your life. Share the wealth. :)

Re:Ask HN: What are the things that you have automated in yo... by A_Person on Sunday June 24, 2018 @04:11PM (#17388176)
Well put :-) I was actually taking a subtle dig at folk who say, "just ditch IE, how hard can it be?" If you have an enormous corporate or personal application, the answer could be: "impossibly hard", unless a good fairy appears from the sky and rewrites it for free!
Re:Ask HN: What are the things that you have automated in yo... by test6554 on Monday June 25, 2018 @09:18AM (#17392140)
You might look into Yodlee. Services like Mint use their non-free apis to get financial data.
Re:Ask HN: What are the things that you have automated in yo... by delbel on Sunday June 24, 2018 @07:51PM (#17389002)
If you wanted to re-write that for Chrome or whatever (including headless) I would suggest Selenium or WebDriver suite, including PhantomJS. However, my knowledge (expert level at the time) is a bit outdated and there are probably new toolkits based on the Selenium/WebDriver protocol, it is a good direction to point you to.

I have had my bank remove records from my online account, or not have the ledger balance add up, so your idea interest me. I contacted the controller of the currency about my bank messing with my records, and they did absolutely nothing.

Re:Ask HN: What are the things that you have automated in yo... by test1235 on Monday June 25, 2018 @06:45AM (#17391314)
PhantomJS has pretty much gone the way of the dodo since Chrome got its headless mode:

https://github.com/ariya/phantomjs/issues/15344

https://developers.google.com/web/updates/2017/04/headless-c...

Re:Ask HN: What are the things that you have automated in yo... by A_Person on Sunday June 24, 2018 @08:38PM (#17389171)
Thanks for the reference to sekenium etc. I know nothing about other browsers' automation capabilities, so i'll takee a look at some stage.
Re:Ask HN: What are the things that you have automated in yo... by sonar_un on Sunday June 24, 2018 @11:14PM (#17389681)
I’ve automated a bunch of tasks thanks to selenium. It’s really pretty simple once you get the hang of it. I’d totally recommend.
Re:Ask HN: What are the things that you have automated in yo... by A_Person on Sunday June 24, 2018 @11:50PM (#17389779)
I'll certainky take a look at it
Re:Ask HN: What are the things that you have automated in yo... by ntzm on Monday June 25, 2018 @06:34AM (#17391268)
Wouldn't go for PhantomJS
Re:Ask HN: What are the things that you have automated in yo... by 13of40 on Sunday June 24, 2018 @05:31PM (#17388497)
Honestly curious: Why wouldn't you just liquidate all of it and invest in some mutual funds?
Re:Ask HN: What are the things that you have automated in yo... by A_Person on Sunday June 24, 2018 @06:01PM (#17388610)
A lot of it is in fully-franked shares in a self-managed superannuation fund. The fund is now in pension mode, which means it pays no tax on earnings. Where I live, fully-franked shares, in a zero tax environment, get a refund, from the tax office, of all the tax that the issuing companies payed on those earnings!! It's a sweet deal, but actually makes me feel a bit guilty. It's all 100% above board (and managed by one of the big 5 accounting firms), but it means I pay less tax than all my friends and acquaintances, all of whom earn much less than me. Reference: Warren Buffet's secretary :-)
Re:Ask HN: What are the things that you have automated in yo... by whatsstolat on Sunday June 24, 2018 @06:14PM (#17388661)
Fellow Aussie
Re:Ask HN: What are the things that you have automated in yo... by A_Person on Sunday June 24, 2018 @06:21PM (#17388686)
Onya!!
Re:Ask HN: What are the things that you have automated in yo... by jacobush on Sunday June 24, 2018 @03:40PM (#17388027)
No problem. I will need your bank and website passwords... :-D
Re:Ask HN: What are the things that you have automated in yo... by A_Person on Sunday June 24, 2018 @04:14PM (#17388189)
I've no idea what they are! They're all random strings within the integrated password manager that I wrote as part of the package!
Re:Ask HN: What are the things that you have automated in yo... by egypturnash on Sunday June 24, 2018 @08:14PM (#17389064)
This is kind of perfect.
Re:Ask HN: What are the things that you have automated in yo... by A_Person on Sunday June 24, 2018 @08:39PM (#17389177)
The rubber hose would fail!
Re:Ask HN: What are the things that you have automated in yo... by helsinki on Sunday June 24, 2018 @03:27PM (#17387975)
Hahaha, oh wow, that's a lot of work. I think you should learn Python and some Requests...
Re:Ask HN: What are the things that you have automated in yo... by A_Person on Sunday June 24, 2018 @04:30PM (#17388247)
I don't think that would make it much simpler (if that's what you're getting at). There's tons of grunt work that just has to be done - eg. calculating expected dividend payments (by automatically matching portfolio details to dividend schedules from corporate websites), and automatically matching those to bank account credits - all through code. Despite the laughs that the VB languages get on HN, VBScript is in fact, a fully featured traditional imperative programming language, and has all the features you'd expect in such a language (but no IDE :-(). So I doubt that rewriting in python would make much difference. And I'd still be bound to the IE automation model.
Re:Ask HN: What are the things that you have automated in yo... by A_Person on Sunday June 24, 2018 @06:17PM (#17388671)
To wafflesindeed (there's no reply button):

> I'm not OP, but I suspect they meant that Python and some libraries would have solved your problem in a few days of work.

I wrote software professionally for 30 years. I have a good understanding of the work required to implement a business requirement! I cant really see myself spending 10 years on a one-week task :-)

PS. I didn't downvote you.

Re:Ask HN: What are the things that you have automated in yo... by PurpleRamen on Monday June 25, 2018 @05:21AM (#17390994)
> And I'd still be bound to the IE automation model.

Why? You fetch data, calculate stuff and give out reports. Unless there is heavy javascript involved, you don't need a browser for any of this. But now you are dependent on the fragile setup you have created, hopping that VBScript, IE and your datasources will continue to work as long as you live.

Becoming independant and gaining a future ground would be a valid reason to change to a better environment. It doesn't need to be Python, but considering the requirements, Python is a very good choice for this it seems.

Re:Ask HN: What are the things that you have automated in yo... by A_Person on Monday June 25, 2018 @05:30AM (#17391033)
All of the websites concerned have strict authorization requirements and none have public APIs afaics. At the start, it just seemed easiest to automatically scrape them. I guess I could have reverse engineer their private APIs and used those - but as the doctors say, the retrospectoscope is a rare and expensive piece of equipment!
Re:Ask HN: What are the things that you have automated in yo... by PurpleRamen on Monday June 25, 2018 @05:38AM (#17391053)
Is authorization done with some activeX-applet? Otherwise I don't really see what IE is offering that Chrome offers too, but scrapping-libs can't.

BTW There are libs to use chrome programatically for those things. It's called headless-mode or chromeDriver. Similar things exist for other Browsers. So you could probably rewrite your app to become more indipendant without lossing the existing functionality.

Re:Ask HN: What are the things that you have automated in yo... by A_Person on Monday June 25, 2018 @12:24PM (#17393623)
I just don't see enough cost benefit from rewriting hundreds of thousands of lines of working code at this time. I can't see IE going away, any time soon, due to corporate tie-in. Hopefully VBScript ditto. If it all falls over down the track, I'll review my options then :-)
Re:Ask HN: What are the things that you have automated in yo... by wafflesindeed on Sunday June 24, 2018 @05:36PM (#17388518)
I'm not OP, but I suspect they meant that Python and some libraries would have solved your problem in a few days of work.

This is particularly true in the UK where there's a big push for "Open Banking" where banks must provide APIs for developers.

This would remove your dependency on IE if it was all done from the command line :)

Re:Ask HN: What are the things that you have automated in yo... by nanospeck on Monday June 25, 2018 @07:33AM (#17391497)
I’m interested to work on it. Can u share the details in [email protected]
Re:Ask HN: What are the things that you have automated in yo... by anonnel on Sunday June 24, 2018 @02:49PM (#17387779)
You mean I can rewrite several hundred thousand lines of VBScript for you for free? Sounds awesome. Do reach out.
Re:Ask HN: What are the things that you have automated in yo... by dang on Sunday June 24, 2018 @03:22PM (#17387952)
That was obviously a good-humored joke.

Could you please stop posting unsubstantive comments to Hacker News? You've done it a lot, and it's not what this site is for.

https://news.ycombinator.com/newsguidelines.html

Re:Ask HN: What are the things that you have automated in yo... by sudouser on Sunday June 24, 2018 @11:56PM (#17389792)
ever tried sikulix? it clicks the things
Re:Ask HN: What are the things that you have automated in yo... by A_Person on Monday June 25, 2018 @02:01AM (#17390192)
Nope, never heard of it. My software certainly has to click lots of thing! But it also has to do lots of data processing behind the scenes. So the clickety click is only part of the story :-)