News for hackers. Stuff that matters.
Re:AWS CodePipeline by darksaints on Thursday July 09, 2015 @01:00PM (#9859164)
Internally at Amazon, Pipelines (which inspired this service) was a lifesaver. Apollo (which is the inspiration for CodeDeploy) was also helpful, but should probably just be replaced by Docker or OSv at this point.

But if they ever release a tool that is inspired by the Brazil build system, pack up and run for the hills. When it takes a team of devs over two years to get Python to build and run on your servers, you know your frankenstein build system is broken. It could be replaced by shell scripts and still be orders of magnitude better. Nobody deserves the horror of working with that barf sandwich.

Re:AWS CodePipeline by mirceal on Thursday July 09, 2015 @01:08PM (#9859214)
hah. I believe you either don't understand Brazil (the build system) or you had to deal with inexperienced people that don't get it. IMHO the internal Amazon tooling is THE reason Amazon is able to build things so fast and to iterate so quickly.
Re:AWS CodePipeline by SpikeGronim on Thursday July 09, 2015 @01:36PM (#9859420)
I love Brazil and Apollo. As an ex-Amazonian I miss them on a weekly basis.
Re:AWS CodePipeline by saosebastiao on Thursday July 09, 2015 @01:49PM (#9859510)
If you are on a team that does nothing but Java on a MAWS or corp platform, brazil was built for you and I wouldn't be surprised that you like it. But...if you deviate slightly, you run the chance of not getting what you need. If you stray away from the JVM or Ruby, chances are you will never get anything at all, let alone what you need.
Re:AWS CodePipeline by mirceal on Thursday July 09, 2015 @02:48PM (#9859922)
ruby, java, perl :S , python, roll-your-own have done it all. :) yes, the learning curve can be pretty steep, but one thing that always helps is that the probability that there is somebody out there that did it already is very high. you just need to be curios and invest into learning it and looking at what others are doing.
Re:AWS CodePipeline by coreyoconnor on Thursday July 09, 2015 @03:04PM (#9860018)
Check out nix / nixpkgs. :-) That's the closest to Brazil IMO.
Re:AWS CodePipeline by mirceal on Thursday July 09, 2015 @03:39PM (#9860222)
look pretty awesome. never heard of this before. I'm sure it has some caveats...
Re:AWS CodePipeline by coreyoconnor on Thursday July 09, 2015 @04:06PM (#9860342)
Yep. Definitely! One caveat, in short, is: Everything involved in the build must be placed under the nix store (/nix/store). Which, for some software, can cause issues: They might have a hardcoded runtime path that references /usr/lib. Nix provides tools to resolve these assumptions, but they can still be a sticking point at times. There is also, more or less, a unsafePerformIO that can be used. Which is discouraged. Still, for private builds the cost can be acceptable.
Re:AWS CodePipeline by mirceal on Thursday July 09, 2015 @05:27PM (#9860794)
do you know if there is a way to cache things if used across multiple apps?
Re:AWS CodePipeline by coreyoconnor on Thursday July 09, 2015 @07:16PM (#9861346)
Assuming I understand the question correctly: Yes.

Nix uses a pure evaluation model so it enjoys the property: An artifact in the nix store is uniquely identified by the closure used to build the artifact. For any artifacts A, B if and only if the closure used to build the artifact are equal then the artifact file path will be equal.

This creates opportunities for sharing between builds that can be hard to achieve in other systems. One form of sharing "referencing a derivation in multiple apps" works as expected, just like other systems: Each app will reference the same artifact.

(a derivation is the closure to be evaluated to build an artifact in the nix store. Well, attribute set of closures.)

Suppose a derivation is assigned to the variable "commonData" and two derivations "appX" and "appY" reference this closure. "commonData" will be built once and both app derivations will receive a path to the same file in the nix store.

The other form of sharing comes from the equality comparison being based on the closure and not the name used to reference the closure.

Ehh.. I'm butchering the explanation... I think there is a succinct PL term that covers this.

Suppose we have a derivation:

let x = mkDerviation { name = "foo"; builder = aBuilder; src = /share/src/foo; }

which is referenced by another derivation

let y = mkDerviation { name = "bar"; builder = aBuilder; src = /share/src/bar; inherit x; }

"y" will force the evaluation of the "x" derivation's closure. The source directories, since they are not in the nix store, will be copied to the nix store first. (By an implicit conversion between local files and nix store paths)

So far so good, but what happens if there is another derivation like so?

let z = mkDerviation { name = "zab"; builder = aBuilder; src = /share/src/zab; somethingLikeX = mkDerviation { name = "foo"; builder = aBuilder; src = /share/src/foo; }; }

"somethingLikeX"'s equation is equal to "x" but not the same reference.

What will happen if z is evaluated after y? (assuming aBuilder is the same) First, the derivation "somethingLikeX" will be evaluated. Ah ha! That closure is equal to the closure for "x" above! Which has already been evaluated. So that evaluation result will be shared. Even though "z" does not directly reference "x".

This can result in more sharing than the developer explicitly requested: Equal closures are shared.

Re:AWS CodePipeline by aphexairlines on Saturday July 11, 2015 @05:58AM (#9869220)
Re:AWS CodePipeline by Rapzid on Thursday July 09, 2015 @06:38PM (#9861156)
I thought that was because they MVP everything? Still waiting on a NAT service and a slew of ELB features that are "coming" so.. I'm not aware that they iterate particularly quickly across the board. Tags are still "kinda" supported to differing degrees or not at all across services.
Re:AWS CodePipeline by felipesabino on Thursday July 09, 2015 @02:41PM (#9859869)
I wonder why GitHub specifically and not just Git repos in general? Isn't it weird?

It means they don't even support their own new "Git" product AWS CodeCommit [1]

[1]https://aws.amazon.com/blogs/aws/now-available-aws-codecommi...

Re:AWS CodePipeline by andrewguenther on Thursday July 09, 2015 @02:46PM (#9859901)
My guess would be WebHooks support.
Re:AWS CodePipeline by felipesabino on Thursday July 09, 2015 @02:51PM (#9859936)
That could be, just noticed that CodePipeline's Post-Receive Hooks are still a work in progress
Re:AWS CodePipeline by shekhargulati on Thursday July 09, 2015 @03:39PM (#9860221)
Yes, you have to poll CodePipeline every second or so to get the jobs and perform your task. Currently, web hooks are not implemented or they are WIP.
Re:AWS CodePipeline by atmosx on Thursday July 09, 2015 @02:03PM (#9859623)
This is interesting for lone developers, but I'm not sure about the pricing:

You’ll pay $1 per active pipeline per month (the first one is available to you at no charge as part of the AWS Free Tier). An active pipeline has at least one code change move through it during the course of a month.

Does this mean that every time you run a session you pay 1 EUR no matter how many stages the session has (pull, compile/build, test (multiple tests) and deploy?

Re:AWS CodePipeline by BillinghamJ on Thursday July 09, 2015 @02:16PM (#9859707)
No, you pay $1/month/pipeline for the pipelines which you've used during the month.

Pipeline used 0 times - $0, 1 time - $1, 500 times - $1

Re:AWS CodePipeline by mirceal on Thursday July 09, 2015 @02:51PM (#9859940)
10000 times - $1 :) this is not to make money. It's to drive adoption of AWS and get [even] more people on it by making their lives easier.
Re:AWS CodePipeline by jtwaleson on Thursday July 09, 2015 @11:49AM (#9858576)
Is there any way to integrate this with ECS? That would be a great feature for me.
Re:AWS CodePipeline by jayonsoftware on Thursday July 09, 2015 @12:02PM (#9858701)
Can we build .NET code ?
Re:AWS CodePipeline by cschneid on Thursday July 09, 2015 @04:04PM (#9860329)
From my reading, it appears to hook up to a Jenkins server, potentially that you control, so it can do anything Jenkins can, which includes building windows artifacts.
Re:AWS CodePipeline by pragar on Thursday July 09, 2015 @11:22AM (#9858350)
Thanks. I was eagerly waiting for this :)
Re:AWS CodePipeline by maikklein on Thursday July 09, 2015 @06:24PM (#9861103)
Could I install Unreal Engine 4 on CodePipleline so that I can build my game remotely?
Re:AWS CodePipeline by jordanthoms on Thursday July 09, 2015 @09:33PM (#9861845)
As far as I can tell this is more about orchestrating builds, to actually build your game you'd need to run jenkins on an EC2 instance and connect it to CodePipeline.
Re:AWS CodePipeline by dynjo on Thursday July 09, 2015 @11:22AM (#9858348)
Amazon seriously need to hire some good UI designers. They produce great stuff but it all looks like it was designed by developers in 1980.
Re:AWS CodePipeline by twistedpair on Thursday July 09, 2015 @01:40PM (#9859448)
AWS sells an API. Serious users don't use the web console. Need to tend your cluster of 1000+ EC2 servers? The Web console will fail for that. The console fronts the API, just like the CLI tool.

AWS just builds the basic UI's to make it possible for people to do simple things. If you get deep into the weeds, you'll need to use the API directly, i.e. for installing an SSL cert on your CloudFront distribution, or launching a 2K node computing cluster.

If you don't like the UI, make your own and sell it to people. That has become a viable business model for various startups, or launch your own hosting company on top of EC2 like Heroku.

Re:AWS CodePipeline by dchest on Thursday July 09, 2015 @06:38PM (#9861157)
If you don't like the UI, make your own and sell it to people.

Or ask Amazon to hire some good UI designers.

Re:AWS CodePipeline by twistedpair on Thursday July 09, 2015 @09:10PM (#9861774)
Why? I'm amazed people expect a light and airy, responsive, material designed enterprise control panel that is redesigned yearly. These are backend tools. The UI works and was (1) not meant to be sexy and (2) not meant to push sales.

AWS is in a race to the bottom, cutting prices all the time. Hiring 50 UX devs to rewrite their web interfaces doesn't comport with Jeff's lean vision.

Re:AWS CodePipeline by brianwawok on Thursday July 09, 2015 @09:55PM (#9861901)
You can both be pretty AND powerful. It is hard to do, but it can happen.
Re:AWS CodePipeline by dchest on Friday July 10, 2015 @07:32PM (#9867647)
Looks like you are confusing UI designers with bullshit designers, who make such things "material design". It's not a beauty contest.
Re:AWS CodePipeline by softawre on Thursday July 09, 2015 @11:26AM (#9858375)
It just needs to be usable, which it is in my experience.

Not that it matters, but I think they use GWT.

Re:AWS CodePipeline by andybak on Thursday July 09, 2015 @12:28PM (#9858903)
The initial impression of a new AWS console is pretty overwhelming. I've stuck with DigitalOcean and CloudFlare over Amazon alternatives partly because every time I sign into AWS all I can think is "Heck. Which of these product names and acronyms matches the thing I came here to do?"
Re:AWS CodePipeline by Someone1234 on Thursday July 09, 2015 @01:04PM (#9859193)
I will fully admit that when I started using AWS, it took me a while to learn Amazon's "language." Unfortunately Amazon does a bad job of distinguishing between their core offerings (e.g. S3, EC2, Route53, etc) and their very niche ones (e.g. EMR, Kinesis, SWF, etc), so new users are left scrambling to figure out what they need to know.

I understand their desire to create unique "products" that people can use in a conversation (e.g. "Have you considered Route53 for your DNS?"), but ultimately mixing common and niche things together and giving everything confusing names is likely doing Amazon more harm than good.

That all being said, Amazon are slowly improving. See this page[0]. They now have a list of their products and how they fit into different categories. But the console can still be a jumbled mess of different acronyms and made up words.

[0] https://aws.amazon.com/

Re:AWS CodePipeline by rjbwork on Thursday July 09, 2015 @02:02PM (#9859609)
This is why I REALLY like the MS/Azure way of doing things.

"I need to host a web app" "Okay there's Azure Web Apps for that"

"I need to store lots of files" "There's Azure Storage/Blob Storage for that"

"I need a SQL Database" "There's Azure SQL for that"

"I need a VM" "There's Azure Virtual Machines for that"

"I need a Data Lake" "There's Azure Data Lake for that"

"I need a Data Lake" "There's Azure Data Lake for that"

"I need a Data Warehouse" "There's Azure Data Warehouse for that"

"I need a Cache" "There's Azure Redis Cache for that"

I could go on, but you get the picture. Cute names are not the way to go when you're offering dozens of services which may overlap with eachother somewhat. I can just scroll down a list of things MS offers on Azure and be able to easily pick out the things I need to use by their names alone.

Re:AWS CodePipeline by Artemis2 on Thursday July 09, 2015 @03:06PM (#9860028)
As someone who does not uses Azure, what is a Data Lake? Is it like a Cloud but liquid?
Re:AWS CodePipeline by redwards510 on Thursday July 09, 2015 @04:06PM (#9860341)
"A massive, easily accessible data repository built on (relatively) inexpensive computer hardware for storing "big data". Unlike data marts, which are optimized for data analysis by storing only some attributes and dropping data below the level aggregation, a data lake is designed to retain all attributes, especially so when you do not yet know what the scope of data or its use will be."
Re:AWS CodePipeline by Artemis2 on Thursday July 09, 2015 @05:14PM (#9860738)
> data marts

So it's just deeper down the rabbit hole.

Re:AWS CodePipeline by rjbwork on Thursday July 09, 2015 @05:30PM (#9860819)
The intended use is to be able to use things like Hadoop or other tabular text processing systems to glean information from enormous amounts of data, then once valuable insights are found, use the Data Lake source to process it into a form suitable for a data mart, or preferably, a data warehouse.
Re:AWS CodePipeline by aikah on Thursday July 09, 2015 @01:38PM (#9859433)
Do you really find the AWS console usable? I was shocked the first time I logged into that thing, I said to myself, "what the hell is that?". AWS is certainly one of the most complicated app available online, and i'm sure it is maintained by a lot of different teams but a serious designer didn't do the UX/UI. It feels SO clunky.

> Not that it matters, but I think they use GWT.

the problem is I think it's much more difficult to iterate with this kind of abstractions.

Re:AWS CodePipeline by mcintyre1994 on Thursday July 09, 2015 @11:27AM (#9858388)
I find this comment interesting because I thought it was Bootstrap - I think a few versions/years old but not any older than Bootstrap.
Re:AWS CodePipeline by uberdog on Thursday July 09, 2015 @12:26PM (#9858882)
Have you used Asgard? The AWS console is gorgeous comparatively.
Re:AWS CodePipeline by softawre on Thursday July 09, 2015 @12:31PM (#9858931)
Haha. Which is funny, considering Netflix has some pretty good UX all around. That's what happens when developers builds UIs.
Re:AWS CodePipeline by ceejayoz on Thursday July 09, 2015 @12:51PM (#9859088)
Netflix has long had bits of horrid UX. They just finally replaced the agggggggonizingly slow scroll left/right as you're browsing through the suggested content carousels.

An internal tool like Asgard doesn't have the same need a customer-facing tool like the AWS Console does to be user-friendly, either.

Re:AWS CodePipeline by jon-wood on Thursday July 09, 2015 @12:01PM (#9858693)
I don't think anyone on the AWS team really thinks of the console as being the primary UI for AWS. The primary UIs are the API and CLI interfaces.
Re:AWS CodePipeline by ceejayoz on Thursday July 09, 2015 @12:14PM (#9858796)
> The primary UIs are the API and CLI interfaces.

This used to be the case - new features usually launched API-only and console came later - but I think Amazon has woken up to the fact that many customers want to play with it easily before they go automating everything using the API.

Re:AWS CodePipeline by solve on Thursday July 09, 2015 @12:23PM (#9858866)
Strongly disagree. Beautiful design is a failure anti-pattern. It's no coincidence that many of the best tools are ugly. AWS is a set of tools.
Re:AWS CodePipeline by bgibson on Thursday July 09, 2015 @01:09PM (#9859228)
> Beautiful design is a failure anti-pattern.

I've been wondering about this recently myself. It does seem to be the case sometimes. Any idea why, or if this has ever been studied?

Re:AWS CodePipeline by brianwawok on Thursday July 09, 2015 @09:56PM (#9861904)
Pretty app - Ehh, I could add this new feature but it would make my menu go off screen, maybe I will skip it and work on making the profile icon sparkle.

Function ugly app - Ehhh I could research better fonts, but sans serif will work - let's go code some new featues.

Re:AWS CodePipeline by robotnoises on Thursday July 09, 2015 @11:48AM (#9858571)
I agree it doesn't look great, but I assure you they have A/B tested the shit out of every page they have ever rendered. Aesthetics aside, I am sure there is a justifible reason they do everything in their UIs.
Re:AWS CodePipeline by whafro on Thursday July 09, 2015 @12:04PM (#9858716)
Eh, I'd agree about pages on the public/commerce side, but I highly doubt anything approaching that same level of rigor applies to the AWS console.
Re:AWS CodePipeline by robotnoises on Thursday July 09, 2015 @12:06PM (#9858731)
That's a very good point. Maybe a more accurate statement is "any page that could potentially drive a sale has been A/B tested to death."
Re:AWS CodePipeline by ceejayoz on Thursday July 09, 2015 @11:25AM (#9858370)
Yeah, the console feels cobbled together with whatever a particular team decided they'd use for their small part.
Re:AWS CodePipeline by ebbv on Thursday July 09, 2015 @11:56AM (#9858646)
What's up with the Amazons spam? There's 5 different submissions on the front page right now. They could have all been one. Bad form, AWS team.
Re:AWS CodePipeline by dang on Thursday July 09, 2015 @12:54PM (#9859109)
The major tech cos bundle their big announcements on their annual conference day. Amazon, Apple, Google, Microsoft all do this, and all have had multiple stories on HN's front page at the same time. As long as the announcements are significant and HN-worthy and they do it rarely, it seems ok. A story doesn't become spam just because it's coeval with others.

If one or two of the stories are noticeably weaker than the others, or especially if they've appeared on HN before, we'd appreciate users pointing that out, since we can penalize those.

Re:AWS CodePipeline by andyjohnson0 on Thursday July 09, 2015 @12:01PM (#9858692)
AWS Global Summit [1] is taking place in NY at the moment. Hence the new product announcements.

[1] https://live.awsevents.com/

Re:AWS CodePipeline by ebbv on Thursday July 09, 2015 @12:09PM (#9858764)
That is not addressing my question; why are they posted as 5 separate submissions instead of being 1 LiveBlog post or something?
Re:AWS CodePipeline by ceejayoz on Thursday July 09, 2015 @12:13PM (#9858784)
Each one is a separate post, about a separate service, which would logically generate separate, product-specific discussions.
Re:AWS CodePipeline by andybak on Thursday July 09, 2015 @12:29PM (#9858911)
I agree. A single post would just end up unmanageable and dominated by whoever had the top comment. (I always yearn for a 'by date' sort for HN comments as often the top comment hijacks the discussion on a tangent and it's annoying to find people discussing the primary topic)
Re:AWS CodePipeline by dang on Thursday July 09, 2015 @01:21PM (#9859316)
> I always yearn for a 'by date' sort for HN comments

We can't make a big flat list and sort those by date without losing the threadedness of the comments and thus their context. But we could sort siblings by date at each level of the tree, including the top level. Is that what you mean?

Re:AWS CodePipeline by andybak on Thursday July 09, 2015 @02:02PM (#9859611)
> without losing the threadedness

Sometimes this would be a good thing. Top-level (and in general 'higher up the tree' comments completely dominate. It's rare for the direction of a HN thread to shift much after a reasonable number of comments as everything gets buried quite far down.

I'd really like a 'what changed since i was last here' button. Quite often I come back to a long thread I was especially interested in and I'm completely unable to find out if anything of value has been added.

> But we could sort siblings by date at each level of the tree

More options = better. The intended audience of HN can cope with a few more things to click. Treat us like power-users.

Re:AWS CodePipeline by dang on Thursday July 09, 2015 @04:52PM (#9860607)
HN has always treated users like power users. That's part of its design. But there are other considerations, such as its minimalism and the fact that there is a single shared view of the content.
Re:AWS CodePipeline by andybak on Friday July 10, 2015 @04:13AM (#9862809)
> a single shared view of the content

So is the lack of options to resort and reorder comments an intentional thing?

It would be interesting to hear the reasons for it and what benefits you consider it brings.

Re:AWS CodePipeline by derwiki on Thursday July 09, 2015 @12:13PM (#9858786)
I don't think it's fair to assume that this is spam from the AWS team. Many of us use AWS services daily and like to keep up with new offerings.
Re:AWS CodePipeline by ebbv on Thursday July 09, 2015 @12:15PM (#9858798)
If you look at the profiles of the submitters, at least one admits to being an Amazon employee.

Yes, lots of us are interested in AWS (I am too), but that doesn't mean each separate announcement should be its own post. And I don't think the nearly uniform submissions we see would have happened organically.

Re:AWS CodePipeline by andyjohnson0 on Thursday July 09, 2015 @12:11PM (#9858776)
Because they were posted by different people who didn't take your preference for a single post into account.
Re:AWS CodePipeline by ebbv on Thursday July 09, 2015 @12:13PM (#9858785)
At least two of them are posted by the same guy. And at least some, if not all of them, are being posted by Amazon employees.

So again, I say, the AWS team is spamming HN and it's in poor form. It should all be one submission.

Re:AWS CodePipeline by WaltPurvis on Thursday July 09, 2015 @01:37PM (#9859428)
No, it should not be all one submission. These are separate and distinct products, and I want to read separate and distinct HN discussions about them.
Re:AWS CodePipeline by kordless on Thursday July 09, 2015 @12:47PM (#9859055)
Note that you are also getting voted down. Lot's of Amazon employees to hit down votes on people who say anything against them. This text may be grey. :)
Re:AWS CodePipeline by WaltPurvis on Thursday July 09, 2015 @01:39PM (#9859441)
He's getting voted down because he's completely wrong and semi-slandering Amazon employees with no factual basis, as are you.
Re:AWS CodePipeline by tedunangst on Thursday July 09, 2015 @12:57PM (#9859136)
Welcome to the future, where it is now required that you organize your website to conform with the HN submission process.