News for hackers. Stuff that matters.
Re:Andreessen-Horowitz craps on “AI” startups from a great h... by bob1029 on Monday February 24, 2020 @04:39PM (#22408738)
I find it fun how the cost of the cloud is forcing people to consider what absolutely must run in the cloud (presumably for stability and compliance reasons) and what can be brought back on-prem.

We don't train ML models, but we are in a similar boat regarding cloud compute costs. Building our solutions for our clients is a compute-heavy task which is getting expensive in the cloud. We are considering options such as building commodity threadripper rigs, throwing them in various developers' (home) offices, installing a VPN client on each and then attaching as build agents to our AWS-hosted jenkins instance. In this configuration we could drop down to a t3a.micro for Jenkins and still see much faster builds. The reduction in iteration time over a month would easily pay for the new hardware. An obvious next step up from this is to do proper colocation, but I am of a mindset that if I have to start racking servers I am bringing 100% of our infrastructure out of the cloud.

Re:Andreessen-Horowitz craps on “AI” startups from a great h... by blt on Monday February 24, 2020 @05:37PM (#22409177)
If I worked from home and my employer asked me to install a server in my home, I would tell them to go fuck themselves.

It's noisy, it takes up space, and presumably I'm on call to fix it if it breaks.

You should pay them an extra 24x(PSU wattage)x(peak $/Wh in area) per day for the electricity too.

I'm alarmed that someone in your company felt this idea was appropriate enough to propose.

Re:Andreessen-Horowitz craps on “AI” startups from a great h... by bob1029 on Monday February 24, 2020 @06:11PM (#22409424)
We would certainly compensate employees. I didn't feel it appropriate to disclose every last detail regarding the arrangement in a thread which is only tangentially-related to the OP.

This was my idea. I am a developer in my company. We are a flat structure. We have a lot of respect for each other. I am on a standup with the CEO every day. We all believe in our product and would happily participate in whatever activity brings it to market more quickly. We do not hire or retain the kind of talent that would flatly refuse to participate in experimental projects like this. At least not without some sort of initial conversation about why it's not a good fit for a particular individual.

I certainly see how someone might share your perspective. I used to work for a souless megacorp and I could have easily found myself telling my former employer to "go fuck themselves" if a proposal similar to this was imposed upon me.

Re:Andreessen-Horowitz craps on “AI” startups from a great h... by starfallg on Monday February 24, 2020 @07:21PM (#22409798)
Try the value data center providers like he.net.

A 42u rack and 1 Gbps connection is $400 per month.

Put cheap supermicro EPYC servers in rather than threadrippers (or build your own). High capacity RDIMMs are cheaper than UDIMMs.

This will give you a much more maintainable solution than workstations at employee's home via overlay VPN.

There is a maintenance cost for infrastructure that people tend to forget these days.

Re:Andreessen-Horowitz craps on “AI” startups from a great h... by xemdetia on Monday February 24, 2020 @07:04PM (#22409717)
I think it was funny to presume that the employer wouldn't pay for it. At one of my old roles I had a second laptop I used solely for VMs just because I ran out of space on primary workstation. A simple fix orders of magnitude cheaper for a developer compared to more ESXi or cloud hardware.
Re:Andreessen-Horowitz craps on “AI” startups from a great h... by Aeolun on Monday February 24, 2020 @07:10PM (#22409745)
I guess the ‘go fuck yourselves’ is heavily dependent on how able the company is to do this sort of stuff without involving you.
Re:Andreessen-Horowitz craps on “AI” startups from a great h... by blt on Monday February 24, 2020 @08:01PM (#22410031)
Do you have an ownership stake in the company?
Re:Andreessen-Horowitz craps on “AI” startups from a great h... by breischl on Monday February 24, 2020 @06:05PM (#22409396)
Tangentially, I once joined a startup where a week after I started the CEO demanded that I run their crawler software on my own personal hardware, on my personal internet connection, 24x7. I (politely) told him to go fuck himself.

I later learned that the CTO had to spend a couple hours talking him out of firing me. I probably should've just quit on the spot anyway.

Re:Andreessen-Horowitz craps on “AI” startups from a great h... by jiofih on Monday February 24, 2020 @06:07PM (#22409405)
You can stop at “go f* themselves”, plenty of jobs to choose from.
Re:Andreessen-Horowitz craps on “AI” startups from a great h... by m0zg on Monday February 24, 2020 @04:52PM (#22408860)
This is not a new phenomenon. As early as in 2009 I worked for a company (ads, but not Google) which outgrew the typical "cloud" cost structure at the time, and moved everything to a more traditional datacenter, and saved substantial money even considering 3 more SREs they had to hire to absorb the increased support needs. AWS charges what the market will bear, and as such it was never designed to make sense for everyone. One needs to re-evaluate on the back of the napkin from time to time.
Re:Andreessen-Horowitz craps on “AI” startups from a great h... by buckminster on Monday February 24, 2020 @05:14PM (#22409011)
I once had a borrowed Sun blade server in my home office. The fan in it sounded like an industrial vacuum cleaner. It got moved to a different room and was powered on as little as possible.

Your plan makes sense but be mindful of the acoustics or your devs may grow to hate you.

Re:Andreessen-Horowitz craps on “AI” startups from a great h... by bob1029 on Monday February 24, 2020 @05:39PM (#22409201)
Excellent point. If we are building these rigs by hand (which is a likely option considering the initial usage context), the cooling solution would probably be a Noctua NH-U14S or similar. I already have one of these in my office attached to a 2950X and it is dead silent. You can definitely hear it when every core is pegged, but it's hardly noticeable over any other arbitrary workstation. The sound is nowhere near as intrusive as something like a blower on a GPU (or god forbid a sun microsystems blade).
Re:Andreessen-Horowitz craps on “AI” startups from a great h... by maremp on Wednesday February 26, 2020 @09:37PM (#22429855)
What you’re saying sounds like a PC custom build enthusiast. I respect that, I like it too.

Please be mindful of the fact that consumer products are not designed for the workstation/server type of load. It’s related to why the hardware is cheaper compared to the server hardware. Also, the consumer ISP connection is most likely not as reliable as that of a data center. I’m working remotely from home and I experienced this many times, bad performance in peak times or a half of a day of downtime can happen, without any warning. And account for maintenance, everyone on the team must be able to figure out a problem or deal with getting someone else in to fix it.

I know I sound like a buzzkill. I am writing this with good intentions.

Even if this works right now, it’s not a reliable long-term solution. Maybe instead of dumping a couple of grand on consumer PCs to handle server’s work, look into building a proper server. Or you could find a datacenter provider to rent their hardware, something that is not as shiny and full of features like AWS.

Re:Andreessen-Horowitz craps on “AI” startups from a great h... by m0zg on Monday February 24, 2020 @05:21PM (#22409061)
BTW, the only reason why consumer vacuum cleaners are so loud is because consumers associate loudness with suction power.

"Backpack" style commercial vacuum cleaners have more suction, and are barely audible in comparison.

Re:Andreessen-Horowitz craps on “AI” startups from a great h... by buckminster on Monday February 24, 2020 @05:28PM (#22409117)
I stand corrected. It was much louder than a consumer vacuum but my analogy skills are weak.
Re:Andreessen-Horowitz craps on “AI” startups from a great h... by ggm on Monday February 24, 2020 @05:36PM (#22409176)
Your analogy skills were strong, because analogy is rooted in myth, not fact. Achilles did not actually have an Achilles tendon because Achilles did not exist

There does not have to be an increadibly loud functional industrial vacuum cleaner, for figuratively everyone to get your analogy, because the Herculean reality of vacuum cleaners is that you cannot clean an augean stable of lego on the floor, without a lot of noise. If you get my analogy.

Re:Andreessen-Horowitz craps on “AI” startups from a great h... by ColanR on Monday February 24, 2020 @07:07PM (#22409735)
To continue the pedantry, I don't think we know for sure that Achilles didn't exist. Troy certainly did, and we have the Iliad to thank for knowing to look for it.
Re:Andreessen-Horowitz craps on “AI” startups from a great h... by ggm on Tuesday February 25, 2020 @01:30AM (#22411319)
No, we have Schliemann to thank, for digging through it in his lust for anachronistically Indy-Jonesing the archeology. Troy only exists because Schliemann stole the gold. Otherwise its just some hill in Turkey.
Re:Andreessen-Horowitz craps on “AI” startups from a great h... by ColanR on Tuesday February 25, 2020 @11:11AM (#22414305)
I know the story...Schliemann went looking for Troy because the Illiad told him it existed.
Re:Andreessen-Horowitz craps on “AI” startups from a great h... by ggm on Wednesday February 26, 2020 @03:00AM (#22421245)
Didn't work on Ararat. False in unum, false in omnibus. Anyway, he went to the wrong hill first. But overall yes, myth informs reality sometimes. Shame he didn't find the bones of the giant wooden platypus the Greek ice cream salesmen hid in.
Re:Andreessen-Horowitz craps on “AI” startups from a great h... by roel_v on Monday February 24, 2020 @07:00PM (#22409692)
I don't know about backpack style vacuums (the few I've seen seem plenty loud), but the idea that manufacturers deliberately create loud vacuums to make them seem more powerful is ridiculous. There is an entire market segment of vacuums that are marketed as 'quiet', so there is plenty of demand for quiet vacuums. There is simply a trade off between suction / loudness / price, and various models make different trade offs (corresponding to various points in the market).
Re:Andreessen-Horowitz craps on “AI” startups from a great h... by wpietri on Monday February 24, 2020 @07:12PM (#22409753)
That some people want quiet vacuums is not proof that some people don't want loud vacuums. Cars and motorcycles are good examples of markets where you see both deliberately quiet and deliberately loud examples.
Re:Andreessen-Horowitz craps on “AI” startups from a great h... by steev on Monday February 24, 2020 @08:00PM (#22410023)
This doesn't seem any more ridiculous than companies setting a higher price and then always having the item on sale. I think JCPenny famously tried simply lowering their prices and sales went down. I guarantee if you ask the average person "would you rather companies cut the BS and just lower their prices rather than have items on sale in perpetuity" they would of course say lower the prices. But as the anecdote would demonstrate, that isn't how things actually play out.

The point is, when it comes to consumer behavior, I don't think anyone has a clue what to expect. It would not surprise one bit if vacuum companies make louder vacuums because the consumer thinks it works better.

Re:Andreessen-Horowitz craps on “AI” startups from a great h... by m0zg on Monday February 24, 2020 @08:12PM (#22410083)
Is BMW piping in fake "engine noise" through the speakers also "ridiculous" in your opinion?
Re:Andreessen-Horowitz craps on “AI” startups from a great h... by linksnapzz on Tuesday February 25, 2020 @12:29AM (#22411132)
Once upon a time, a company I worked for convinced Sun Microsystems that it would be cool to provide us with hardware at a discount. The discount hardware was six UltraSPARC E4000 servers, each with 8 cpus and 8GB of memory, and a creator3d card in the io mezzanine upa slot.

This company was using them as desktop workstations, in an open office.

One was used as a build host. Often, the shopvac wail of E4000 fans would be cut short by some poor dev going berserk and unplugging the thing when nobody was looking...

Re:Andreessen-Horowitz craps on “AI” startups from a great h... by echelon on Monday February 24, 2020 @08:00PM (#22410025)
> I find it fun how the cost of the cloud is forcing people to consider what absolutely must run in the cloud

Honestly why ever go to the cloud? It seems like a Larry Ellison boondoggle with the absurdly high costs and lock-in. (Ever look at moving your data?)

Running your own metal is cheaper if you actually fund it.

Re:Andreessen-Horowitz craps on “AI” startups from a great h... by CoolGuySteve on Tuesday February 25, 2020 @01:13AM (#22411271)
In my experience, teams will rack up thousands in monthly expenses just being parked in a shell on very large On-Demand or Reserved EC2 instances. Basically using them as development boxes without realizing how much they cost.

I've saved a ton of money just giving them dedicated workstations to develop on and then having everyone use a shared EC2 instance to push jobs to a fleet of spot instances for large scale training.