This involves under-clocking and under-volting the cards, while also ensuring they're cooled well.
I think an argument could be made that mined-on cards are more well cared for than those owned by your average gamer.
There's not some 'destroy the ASIC' instruction. Work is work.
Mined on cards undergo fewer duty cycles, I wonder how the calculus works out between one long mild duty cycle compared to many more that are also more extreme.
If the card crashes due to tuning, the machine crashes, and the rest of the cards are taken offline until the machine reboots. In order to change settings, you have to reboot the entire machine too. It is a pain in the ass to find optimal tuning at scale.
There is no publicly available software to 'autotune' cards at scale. It took me a long time to first learn how to tune cards (by hand tuning thousands of cards and getting a feeling for things) and then once I figured that out, I built software to do this. It required changes in the AMD driver to get it to output in the kernel messages, which card was crashing. It required updating the mining software itself. It was a nutty insane project, but it worked well. We hit our target hashrate / power usage.
That was another reason to go with the RX4/5 series... fastest route to ROI.
The only 'secret', which isn't a secret, was that I realized that since this is hardware, we know the maximum settings (hash/watt) that should be possible. Therefore, I set the cards to that best setting and then tune down from there. This is the opposite train of thinking than most other miners.
Most people think, let's start low and then tune up from there to make them 'faster'. The cards crash when they can't handle the settings, so it turns out that tuning down is a better way to tune since they stop crashing when they are stable... and thus don't need tuning from there. There are 3 different sets of 'knobs' to tweak, so I had to build an algo to adjust the knobs in the right order to tune things down. I just had the concept of 'current -> next settings'.
Temperature and power fluctuations can make the cards crash too... so by always tuning down, you're always heading towards more stability instead of instability. Since neither of those could be controlled, machines would reboot randomly all the time.
The software I built was a golang daemon that ran on each machine and watched for these crashes and modified the tuning of each card individually. The daemon is pretty cool as it is effectively a task runner. I had different tasks to configure and monitor the machines as well. The machines are all independent, idempotent and self-healing workers. Reliably distributing the software to 20k different workers, is a fun challenge. There are a ton of unit tests, so that helped a lot.
If I have the energy, I may rip out the tasks from the daemon, turn it into a library and open source that. It is kind of a fun project that could be useful for others trying to manage large scale individual workers. Tasks could easily be 'apt install' or monitoring utilities. I even bundled node_exporter into the binary, so that we could monitor the machines with prometheus.
This comment is just your continued trolling with your anti-crypto trope. Please try another tactic.
Update: I got downvoted to -4 on this one after getting several upvotes. Makes me think that articbull has multiple accounts and is just gaming the system.
You are very likely mistaking normal HN herd behaviour - plenty of people are going to think bullshit to “unused hydro”.
Also “Please don't comment about the voting on comments. It never does any good, and it makes boring reading.” and “Please don't post insinuations about astroturfing, shilling, bots, brigading, foreign agents and the like. It degrades discussion and is usually mistaken. If you're worried about abuse, email [email protected] and we'll look at the data.” - https://news.ycombinator.com/newsguidelines.html
If you assume that HN is full of good people, and respond accordingly, you will learn more and have more fun. If you assume HN is full of bad people causing trouble, why would you stick around? Good communities are built upon trusting cooperators and learning the valid signals of trust.
> plenty of people are going to think bullshit to “unused hydro”.
Maybe... but the way I went up and then suddenly went down to -4, was super odd.
> If you assume that HN is full of good people, and respond accordingly, you will learn more and have more fun..
I've been around here, with an account, since 2009. It is the only community I participate in any more. I believe that HN is full of great people and that the content is about as good as you're going to get these days. This is why accounts such as articbull are becoming boring to me. All it is is anti-crypto hatred that is often not fully factual. Equating me to a tobacco exec is just icing on the cake. Sorry for calling that out.
This person is a successful troll, and a pathetic thinker.
Transmission of power is expensive and lossy. These data centers are in a very remote location that isn't heavily populated. That's why they are there to begin with! The power would absolutely go to waste otherwise. That's also why we picked that area... power was cheap because it wasn't being used.
Our GPUs are 'new old stock'. They were already produced, sitting in warehouses not doing anything, and would have gone to waste anyway.
Update: Similar with the downvotes...
First question, where do you think I got most of the karma? Sentiment here is broadly against crypto and has been for years (although it comes and goes). The pro side is loud, but enjoys far less support than you might imagine.
> Transmission of power is expensive and lossy.
No, it isn't. Transmission of power is super efficient. The EIA says the US lost only 5% of its electricity to T&D over the 2016-2020 period. [1] Expensive it may be, but that's not a reason not to do it. A single long power line from Boston to LA would only see ~25% loss from one end to the other. The whole Ohm's law thing (P=IV), we use transformers that are 99%+ efficient to raise the voltage to ~345kV and big fat wires to reduce current and resistance and hence transmission line losses.
> That's also why we picked that area... power was cheap because it wasn't being used.
It should be used to do something productive, not generate e-waste.
> Our GPUs are 'new old stock'. They were already produced, sitting in warehouses not doing anything, and would have gone to waste anyway.
Or, prices would have gone down and more people would have bought them. I can't see 3060s getting scrapped due to lack of demand when they were going for $850 a pop.
Agreed. However, I'd hope that 'against crypto' was less of one dude ranting endlessly, which is really boring and should get down voted.
> Transmission of power is super efficient.
We're not talking about efficiency. Transmission is expensive. Which goes beyond efficiency. You have to factor in the cost of the lines themselves, which are based on distance. I've seen these costs first hand, have you?
> It should be used to do something productive, not generate e-waste.
Oh yes, please define productive for me.
> Or, prices would have gone down and more people would have bought them.
They weren't for sale to the public. Nor would they ever have been.
> I can't see 3060s getting scrapped due to lack of demand when they were going for $850 a pop.
Huh? This whole thread is about RX4xx, which is what my cards are.
It would actually be less than that using a modern HVDC transmission line. Losses for HVDC are around 3% per 1000 km of cable, and around 0.7% for the converter stations at each end. So around 15% loss for a theoretical Boston-to-LA HVDC.
When those plants shut down, the power, which was transmitted directly to the plants, had nowhere to go. Sending it somewhere else requires building high voltage transmission towers, which is very expensive to build as well as maintain.
Coinmint [1] took over the old Alcoa plant and is now pulling power from that dam. As it was explained to me by the head electrician when I visited, Coinmint is doing the dam a favor by having a constant draw of power to replace what was once the smelters. They also are paying for the electricity, which helps with maintenance of the dam, which is effectively a national resource.
This is true for nuclear and some types of fossil-fuel power plants. But for hydro, all you have to do to vary the power output is adjust the amount of water running through the turbines. Plus, hydro plants that have their own storage lakes/dams have built-in storage: any water you're not using for generation can be saved in the lake for when it's needed.
(OK, some rivers have minimum flow requirements so that fish don't die and water can be used downstream etc...)
What strikes me though is people are rarely discussing mileage, only nature of work.
The video card in my gaming computer works, on average, maybe 1hr/week (some weeks more, most weeks 0). So right out of the gate, my video card has been working basically 168 times less long than any mining card running 24x7. We are talking more than two orders of magnitude difference in "mileage", before we discuss how well or poorly it has been taken care of.
The 3090 is notoriously bad for mining because it has memory chips on both sides of the board, leading to more heating in one spot and poor cooling for the 'back side' chip. Other cards tolerate it much better, but it can be an issue.
So while overall mining cards should be good, there are some specific issues to watch out for. A failed memory chip is not an easy fix, requiring a reball/replacement of that chip.
Maybe, as you suggest, it is just something that is more card specific.
Furthermore, flashing custom mining BIOS, what I’m told is that every miner here has to do or it wouldn’t be profitable, voids warranty.
I think the used goods buyers' attitude is similar to those in the car market. Never mind the miles, if a car had heavy users, like a pizza company's delivery vehicle, it looks different in the market than a garage-stored family car used to visit the relatives.
To the pizza car analogy, when I was in that game I put a hard 60k miles/yr on such vehicles, but I also regularly change the oil, keep great tires and brakes on them, replace bushings and other suspension components at the first sign of issue, etc... Partly because driving that many miles makes issues easy to spot due to familiarity and temporal coherence and partly because it is cheaper to fix many such things sooner rather than later after they've jostled something else out of place. Nobody ever asked if it was a pizza car. I listed it as well maintained and got top dollar each time because it was and their inspections showed that to be true.
Sorry, but this is pretty much incorrect. Copper migrates, even in solid state devices, and those copper migrations cause breaks and shorts, which cause errors and failures. The smaller the scale, the faster this happens. There is absolutely a MTBF even for solid-state devices.
This entire slide deck is worth the perusal, if only for the "whoah, that actually happens?!" moments. But here's a relevant slide for this claim:
https://c3.ndc.nasa.gov/dashlink/static/media/other/Observed...
The money quote: "Commercial semiconductor road maps show component reliability timescales are being reduced to 5-7 years." (with attribution)
Another fascinating slide is this one, which shows how an IC can start to have a timing issue after being in service for a time.
https://c3.ndc.nasa.gov/dashlink/static/media/other/Observed...
If you're buying something for like half of retail and rolling the dice on it, I think it is a fair bet. Even if you only get 1-2 more years out of a card that has been used before (for any purpose), then you've done well.
For clarity, because @derac is confused below... this is a joke. Hence the winky face.
If we sell them for pennies on the dollar or e-waste them, doesn't matter to me.
We ROI'd these cards already.
You are sitting on 10m+ of stock and you don't care about the money. OK bud guess I must be schizophrenic, damn.
You are literally trying to sell your cards to a sister comment...
"Ok, $10, but you have to pay for shipping and min order is 10k. ;-)"
You are literally making stuff up now. The 10m+ of stock is worthless if nobody would buy it. If anything, it'll cost us to e-waste it. So, the price ceiling, if we were to sell it, is simply the cost e-waste and moving it around.
That was a winky face... a joke. I was kidding because the OP was unhappy with the statement of half price, which I also made up as a random baseline.
Edit: to me it's clear that you are acting in bad faith since you called me crazy for pointing out an obvious conflict of interest.
2. Tell me exactly how would it benefit me to change public discourse around the long term reliability of GPUs? The most I can think of is that it might drive up the prices of used cards a tiny bit in a very saturated market, but I doubt it. Especially since I'm just one data point and I'm only posting here on HN. Plus, as I said, I don't care what our cards sell for. Our baseline cost at this point, is to offset any price of e-waste and shipping.
3. I shared my findings about long term reliability because this is a tech community and I thought they'd be interested in it! OMG, I'm such a horrible person for thinking this way! That is sarcasm.
4. Allowing people to realize that mined on GPUs are actually ok (not all, but generally better than common thought) might mean that a few more find useful homes and not end up as e-waste. That's a good thing! I even suggested that people use it as a way to negotiate the price down!
Now you're at it again with the criticism. I'm not handwaving anything. I'm stating what I saw after having the fairly rare experience of running 100k+ GPUs over multiple years in harsh environments.
Let's turn that around, do you have any experience with this at all? Or do you just criticize people?
It also doesn't benefit me in any way to share information like this. I'm only sharing this information now because we're pretty much done with this phase of the business, so I can talk about things a bit more openly, and it seems relevant to share on a thread about RX cards.
You did handwave that, we can agree to disagree.
I didn't even say you are or aren't doing anything nefarious. I pointed out a conflict of interest (incentive to propagandize) and you responded by calling me crazy. Now you are saying "All I do is criticize". Boohoo to you sir.
You can care for that car (or chip) all you want, but chips aren’t cars. It’s a slow process, and may not happen on the timescale of mining, but it happens.
https://en.wikipedia.org/wiki/Transistor_aging
https://en.wikipedia.org/wiki/Time-dependent_gate_oxide_brea...
It kinda does. The effects are small, but increase as the fab node gets smaller.
Either I'm reading this sentence incorrectly or there is something else I don't understand. How can that statement be true?
Whether it's true for light bulbs I don't know.
Other algos take advantage of more of the GPU side of things as well. ProgPoW, if it had passed, would have done that. From what I understand, RVN also does as well... and uses more power too.
PSU's, on the other hand, are mostly hand soldered in China and have all sorts of issues.
very different than a car
but valid questions to ask to whoever you are buying from used
I feel like people dislike cryptominers (IMO reasonably, and for all sorts of reasons) and want to punish them all the way by preventing them from liquidating their stocks of cards, but I don't think it's a good reason to make stuff up.
I wouldn't buy a card that I know was used for mining because I'm effectively boycotting miners, not because of some made up technical excuse.
You could tell me that either of those things dominate, I don't know. But I would want to have atleast an idea of this before I bought a used miners GPU.
Having said that, I wouldn't buy a gamer's old GPU.
You can't run the cards at high temps or they crash. They also have built in thermal protection where they shut down at about 85c (depends on the settings).
Miners also undervolt/underclock the cards. Lower power is better for opex in the equation above. =) This means they aren't run as hard as you might think. ethash as an algo is also memory hard, so the GPU portion isn't 100% at all. That's why older cards are still in use and more ROI efficient.
Also, PC gamers are the last group I would expect to successfully boycott something. They'll install some spyware-ridden game launcher just to get a $5 discount on a game, while complaining about it. What discount will they resist for a mined-on GPU? $50? Assuming they even know it was mined on.
We have zero cards for sale right now. Nobody wants them because they are heat sinks and not fans.
seeing more and more people just waiting out to buy the newer cards so it seems like the price needs to come down a lot more to convince people to buy a gpu used in mining.
just my two cents based on observing reddit threads
I was just thinking today about many years ago when I test drove a used car and heard whistling from the roof. Saleswoman said it was easy to fix. It must've happened during every test drive and possibly dissuaded some buyers, so why hadn't the dealer already fixed it? I bought the car, and guess what, the whistling wasn't fixable.
What kind of question is that.
We use a case based cooling system. ie: an enclosed box with fans. 12 cards in a box. This is lower power, fewer fans (5 instead of 12), and better cooling than putting fans on each card. Also easier to replace a fan on a case than a card.
At my scale, water cooling doesn't work and isn't cost efficient. Air is a surprisingly great cooling system.
It might work, I don't know... airflow on the box itself wasn't enough, but there were 12 cards in the box, we also had to have external airflow as well. A single card probably wouldn't be too much of an issue though.
That said, we aren't parting out single cards... it would be more like if you wanted to buy 5-10k of them. ;-)
The paste goes bad, but is easily replaced (majority of our cards have been running just fine with OEM old paste). We don't have fans on the cards, but if we did those would go bad and easily replaced. But the cards themselves, really don't go bad very often.
If you're thinking about picking some up, don't worry about the mining part, but definitely use that to try to get a discount.