News for hackers. Stuff that matters.

ColFrancis

Re:I've Stopped Using Box Plots. Should You? on Monday June 24, 2024 @11:39PM (#40784227)
As a first pass definition it does well to explain the concept. Even if you're interpolating you will need to rank the samples and find the two nearest neighbours to interpolate between.

It serves to distance it from the moment-based statistics like mean and variance at least.

Re:I've Stopped Using Box Plots. Should You? on Monday June 24, 2024 @11:37PM (#40784209)
The quartiles are defined by doing the sorting and splitting algorithm. So if you want quartiles (or any other quantile generally) you need to calculate it that way. The mean and standard deviation (sigma) are fundamentally different, which is why the image you linked shows them to contrast against the quantiles.

If you want to represent the standard deviation with your box plot, you can calculate it using standard formulas, many maths libraries have them built in. I don't know how to plot it using any graphing package though. ggplot, plotly and matlab all use the quantiles (the ones I have experience with). Perhaps where ever you learned to read them as mean and standard devation has a reference you could use?

> They are too random to be meaningful. It does not make sense to draw a box plot from that.

This can be a problem. In practice, the distributions I see don't go too crazy and are bounded (production rates can't be negative and can't be infinite). I prefer to use the 10th and 90th percentiles which are well defined and better behaved for most distributions. I do make sure it's very clearly marked on each plot though as it's not standard. Using the 1.5 x IQR cutoff is no better though as when you have enough samples you find that the whiskers just travel out to the cutoff.

Re:I've Stopped Using Box Plots. Should You? on Sunday June 23, 2024 @07:32PM (#40771513)
For what it's worth, you've convinced me that my beloved box plots need to be explained if I want to use them again.

The SVG you've provided clearly shows that the box plot splits the data in 4. The interquartile range (IQR) is clearly marked and it even has a comparison for what the standard deviation (variance) measure would be.

Secondly, if the data truly came from a normal distribution, there are no outliers. Outliers are data points which cannot be explained by the model and need to be removed. Unless you have a good reason to exclude the data points they should be included. This is why I like the IQR and the median, they are not swayed by a few wide valued data points. The 1.5*IQR rejection filter I think is lazy and unjustified. Happy to discuss this point further as it is a bug bear of mine.

Re: on Monday June 26, 2023 @12:39AM (#36475781)
As I understand it, it's more that the mathematical functions aren't specified by the standard.

There is no sqrt instruction. So you stard with a taylor approximation, then maybe do a few rounds of Newton's method to refine the answer. But what degree is the taylor series? Where did you centre it (can't be around 0)? How many Newton's methods did you do? IEEE 754 might dictate what multiplication looks like, but for sqrt, sin, cos, tan you need to come up with a method of calculation, and that's in the standard library which usually comes from the compiler. You could make your own implementations but...

Floats are not associative: (a+b)+c != a+(b+c). So even something like reordering fundamental operations can cause divergence.

Re:UK pensions hit with margin calls as gilts and sterling slide on Thursday September 29, 2022 @10:08PM (#33030032)
Sure, punish the recklessness to remove the moral hazard. You can both make sure people don't starve and align the incentives. If you don't then next time it will be worse.
Re:Guido was right, there should be no lambda in Python (2018) on Thursday September 29, 2022 @10:03PM (#33029995)
I know this is a very late reply, but I'm curious. How often does it align that the functional argument that you want to specify is in the correct position of the argument list that you get to make use of this feature? The times that I need to partially apply are slim, and I don't believe it's consistent that they're on the same side all the time.

The lambda keyword is long which makes it a pain, but currying really is equivalent to specialised partial application right? People seem to love currying so much I feel like I don't understand something here.

For context I work a lot with java, R, and Python which all have anonymous functions and higher order functions, but are all decidedly _not_ "functional languages".

Re:Guido was right, there should be no lambda in Python (2018) on Tuesday September 27, 2022 @07:50PM (#33002659)
> l_add needs the arguments passed one at a time. We never see this syntax with multiple parenthesis in Python. It works and it has many benefits built in, but it would be a paradigm shift to expect pythonistas to start writing their programs this way, it’s just not Pythonic.

He makes a big point about currying in his particular manner, but what's wrong with

    f = lambda x,y: x+y
    g = lambda y: f(3,y)
That's also currying right? I've always been a little confused as to why the style he advocates is necessarily better. I'm not a functional guy so I'm maybe I'm missing something, what is the benefit of syntactic sugar for partial application to the first (or last) argument of a function? What if I want partial application of a function with the middle argument specified? Now you're out of luck and have to do the above anyway.

ETA: on the main point

> In Python, we do not use lambda with the intention of using lambda calculus. Our syntax needs to encode our intention in the places we have used it, and where we will use it. These use cases are inline, anonymous functions.

That seems needlessly nitpicking. Wikipedia has "lambda function" as a synonym for anonymous function so it seems pretty widespread in our language at this point. https://en.wikipedia.org/wiki/Anonymous_function

Re:Tutors / tips to change your English accent on Sunday August 07, 2022 @08:43PM (#32381351)
It's tricky. I find it a little sad that regional accents are disappearing but it's a process that's been happening for a long time.

The trouble is that there's real benefits for having certain accents and you can't fault someone for wanting to be seen in a different way.

Re:What happens when a reservoir goes dry on Tuesday July 19, 2022 @07:58PM (#32159557)
> Majority of the streams are gone.

In Australia that's typical for many reasons. Are you sure they're not just seasonal flows?

Yikes, 260 character path names can be a real pain. Please support sensible path names or at least tell me why you can't read or write to a file. Often you just get a "can't write to that file" error message, or worse, "forbidden" so you spend an hour trying to debug the mess that is folder permissions on windows.

I'm not going to convince you to change your tool chain, userbinator, but for the sake of the discussion: Once you have multiple projects going on, with multiple components, and then those components have a small directory structure themselves, you can easily reach 260 characters. Add to that, if theres data coming from another org, a long file name can be very helpful to keep track of what it is (and don't forget 10-12 characters for a date!). And finally, the nail in the coffin: most users don't think about path names, I struggle to get people to not put periods in their filenames which messes with some tooling, how am I going to convince the guy in finance who gave me this data that he should use short file names? Should I modify the file name and make it untracable?

ETA: The "if you have to ask you've messed up", I don't ask, I expect and then get annoyed it broke. I had 10,000 files collected into a folder. Why can every other program tell me the list of files in an instant but windows explorer crashes (the whole desktop environment) because I opened that folder to see it. I'm not meant to do that? Then why can the kernel, the disk, the file system, and all other programs handle it with ease?

Re:A better way to divide the pie on Monday March 21, 2022 @01:09AM (#30750530)
I think the example problem has many issues with it which confuse the issue. There is a pervading assumption that more slices is better. What if Alice only wants 5 slices? Then her fall back is quite close to what she needs but also she doesn't mind how the pie is divided so long as she gets one slice.

It's not clear what everyone's utility function is. If a sales person is nearing a quarterly target, their utility function might have a giant kink in it, or if they're at the start of a period it might be very linear. The marginal pie, more often than not, has multiple dimensions which different agents care different amounts for.

Thankfully I've never been in a negotiation without a decent BATNA and so just chat about it until either we decide it's not going to work out, or we are both happy. People who negotiate too hard by playing stupid games and trying to max out themselves without any quarter given are tiring and push me to the first option. Why would I want to work with you if you're a pain to deal with before we even start?

Re:Imprisoned for Your Safety on Saturday February 19, 2022 @07:29PM (#30401952)
I cannot run it "permanently". Basically, I can run it along the floor but not hidden at all as that counts as installing.

https://whirlpool.net.au/wiki/diy_cabling_-_read_this_before...

Re:Imprisoned for Your Safety on Saturday February 19, 2022 @07:27PM (#30401934)
A genuine question, from someone who doesn't have the option: would you prefer to have it banned?
Re:Imprisoned for Your Safety on Saturday February 19, 2022 @07:24PM (#30401917)
I've noticed a lot of turns at traffic lights are getting explicit red arrows. A solid green light without an arrow means that you need to give way to pedestrians and other vehicles if turning. It feels to me no one is looking any more, they just think green means go. I think it's because we've trained them out of thinking with too many constraints.
Re:Imprisoned for Your Safety on Saturday February 19, 2022 @07:21PM (#30401898)
Warning labels, perhaps not. But in Australia I cannot run Cat6 for "safety". I might electrocute someone with the 12V and must get a licensed professional to do the work for me.

A label warning that high voltage is inside is fine, a required label is quite ok, a fine because you didn't put the label on (even though it wasn't explicitly required) is a little bit of a stretch, banning modification is clearly an overreach (in my book).

There is a line somewhere, and drawing it too close to either end is harmful. So now we're arguing degrees, and not absolutes.

Jobs shouldn't be a limited resource.
Funnily enough, I really struggled going to Java and python coming from C++ originally. What do you mean I don't know if I'm getting a copy? The semantics of passing something to a function were so confusing and I was never sure if I got a copy or not. Pointers made so much sense.
Re:Technology Traps (1978) [video] on Sunday July 04, 2021 @07:59AM (#27729023)
He almost got there in the end, but he stopped short. The plough will break. Then what? Do you know how to make charcoal and how to make a forge? What about replace the steel that's rusting around you?

Beyond the technology traps, we're in a "society trap", we need each other and the skills each other has. The trouble for the fictional scenario is not the technology, it's the lack of a community to pool knowledge. The little old lady who loves her garden and knows what plants are what, even some of the exotic ones typically imported, the fitter and turner who might not know how to exactly use a forge but understand metallurgy enough to shape it sensibly, the farmer who knows what the plants and animals need even if they don't have on hand the tools they normally use.

I for one don't want to end up by myself, I quite like all the benefits society and specialisation bring. In no part of history or prehistory I know of (not a lot, granted) has much of society known enough to survive alone without at least some sensible community.

Re:On the bare necessity of psychological safety on Wednesday April 21, 2021 @06:38PM (#26896305)
Regardless of reality, perception of reality is more important for this. The boss can think the worker is wrong, and can react with a

> That's stupid, it won't work due to X, we're going with Y

or a

> Thanks for the feedback, but this time we're choosing Y due to X.

One is going to leave the worker happier than the other and feeling safer despite the boss thinking the worker is wrong. Unless you can measure the counter-factual (often not) you will only guess as to which was better.

Re:Just Be Rich on Tuesday April 13, 2021 @10:11AM (#26792363)
> The poor get poorer not because tech startups get richer.

Not that Uber has made any money, but it's certainly dropped the price of taxis and their market. Someone got poorer.

Re:GitHub, fuck your name change on Wednesday March 17, 2021 @07:57AM (#26489827)
I get the feeling sometimes that language is unstable in the absence of outside forcing.

Stupid, moron, retard. People start using them as a pejorative and so a new neutral term is required.

Lavatory, toilet, water closet, bathroom. People don't like talking about poo so they keep using euphemisms to describe the room where it happens. This one is a favourite of mine as some people find 'toilet' distasteful despite it being the furtherest from the actual action taking place.

Very, literally. Not sure where this one is headed, but if it's happened at least twice, it'll shift again some time.

Are we doomed to keep shifting language to keep up taboos and keep the meaning of words which naturally will shift?

Re:I wrote one of the fastest DataFrame libraries on Wednesday March 17, 2021 @03:21AM (#26487586)
Thank you, my brief research led to a list of versions that had R 1.0 as 2000, but it appears that v0 lasted a good many years. Pandas as well was in v0 for many years so it is the better comparison to use like-for-like.
Re:Audi abandons combustion engine development on Wednesday March 17, 2021 @03:16AM (#26487561)
Genuine question: How is this different from a F1 engine where each piston head is crafted for each cylinder in the block? When you go high performance things that look like they're a commodity become one-of-a-kinds as you push them right to the limits.
Re:I wrote one of the fastest DataFrame libraries on Sunday March 14, 2021 @07:29AM (#26454453)
R is from ~2000, while pandas started in 2011. Is it possible that the lack of compute power had an effect on the required performance characteristics?
Yes, it would. But right now the long term maintenance of the A380 is shaky despite Airbus still being around. Devils advocate to long term maintenance with a large company: they could just stop caring about you as a customer or refuse to support you.

It's a difficult space to make decisions in and I think it's difficult to make any broad statements about reliability of product or company. Someone would need to actually run some numbers.

The idea is that if it's small enough to fail, then the incentives are much stronger to avoid failure. If it's too big to fail, why would they put effort into avoiding failure? So yes, buying from a small enough to fail company could be seen as safer.

Note that too big to fail, and small enough to fail has less to do with the size of the company and more to do with market position and government perception.

Re:Accidentally quadratic: When Python is faster than C++ on Thursday March 04, 2021 @05:02PM (#26349049)
Just checking, even though you likely know. The numpy arrays really only pay off when you vectorise your calculations. Don't expect any speed up if you're still using list comprehensions.

A vector operation is just where instead of having two lists, U and V which add to make W = [x+y for x,y in zip(U,V)] you directly operate on the arrays, W = U + V.

This allows the inner loop to be completely in native code. Sometimes some curious things happen where it's easier to do more vector operations resulting in more looping just because each of those operations is a vector operation and so is faster than the loop. For example, increment a number every time you see a particular element (time series breaking out subsequences) might look like np.cumsum(U == value) which is two loops in practice, but much faster than the iterative approach.

In this case if they create enough supply to match demand the price will fall as they are no longer "limited supply". They would far rather leave money on the table than push it too far and lose the value proposition entirely
Re:In-car touchscreens are dangerous on Friday February 26, 2021 @08:04PM (#26281707)
> At those Texan speeds, managing to stop a sixteen wheeler on ice would be a miracle.

So what you're saying is they should have been going slower due to inclement weather?

> constant logarithmically incrementing indicators

Exponentially increasing? 10^3, 10^6, 10^9, 10^12

(I kind of hate being that guy, but I mean it in a positive and helpful sense)

Older »