News for hackers. Stuff that matters.
Re:Our journey to type checking 4M lines of Python by stmw on Saturday September 07, 2019 @07:59PM (#20907543)
This is impressive. But IMHO, anyone starting today should just use Rust from the beginning, and avoid having to endure all of the problems for years and then taking on a big HN-worthy post about 4M lines.
Re:Our journey to type checking 4M lines of Python by MoronInAHurry on Saturday September 07, 2019 @09:52PM (#20908074)
Why do so many people think "just rewrite it in Rust" makes sense to post on every single topic about a dynamic language, even though nobody ever does it because it doesn't actually work in reality?
Re:Our journey to type checking 4M lines of Python by Pimpus on Sunday September 08, 2019 @04:14AM (#20909224)
That's not what he said? You live up to your name though.
Re:Our journey to type checking 4M lines of Python by vonseel on Saturday September 07, 2019 @09:04PM (#20907846)
Maybe if you're Dropbox, Rust is an option, but Rust is much, much more complicated than Python and not nearly as approachable - especially for beginners.
Re:Our journey to type checking 4M lines of Python by ajxs on Sunday September 08, 2019 @12:32AM (#20908623)
I'd argue that how complicated a language is shouldn't be a factor for a company at that scale. If it's the right tool for the job, and I'm not arguing that it is, then they have the ability to hire engineers capable of using it.
Re:Our journey to type checking 4M lines of Python by oscargrouch on Sunday September 08, 2019 @12:58AM (#20908703)
I dont know the experience of others with this, but in reality is not that easy to have people with enough brain power or compromise for languages like C++ and Rust.

Sure theres "enough for the world", if you sum everybody, but i bet that even a company of the size of Dropbox would have a hard time hiring everyone they need if the complexity of the language is the same as Rust or C++.

For instance, if you read the Go manifesto to why the language was created, you can read between the lines that C++ getting in the way where enginners were fighting hard to deal with language complexity.

So the same way as Java before them, they created a language for the "average programmer". And you can see how the language is very pragmatic and how they fight to no put more complexity on the language.

By the way, i know its a personal opinion, but i think Rust will not be a great fit for the cloud backend exactlya because of this kind of thing. It will be hard to create codebase as complex as the ones in Java, because there will be much less man-hours available.

Maybe if generations are getting smarter, IDK, but needing to rely on a big head-count of C++ or Rust enginners for any company, i bet they will have a hard time to fill all the positions they need.

Theres much more C++ enginners than Rust right now, and companies have a hard time hiring them (and i bet that for Rust it will be no different giving its complexity).

Re:Our journey to type checking 4M lines of Python by speedplane on Sunday September 08, 2019 @06:50AM (#20909718)
> if you read the Go manifesto to why the language was created, you can read between the lines that C++ getting in the way where enginners were fighting hard to deal with language complexity.

My understanding was that the push from C++ to Go was largely motivated by long C++ compile/linking times. Go is designed to compile large code-bases much faster than C++. I've experienced this myself, it's pretty frustrating to wait 2-5 minutes for a C++ app to compile / link just so you can debug it for 30 seconds and start all over again.

Re:Our journey to type checking 4M lines of Python by hermitdev on Monday September 09, 2019 @12:48AM (#20915269)
> I've experienced this myself, it's pretty frustrating to wait 2-5 minutes for a C++ app to compile / link just so you can debug it for 30 seconds and start all over again.

That's quaint. When I first started working in C++ professionally, we were at +16 hours for a clean build and link of just one of our systems (circa 2004). Over the years, reduced the amount of code (without sacrificing functionality), shared common code (object model) between client and server (went from about 2M LOC to about 250k LOC), reduced the number of external dependencies, introduced precompiled headers on both windows and Linux, and a clean build was down to about 30 minutes. An incremental build and link could be less than 30 seconds, depending on the scope of the change. But, yeah, C++ leaves a lot to be desired in terms of edit, build, test, repeat cycles.

Re:Our journey to type checking 4M lines of Python by oscargrouch on Sunday September 08, 2019 @02:04PM (#20912185)
But this is a by-product of their initial motivation.

I mean, how could they know their link times would be that much better?

I know that Pike et al. worked on compilers for Inferno, and the Ken´s C compiler used some tricks to know beforehand they could make it compile and link very fast, but they couldnt know if the language they were designing would be as fast (to compile) as C or Limbo.

They could say confidently: "we could do better", but they would need more pragmatic motives, even to convince Google to support that effort.

And i bet Google was struggling with C++ enginnering, specially within the newcomers on the team.

I think, both things are true, but developer ergonomics play a very important factor into the why of a language and where it fits in the scheme of things.

Re:Our journey to type checking 4M lines of Python by hermitdev on Monday September 09, 2019 @12:13AM (#20915161)
But why rewrite a 4M LOC system that works in a new language, just because of a new shiny? How do you justify that from a business standpoint? Especially when your current engineering staff are already experts in the currently used technology, but likely novices or mediocre in Rust? Do you freeze new features during the rewrite? Do you develop new features in parallel? Rewriting, just because, Rust, is a waste.

Sure, you can hire new talent that knows Rust, but how long for them to pick up the relevant domain knowledge? How long for the existing talent that has the domain knowledge to become equivalently proficient in their current tech stack?

Plus, with a rewrite, you risk falling ill to second system syndrome.

Economically and from a practical engineering standpoint, it seems to me that Dropbox is taking a very pragmatic approach. Gradually make working with the exist code easier rather than throwing the baby out with the bath water.

I'm not hating on Rust, but this fervor to rewrite everything in Rust is obnoxious and often short sighted, in my opinion. If you're talking brand new, green field development, by all means, it should be considered.

Re:Our journey to type checking 4M lines of Python by speedplane on Sunday September 08, 2019 @06:44AM (#20909698)
> I'd argue that how complicated a language is shouldn't be a factor for a company at that scale.

I'd agree with you re Dropbox, a relatively "enlightened" modern software company. But they are the exception, not the rule. Most companies, big and small hire the cheapest developers they can find. I'm not talking about silicon valley tech companies, but think e-commerce and corporate sites (Nordstrom, General Motors) and old school companies (e.g., Goldman Sachs, power utilities).

These companies don't care about the "right tool for the job", they care about the right people for the job. If the "people" are adept at antiquated technologies, then they'll still pass muster.

Re:Our journey to type checking 4M lines of Python by vonseel on Sunday September 08, 2019 @04:04AM (#20909203)

  shouldn't be a factor for a company *at that scale*
Exactly.
Re:Our journey to type checking 4M lines of Python by 0815test on Sunday September 08, 2019 @05:28AM (#20909462)
How so? Rust is much, much less complicated than C++, a language that's routinely taught to beginners. Python is far more complex than either of those, especially once you account for the incidental complexity in whatever "packages" you're interfacing with - Rust simplifies a lot of that stuff without giving up on performance.
Re:Our journey to type checking 4M lines of Python by m0zg on Sunday September 08, 2019 @02:47AM (#20908999)
I'm fully in support of the beginners near coming anywhere near my data storage and synchronization.
Re:Our journey to type checking 4M lines of Python by bsder on Saturday September 07, 2019 @09:27PM (#20907954)
Uh, no.

Rust crates are still full of the type of bugs that Python libraries killed years ago. Garbage collection overall really does make things easier. And Java still does concurrency better than practically everything else out there.

If you are doing systems programming, go for Rust instead of C++. Maybe even Rust instead of C. But, I would reach for Python, Ruby, Kotlin, Clojure or even modern Java before Rust for most non-Javascript projects.

Re:Our journey to type checking 4M lines of Python by pjmlp on Monday September 09, 2019 @05:04AM (#20916407)
Agreed, any strong typed AOT/JIT compiled language should do the job.

As for systems programming, I would argue that Rust still isn't an alternative to C++ in several deployment scenarios that many of us care about, although it should eventually get there.

Re:Our journey to type checking 4M lines of Python by j88439h84 on Saturday September 07, 2019 @09:30PM (#20907973)
I'm not familiar with Java's concurrency. In Python I'm excited about the idea of "structured concurrency", embraced by its wonderful Trio library. https://trio.readthedocs.io
Re:Our journey to type checking 4M lines of Python by bsder on Saturday September 07, 2019 @09:39PM (#20908004)
java.util.concurrent is sheer utter brilliance condensed to code.

If you're doing concurrency, you should look at how java.util.concurrent does it. And, then, if you aren't doing it that way, you should start asking really hard questions.

Re:Our journey to type checking 4M lines of Python by rixed on Saturday September 07, 2019 @11:53PM (#20908513)
Such a strong wording lead me to check this library doc but I couldn't find anything in there unexpected or that looked particularly original, but only classical message queues, semaphores, futures and locks. Could it be that I looked at the wrong thing?

Could you be more specific about what triggered such an enthusiastic description?

Re:Our journey to type checking 4M lines of Python by bsder on Sunday September 08, 2019 @12:30AM (#20908615)
Reference here: https://docs.oracle.com/javase/8/docs/api/java/util/concurre...

Take a good look at things like ConcurrentLinkedDeque or ConcurrentSkipListSet and then look at what guarantees they do and do not make and think about the rationale behind those decisions.

And all you have to do in java is create the collection, hand the reference to multiple threads, and IT JUST WORKS like a normal Java collection.

You do NOT want to be using semaphores and locks directly if you have java.util.concurrent at your disposal. Look at the "Concurrent Collections" section:

    Most concurrent Collection implementations (including most Queues) also differ from the usual java.util 
    conventions in that their Iterators and Spliterators provide weakly consistent rather than fast-fail traversal:

        they may proceed concurrently with other operations
        they will never throw ConcurrentModificationException
        they are guaranteed to traverse elements as they existed upon construction exactly once, and may (but are not guaranteed to) reflect any modifications subsequent to construction.
These are remarkably difficult properties to get correct. Then you have to think about fairness vs progress (the java.util.concurrent folks have thought about this a LOT).

That library is something I sorely miss when I'm operating in another language.

You should also dive into the code itself. Sometimes you'll look at a function and marvel at how much work it really takes to get something right. And sometimes you'll look at a function and wonder how it could possibly be so simple.

Re:Our journey to type checking 4M lines of Python by rixed on Monday September 09, 2019 @01:31AM (#20915427)
I will have to look at the implementation some day, but I think I am starting to see what you mean with the API.

Is the typical implementation of those days structures relying on many small scale mutexes to avoid contention, or RCU techniques ?

Re:Our journey to type checking 4M lines of Python by bsder on Monday September 09, 2019 @05:54AM (#20916685)
Mostly compare-and-swap mechanisms at critical points so that everything is always consistent (for very specific definitions of consistent).

I think there are alternative paths when contention starts becoming too high.

It's been a while, but the code is the real reference. You can go back and look at the mailing list archives as well.

http://cs.oswego.edu/mailman/listinfo/concurrency-interest

Re:Our journey to type checking 4M lines of Python by wcdolphin on Saturday September 07, 2019 @09:40PM (#20908011)
You’ll like Kotlin’s concurrency model then! It not only allows concurrency, but also parallelism, with rich language support. It is really pretty great, check it out!
Re:Our journey to type checking 4M lines of Python by viraptor on Saturday September 07, 2019 @09:44PM (#20908033)
> Rust crates are still full of the type of bugs that Python libraries killed years ago.

That's an interesting claim. What do you mean specifically?

Re:Our journey to type checking 4M lines of Python by bsder on Saturday September 07, 2019 @11:32PM (#20908443)
Whenever I pull a crate, I invariably have to file a bug because I don't have the exact OS, OS version, Rust version, and crate version that the author develops on.

I rarely file bugs on Python packages no matter how obscure my setup is.

Re:Our journey to type checking 4M lines of Python by viraptor on Sunday September 08, 2019 @01:42AM (#20908810)
Is that just a forward compatibility only? As in you're trying to compile on stable what is only available on nightly? Or are there actual backwards compatibility issues?
Re:Our journey to type checking 4M lines of Python by bsder on Sunday September 08, 2019 @03:04AM (#20909040)
Mostly it's OS combos: Package X has issues on the latest Windows 10. Package Y runs on Ubuntu but not RedHat and fails on the latest release of Debian but not the prior one. Package Z crashes on OS X any older than yesterday.

Now that Rust stable is actually reasonably solid, I just ignore crates that demand nightly.

Re:Our journey to type checking 4M lines of Python by j88439h84 on Saturday September 07, 2019 @08:36PM (#20907721)
If you're starting now, you could use typed Python on PyPy from the beginning.
Re:Our journey to type checking 4M lines of Python by speedplane on Sunday September 08, 2019 @06:46AM (#20909706)
> If you're starting now, you could use typed Python on PyPy from the beginning.

And if you started 5 years ago, you're now likely spending half of your time over the next few months/years porting your code from Python 2 to 3. What a disaster.

Re:Our journey to type checking 4M lines of Python by StavrosK on Thursday September 12, 2019 @06:00PM (#20956917)
In what world does it take years to port a Python 2 project to 3? This FUD gets tiring after a while.
Re:Our journey to type checking 4M lines of Python by jdm2212 on Saturday September 07, 2019 @08:03PM (#20907565)
Having never used Rust: how's the ecosystem around it like for maintaining it on production servers? Are there equivalents of Java Mission Control and `jcmd`?
Re:Our journey to type checking 4M lines of Python by stmw on Saturday September 07, 2019 @08:16PM (#20907628)
The ecosystem is good and growing, but not as extensive as java, of course. We use generic cloud monitoring/SRE tools (think statsd, not jcmd, for example). From prior experience, I also think JVMs somehow tend to require more of this kind of tooling, compared to Rust - or, for that matter, other languages & runtimes.
Re:Our journey to type checking 4M lines of Python by jdm2212 on Saturday September 07, 2019 @09:53PM (#20908080)
I'm curious how much Java needs the tooling in an all else equal situation (totally possible 'cuz Java is _old_) vs how much Java has all this cool tooling that enables/encourages people to do things that aren't possible in Go but which require the tooling.

I've mostly used the Java, C++ and Go in my career, and it's pretty amazing to me that I can write write safe, readable code (like Go) on top of high-performance libraries (like C++) and step through code on a prod machine with a remote debugger, get live performance metrics with a single CLI command, etc.

Re:Our journey to type checking 4M lines of Python by viraptor on Saturday September 07, 2019 @09:39PM (#20908005)
Not in the same way. Since there's no significant runtime behind rust, there's no specific way to query it in an abstract way. On the other hand, native code means that you can use any common way to query the system. Anything you'd use for C or C++ will work as well. Profilers, stack querying, monitoring the process itself, user-level probes, etc. will work just fine. There's lots of tools that will work for this.