It's like Jonathon Blow asks: why does Photoshop take longer to load today than it did in the 90s despite the (insane) advances in hardware?
I believe it's due to a bunch of things, but over complicating the entire process is one of the big issues. If people (developers/engineers) would only sit back and realise just how much computing power they have available to them, and then realised that if they kept things simple and efficient, they could build blazing fast solutions over night.
I cringe thinking about the wasted opportunities out there.
This is a problem with the software industry today. we have forgotten how to do the simple stuff, that works and is robust.
... computational power has grown ~%259,900 since the 90s.
The point being made is this: Photoshop does one job and has one focus should), yet it has gotten slower at doing that one job and not faster. Optimising the code AND introducing incredibly hardware to the consumer market should see Photoshop loading in milliseconds, in my opinion.
Has it though? Without measurements this is just idle talk.
And I've used Photoshop in the 90s and I use it today ocassionally. I remember having 90s sized web pictures (say, 1024x768) and waiting for a filter to be applied for tens of seconds - which I get instantly today with 24MP and more...
And if we're into idle talk I've always found Photoshop faster in large images and projects than competitors, including "lightweight" ones.
It's hella more optimized than them.
In any case, some e.g. image filter application (that takes, e.g. 20 seconds vs 1 minute with them) just calls some optimized C++ code (perhaps with some asm thrown in) that does just that.
The rest of the "bloat" (in the UI, feature count, etc) has absolutely no bearing as to whether a filter or an operation (like blend, crop, etc) runs fast or not. At worst, it makes going around in the UI to select the operations you want slower.
And in many cases the code implementing a basic operation, filter, etc, hasn't even been changed since 2000 or so (if anything, it was optimized further, taken to use the GPU, etc).
https://en.wikipedia.org/wiki/Kentsfield_(microprocessor) Modern processors are nowhere near 10x faster single core. Maybe 4x.
If you take into account performance/watt you do get 10x or better.
If the trend had continued we would have 1500Ghz cores by now.
Here's the single thread raw data from that repo. If you take into account clock speed increase (which, as you agree, have plateaued) we're looking at maybe a 2x increase in instructions per clock for conventional int (not vectorized) workloads.
Is there even another 2x IPC increase possible? At any time scale?
https://github.com/karlrupp/microprocessor-trend-data/blob/m...
I did not know how to use Winamp playlists because Winamp has been "an instant" app for me, I just clicked on a song and it played within miliseconds. That was my flow of using Winamp for years. This did not change between the Celeron and Duron, the thing was instant on both Celeron and Duron.
Then Winamp 3 came out and I had to use playlists because a song once clicked took good second or two to start playing. Winamp 5 from 2018 still starts slower than my beloved 2.73* did 20 years ago. On Celeron 333 and 5400 RPM HDD with 256 MB of RAM. I think even the good old Winamp 2.x is not as fast as it was on Windows 98/XP.
Something went wrong.
* not sure if it was 2.73, but I think so
Note: I realise Winamp 3 was crappy as hell, but still...
I learned that every time you gain something, you also loose something without realizing it, because you take it for granted.
I remember having a 486 my self and everything being so damn snappy. Sigh.
This was AMAZING.
Photoshop at the time would take many many seconds to zoom in/out.
One person in the group asked, "Yeah, but how much memory is in that machine?"
The guy hemmed and hawed a bit and finally said "It's got a bit, but not a lot, just 24mb."
"Yeah, well that explains it." Nobody had 24mb of RAM at that time. Our "big" machine had 16mb.
A feature that Photoshop has since added, but it appeared in Live Picture first.
Photoshop can grow them itself and the below looks no and thread are amazing. https://news.ycombinator.com/item?id=26448986
The Apple question "why isn't this easy" is missing from 95% of UX in modern software. Stop acting as if software devs are the only users of software. And even then: computers should do work and not produce work or mental overhead.
I switched to a MacBook at work a few months ago, and it's been an epic of frustration and Googling: e.g.,
1. I set the keyboard layout to UK PC, but it kept switching back to default. (I was accidentally mashing a keyboard shortcut designed to do just that. I've since disabled it.)
2. Clicking on a link or other web-page element will, occasionally and apparently randomly, take me back a page or two in my history rather than opening the link or activating the element. (At least in Chrome: I've not used another browser on macOS yet.)
3. Command-tabbing to a minimised application does not, as one would naively expect, automatically unminimise it. Instead I'm left staring at an apparently unchanged screen.
4. If I open Finder in a given folder, there is, as best can tell, no easy way to navigate to that folder's parent.
Now arguably #1 was the fault of my own ignorance (though some kind of obvious feedback as to what was happening and why would have been nice), and #2 may be down to Google rather than Apple.
But #3 and #4 are plain bad design, bordering on user-hostile.
So far I'm not seeing that Apple's reputation for superior UX is justified, at least not in laptops.
Finder is just god awful and does all it can to obscure your actual filesystem location but the go to folder (cmd+g?) can get you where you need to go.
At this point I think Apple still deserves their reputation for superior UX, however that's a result of how epically bad Google, MS, and Facebook are at UX, not Apple doing a great job like they used to.
You can add a button to the toolbar in Finder (customize it from options) that when dropped down will show the complete path to the current folder as a list. You can use that to move up the tree.
I think that's marketing, and not borne out of real experience.
Try setting up a new Apple device, there is very little in IT that is as frustrating and confusing.
It's a 5 minute process, and there's very little to it that it's not optimized for ease. Any champ can do it, and millions do.
Not sure what you're on about.
>I think that's marketing, and not borne out of real experience.
Yes, all those people are delluded.
And good luck if you're setting up a family account for several devices with different iOS versions. You're gonna really need it.
I've tried it tons of times, have over 20 iOS/macOS devices over the years, and for some pervese reason, on macOS/OS X I like to install every major update on a clean disk too (and then re-import my data. It's an old Windows 95/XP-era reflex), so I do it at least once every year for my main driver (plus different new iOS devices).
What part of this sounds difficult?
https://www.youtube.com/watch?v=70u0x8Kf6j4
And the whole "optimized for selling you bullshit Apple services" is a couple of screens you can skip with one click -- and you might want to legitimately use too.
So the kind of minds we want?
"Why isn't this easy?" should be the bread and butter question of a programmer...
I dunno, personally that's why I've used Apple products for the past decade, and I think it's also maybe part of why they have a 2T market cap, and are the most liquid publicly traded stock in the world?
Most people don't really know how their car works, and shouldn't have to. Same goes here.
You gotta be careful about these arguments. They often have a slippery slope to a superiority complex (of "leet" NIX users over the unwashed "proles") hiding deep within.
Needlessly complex or powerful user interfaces aren't necessarily good. They were quite commonplace before Apple. Apple understood the value of minimalism, of cutting away interaction noise until there's nothing left to subtract. Aesthetically speaking, this is approach has a long, storied history with respect to mechanical design. It's successful because it works.
What Apple understood really acutely and mastered is human interface design. They perfected making human centric interfaces that look and feel like fluid prosthetic extensions of one's body, rather than computational interface where power is achieved by anchoring ones tasks around the machine for maximal efficiency. Briefly, they understood intuition. Are you arguing that intuition is somehow worse than mastering arcana, simple because you've done the latter?
Now, I'm not going to say that one is better than the other. I love my command line vim workflow dearly, and you'll have to pry my keyboard out of my cold dead hands. But there's definitely the idea of "right tool for the right job" that you might be sweeping by here. Remember, simplicity is as much a function of cherished *NIX tools you probably know and love. It's where they derive their power. Be careful of surface level dismissals (visual interfaces versus textual) that come from tasting it in a different flavor. You might miss the forest for the trees!
As an example, read how the one-man-band of Execution Unit manages to, in house, write and test his game on three operating systems. https://www.executionunit.com/blog/2019/01/02/how-i-support-...
It's not a matter of "this is too hard", Blow just does not want to do it, let's be honest.
preferably over a twitch stream..
One is that there’s a minimum performance that people will tolerate. Beyond that you get quickly diminishing user satisfaction returns when trying to optimize. The difference between 30 seconds and 10 seconds in app startup time isn’t going to make anyone choose or not choose Photoshop. People who use PS a lot probably keep it open all day and everyone else doesn’t care enough about the 20 seconds.
The second problem is that complexity scales super-linearly with respect to feature grown because each feature interacts with every other feature. This means that the difficulty of optimizing startup times gets harder as the application grows in complexity. No single engineer or team of engineers could fix the problem at this point, it would have to be a mandate from up high, which would be a silly mandate since the returns would likely be very small.
(I like anything that echos well in this chamber of mine... just kidding ;)
To solve the problem you're describing we need to be better at protecting all members of society not in (high paying) jobs, such as universal basic income and, I don't know, actually caring about one another.
But I do see your point, and it's an interesting one to raise.
Most orgs (and most devs) feel developer productivity should come first. They're not willing to (and in a lot of cases, not able to) optimize the apps they write. When things get hard (usually about 2 years in) devs just move on to the next job.
It is a matter of incentives. At many companies, developers are rewarded for shipping, and not for quality, efficiency, supportability, documentation, etc.* This is generally expected of a technology framework still in the profitability growth stage; once we reach a more income-oriented stage, those other factors will enter incentives to protect the income.
I think you can build something complex quickly and well at the same time. I built opskit.io in three weeks. It's about 90% automated.
One definitely can. It's a real crapshoot whether the average developer can. For what it's worth, I consider myself a below-average developer. There is no way I could grind l33tcode for months and even land a FAANG interview. I can't code a red-black tree to save my life unless I had a textbook in front of me. Code I build takes an enormous amount of time to deliver, and So. Much. Searching. You get the picture. I'm a reasonably good sysadmin/consultant/sales-engineer; all other roles I put in ludicrous amounts of effort into, to become relevant. Good happenstance I enjoy the challenge.
For the time being however, there is such enormous demand for any talent that I always find myself in situations where my below average skills are treated as a scarcity. Like near-100 headcount testing organization in a tech-oriented business with an explicit leadership mandate to automate with developer-written integration code from that organization...and two developers, both with even worse skills than mine. When a developer balks at writing a single regular expression to insert a single character into the front of an input string, that's nearly the definition of turning one wrench for ten years; while I'm slow and very-not-brilliant, I'm a smart enough bear to look up how to do it on different OS' or languages and implement it within the hour.
This is not unusual in our industry. That's why FizzBuzz exists. That's just to clear the bar of someone who knows the difference between a hash and a linked list.
To clear the bar of "something complex quickly and well at the same time" though, I've found it insufficient to clear only the technical hurdle and obtain consistent results. The developer has to care about all the stakeholders. Being able to put themselves into the shoes of the future developers maintaining the codebase, future operators who manage first line support, future managers who seek summarized information about the state and history of the platform, future users who apply business applications to the platform, future support engineers feeding support results back into developers, and so on. That expansive, empathetic orientation to balance trade-offs and nuances is either incentivized internally, or staffed at great expense externally with lots of project coordination (though really, you simply kick the can upstairs to Someone With Taste Who Cares).
I'd sure as hell like to know alternatives that are repeatable, consistently-performing, and sustainable though. Closest I can think of is long-term apprenticeship-style career progression, with a re-dedicated emphasis upon staffing out highly-compensated technical writers, because I strongly suspect as an industry we're missing well-written story communication to tame the complexity monster; but that's a rant for another thread.
The previous Pi computation record of about 2577 billion decimal digits was published by Daisuke Takahashi on August 17th 2009. The main computation lasted 29 hours and used 640 nodes of a T2K Open Supercomputer (Appro Xtreme-X3 Server). Each node contains 4 Opteron Quad Core CPUs at 2.3 GHz, giving a peak processing power of 94.2 Tflops (trillion floating point operations per second).
My computation used a single Core i7 Quad Core CPU at 2.93 GHz giving a peak processing power of 46.9 Gflops. So the supercomputer is about 2000 times faster than my computer. However, my computation lasted 116 days, which is 96 times slower than the supercomputer for about the same number of digits. So my computation is roughly 20 times more efficient.
We benchmarked it, i was much, much faster.
Technically though once that multi-gig file becomes many hundreds of gigs, my computer would loose by a huge margin.
https://web.archive.org/web/20200414235857/https://adamdrake...
I have tried this, I literally wanted to create a simple web app that is powered by the cheapest solution possible, but it had to serve from a database that cannot be smaller than 150GB. SQLite failed. Even Postgres by itself was very hard! In the end I now launch redshift for a couple days, process all the data, then pipe it to Postgres running on a lightsail vps via dblink. Haven’t found a better solution.
For a 32 core processor, that means that it can process a data set of 100G in the order of 30 seconds. For some types of tasks, it can be slower, and if the processing is either light or something that lets you leverage specialized hardware (such as a GPU), it can be much faster. But if you start to take hours to process a dataset of this size (and you are not doing some kind of heavy math), you may want to look at your software stack before starting to scale out. Not only to save on hardware resources, but also because it may require less of your time to optimize a single node than to manage a cluster.
This is a great phrase that I'm going to use more.
Yes, you can. Without indexes to slow you down (you can create them afterwards), it isn't even much different than any other DB, if not faster.
>Even Postgres by itself was very hard!
Probably depends on your setup. I've worked with multi-TB sized Postgres single databases (heck, we had 100GB in a single table without partitions). Then again the machine had TB sized RAM.
The PostgreSQL database for a CMS project I work on weighs about 250GB (all assets are binary in the database), and we have no problem at all serving a boatload of requests (with the replicated database and the serving CMS running on each live server, with 8GB of RAM).
To me, it smells like you've lacked some indices or ran on a rpi?
Anyway, I’m curious what kind of data the op is trying to process.
But this means there are a hundred million entities, publishing 3x number of papers and a bunch of metadata associated. On redshift I can get all of this loaded in minutes and takes like 100G but Postgres loads are pathetic comparatively.
And I have no intention of spending more than 30 bucks a month! So hard problem for sure! Suggestions welcome!
By default you get a commit after each INSERT which slows things down by a lot.
*Obviously not really. But very very many things do, even doing useful jobs in production, as long as you have high enough specs.
Me: "No, absolutely not, at some point we'll need to hire someone who knows what they're doing."
As time passed and we got more data, I significantly improved the data cleaning portions so that most of it was automated, and the parts that weren't automated would be brought up as suggestions I could quickly handle. I learned the very basics of performance and why `eval` is bad, set up my script so I didn't need to hard-code the number of files to process each day, started storing data on a network drive and then eventually a db...
I still don't know what I'm doing, but by the time I left it took maybe 5 minutes of manual data cleaning to handle thousands of jobs a day, and then the remainder could be done on a single machine.
Feels like toxic optimism.
Those come out at something like 1GB and 10TB which are obviously rather different.
Then, later, probably someone would check out the scripts from a shared repo. Then, read an outdated README, try it out, swear a bit, check for correctness with someone dependent on the results, and finally learn how to do the task.
There is a lot of business processes that can tolerate days or weeks of delay in case of such a tragic (and hopefully improbable) event. The trick is to know which of them can't.
This is really true BUT that kind of problems are OK - nobody cares - until somebody starts caring and then all of a sudden it is urgent (exactly because they were undetected for weeks/months due to their periodicity)
E. g. we have less than 1% chance per year that a given person leaves us on bad terms or suffers a bad accident or illness. In case it really happens, it will cost us X in delays and extra work. To lower the probability of this risk to Y% would cost us Z (money, delay, etc).
If you do this math, you can tell if it's a good idea to optimize here, or if you have more pressing issues.
In my experience, this sort of one-man jobs gets automated or at least well described and checked for fear of mistakes and/or employee fraud rather than "downtime".
Eventually I built out some more infrastructure to run the jobs automatically on a dedicated machine, but last I checked everything still runs on one instance.
He then mentioned that he'd noticed that somehow my employer had been processing thousands of jobs, much faster than his, and asked how many machines we were using.
I didn't have the heart to tell him we were running everything manually on my two-year-old macbook air.