beepbeepbeep1
This was a bit back but not to far back.
Took a significant pay rise. elsewhere and got to work with some other ex atlassian colleagues.
The JVM is just getting warmed up load testing for that short a period.
Try benchmarking Kafka from 0 partitions to a few thousand partitions in 100 partition increments. The benchmark only needs to write to a single topic, using their provided producer perf tool while all other topics are inactive with zero data.
As the partitions increase there is a very noticeable drop in throughout that looks to be linear.
Kafka does not handle a large number of partitions well currently, large even being low thousands. It's easy to hit with just a few hundred topics.
Reading between the lines ehen Linkdin and Netflix advertise several clusters, i am predicting/guessing they shard the data.
It doesn't need to be the only product or have the most users it only needs to be functional and make enough money for it to be worthwhile except with vc funding it has to constantly chase growth and returns.
In the summary section it says the conspicuity of motorcycles is improved with daytime lights although may be less if cars have daytime lights.
It’s been a long time since I read that report but it has lots of real data and research.
Cars are easier to spot regardless of lights due to them being much larger than a motorcycle or cyclist and the reports being biased towards protecting the most vulnerable road users.
Most of it was on the lines of if every vehicle had lights on you would no longer stand out, you are just another light which is also harder to judge distance in the mirror. Where as if yo have lights on but cars don't you are more likely to stand out for thr fact there's less motorcycles so not as many lights to blend in to the background.
In the AU everyone genrally has full beams on regardless, horrible experince driving at night.
Sure they can't afford $1M but in general you are lucky to get $110k+ in the UK even if you have 10 years behind you, even more so outside of London, in London the money doesn't go that far.
It's a no brainer, if you can get a visa and happy to move even if for a few years it can make a big financial life difference.
The only reason i can think of that is they get to push point of sale software out by using K8s from some central system. I cant think of a worse use/abuse of k8 as a software updafe system if that's what they are doing.
The other reason is they distributed their compute and resturants pay the power bill but that sounds just as silly.
Curious to know why you would use k8s at the edge
If you are comfortable at operations you'll be fine. Some people are not good at ops so outsourcing the problem making the ops side someone else's issue can also be useful.
Self hosting will offer far more options when it comes to scaling and tweaking. Overall on bare hardware costs it's cheaper and faster although up front costs will be higher.
Kafka usecases are rarely elastic so don't gain that advantage in the cloud. Also Kafka's missing tierd storage makes it expensive if storing big volumes of data.
I was annoyed when it changed to being subscription based, the price wasn't an issue I think for the value it gives the price is fair, 1Password is the only thing now keeping me on a Mac.
The issue i had with it is you don't own the software and when my credit card expired and i forgot to update it losing access to passwords even if temporarily until updating the card details left a bad taste and showed the downside of being subscription only.
It's been on my list to move away ever since. 12 months on I've still not done it, there's lot's of alternatives but 1Password have the UX and functionality just right however i know i should move as i now don't own my passwords.
Docker as in downloading random images of Dockerhub and plumbing them together. Not so great, quality varies and most people don't actually check what they are really running. Often they are VMs packaged up as a container, badly.
Also running two clusters to handle large volumes of data, it's big money. Even a small modest cluster with around 20TB+ of data was north of 30K a month on AWS. That's a full app cluster though with consumers/producers aswell as brokers.
Growing LVM with XFSs has worked well, 0 downtime and around 60 seconds.
Allows you to over provision just enough you do not have to babysit the drives or pay $$$ for unused disc.
If you stripe the volumes you'll also distribute your IOPs in AWS.
Outside AWS LVM still applies. Kafka's JBOD is useless without easy / auto rebalancing.
This week onsite at a client's I discovered ScaleIO which can present up to a 1PB volume and does clever sharding/replication in the background.
I wasn't involved on the project but a major Telecoms company i was at was prototyping scanning for mobile devices on their provided home routers. If they didn't recognise a mibile as being on their network they would then target you with mobile deals. Devs saw the issues with this, essentially scanning your home network from their routers and sending the data back to be processed. Managers found Apple's mac address rotation annoying and lost the point why Apple do it.
Unfortunately besides quitting the job devs on the project don't have a say as they don't have the power or say in the final decisions at these companies. It's easy to say you would quit but it's harder when you have a mortgage, wife and kids for example.
Data on a topic can be partitioned by a key so you can partition by user id resulting in all user events going to the same partition however there’s no functionality to delete a partition by key.
If each user has their own topic that you can delete you would end up with a lot of topics and Kafka isn’t great with that.
You also have the issue if event sourcing or doing DDD your events are based around domains/aggregates with clear context bounds. The domain is very likely not a user but may involve a user. Its a tricky modeling issue.
Event sourcing is about immutable events. You cannot change an event in the past as its already happened, accounts don’t use rubbers. The ability to go back and change past events or delete them breaks the laws and guarantees of an immutable event system.
It’s a difficult problem. I like event sourcing due to the properties it gives but it may not be suitable for some things due to GDPR.
One thing I have heard talked about is encrypting personal identifiable information in an event with a user specific key. When the user requests deletion you delete the key. The events don’t change they are still there however anything which identifies a user is unreadable.
I wouldn't use it for non backup object storage, as it's in a single data center. S3 and Google Cloud are completely different, I'm not sure on the Google specifics but S3 has data replicated across three AZs.
I mainly use Evernote. I'm trying my best to use org-mode more and sync with Google Drive as Evernote is subscription based and it comes down to do I want to keep all my information with a third party who i don't know what their backup policy is etc. Unfortunately I tend to fall back to Evernote.
The reason I mostly use Evernote and I think key for any alternative is search, the ability and ease of finding content. Evernote is primarily searched based and using tags/free text it's fairly powerful. Most stuff goes in to a singke notebook.
VPS's are probably a no go as they have all moved to SSD's with low sized volumes. Even with Digital Ocean where you can add block storage you are looking at $100 for a week for 4TB + compute and bandwidth. AWS you are looking at $368+ just on data egress.
hetzner.com server auctions look the best option. Current cheapest is €31.93 per month for a Intel Core i7-3770 (CPU-B 9445) with 2x 3 TB drives and 16 GB. Probably simple enough to use lvm and jbod on the drives to give a single 6tb volume. No egress charges but reading they throttle if you go over 20,000GB a month so a non issue.
No setup fees/contract on the auctions although reading I think it's 30 days/1 month notice to cancel so would need to cancel as soon as ordered.
My plan is to probably move to Git Annex https://git-annex.branchable.com/, use my local NAS as the main backup, maintain a copy of media / photos / docs in Backblaze and duplicate files I can't afford to lose to S3 also. Looks simple enough with Git-annex and lets me maintain multiple copies based on cost / importance.
Should be able to get max throughput and cost will be minimal if only have the VPN for the time it takes to seed.