nicornk
nicornk on Hacker News
I hope AWS doesn’t dilute the DuckDB brand by calling their managed DuckDB offering SageMaker<something>…
This would be my bet and hope as well.
In that sense all popular ERP vendors seems to be the same.
I agree. Pixi solves all of that issues and is fully open source including the packages from conda-forge.
Too bad there is nowadays the confusion with anaconda (the distribution that requires a license) and the FOSS pieces of conda-forge. Explain that to your legacy IT or Procurement -.-
If you just want to enable ssh to ec2 instances (through SSM) using ssh i-… you can add the following lines to your ssh config
https://gist.github.com/nicornk/5d2c0cd02179f9b46cc7df459af0...
host i-*
IdentityFile ~/.ssh/id_rsa
TCPKeepAlive yes
ServerAliveInterval 120
User ec2-user
ProxyCommand sh -c "aws ec2 start-instances --instance-ids %h ; aws ec2 wait instance-running --instance-ids %h ; aws ec2-instance-connect send-ssh-public-key --instance-id %h --instance-os-user %r --ssh-public-key 'file://~/.ssh/id_rsa.pub' --availability-zone $(aws ec2 describe-instances --instance-ids %h --query 'Reservations[0].Instances[0].Placement.AvailabilityZone') ; aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"
This will also allow VSCode remote development.
The blog post echos my experience that duckDB just works (due to superior disk spilling capabilities) and polars OOMs a lot.
Snowflake usually unloads data to an internal stage bucket in the same region as your snowflake account. If you use an s3 gateway endpoint getting that data is free of egress charges.
and this is unfortunately the case for Velox.
Yeah, finally a method to figure out the org id of all those snowflake internal stage buckets that snowflake does not want to share for our VPC Endpoint Policies…
So how did you „further optimized [buildkit] to build Docker images up to 40x faster“
Pixi is great, highly recommended!
There are Aldis in Germany with self checkout.
Thanks for sharing your experience. I always think of starting in that space but your post shows me it’s probably better to wait for a few more months.
Isn’t it a little bit unrealistic that your proxy fleet can keep up with the massive traffic that some s3 buckets experience? I think every solution that Proxies traffic to s3 will at a certain point break.
Uploading data into AWS is free.
Really interesting going through your code, thanks for sharing.
Why did you opt for running the parallelization code in a separate python process (s3op)?
You might want to update section ‚Caution: Overwriting data in S3‘ in the docs since S3 offers strong read after write consistency since dec 2020.
https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-rea...
That‘s basically MS Teams.
Posted by
nicornk on Friday August 16, 2019 @08:32AM
from the bloomberg.com dept.