broner
Re:DuckDB is probably the most important geospatial software of the la...
on Monday May 05, 2025 @08:22PM (#43900723)
Nope, just reading from S3. Check this out: https://duckdb.org/2024/07/09/memory-management.html
Re:DuckDB is probably the most important geospatial software of the la...
on Sunday May 04, 2025 @05:47PM (#43889861)
Works fine for me on TB+ datasets. Maybe you were doing in-memory rather than persistent database and running out of RAM? https://duckdb.org/docs/stable/clients/cli/overview.html#in-...
Re:DuckDB is probably the most important geospatial software of the la...
on Sunday May 04, 2025 @05:26PM (#43889729)
You can use DuckDB in ipython to solve the globbing issue. Then you don't have to worry about OOMs with geopandas.
Re:Show HN: BemiDB – Postgres read replica optimized for analytics
on Saturday November 09, 2024 @03:58PM (#42096860)
My workload is 100% read. Querying zstd parquet on s3 standard. Neither clickhouse nor duckdb has a great s3 driver, which is why smart people like https://www.boilingdata.com/ wrote their own. I compared a handful of queries and found DuckDB makes a lot of round trips and Clickhouse takes the opposite approach and just reads the entire parquet file.
Re:Show HN: BemiDB – Postgres read replica optimized for analytics
on Saturday November 09, 2024 @10:44AM (#42094999)
I'm doing this with clickhouse querying parquet files on S3 from an EC2 instance in the same region as the S3 bucket (yes DuckDB pretty similar). S3 time to first byte within AWS is 50ms and I get close to saturating an big EC2 instance's 100Gb link doing reads. For OLTP type queries fetching under 1 MB you'll see ~4 round trips + transfer time of compressed data so 150-200 ms latency.
Re:We migrated from AWS to GCP with minimal downtime
on Friday June 07, 2024 @02:13PM (#40611309)
Switch your blind faith to Go (s5cmd) and get ~1200 MB/s upload speed!
zfs send tank/pgdata@snapshot | pbzip2 > mybackup.zfs.bz2
s5cmd cp mybackup.zfs.bz2 s3://mygooglebucket/
https://github.com/peak/s5cmd/blob/master/README.md#Benchmar...
Re:Show HN: Serverless Postgres
on Tuesday June 04, 2024 @07:06PM (#40579846)
For $1,500 per TB per month? No thanks.
Re:I Sold TinyPilot, My First Successful Business
on Thursday May 30, 2024 @11:53AM (#40525149)
Should be a tax advantage to the non-grind income. $920k with 20% fed + 5% state capital gains tax?
Re:Show HN: Serverless Postgres
on Thursday May 30, 2024 @11:19AM (#40524778)
Could you use this to create 100s of on-demand read replicas and get 100s of GB/s out of S3? Seems like a nice way to cheaply and quickly do batch jobs.
Re:Understanding Parquet, Iceberg and Data Lakehouses
on Saturday December 30, 2023 @03:51PM (#38818991)
This looks pretty neat if you're ok moving to AWS https://www.boilingdata.com/
Re:Gooey: Turn almost any Python command line program into a full GUI ...
on Sunday December 10, 2023 @08:50AM (#38591511)
Reminds me of this previous HN submission Blotato: https://news.ycombinator.com/item?id=38206235
Re:Fastly acquires Glitch
on Thursday May 19, 2022 @09:50AM (#31434630)
Glitch? Oh they're with Fastly now! https://www.tiktok.com/@benedicttown/video/70983764223009620...
Re:How will climate change affect your home?
on Friday October 09, 2020 @05:52PM (#24734796)
As the climate rapidly changes and gets more dangerous, it’s good to know what you’re in store for. This site gives you a picture. https://climatecheck.com
Re:Ask HN: Programs that saved you 100 hours?
on Tuesday April 14, 2020 @02:25PM (#22869573)
I switched from Overcast to Pocketcasts. Everything I liked about Overcast plus playlists I can figure out how to use.
Re:Ask HN: are you 1) skinny and 2) self-employed?
on Tuesday January 28, 2014 @12:54PM (#7138527)
I'd be happy to help :) What age guys are you looking for?