the only thing i can say related to the topic is that s3 multipart outperforms other methods for files larger than 50mb significantly, but tends to have similar or slightly slower speeds compared to s3 regular upload via supabase or simplest supabase storage upload for files with size about and less than 50mb.
s3-multipart is indeed the fastest way to upload file to supabase with speeds up to 100mb/s(115 even) for files>500mb. But for files about 5mb or less you are not going to need to change anything in your upload logic just for performance cause you won’t notice any difference probably
everything mentioned here is for upload only
Generally, you would want to place an upload server to accept uploads from your customers, that is because you want to do some sort of file validation, access control or anything else once the file is uploaded. The nice thing is that we run Storage within the same AWS network, so the upload latency is as small as it can be.
In terms of serving files, we provide a CDN out-of-the-box for any files that you upload to Storage, minimising latencies geographically
A common pattern on AWS is to not handle the upload on your own servers. Checks are made ahead of time, conditions baked into the signed URL, and processing is handled after the fact via bucket events.
we only have plans to keep pushing open standards/tools - hopefully we have enough of a track record here that it doesn't feel like lip service
(This is not a paid promotion)
we receive a lot of community feedback and ultimately there are only a few "primitives" developers need to solve 90% of their problems
I think we're inching closer to the complete set of primitives which can be used to combine into second-order primitives (eg: queues/search/maps can all be "wrappers" around the primitives we already provide)
Thanks for an awesome product. Please also never get bought or make plans to in the future, or if you really, really, really have to then please not by google.
The primary options I can think of that are not full acquisition are: - company buys back stock - VC sells on secondary market - IPO
The much more common and more likely option for these VCs to make the multiple or home run on their return is going to be to 10x+ their money by having a first or second tier cloud provider buy you.
I think there's a 10% chance that a deal with Google is done in the future, so their portfolio has Firebase for NoSQL and Firebase for SQL.
so it would serve Google well if they matched what supabase is doing or bought them out
*Subject to change
Heroku might be a better example of a company that was acquired and the shut down their free plan.
My simple SSG Next.js static site loads much slower than my Jekyll site on GitHub pages.
And I can't figure out how to improve its speed or disable the "ISG" feature that I believe is to be blamed for the poor performance.
https://nextjs.org/docs/app/building-your-application/deploy...
Unfortunately looks like I can't use it now since I am using `next-plausible` which does require a Node.js proxy...
i know that one day these founders will exit and it will be sold to AWS, Google or Microsoft cloud.
so its a bit of a gamble but that risk is offset by its super cheap cost and ability to export out the data and swap out the pieces (more work but at that point you should be cashflow positive to pay engineers to do it for you).
At the time, the most requested feature was a push notification mechanism, because implementing that on iOS had a steep learning curve and was not cross-platform. Then probably some more advanced rules to be able to do more functional-style permissions, possibly with variables, although they had just rolled out an upgraded rules syntax. And also having a symlink metaphor for nodes might have been nice, so that subtrees could reflect changes to others like a spreadsheet, for auto-normalization without duplicate logic. And they hadn't yet implemented an incremental/diff mechanism to only download what's needed at app startup, so larger databases could be slow to load. I don't remember if writes were durable enough to survive driving through a tunnel and relaunching the app while disconnected from the internet either. I'm going from memory and am surely forgetting something.
Does anyone know if any/all of the issues have been implemented/fixed yet? I'd bet money that the more obvious ones from a first-principles approach have not, because ensh!ttification. Nobody's at the wheel to implement these things, and of course there's no budget for them anyway, because the trillions of dollars go to bowing to advertisers or training AI or whatnot.
IMHO the one true mature web database will be distributed via something like Raft, have rich access rules, be log based with (at least) SQL/HTTP/JSON interfaces to the last-known state and access to the underlying sets selection/filtering/aggregation logic/language, support nested transactions or have all equivalent use-cases provided by atomic operations with examples, be fully indexed by default with no penalty for row or column based queries (to support both table and document-oriented patterns and even software transactional memories - STMs), have column and possibly even row views (not just table views), use a copy-on-write mechanism internally like Clojure's STM for mostly O(1) speed, be evented with smart merge conflicts to avoid excessive callbacks, preferable with a synchronized clock timestamp ordered lexicographically:
https://firebase.blog/posts/2015/02/the-2120-ways-to-ensure-...
I'm not even sure that the newest UUID formats get that right:
https://uuid6.github.io/uuid6-ietf-draft/
Loosely this next-gen web database would be ACID enough for business and realtime enough for gaming, probably through an immediate event callback for dead reckoning, with an optional "final" argument to know when the data has reached consensus and was committed, with visibility based on the rules system. Basically as fast as Redis but durable.
A runner up was the now (possibly) defunct RethinkDB. Also PouchDB/PouchBase, a web interface for CouchDB.
I haven't had time to play with Supabase yet, so any insights into whether it can do these things would be much appreciated!
“”” PLV8 is a trusted Javascript language extension for PostgreSQL. It can be used for stored procedures, triggers, etc.
PLV8 works with most versions of Postgres, but works best with 13 and above, including 14, [15], and 16. “””
Is there a Postgres storage backend optimized for storing large files?
We store the metadata of the objects and buckets in Postgres so that you can easily query it with SQL. You can also implement access control with RLS to allow access to certain resources.
It is not currently possible to guarantee atomicity on 2 different file uploads since each file is uploaded on a single request, this seems a more high-level functionality that could be implemented at the application level
So this is like, S3 on top of S3? That's interesting.
Currently, it happens to be S3 to S3, but you could write an adapter, let's say GoogleCloudStorage and it will become S3 -> GoogleCloudStorage, or any other type of underline Storage.
Additionally, we provide a special way of authenticating to Supabase S3 using the SessionToken, which allows you to scope S3 operations to your users specific access control
https://supabase.com/docs/guides/storage/s3/authentication#s...
The announcement is that Supabase now supports (user) —s3 protocol—> (Supabase)
Above you say that (Supabase) —Supabase S3 Driver—> (AWS S3)
Are you further saying that that (Supabase) —Supabase S3 Driver—> (any S3 compatible storage provider) ? If so, how does the user configure that?
It seems more likely that you mean that for any application with the architecture (user) —s3 protocol—> (any S3 compatible storage provider), Supabase can now be swapped in as that storage target.
(user) -> s3 protocol -> (Supabase) -> (AWS S3)
you could fork (or contribute) a database driver for any s3 compatible backend of choice.
(user) -> s3 protocl -> (pbronez-base) -> (GCP Cloud Storage)
In a lot of cases, if your use case is simple, 30% is enough if you’re doing the most common GET and PUT operations etc. But all it takes is one unsupported call in your desired workflow to rule out that vendor as an option until such time that said API is supported. My main beef with this is that there’s no easy way to tell usually unless the vendor provides a support matrix that you have to map to the operations you need, like this: https://docs.storj.io/dcs/api/s3/s3-compatibility. If no such matrix is provided on both the client side and server side you have no easy way to tell if it will even work without wiring things in and attempting to actually execute the code.
One thing to note is that it’s quite unrealistic for vendors to strive for 100% compat - there’s some AWS specific stuff in the API that will basically never be relevant for anyone other than AWS. But the current situation of Wild West could stand for some significant improvement
We are transparent with what's the level of compatibility - https://supabase.com/docs/guides/storage/s3/compatibility
The most often used APIs are covered but if something is missing, let me know!
image transformations, caching, automatic cache-busting, multiple protocols, metadata management, postgres compatibility, multipart uploads, compatibility across storage backends, etc
Also my sympathies for having to support the so-called "S3 standard/protocol".
I hope you like the addition and we have the implementation all open-source on the Supabase Storage server
Does this mean that Supabase (via S3 protocol) supports file download streaming using an API now?
As far as I know, it was not achievable before and the only solution was to create a signed URL and stream using HTTP.
The good news is that the Standard API is also supporting stream!
also some community tools: https://github.com/supabase-community/firebase-to-supabase
we often help companies migrating from firebase to supabase - usually they want to take advantage of Postgres with similar tooling.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-...
I don't think we'll ever build the underlying storage layer. I'm a big fan of what the Tigris[1] team have built if you're looking for other good s3 alternatives
[0] https://github.com/supabase/storage/blob/master/docker-compo...
[1] Tigris: https://tigrisdata.com
1. You buy the storage from Amazon so I pay you and don't interact with Amazon at all or
2. I buy storage from Amazon and you provide managed access to it?
In the future we will likely provide "bring your own s3" (2)
yes, you can store all of these in Supabase Storage and it will probably "just work" with the tools that you already use (since most tools are s3-compatible)
Here is an example of one of our Data Engineers querying parquet with DuckDB: https://www.youtube.com/watch?v=diL00ZZ-q50
We're very open to feedback here - if you find any rough edges let us know and we can work on it (github issues are easiest)
We have yet to make a commitment to any one product. Having Postgres there is a big plus for me. I'll have to see about doing a test or two.
I was hesitant to use triggers and PG functions initially, but after I got my migrations sorted out, it's been pretty awesome.
I think if there was a tightly integrated framework for managing the state of all of these various triggers, views, functions and sproc through source and integrating them into the normal SDLC it would be a more appealing sell for complex projects
You can then take it a step further but opting-in to use Branching [2] to better manage environments. We just opened up the Branching feature to everyone [3].
[1]: https://supabase.com/docs/guides/cli/local-development#datab... [2]: https://supabase.com/docs/guides/platform/branching [3]: https://supabase.com/blog/branching-publicly-available
Let’s say that we are using the Typescript sdk to make our app and need to do some fancy aggregation on a table that isn’t supported by Postgrest natively (specifically we can’t retrieve the data with Postgrest out of the box with its typical setup). Postgrest tells us that in this case we can do two things: create a view or make a Postgres function. Each has their pros and cons, but with either choice now we have this problem: some of our business logic is in sproc/function/view and some of it is in Typescript. In a typical setup using an ORM it would all be in Typescript.
The conventional wisdom is that db’s are dumb state holders and all of the business logic goes in the app - Supabase attempts to turn this on its head and say no actually it’s ok to store business logic in the db. But now if we do that we have a new problem: we don’t have a blessed path forward for where the line is on what goes where anymore. We don’t have good patterns for storing and managing this so other developers understand where to put things and how to work with our app anymore, because it no longer holds the principle of least astonishment. That’s what I mean by framework in this context.
Maybe all that is necessary here is a battle tested example project that demonstrates the “correct” way to make this demarcation. But as-is it steers me away from using Supabase for more complex projects if I even think they will need something that Postgrest won’t support without making a view or sproc/function
https://docs.aws.amazon.com/redshift/latest/dg/t_loading-tab...
[1]: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_...
My only real qualm at this point is mapping JS entities using the JS DB API makes it hard to use camelcase field names due to PG reasons I can't recall. I'm not sure what a fix for that would look like.
Keep up the good work.
Postgres requires you to "quoteCamelCase". There are some good JS libs to map between snake case and camel case. FWIW, I like a mix of both in my code: snake_case indicates it's a database property, and camelCase indicates its a JS property. Try it out - it might grow on you :)
For background: we have a storage product for large files (like photos, videos, etc). The storage paths are mapped into your Postgres database so that you can create per-user access rules (using Postgres RLS)
This update adds S3 compatibility, which means that you can use it with thousands of tools that already support the protocol.
I'm also pretty excited about the possibilities for data scientists/engineers. We can do neat things like dump postgres tables in to Storage (parquet) and you can connect DuckDB/Clickhouse directly to them. We have a few ideas that we'll experiment with to make this easy
Let us know if you have any questions - the engineers will also monitor the discussion