ryanwaldorf

Oh that's really interesting! I imagine there could be a reason for it, for instance the data is distributed differently in the micropartitions so different where values could result in different data lookup patterns as you may skip more/less blocks. But overall this makes a lot of sense!

Really interesting! Could you clarify what difference having two different query_parameterized_hash for similar queries is? Is there a performance hit?

Totally agree. In my last job I was able to create my own ETL jobs as a PM to get data for my own analyses and figured out a fairly minor configuration change could save us $10M per year. It was from one of many random ETL jobs I created myself out of curiosity that, if I had been forced to rely on other people, I may not ever have created.

What motivated this: I kept hitting token limits when working with Claude on tasks with many parts. For instance, Claude would recommend three different areas of code to work on, but by the time I started on the second area, I'd get token limit warnings. This happened because Claude needs to process the entire conversation history with each response, even though the discussion of the first area is often unrelated to the second.

This extension lets you fork the conversation at any point, preserving all context and files up to that point while starting fresh with the token count. You can simply copy and paste it into a new chat or (download it as a file to attach) and continue your chat. Your conversation continues from the forked point as if it were the same conversation.

I've really enjoyed using Claude projects to help generate code drafts. However, one thing that's been painful is uploading new files versions after changes. I'd have to separately go through each folder in my project to upload the files again so Claude could have the most recent versions. It only took a few minutes each time....but it was a bit annoying.

As a result I built a file organizer for Claude projects. It selectively copies files based on extensions and .gitignore rules, organizing them into a target folder for easy uploading. It commits all the files in that folder to a git repository so they're not lost, deletes them, and then copies the updated files to that folder. This allows you to upload your latest file states to a Claude project without having to manually click through all your folders to select and upload specific files.

Hope y'all find this useful!

just replied to chrisjc that I created a branch with externalbrowser auth and pushed it to github. would you mind taking a look and letting me know if it works for you? would love to get your feedback as I don't have SSO set up in my account to test this myself

I just pushed the external_browser branch to github that should offer 1/ externalbrowser authentication and 2/ the ability to use TOML files with profiles with instructions on how in the readme. if you run the following you should be able to test it out.

     git clone https://github.com/ryanwith/melchi.git
     git pull external_browser
     # remaining steps are the same

Would you mind testing it out and letting me know if it works for you? Would really appreciate it!

You can also let me know if it works in discord here: https://discord.gg/bTg9kJ92

This makes sense, thank you!

Ah gotcha! Do you have a use case where you'd look to remodel/transform the data between warehouses?

Thank you! Will take a look at this over the next few days

Would that be to support SSO? If so, I haven't planned it yet but I could. Would that make this more useful to you?

That's really interesting! Could you tell me a bit more of what you're thinking? I'm not the most familiar with SQL Mesh and the typical workflows there.

I wanted to start with duckdb since it's really an incredibly powerful tool that people should try out. The performance you can get on analytical queries running on your local compute is just really impressive. And with snowflake streams you can actually stream live data into it without changing anything about your existing data. On why not other databases, I wanted to focus on OLAP to start as there are already other great tools like DLT that help you load data from OLTP sources like postgres and mysql to OLAP sources already, but OLAP to OLAP is pretty rare.

Have you run into a use case for streaming data between data warehouses yourself yet? If so which warehouses?

Played with this and it did a great job of creating the initial framework for my most-recent react app based on a few paragraphs of text. Would recommend trying it out

You can download one of the NZ government's Annual Enterprise Surveys here: https://www.stats.govt.nz/large-datasets/csv-files-for-downl.... Then simply go to the Excel to SQL tab (https://sqlgenerator.io/#/sql-converter/excel-to-sql) and press upload file. It will generate SQL CREATE TABLE and INSERT INTO TABLE statements for you. You can then download it as a file or simply copy and paste it into your query editor.

Additionally, if you want to make any changes like including only certain columns or changing column types/names you can do that via the inputs below the generated SQL. Any changes you make are immediately reflected in the SQL generated.

What's the benefit of this approach?

Cool site! What are the main use cases for this?

Why were folks concerned about the transaction logs?

Fair callouts. For the first version I was looking to solve the basic use case of "I've downloaded a standard CSV from one database and I want to ingest it into another". I'll look at adding the ability to handle non-standard CSVs as well.

Also I appreciate you pointing out what Snowflake and BigQuery have with this. I'm wondering if there's something Redshift-specific that may be useful here for larger file volumes. It is more performant to do a CSV copy, but it can be a PITA when it comes to having a role with the right permissions to copy data from the right location.

Anything else you'd recommend I look into?

It's helpful for the ones that don't have this yet like Redshift (for full disclosure I'm a PM there). If your system does have this already though you probably just want to use that.

Yup that's exactly it. This is an ease-of-use option for when you need to ingest somewhat small amounts of data into another database for joining. For large amounts of data, copying the CSV via a copy command should be your best bet.