kengoa

Nice list! I remember HN talking about https://students.brown.edu/seeing-theory/ when it came out but sadly it seems like this website was discomissioned.

Added an entry for my data visualisation tool here: https://github.com/stared/interactive-machine-learning-list/....

Edit: found an updated link for seeing theory so I fixed it in the PR above. Feel free to cherry-pick if #24 is not relevant.

Amazing work! Reminded me of LLM Visualization (https://bbycroft.net/llm) except this is a lot easier to wrap my head around and that I can actually run the training loops, which makes sense given the simplicity of the original microgpt.

To give a sense of what the loss value means, maybe you can add a small explainer section as a question and add this explanation from Karpathy’s blog:

> Over 1,000 steps the loss decreases from around 3.3 (random guessing among 27 tokens: −log(1/27)≈3.3) down to around 2.37.

to reiterate that the model is being trained to predict the next token out of 27 possible tokens and is now doing better than the baseline of random guess.

Understanding Python's data model helped me become a better software engineer. I'd recommend watching this PyData talk (James Powell: So you want to be a Python expert? | PyData Seattle 2017 - https://www.youtube.com/watch?v=cKPlPJyQrt4) and then using the official doc as a reference which is also fun to read: https://docs.python.org/3/reference/datamodel.html

> Consequently, knowledge bottlenecks are a form of operational inefficiency in startups.

This resonates with my startup experience, especially when the organisation is growing rapidly (which is almost always the case for VC-funded startups)

I ended up making my app installable as a PWA based on this conversation myself. Changes were straightfoward to add on top of a Vite + React application: https://github.com/visprex/visprex/pull/62/files

Yes as dimava commented above a PWA would be a great alternative.

> in a way it can be more honest for a developer with any commercial intentions to not open source something while offering paid ways to modify the source code

I appreciate that you made your commercial intentions clear in the comments and I agree that you don't want to walk back your license as it definitely leads to backlash. I think developers (i.e. people on HN) tend to trust open-core models more where there's a clear separation between the core open-source components and a paid offering but I guess you don't have to overindex on those given the general audience of this tool.

I don't know what your ultimate monetization goal is, but in my opinion one realistic path is an one-time only payment of some reasonable amount with the full feature as opposed to subscription (I saw your Stripe link in the source).

Great project! I'm also maintaining a fully in-browser data visualisation/analytics tool [1] and I often found it hard to convince people that their data isn't sent anywhere. It's funny I also say the exact same thing that they can inspect the Network tab. As a potential alternative to open-sourcing the codebase, maybe a desktop version of the same tool (i.e. an Electron app) could help address the privacy concerns if it explicitly states that it won't make network calls, although it's a perception problem rather than a technical one.

[1]: https://docs.visprex.com/

Very interesting read, I wish more startups did postmortems like this. It seems like the authors managed to build a set of tools that provided real value that ultimately did not make commercial sense, but would've been interesting to see if this learning:

> Scout was our most-used product. Its users weren’t our target market, but some were. We had vague ambitions to use it as an inbound marketing tool, but we never capitalised on it. This was a missed opportunity.

was applicable in a conservative industry like real estate and government as lots of open-core companies operate on this model (e.g. free open-source software and marked-up hosted solutions).

I encountered it during my masters in 2019 and it still remains my favourite textbook to this day.

> That year, I lost about thirty pounds due not having enough money for food. It was in this context that the lab’s principal investigator told me that his funding for my position was running out. Desperate for work, I applied for a job as a web producer and decided to learn to code. My thinking was that it was a stepping stone towards any sort of scientific career, and it would be possible without more formal education or debt. In hindsight, this was the right move, but at the time, it felt like the only move.

This reminded me of the summer when I accidentally landed my first programming gig at my friend's startup as a part-time web developer during university. I was running out of money for food as my research assistant role was unpaid and I asked around for some odd jobs in the city. As a non-CS major, all I had was data analysis skills in Python but my friend thought I was was good enough to help him out with his SaaS startup, and I ended up learning a lot on software engineering principles on that job.

Nice visualisation! Some entities seem to have "undefined" on the graph that you might want to double check.

Yes now I'm looking at the tests I can see that the coverage won't increase linearly, but 12% (with some of the hard parts) in two weeks is impressive. I starred the repo, good luck with your journey :)

dayjs seems like exactly I was looking for, thanks for the suggestion! I might have tried to write a parser myself otherwise.

I will try to add more support for other data formats like xlsx and parquet in the future. Current CSV parsing is also a bit limited (i.e. cannot deal with timestamps) so I will try to update parsers first.

Also thanks for the error checker! I pushed the fixes in https://github.com/visprex/visprex.github.io/pull/4

Thanks for commenting this, I would say speeding up the iteration between visualisation steps is the main benefit as you might not want to be thinking about matplotlib syntax when trying to get a sense of data distributions.

> had to rely on tools like Google Sheets or Colab + Pandas for quick cleaning and wrangling before exploring different visualizations.

Yes I had the same experience for analytics work some years ago. As others have pointed out, Visprex only works in a happy path where data is a clean CSV file so will definitely need to work on data cleaning. I have a DuckDB integration planned but not sure if this is easy enough for the target audience. Will try to add some predefinied functionalities, thanks for the feedback!

I'm very glad to hear this as this is exactly the target audience and the use case I initially thought of! I hope your students find it useful.

Yes I do have plans for data preprocessing using DuckDB WebAssembly (I have upcoming features secion in this blog: https://kengoa.github.io/software/2024/11/03/small-software....) but this will require SQL which some of the target audience might not be familiar with. I'm thinking of something like visual query builder from metabase.

> With the UI I want to be able to toggle between different strategies quickly - strip characters from a column to treat it as numeric, if less than 2% or 5% of values have a character, fill na with mean, interpret dates in different formats - drop if the date doesn't parse

Those are really good examples and I can make those predefined preproccesing techniques available as toggles in the dataset tab. Thanks for the feedback!

I used d3.js for Visprex and some of those graphs are modified from examples in https://d3-graph-gallery.com/

> how this compares to a spreadsheet CSV import tool such as the one in Excel which is extremely flexible.

I would say the data loading functionality compares very poorly to Excel CSV import for all the reasons you pointed out, and I agree that the users can face those formatting issues which could be resolved in another tool like Excel or Google Spreadsheet for non-technical users and Notepad++ or editors for a bit more technical users. The assumption on CSV files being clean is strong so I will try to surface import errors at least, and in the meantime point to different ways to format the data as those tools will be complementary to Visprex.

> To me, this implies that no steps have been taken to manage user/data privacy.

This is a good point. I fixed the wording and now it simply reads "No tracking or analytics software is used". Thanks!

I haven't considered beeswarm charts for this before, I will add those to a list of upcoming features. Thanks for the feedback :)

Thanks for trying it out! This is unfortunately not possible as of now and is one of th high-priority tasks to parse timestamps and datetimes, which is now incorrectly parsed as a string (Categorical). I'm using Papa Parse to load CSV data and I will likely need to add a custom parser on top of it.

Some of those plans are mentioned in my blog post reflecting on building this app: https://kengoa.github.io/software/2024/11/03/small-software....

Hello HN. I've always found writing data visualisation scripts boring and repetitive in data science workflows earlier in my career, so I built this tool to automate it. The available methods are based on my experience in econometrics where histograms and scatterplots were the starting points to check data distributions.

The link is to the documentation and the app is freely available at https://visprex.com, and if you're curious about the implementation it's open source at https://github.com/visprex/visprex. I'd appreciate any comments and feedback!

> I'm fairly certain I am able to eventually cover 100% of JavaScript spec. Any ideas, questions or critique welcomed!

Do you have the results of test262_runner.rb? I came to know about test262 at a talk by the porffor's author and something like https://github.com/CanadaHonk/porffor?tab=readme-ov-file#tes... in README would be great to show this progress. Great project by the way!

> Some folks feel the need to prove they worked on something yesterday, while others just enjoy talking.

This hits so close to home. The difficult part is that this tendency is not correlated with the level of urgency of the topic or priority within the project.

If I set my own email in someone else's email linx page from lynx.boo/username/edit, can I change others' links? (I haven't tried)