News for hackers. Stuff that matters.
Re:Ask HN: What's a side project you built to make money tha... by 37ef_ced3 on Wednesday December 30, 2020 @04:38PM (#25586456)
NN-512 (https://NN-512.com) generates stand-alone C code for AVX-512 neural nets

The idea was that user companies (or even the hardware companies: Intel, AMD) would pay me as a contractor to implement deconvolutions, transformers, etc., as extensions to the free/open core I released (which includes very sophisticated support for all forms of dense convolution, fully-connected layers, pooling, batch norm, activations, etc.)

The free/open core supplies all the basic operators, but many users would need extensions of the core to support their particular networks (or even just software to feed their parameters into the system and get results out) and hardware companies benefit from the software's existence because they sell chips and NN-512 makes the new vector units on their chips more useful

But it didn't work out that way. NN-512 did generate offers to work on other projects (a generous offer from Intel, for example) but nothing that extends NN-512

Why? There are numerous free end-to-end deep learning tools developed by large, company-sponsored teams (Nvidia, Facebook, Google, Intel, etc.) so a specialized tool that requires integration effort is unattractive. That's my guess. So beware, if you're working along similar lines

Re:Ask HN: What's a side project you built to make money tha... by staunch on Wednesday December 30, 2020 @04:54PM (#25586607)
I saw this in one of your other posts and it seems super interesting to me.

1. Improve the web site slightly (add a contact method and pricing info) and make it clear that it's a company offering an enterprise software tool. Programmers will be viewing the site, so it doesn't need much more than is there already.

2. Charge a flat $7,500/year for a commercial usage license and increase it as the functionality increases. Offer email support and updates. Many programmers can buy a $7,500/year piece of software without their management even asking a follow up question.

3. Write some blog posts showing how to use it and why it's good. Some benchmarks (LOC, performance, etc) vs alternatives would be a good source of posts. Real world examples that people can copy/paste to start with is probably the most useful.

4. Submit the blog posts to HN and some subreddits.

5. Profit (maybe)

Re:Ask HN: What's a side project you built to make money tha... by Aeolun on Thursday December 31, 2020 @02:06AM (#25590329)
> Many programmers can buy a $7,500/year piece of software without their management even asking a follow up question.

I’d like to know where you work.

No place I’ve ever worked would approve that without a lot of questions.

Re:Ask HN: What's a side project you built to make money tha... by unixhero on Thursday December 31, 2020 @02:56AM (#25590588)
> He is not wrong
Re:Ask HN: What's a side project you built to make money tha... by 37ef_ced3 on Wednesday December 30, 2020 @05:56PM (#25587274)
Thank you for this thoughtful reply!
Re:Ask HN: What's a side project you built to make money tha... by Smerity on Thursday December 31, 2020 @02:09AM (#25590340)
I work in the space and was impressed with NN-512 as there's a painful gap in inference cost between CPU and GPU that doesn't have to exist. Intel and AMD are really missing a boat here, most other companies have enough cash they just go to GPUs, academics rarely sling low level code even in CUDA let alone AVX-512, and other than Fabrice Bellard's work few I've seen few go that low level.

My suggestion would be to focus on an initial use case where a very limited low cost / high efficiency CPU model can provide massive advantage. NN-512 should be the framework that expands from that Redis like core. The limited use case tactic is what I'm focusing on[1], mainly as I have a particular application and have less technical brilliance than yourself so need to focus ;)

An aged but still relevant example is the early word2vec work which was (and still is) frequently better to throw onto CPUs than GPUs. A well tuned implementation is not only advantageous on CPU but can win out in many scenarios where cost / latency / ... are important.

Congrats on the project though! I'd be curious for your thoughts for the future if you ever want to chat =]

[1]: Initial experiments written up as a tutorial with Rust and ISPC for a specific CPU based NN task - https://state.smerity.com/smerity/state/01E8RNH7HRRJT2A63NSX...

Re:Ask HN: What's a side project you built to make money tha... by 37ef_ced3 on Thursday December 31, 2020 @04:17AM (#25591020)
In your [1], are your input/output arrays aligned the same for the fastest SIMD and ISPC runs?
Re:Ask HN: What's a side project you built to make money tha... by Smerity on Thursday December 31, 2020 @04:38AM (#25591144)
Not in that codebase as it was a tutorial / wanted to ensure it's callable from safe Rust code so stuck with `_mm256_loadu_ps`. That code was just playing with dot product like lookup over vectors on CPU. The code I'm more interested in is trying to cram models into ~L2/L3 cache such that a CPU optimized model can be trained on GPU to be deployed on CPU.
Re:Ask HN: What's a side project you built to make money tha... by d_burfoot on Wednesday December 30, 2020 @05:20PM (#25586887)
How about a tool that generates C (or Java or X) code from TensorFlow or PyTorch models? That way you can train a model offline, and then export it to your preferred language for serving/production. For bonus points, figure out how to get the generated code to use the GPU properly.
Re:Ask HN: What's a side project you built to make money tha... by 37ef_ced3 on Wednesday December 30, 2020 @05:35PM (#25587055)
Addressing your last sentence, I am actually a GPU programmer by trade. GPUs are my specialty, and much of my professional work has been in Nvidia's CUDA

But AVX-512 is a real step forward, if you look at the price/performance ratio for real-world workloads. So AVX-512 is the focus of NN-512

GPU cloud compute is almost unbelievably expensive. Even Linode charges $1000 per month, or $1.50 per hour (look at the GPU plans: https://www.linode.com/pricing/#row--compute). It's really hard to keep that GPU saturated, which is what you need to do to get your money's worth

An AVX-512 Skylake-X cloud compute instance costs $10 per CPU-core per month, or 2 cents per hour, at Vultr (https://www.vultr.com/products/cloud-compute/). It's easy to saturate that core

Think about it this way: a GPU is really just a cluster of SIMD units with fast memory. A GPU thread variable is just like a SIMD lane, and a GPU warp variable is just like a SIMD vector register

Nvidia's SIMD pseudo-instructions are called PTX and they are like a 1024-bit version of AVX-512. An AVX-512 core is like a general purpose CPU with a 512-bit GPU core built in

So paying for a single AVX-512 core is like paying for part of a GPU (16 threads), plus the general purpose compute you need to keep that GPU part supplied with work

Re:Ask HN: What's a side project you built to make money tha... by bravura on Wednesday December 30, 2020 @07:02PM (#25587883)
I've been interested in your project. I don't think you're realizing the potential of the parent's suggestion.

You motivate the work through bottom-up construction, but I don't think you have a clear explanation for who the users are. They must be:

1) Sure they are only going to use models that are supported by NN-512. This is a big jump for me and many people who want to do transformers, LSTMs, etc. and not just what NN-512 costs. 2) They have to be price sensitive. But if you want larger companies, they will rarely just have one use-case that NN-512 supports; they will probably have a ton of use-cases that NN-512 supports. 3) They want to target AVX-512. If they are a medium or larger company they probably want to have tooling around this, to support model development and later deployment.

Those are my thoughts in general.

I currently need to be able to target and deploy on PC and Mac CPUs from the past 10 years. (I know, this is specific.)

I think parent is saying that individual devs are more likely to try NN-512 if they don't have to learn a new API, since most of them want to have the flexibility to fool around before committing to a new tech or bringing it in-house.

I guess my suggestion is really try to specifically define who the user is for NN-512 and what NN problems they are trying to solve, and let that shape the roadmap.

Re:Ask HN: What's a side project you built to make money tha... by 37ef_ced3 on Wednesday December 30, 2020 @07:26PM (#25588063)
My original thinking was that companies needing to use transformers, LSTMs, etc., would want to fund the development after doing the simple work to evaluate NN-512's ResNeXt or DenseNet performance on their hardware

Same with development of tooling (network conversion, parameter import). It just takes a lot of time/effort for me to write the code. It's easy to say that support for AVX, AVX2, etc., would be nice, but that's months of hard work (unfunded)

I guess we agree that a niche tool like NN-512, that has a reduced feature set (trimmed to allow one person to complete it and release it for free), has trouble competing with the huge, free, flexible, multi-target tools funded by large companies

In retrospect this is obvious!

Re:Ask HN: What's a side project you built to make money tha... by m3at on Wednesday December 30, 2020 @07:26PM (#25588067)
TVM can do that for you: http://tvm.apache.org/
Re:Ask HN: What's a side project you built to make money tha... by exged on Thursday December 31, 2020 @04:30AM (#25591088)
Just wondering, what makes NN-512 superior to Intel's oneAPI DNN library?
Re:Ask HN: What's a side project you built to make money tha... by 37ef_ced3 on Thursday December 31, 2020 @08:35AM (#25592511)
For example: try to find a deep learning framework that uses Intel's oneAPI DNN library and achieves 80% of the performance of this: https://nn-512.com/browse/DenseNet121

Tell me if you can find one

Re:Ask HN: What's a side project you built to make money tha... by overkalix on Thursday December 31, 2020 @06:24AM (#25591692)
I had a similar idea last year, but clearly I'm not talented nor experienced enough to do it. Congratulations dude, this rocks.
Re:Ask HN: What's a side project you built to make money tha... by reitzensteinm on Wednesday December 30, 2020 @09:38PM (#25588984)
Your project is seriously cool, I'm sorry it didn't work out for you.
Re:Ask HN: What's a side project you built to make money tha... by 37ef_ced3 on Wednesday December 30, 2020 @11:16PM (#25589530)
Thanks dude
Re:Ask HN: What's a side project you built to make money tha... by person_of_color on Thursday December 31, 2020 @08:22AM (#25592396)
Where did you land?