wiresurfer

A quick note, on platform.searchaf.com The account creation process hits a snag with verify-email links received on email giving a 404. hope it helps.

On a parallel note, It would be nice to put an architecture diagram in the github repo. Are there particular aspects of the current implementation which you want to actively improve/rearchitect/change?

I agree with the goals set out for the project and can testify that elasticsearch's DX is pretty annoying. Having said that, distributed indexing with pluggable ingestion/query custom indexes may be a good goal to aim for. - Finite State Transducers (FST) or Finite state automata based memory efficient indexes for specific data mimetypes - adding hashing based search semantic search indexes.

And even changing the indexer/reranker implementation would help make things super hackable.

So a little backstory, I have worked in Logistics and warehousing, albeit as a senior dev on robotics solutions and its needed implementations.

What you don't want to do is hire a team inhouse if your pain timeline needs fixing in < 1 Financial Year.

As someone mentioned. Senior freelancers preferably with some domain exposure.

Don't think of this as a software problem, but a house rennovation project!

Here is the playbook we have recommended and executed for many clients,

- Hire a documentation guy. Start writing/drawing out all business processes your current third party system serves. Use BPMN if possible.

- Sit down with a senior dev to draw a High Level Design of this system. And find subsystems or funcitons which you can live with, vs things which absolutely need retrofit/rewrite.

- See how you can define a scope of work where what's good keeps running, and a new smaller better subsystem starts taking over functions you need replacement.

- rinse repeat.

--- Strategically speaking, Hiring is a bit painful. Depends on geography. Retaining someone beyond 3-5 years is difficult. People churn out of boredom as the work gets repetitive and they don't get to flex their tech stack itch.

Money wise, is the third-party company a potential acquihire target? If its small, why not take it over and inhouse the team. You would get the "catered to us" part sorted and you can still sell to other customers if you so desire. This has its own risk-reward. But I will let you be the judge of it.

And in anycase, get an independent technical consultant to cover your blind side.

I have grappled with the same conundrum. Here are my takes on documentation writing. First is the X axis of type of document we are called on to write. 1. There is personal writing. books included. 2. Personal Blogging and Dev Rel publishing. 3. Then there is developer tool documentation. [code snippets, sdks, libraries] 4. Then there is product documentation. [images, steps, how tos]

Then there is Y axis of tools. 1. markdown and with it Z axis of mkdocs, hugo, KMSs like obsidian <name everything markdown here> 2. rst 3. org? 3. wysiswg editors or anything custom.

I find - markdown both ubiquitous and somewhat approachable. for most beginners. with obsidian/logseq/... it can also work for non tech writers too. its also more widely supported. [Github i am looking at you] - dev rel/dev tools docs with sphinx and rst is absolutely godsend! also a vibrant ecosystem. - product docs with mkdocs x markdown if its a consumer facing docs with lots of images and need for customizing. rst if its a developer product.

---

I write my blog in markdown, https://blog.shaishav.kr

My team maintained https://userdocs.rapyuta.io in markdown for the last 4 years. https://github.com/rapyuta-robotics/rapyuta-io-docs

- we also maintained https://sdk.rapyuta.io/ & https://cli.rapyuta.io/ both in semiautomated rst + github actions setup

Hey Brendan,

> If your company is paying for commercial software that includes kernel drivers or kernel modules, you can make eBPF a requirement.

Windows soon, may still be atleast a year ahead. Would that be a fair statement? atleast being the operating keyword here.

Specifically in the context of network security software, for eBPF programs to be portable across windows/linux, we would need MSFT to add a lot more hooks and expose internal kernel stucts. Hopefully via a common libbpf definition. Otherwise, I fear, having two versions of the same product, across two OSs would mean more secuirty and quality issues.

I guess the point I am trying to make is, we would get there, but we are more than a few years away. I would love to see something like cilium on vanilla windows for a Software defined Company Wide network. We can then start building enterprise network secutiry into it. Baby steps!

---

btw, your talks and blog posts about bpftools is godsent!

I'm Shaishav (@wiresurfer) and I am seeking out my next challenge. I am on a career break after working for the last 15 years, so I am putting this out here to let serendipity do its thing.

A bit about me: >15 years of experience in the industry. Microsoft Research -> startup -> Head of Technology for a GigaWatt scale Wind+Solar Renewable Energy Firm -> Cloud PaaS +Industrial Robotics for a Series B+ Japanese Robotics Firm.

I have put on many hats over the years. I can setup business units, run product, or build out engineering teams + products

Thrive in deep tech environments. Healthcare/SpaceTech/Modern Datacentre stacks excite me. AI/ML/Renewable, Industrial Iot, Logistics and Robotics are things I can help on day 0.

----

- Location: Bangalore, India

- Engagement: Fulltime or Contract [if you have a welldefined need which I can help with]

- Remote: OK

- Willing to relocate: Depends on the Opportunity.

- Technologies: Python,Golang, Cpp. ML[NLP + GenAI], ROS, Linux [Kernel + Networking + eBPF], k8s+docker [as a PaaS operator]. Worked fullstack but these are my preferred tools for deep work.

- Resume/CV : https://shaishav.kr/

- Email: [email protected]

Now that's one tricky rabbit hole. If you are planning on buying a camera, the deliberation seems to be a part of the shopping process. Post buying though, the differences seem either very important, or don't matter at all. Depends on how serious you want to get with the cameras.

did you chance upon @MentourPilot on youtube and his breakdown of the MH370 investigation + recent update? If not, I am afraid you may have another hour to spend, but its very likely you have already seen this.

So, I was looking at some performance issues which seemed to be stemming from linux networking stack. We eBPF with docker/k8s a fair bit as a PaaS, and I ended up getting into the weeds with linux sys/procfs and kernel tracing. One thing led to another and after quickly dispatching the perf issue, I was supposed to be on a planned week long holiday which turned into a deep dive into linux kernel + networking.

Intermediate step - I feel pretty confident with the gory details of the kernel code now. can possibly build a custom kernel, boot qemu with both a simple C+assembly bare metal kernel, or the self compiled kernel. I feels like the clouds have cleared and I can see the sun. - Incidentally the kernel source code is pretty well documented, but one thing which is missing is a much smaller list of files which are most important. true pareto here. 20% files carry the weight. You also need to know the subsystem you want to touch. Chances are that subsystem is much lesser number of files.

Finally - Got to reading about kernel packet handling. at the L2/L3/L7 level. from nic hardware to userspace. Turns out that eBPF [hello old friend!] has a networking avataar called XDP which is pretty recent [<5 years] way of doing high performance networking on the linux kernel. Along the way, got to know about network performance optimizations specially in modern multicore systems in the kernel like RPS/RSS/aRFS, DPDK/fd.io/VPP.

And now I feel the itch to apply this to some of our networks. Particularly, baremetal servers on equinix metal + aws ec2 + azure can be peered with either VPP/Bird to make a p2p connection which is a factor more performant than the vpc interconnect/gateways which are provided off the shelf.

I might extend the holiday by a few days. and I would love to talk to people who have hands on experience with any of this. Its hard to contain my excitement tbh.

out of curiosity, if the blog post itself an org-mode output? if yes then : 1. What tools did you use for the images? 2. That's a neat sidebar. custom ya-snippets? 3. Would love to take a leaf out of your dotfiles/.emacsd

Totally love the aesthetics of the page.

Sure. So a lot of what you want to grow depends on the weather prevailing in your area. If you have warm and humid climate with atleast 60% sunlight days. you should be able to grow about anything.

Top 3 suggestions 1. start small. preferably with herbs. Their time to harvest is low and as with anything, you need iterations and immediate gratification to keep you going. If this is your first time gardening, then i would say, go with plain gardening and graduate to hydroponics with the healthy plants from your first produce coriander, cilantro, lettuce are good first candidates.

2. There are hydroponic starter kits with plumbing pipes, and minimum hardware. If you want to hit the ground running, then go for these. Otherwise, DIY is possible.

3. Use grow lights if you have variability in sunlight in your area.

Location: Bangalore, India Remote: Remote only Willing to relocate: Maybe. Depends.

Technologies: python/golang/kubernetes | ROS & ROS2 robotics | ML

Overview: Open to Projects/Consulting Work for now.

I am seeking out interesting projects/people to work with. Robotics, Space tech and Energy/Industrial IoT are areas where I may add value

Most recently, running a 35+ eng team for a Cloud Robotics Platform. As a Platform Operator/Architect, run a 24x7 cloud k8s platform for ROS. Simulation + Connected Robots. ROS communication, CI/CD, Metrics Collection and Remote Debug tooling.

Before this, I was the architect and tech founder for an Industrial IoT Saas product. From prototype to market, built a platform for Energy Optimisation across three verticals. - Renewable Energy [Wind farm optimisation, predictive maintenance] - Industrial automation [Factory Management Systems, Building Management System, Energy Efficiency Programs] - Rooftop solar Monitoring

Early background is in ML research specialising in semantic hashing at a FAANG company. Built services with <30ms resp time, 10Mil req/sec back in 2013-2015

Have been in hands-on product and team building roles since 2015. Both as a two time founder and as an Engineering Manger.

Email: [email protected] Mention "From HN:" in the subject for easier tracking.

I've been burnt out with the current state of work. I work for a japanese cloud robotics company and have been witness to a great long term idea being plagued by bad management and strategy.

To manage my headspace, have been doing the following

1. Writing. Realising the value of communication for teams, have been writing up a lot of my learning into a series of documents which talk strategy and decision frameworks for CXOs. I share with a close-knit group of startup founders and product managers in Bangalore. The problems seems pervasive in Indian startups in their growth stage after their series a. specially with the funding spree and the need for hyper-growth. So talking to peers and collating knowledge has been the only way to stay sane :)

2. Gardening and hydroponics. Health in general, but the issue of getting good produce for a clean diet has always been a neglected aspect of my life. My wife and I decided we wanted to spend some time disconnected from devices. Gardening just happened and we haven't looked back since. Its become a routine thing and there is satisfaction in plucking your produce and tending to it.

3. Reviving some of the projects I had worked on in the past but got shelved and opensourcing them.

  - Low cost IoT WiFi connected 4 node Energy Monitor + Switch.  This was a PCB design + firmware + plus the Backend + Frontend+Analysis service + ML models to detect anomalies as well as predict usage. 

  - A self hosted algolia alternative for User Docs.

This is not entirely true. Speaking from my rather long sojourn with CS Research in an industrial research lab setting. While it helps to start out with researchers, many fields of CS research are as much individual pursuits as they are a team effort. If you have a network of domain experts you can communicate with, and access to publications, a sufficiently motivated person can definitely make inroads into research. A couple of other points to be noted, 1. open access to publications is still a utopian dream. Being affiliated with an organization or academic institution gives you access. 2. Infrastructure and Equipment is something you will have to arrange yourself if you want to do things on your own. This is prohibitive when it comes to the latest and greatest in deep learning, or systems engineering or anything biomedical. 3. Data. Organizations have access to data. period. Or atleast can arrange access to data. 4. It helps to have more hands on deck. So working in teams can accelerate the pace of your results.

Some fields which are conducive to lone work would be around cryptography, algorithms, essentially anything theoretical CS. or fields like ICTD where its less about a break through and more about a novel application.

ps: this is a very CS perspective.

My experience stems a relatively brief period (~8months) of managing a resource strapped team of 5 devs to deliver pretty complex and largely vaguely specd out products for my venture. (For reference this included a real estate platform, a client facing android app, plus an enterprise management system all complete with Data ingestion / Machine Learning pipelines, built from the ground up.) Here are my learnings 1) chart your plan. Sprints have feature groups, feature groups have features each with defined dependencies. and each feature has tasks. How clearly you map this dependency graph not just in terms of to-do items, but also in time, and by human resource required will define how on time your product gets shipped.

2) Rightfully Estimating Time Overestimate keeping in mind shipping a feature is usually making it + testing it out a bit.

3) Know what features can be chopped off when push comes to shove.

4) communicate clearly and with as much supporting documentation as possible. We still can't vulcan mind meld. So as a PM you need to convey your prioritised vision to your devs. motivation, how a feature fits on the roadmap, dependent features, Screenshots/ mocks, process flows, test cases to check against, all this will make sure features shipped don't need reiteration.

4) Architect for longevity If you are working with an architect, he will probably be responsible for this. But remember, its ok to delay the initial stages of your product if you are spending time building up the core. Think build systems, continuous integration and deployment cycles, just the API layers and business logic, UI work happening totally independent as functional mocks etc etc.

5) Trying to stay away from the temptation of coding. Code when you need to, assign tasks to yourself as and when required but not as a norm. It takes a lot to see the bigger picture and keep track of so many moving parts of a project. Doesn't help overloading your system with dev tasks too. These two worlds are pretty different beasts.

PS: One tool which I relied heavily on over the last year was Omniplan. We not just tracked dev items, but also things like deployment timelines, adoption rate within the org, training schedules for employees and how all this tied up into the features which needed to be rolled at and at what time. A sample of dev wise sprint plan which helps people plan their workload , keeps meetings to the point, and lets people take vacations when they are relatively less occupied :) http://imgur.com/a/1b2mH

The most critical step is to get the lib in your workflow , preferably with (build-introspect-debug) capabilities. This increases the upfront time to start, but leads to much quicker "code understanding in my opinion.

TL;DR; Start with the minimum exposed surface area of the project (API), dig through these functions first. Definitely know the initialization sequences the library needs.

This is my approach concerning JS projects or for dealing with other peoples code in general.

First, I make a mental model of what I want to do. !important. Then I write the smallest wrapper needed to start fledging out points where "separation-of-concern" happens.

At this point I should have an idea of what the other persons libraries expose as API. I also should have an idea of what can be done with a unmodified library, and what would need patching.

Then comes monkey-patching the lib at individual function levels with a healthy dose of TODO markers and NotImplemented Method signatures.

By this point I should have a good picture of what goes on in the library apart from what gets exposed and would probably have forked a branch by now.

This strategy has been useful not just for JS projects but bigger codebases of java/scala libraries like Lucene Core/Solr or Play framework, Django in the python realm and to limited success with Research code releases like Stanford Core NLP.

I have been a fan of Pipes for a while. Kudos to @psadri and others who made it possible with the state of web tech back in the days. It was a very well made tool both frontend and backend wise!

I have a emotional connect with Pipes as a tool. Right out of college, and before the whole API thing was pervasive, I had leveraged pipes for so many small projects and hacks. Infact I used it in a Yahoo Hackday hack and it landed me a gig at Y!

More relevant to the present, It has been something I have mentioned to so many people who have come to me discussing UIs for automation/control systems/or to manage workflows, processes. In a way it was what noflow's UI looks like, but years in advance. In a different parallel universe, it may be the way people use APIs, or right big Haddop/Spark/Storm jobs/topologies, and the front end was open-sourced back in the day, with full integration with hadoop for job management.

Thats a tough question to answer. Between bukowsky , Edgar Alan Poe, Shakespeare, a few of miltons work, Gulzar, Tolkien, you can imagine the choice to be tough! But here is an unlikely candidate I find particularly interesting because of his simplicity and yet intricacy of expression. Vikram Seth. His work "golden gate" is just a masterpiece with so many hidden gems. To have a story spannig a good 700pages in verse itself is astounding. Here are some of his smaller beauties

Awake

    *Awake for hours and staring at the ceiling 
    Through the unsettled stillness of the night 
    He grows possessed of the obsessive feeling 
    That dawn has come and gone and brought no light.*

And this, which happens to be the name of the small book of his selected short poems

    *All you who sleep tonight 
    Far from the ones you love, 
    No hand to left or right 
    And emptiness above -

    Know that you aren't alone 
    The whole world shares your tears, 
    Some for two nights or one, 
    And some for all their years.*

PS: pardon the bad formatting. Long form text entry via a tiny screen is still a pain point :)

I can vouch for F.lux. Its such an essential tool for me now and I have made it a point to get it on all my devices. Apart from the advantages it has in not disrupting my sleep cycle, I actually enjoy the warmer tones. Even when it comes to home lighting, I have always preferred the yellow shades over the incandescent whites. Never tried melatonin, either stand alone or coupled with 5-HTP. Let me read a little more. Thanks for the pointers.

As an aside, I would definitely say get your sleep cycles right and more disciplined definitely puts you in a much higher efficiency bracket. Coupled with some simple lifestyle hacks ( playing a team outdoor sport, if you are not the workout type, no devices in bedroom rule, boring but informative tech talk to go to sleep, light and early dinners, no coffee post sunset) you would be surprised how many extra hours of high concentration work you are capable of.

I see you have mentioned TF-IDF as something which you are planning to try. That should be interesting.

The way I see it, (and i may very well be slightly off point) you have a corpus of 2000 docs 2 lists -> [Wpos] & [Wneg] with count[Wneg] a factor more than count[Wpos]

if you compute a [0-1] normalized tf-idf score for each term in the set [Wpos] & [Wneg] and sum them up for all words in each of those two sets, you get a score proportional to the count of positive words & negative words. Normalized here would mean using relative frequencies, rather than absolute freq. [I prefer calling the latter term counts]

This puts document_word_count based normalization out of picture and makes it implicit in the tf-idf step.

Now you have Two numbers, Sum(Positive normalized TF-IDFs) and Sum(Negative Normalized TF-IDFs) which you can individually normalize for your list sizes, and then use the two scores for sentiment classification. A dirty hack, and somewhat inefficient if you don't maintain a reverse index.

Second approach could be this. Use your Word List, both positive and negative, to do a Okapi BM25 scoring against your docs using the list as the query set. So you would get a BM25score for your docs. and you can use that to define sentiments.

Corpus - D Di = Document in the corpus you want to classify Query1 = {set of positive words} Query2 = {set of negative words}

PositiveScore = BM25(Query1, Di ) NegativeScore = BM25(Query2, Di )

Some Combination to do classification. if Positive > Negative Score : call it positive!

Just a thought. BM25 has some flexibility in tuning it for length normalizations. Check footnote.

PS: There is the British National Corpus too for word frequencies :)

[1]BM25 and normalizations. http://nlp.stanford.edu/IR-book/html/htmledition/okapi-bm25-...

@mahouse, i edited the description in light of the new information. What he pointed out here was perfectly right (at that time instant)

Hey @apaprocki, The archive pages for bloomberg (at businessweek.com) seem to have dead links and redirecting to businessweek.com instead of the content article. Is that intentional? or maybe just in the process of being fixed.

I totally agree @at_10. Have updated the title to reflect that.

hey james, Thanks for pointing this out. this indeed is awesome!

OP here Didn't know they had a panache for awesomely creepy Error pages. Seems to fit in with their personality, in which case, the title of the post is my bad.

Anyways, I came across this while trying to resolve their archive page links, which seems to be broken, possibly after they changed everything to businessweek.com Although a small tweak to the url can get you to the article, a better tweak takes you to this gem of a 404! :)

Till the time its seen by the right pair of eyes :) Will try and see if I can fork a version and add the "inferred " touchend hack to it. Have a good day.

A whole lot of what you said resonates with my experience. The whole workplace being the place where you find genuinely honest "friends" seems to be a paradox. But its really true. Socializing on the other hand is a hit and miss thing. Its like random sampling. You may get a bunch of honest friends but a whole lot of the people at gatherings are busy doing their own piece in the grander social dance. On a sidenote, social networks as they exist are flawed. But there are certain micro-interactions out of the realm of a regular social network that sometimes have more meaning. Take for example your comment. Striking the same chord at the least makes me feel its not just me :) I have been lucky enough to have interacted virtually with a bunch of people online (irc,quora, HN) which has turned into fruitful friendships. There is no pressure of constant communication but whenever we interact, it starts from the point where we left last time. Its almost like a breath of fresh air. So i wouldn't outrightly saw virtual interactions are futile. BTW that SMS bit is so true. you hit a nerve there :)

Looking at the comments, and the HTTP 503 which i was getting, this seems to be the problem. Worked after a bunch of reloads.