tom_b

My BS in CS is from NC A&T.

My MS in CS in from UNC-CH.

Anybody who thinks the hackers from Duke/UNC consistently outshine the hackers from NC A&T is a damn fool.

I was regularly FLOORED by the skills and raw programming chops from peers at NC A&T. Some of the smartest young men and women in the CS field.

I worked in a small research group at NC A&T. Maybe 10 undergrads. Last time I checked, there were something like 6 MS in CS folks from that group and a couple of PhDs.

When I was an undergrad at NC A&T, we had a weekly colloquium for all CS students. We regularly had alumni from NC A&T and other HBCUs on stage who talked openly about navigating the hiring world, which companies you could expect racism in, and what the working world of programming was like for "people who look like us."

If some shitty recruiter event occurred, we all knew about it.

Want to score some of the best engineers from HBCUs? It's easy. Hire the best ones you can convince to take your mega-FAANG package from. Then send them back every freaking year with a corporate card and tell them to impress people.

I hate these "stupid recruiter" stories. Somebody should get their ass fired.

I think this a much more nuanced question than a yes-or-no answer can cover.

Looking at LinkedIn or indeed.com with just a simple search for Clojure with no geographic qualifier (in the US), we see a very small number of listed openings - 252 over the last month at LinkedIn or 44 in the last 14 days at indeed. A plain google job search on Clojure jobs in the US returns an even smaller number.

Some of those job listings will almost certainly have Clojure listed as part of a "laundry list" of technologies and the actual job will probably not be Clojure oriented.

So just from a raw numbers perspective, the job sites to suggest that there aren't a huge number of opportunities. Comparatively speaking, there are like 60,000 Java job postings on LinkedIn in the last month.

However, if you attend a Clojure/conj (one of the "primary" Clojure programming conferences), you will find that many attendees are part of some (not super-large?) number of companies that use Clojure.

My best advice on building a Clojure career (that I failed to follow) is that a programmer who wants to make a Clojure career path is to be prepared to spend a significant amount of time trying to make connections inside the Clojure community. On top of that, you probably need to build a portfolio of project experience that you at minimum blog about and preferably speak at Clojure conferences, user groups, or meetups.

<edit: removed off-topic thought>

I would gently observe that long-time Clojure developers are maybe too close to the tree to see the forest when it comes to judging Clojure career opportunities.

But I would love to be wrong about my impression of Clojure job opportunities! Perhaps fellow-HN readers could post (anonymous?) numbers of Clojure job openings for their company to give us a better idea?

I was unable to bring Datomic in-house. Mainly because our organization has a huge agreement with Oracle and all our database servers are essentially "free" to teams.

So I cheated. I created some schema designs that were immutable. I added a GUID, timestamp, and a deleted flag (value of 'Y' or 'N') to tables. Basically, all selects are against views that are defined over the tables to select the tuple associated with the max(timestamp) for that tuple along with the tuple having a deleted flag value of 'N'. This means that any select only sees the most recent tuple value for a GUID if it hasn't been "deleted".

There was a little bit more hiding in there to handle dirty writes.

But this worked very well for my requirements. By really only doing inserts, I was able to do similar "point in time" looks at the db as an immutable value.

Fun stuff.

Congratulations to Cognitect. Hopefully the acquisition will help Rich's retirement funding. (see 'History of Clojure' for the not-so-insider joke here)

I have had the pleasure of "using Clojure in anger" and for a few years was very dedicated to learning how to use it effectively. This was a significant step up for my general programming skills and the simplicity of the language often saved me from myself with respect to bad design or choices in code. Whatever programmer "maturity" I can claim is probably due to Clojure and teaching introductory programming and databases (not with Clojure and Datomic to be clear).

This acquisition is more abstractly interesting to me in that a VERY dedicated Clojure/Datomic company made it. We didn't see here, for lack of a better example, a FAANG company grabbing Cognitect. That may have been a choice! Maybe Cognitect went with an acquisition where they retain a high degree of autonomy with the benefits of being acquired.

I never made the leap into a larger organization with Clojure at its core. I was able to use Clojure only in my small shop where I have a small degree of autonomy on development language choice. For a few years, I was somewhat actively seeking an opportunity to join a larger Clojure-oriented team but struggled to gain much traction when applying to openings. It seems possible to build a Clojure career, but I've not been very good at that.

I think much training for dog aggressiveness is more about helping people to understand drives and managing "triggers" than training them out of the animal . . . it seems underappreciated by many normal dog owners.

My last dog had super high-prey drive and was also a fear biter.

Both drives were incredible. You could play train her to do almost anything in a very short period of time with classic behavior shaping. It was unfortunate that her fear drive was higher. But this mostly meant proper space management and patience were required along with never assuming that a situation was "ok" by default.

We worked with a number of animal behavioral specialists (including university veterinary behavioral practices), all of whom concluded it was just her default personality. Much of our success in keeping her with our family was building environments that minimized her fear drive impulses.

Still a great dog in many situations. She passed after a pretty healthy 15 1/2 years and I'm currently on a break from pets.

Thelios Major from System76.

  64GB memory
  8 GB GeForce RTX 2070 Super with 2560 CUDA Cores
  A couple of NVME SSDs.

I install Centos as the OS. Setup, including Nvidia drivers was very easy.

Probably could be built from components cheaper, but I appreciated just making an order and having a box show up later.

My past machines have been Thinkpads, then six years with a Macbook Pro that I wiped and installed Debian on. I made the move back to desktops for a variety of work-related reasons.

Tax policies to reduce smoking rates seem to show reasonable effectiveness:

"the rule of thumb in the United States is that a 10 percent price increase on a pack of cigarettes results in anywhere from a 2.5 percent to a 5.0 percent overall decline in smoking, with most studies showing an average 4.0 percent drop" from https://www.healthaffairs.org/do/10.1377/hpb20160919.471471/...

Data and analytics for this are worth reading about - I find geographic issues (low-tax areas across state lines, total numbers of retail locations selling tobacco products, etc) along with concepts of price elasticity to be interesting modeling problems.

You might want to check out:

https://publications.iarc.fr/Book-And-Report-Series/Iarc-Han...

It has a bunch of freely-available chapters talking about this area of research.

I remember being at the first Clojure/conj - I had just spent some time learning a bit of Common Lisp and stumbled onto the conference announcement somehow.

This was very fortunate for me - after attending the conj, I was able to use Clojure at work to sneak in some FP on the readily-available JVM platforms. I later used it to do some internal REST-API work. The code for that project has run for years without modification or error.

For whatever reason, I also discovered that thinking functionally with Clojure worked so much better and naturally for me than object-oriented design methods. While I have drifted away from Clojure over the last couple of years, I find that my problem solving mind is much better in other languages because of the time spent thinking functionally with Clojure.

I have enjoyed all of those and the Stockfighter attempt in this space as well. They all seem to have hit a hard obstacle in convincing the general, non-FAANG hiring market that they can provide "better" candidates that any old staffing agency. I have definitely had hiring managers in non-software companies tell me that they will just pick from three resumes from a staffing agency of a standard enterprise developer job without much actual consideration of the candidate's resume or portfolio of previous work.

I do suspect there exists a category of higher-end headhunter agencies with a stable of high-quality, proofed candidates that those agencies can more successfully place than other attempts. Just an anecdote, but I have seen in my area some things that look more like a "small team" placements where that team has some history of helping companies radically improve their software quickly. I am specifically thinking about an actual team that gets hired as a group but into fulltime roles at a company as a unit. This seems to be a prequel to an acquisition of the company that hired them being the end goal. Then that entire team seems to repeat the process elsewhere . . .

The response is enthusiastic, but kind of misses the point my original comment was making. All I was saying in that comment is that as a hiring organization, we bring in candidates who on paper have the skills and experience we are looking for, but flame out in spectacular fashion when asked to talk about their skills, experience, and how they might apply to our specific problems and situation.

Note that I am carefully NOT saying, oh the candidate knows javascript, but not vue.js, so they don't fit for us. I am also not saying "oh, if only candidates knew we wanted them to get vue.js certified and it would be more likely we would put them in our interview loop."

I don't think the problem of how candidates can better understand why they are or are not getting callbacks from employers maps very well to any stats-based measurement of job posting descriptions against applicant resumes/CV/LinkedIn profiles.

My only thoughts on the hiring process are really anecdotal, but I believe a few things to be true.

  1 - Most job descriptions are crap.
  2 - Most jobs for software engineers can be done by any
  reasonably trained and/or experienced candidate 
  who can pass a work-sample test.
  3 - Totally untrained and inexperienced candidates 
  seem to apply to lots of jobs.
  4 - Software engineer hiring should be ideally 
  based on work-sample tests.
  4a - I have a *very* hard time finding companies 
  that do #4.

But stats like that also can signal being over-selective.

I would gently suggest that it is a mistake to think in terms of unqualified vs highly-qualified candidates and ML for "callback probability". I don't want a good, but not FANG-level, candidate to NOT apply for our standard software engineer opening just because 74 people already did.

Why? We had 60+ candidates for a recent posting for a junior developer role with some experience - think 1-2 years or some good school projects. Somewhere around 45-50 of those applicants were easily HARD-FLAGGED don't interview at all after a quick cover-letter and resume review by a technical person using a rubric. That rubric was scoped only to weed out applicants with absolutely no meaningful experience.

While I also hate the laundry-list approach for job listings (Must Have: Expert Java and C#, Spring ORM and ActiveRecord skills with deep understanding of React internals), I think the best way to attack that is with a short, but focused cover letter. That should include a short paragraph or two about a candidate's hands-on experience and speculate how their experience might apply to the position. I guess what I am driving at is that this idea of ML assigning importance weights to "soft" vs "hard" requirements is, at best, just another low-value signal in job postings that are already full of low-value signal.

Matches my experience on the hiring side.

In the group between "don't qualify" and "can't afford", we have a hard time with candidates who are unable (or maybe lack the communication skills) to effectively talk about how their experience+potential are a good match for even a generic job opening.

We don't do any algorithmic/challenge screening during interviews.

A common interview fail for us is to have a candidate that can't generalize from or build on something from his/her resume. This usually comes out during "tell us about project X that you worked on" and comments from our side like "oh, that sounds like project Y here. But Y is different this way - what are your thoughts about how to handle <insert small-scoped, specific difference>?"

YMMV, but in my non-FAANG experience in the mid-/south- eastern US, tech managers seem to make at least 1.25X for X=senior dev salary. Maybe keep that in mind?

My dev friends over the last 10+ years have included people who made a management/dev track career decision and anecdotally speaking, those who chose management have earned more and seemed to have more employment stability. In a couple of cases, I have friends in my personal network earning 3X a senior dev salary managing dev teams.

Personally, I decided to stick with the dev track and have been somewhat dismayed to find (after 10+years with one employer) that I am often pulled into decision/management-type situations but am consistently evaluated as a lower-level employee because "coder." It's even more annoying when I chat with my peer group who made the mgmt track choice and I find that the overlap between what I do and they do is actually very large.

We've found that graph-based community approaches have some really nice benefits in our bioinformatics data.

In particular, we have found that these approaches seem to preserve very small cluster structure "better" than traditional approaches. Meaning, we have a small group of cells that we know belong to their own cluster group and the graph-based community approaches preserve these "small" groups outside of other clusters nicely.

But we have also noticed (and had some feedback) that we windup with final modularity scores that are very high - greater than or equal to 0.90 (on a scale of -1 to 1). Applied math folks in the graph algorithms world kind of seem to look at that and go "eh, that is so high you should probably just do PCA and move on . . . "

Especially given that you could (and people seem to) use UMAP as a precursor to louvain methods, I'll probably be looking into UMAP to see how it goes. But our current performance bottleneck is that the clustering (or community approach of graphs with the louvain method) is our computational bottleneck, so we'd like to whittle that runtime down as much as possible.

Nope. But I think UMAP would be a dimensionality reduction step that you fed into a graph-based community extraction algorithm? In our current code experiments, we have been starting with a k-nearest-neighbor approximation algorithm to build the graph. UMAP would, at first glance, replace that.

I think there is even a louvain clustering method (really a graph-based community extraction method) built into several UMAP libraries floating around . . .

Glad to see the article mention dropping in the fastcluster package to replace the default R hclust. I would suggest using the parallelDist library as well instead of the standard dist.

Clustering in general and hierarchical clustering have been something I have spent some recent time trying to come up to speed on. Current state of the art seems to be using graph-based approaches of community detection (Louvain method) where graphs are built for a set of samples with N features by starting with a graph of K-nearest-neighbors and assigning some weight to the edges.

Late 30's? Get off my lawn, youngster . . .

I have had several conversations this year with project manager- or dev team manager-types that lead me to believe that non-software companies are beginning a mass move into cloud services.

I hear AWS and Azure mentioned much more than Google Cloud Platform (GCP). Job search platforms seem to return job posting result counts that match that ( AWS >> Azure >>>> GCP ).

I have also been surprised to have hiring managers refer to Java/C# programmers as a pure commodity in non-software companies. In one case, a project manager for technical dev in a nationally (US) large insurance company said that they mostly just ask one of the large body shops for three resumes and pick one almost at random.

Based on the above and your financial services experience, I would stay in financial services to leverage your industry knowledge. I would try to shift into AWS (or Azure if you think financial services companies would prefer Microsoft to Amazon) and Kubernete/Docker and get deep experience in effectively using those techs. Try to avoid taking a lower paying job unless you have to?

I'm really torn about programming advice. I love coding and just writing software, but I think non-software companies see coders as low-value employees. Of course, that is a total over-generalization. I don't think having some magic number of years in Python is particularly game-changing.

I have discovered that in my region that tech managers are making salaries at least 25% more than programmers and often 50% more. Personally, I am having some second thoughts about spending the last few years as an individual contributor rather than making the jump to management. Also, the highest, most surprising salaries I have heard all belong to technical managers. In some cases, I have confirmed manager base salaries that approach 3X what developers are earning. Mind blown. However, I will also say that the few folks I know in this position were reasonably successful software engineers that moved into systems architecture and with some consulting background. So part of what made the salary happen was understanding how to make a value-proposition pitch to a specific industry need that was acceptable to hiring orgs. Meaning, patio11 is again right - charge more.

I have been happily running Debian (with xmonad) on a 13 inch Macbook with 16GB of memory and the retina display for almost five years now. The screen works great and I have had great success plugging into multiple external monitors including a recently purchased 3840x2160 monitor.

I think it might still dual boot into MacOS, but I don't think I have tested that for four years . . .

Yep, my family had to do the same ER visit for our treatments. Immune globulin is a blood product and I was told it is atypical for any medical center outside of a hospital to have it available. Having to go the ER added a not insignificant amount to our total bill, but we did only have to pay for a single visit to the ER - our followups, we just walked in and a nurse would give us the shot.

On the somewhat funny side, it's weird walking into an ER and being greeted by first name and a wave . . .

Yes, post-exposure costs in the US for human rabies immune globulin (antibodies isolated from the blood of a vaccinated person) in the United States is roughly a $10,000 to $15,000 expense per person (dosage is based on body weight). The vaccine cost is much less, well under $1,000 total per person.

I know these costs from direct experience. I also know that insurance coverage for this cost varies widely as well.

My entire family had a nighttime (meaning the family members were in bed asleep) exposure to an untested bat. Following the U.S. Center for Disease Control and local health department guidelines, plus the advice of infectious disease MD specialists, we had to receive both the immune globulin and rabies vaccine. These costs were not covered expenses under my (otherwise pretty solid) medical insurance plan - we wound up having to pay approximately 30% of the total cost.

In the end, I felt fortunate that at least our insurance kicked in to cover 70% of the costs and that we were in a financial position that allowed us to pay the low five-figure out-of-pocket expense for our treatment without undue hardship.

I'm actually debating keeping our vaccinations up-to-date going forward - I think the booster vaccine, while still uncovered by my medical insurance plan, is relatively low cost and incidences of rabid animal encounters seems to be steadily rising in my home state.

That sounds quite depressing. Would you be willing to share your problem description as given to candidates here? I will say that I have several friends who work in various industries addressing your real question and none of them would be particularly identified as a "data scientist."

The education model (immersive, time-intensive, mastery learning) Lambda School has in place seems like a very sweet spot between bootcamps and traditional university CS programs.

But some of that success has to be attributed to riding a boom wave of demand for software engineers. I'm a mid-career software engineer and have seen at least two "bust" cycles where experienced, well-credentialed, and just smart engineers went 12+ months between jobs. The effects of that have been reflected in undergraduate CS enrollments - we see big dips in the mid 90s and prior to 2006 (where a huge push up in enrollments begins again - https://cra.org/data/generation-cs/phenomenal-growth-cs-majo...). How well positioned the Lambda School revenue model to handle a 50% drop in demand and significant drop in starting salaries?

I am a big believer in the trade school process for software engineers, but I also think that the long term view for programmer careers is one that includes greatly reduced salaries in the average case and one reason I feel that way is that a learner can be productive in the workforce with a shorter-term and less specialized educational background.

FTA (examples of data science):

  targeting health-care customers for hospitals 
  people who can turn social-media clicks and user-posted 
    photos into monetizable binary code is among the biggest 
    challenges facing U.S. industry
  “sentiment analysis,” or finding a way to quantify how
    many tweets are trashing your company or praising it
  determine how customers prioritized paying bills
  “recommendation engines” - those programs that predict
    what you may want to buy next
  advertising

My background is traditional business intelligence, finding actionable data for high level leaders.

A common response on this thread is that little data science is actually occurring in the business world. It would be incredibly useful to me (and I suspect other readers of HN), to hear from other participants on the thread what data science and methods they are using.

I'll kick it off with data analysis examples from my workplace:

  1 - Analysis of patient accruals to various clinical trials.  
    Mainly tracking against goal numbers.  No statistics or ML.
  2 - Analysis of tissue collection opportunities to answer the question:  
    Are we obtaining samples useful to future research opportunities?  No statistics or ML.
  3 - Creating models to accurately predict patient accrual rates for individual studies across various 
    different variables (race, ethnicity, gender, age).
    Simple statistics, probably just a linear regression model.  This is a new effort.
  4 - a long, on-going, and currently unsuccessful attempt to extract useful data from pathology 
    reports (free text descriptions from pathologists examining various collected tissues for 
    both medical treatment and research purposes)

In addition, I know of a few NLP motivated efforts to train classifiers - say, given a set of 500 manually labeled papers, can a classifier be built that would be effective in bucketing an additional 8,000 papers?

Yes. I have taught introductory courses to undergraduate and graduate students with only a MS in CS. I think you would probably be able to teach upper-level courses if you could demonstrate expertise in the subject matter.

Workload varies. I preferred project-based assignments and tried to give students freedom to choose projects in a personal area of interest to increase their commitment, but it makes grading terribly time-consuming. I found that even with a well-defined rubric, individual projects cost me somewhere between 30-60 minutes of grading time per student.

I have taught using a provided syllabus and created my own. I was vastly happier when I created my own, but it adds time. With a provided syllabus, I probably spent 3-4 hours reviewing material and slides per lecture. That included reading the course textbook materials and working through most of the textbook problems. Maybe double that time when working from the syllabus I created, making new slides, and prepping for in-class programming work.

Grading sucks. If you teach a required course, you will also have to deal with some number (hopefully small) of unmotivated students. One of my courses was an elective course and I found the students .... happier?

And no matter how much I emphasized not waiting to start programming projects (including by having graded check points), a significant number of students wait until the day before projects are due to start. And then flame out.

Don't teach a course as project-only if it is a programming-oriented course. A meaningful number of my students submitted rather well-done programming projects, but were unable to write a for loop on an open-book exam. I also had students who were suspiciously unable to even use a required editor to demo small programs in one-on-one meetings.

Otherwise, teaching was fun and it's definitely worthwhile. All the warnings you hear about adjunct pay being low are also true.

Despite only occasionally hacking around with array programming languages, I have always enjoyed the terse syntax. But I admittedly find weird syntax somewhat engaging.

I have wondered what was happening with statistics in the larger group of array languages. Do other HN readers know of J, Q/KDB+, Klong, or others in use by teams outside of the normal financial folks using Q/KDB+? I'd be quite interested in hearing any stories from the business world where it isn't a "lone hacker" situation, but rather even a small group of programmers were using array languages in production situations.

And it's only worse at the staff level. Even getting patient data variables for patients consented to IRB-approved studies is hit-or-miss. Groups build barriers by wanting to "work with" principal investigators (PIs) instead of providing data to PIs and study staff. Data moats are everywhere.

My own group (at a research center attached to a university school of medicine) has switched our focus from helping PIs with clinical data management and applications to helping manage research samples and sample processing, mainly by doing housekeeping around genomic processing across different lab teams.

It's kind of hard to convey just how much I would warn devs away from healthcare and healthcare research IT gigs.

I have not found many-to-many relationships to be particularly problematic in RDBMs from a modeling and usage perspective.

Just curious, are you asking about a multi-way relationship between more than two entities? A many-to-many between Entity1 and Entity2 is a pretty normal model. A many-to-many relationship between Enity1, Entity2, and Entity3 (or even more entities) has been rare in my experience. When I taught RDBMs, I made a point of emphasizing that these types of multi-way relational models should be carefully examined just to ensure the 3-way, 4-way, etc relationship was absolutely necessary.

Honestly though, I think much of this depends on what your experience with both techs is. If you have spent a ton of time using a graph db and are comfortable using that as a backing store for your project, roll with it.

Re: Clojure - something seems off with Clojure not being in any of the sub-regions at all.

I also noticed that Java is not in the world-wide list and yet is in all the sub-regions.

Given that the JVM is such a big part of the Clojure ecosystem, I want to dig into the data once they make it available. I'm totally willing to believe that developers with Clojure skills are in a high-salary bucket, but I'm a little suspicious of what they are showing here.

BTW, for some years I chased specific languages as the cause of the high earnings. I believe this is much less true than I originally assumed . . .