datahead
I want to start playing with models, sims and collected data for sailboat racing- I know the RL/data science stuff, and I assume a good model of your craft takes time to build, and can be improved with collected data. What are some areas to explore when chaining model -> sim -> RL for performance?
I realize this is an extremely complex topic, with several PhDs worth of potential input- if you had to explain to someone technical what it looks like and where to keep digging, what would you say?
IME crossplane has been a "much better terraform" and also borrows from some of tf's open source provider code. Seems to be one of the best IaC pattern for k8s centric shops.
Pros: - adoption of k8s core engine, state mgmt
- model everything as a CRD, consistent definition pattern both infra and app
- open source
- great UI when layered with argoCD
- declarative
Cons:
- steep abstraction learning curve (for me anyway)
- docs lacked key context for newbs (also getting way better, big efforts here)
edit: formatting
@firesteelrain said, "you can do that through abstraction. You "include" your Terraform Azure Provider or Terraform AWS Provider. At the end of the day, your module needs to know what it’s interacting with but not the higher level of abstraction. We have done it at my work to make it cloud agnostic just in case we need to go to another CSP"
Single ops eng in a 3 person startup here. Ops eng is only one of my hats right now :) I found crossplane to be a solid tool for managing cloud inf. My assertion is that "the only multi-cloud is k8s" and crossplane's solution is "everything is a CRD". They have an extensive abstraction hierarchy over the base providers (GCP, TF, Azure, AWS, etc) so it's feasible to do what firesteelrain did. My client requirements span from- you must deploy into our tenant (could be any provider) to host this for us.
I can setup my particular pile of yaml and say - "deploy a k8s cluster, loadbalancers, ingress, deployments, service accounts (both provider and k8s), managed certs, backend configs, workload identity mgmt, IAP" in one shot. I use kustomize to stitch any new, isolated environment together. So far, it's been a help to have a single API style (k8s, yaml) to interact with and declaratively define everything. ArgoCD manages my deployments and provides great visibility to active yaml state and event logs.
I have not fully tested this across providers yet, but that's what crossplane promises with composite resource definitions, claims and compositions. I'm curious if any other crossplane users have feedback on what to expect when I go to abstract the next cloud provider.
cyberax's note on state management is what led me away from TF. You still have to manage state somewhere, and crossplane's idea was- k8s is already really good at knowing what exists and what should exist. Let k8s do it. I thought that was clever enough to go with it and I haven't been dissapointed so far.
The model extends the k8s ecosystem, and allows you to keep going even into things like db schema mgmt. Check out Atlas k8s operator for schema migrations- testing that next...
I also like that I can start very simple, everything about my app defined in one repo- then as systems scale I can easily pull out things like "networking" or "data pipeline" and have them operating in their own deployment repo. Everything has a common pattern for IAC. Witchcraft.
https://www.factual.com/blog/introducing-drake-a-kind-of-mak...
A barbell and some plates w/ rack will cover the main lifts. Squats, deadlifts, presses (chest and shoulder) are the first ones to build. I recommend Starting Strength for an on-ramp to get going. Adjustable dumbbells are great too- plenty of options out there. Some of this equipment is expensive, look for used and consider it an investment in health.
Happy Lifting!
https://www.canadianbusiness.com/the-last-days-of-target-can...
I used inclusive in this context to mean that a junior engineer is expected to speak up and offer input in an executive meeting if they have something to offer. Cutting through hierarchy, "good ideas come from everywhere". This could be idealistic but it helped me in eliciting input.
I don't know what it's like to be on the inside or close periphery to daily core operations at RH. I agree and understand completely what you are saying. "Free speech" has (recently?) been a guise to act terribly toward each other in a number of settings. Trolls gonna troll. It's important to maintain decorum in the workplace and a culture that defies that can turn toxic quickly.
Honestly, I thought IBM would be feel less rudderless with him at the helm. I wanted IBM to become more 'red' and not RH to become more 'big blue'- but it appears this isn't the case after all... seems like a short tenure.
Anyone know more about the new leadership?
Have you ever been in a room with a DJ playing low quality music on a decent sound system? It's awful, grating. Now at least your everyday aspiring DJ has access to higher bitrate tunes and doesn't need to blast low quality junk out of the PA system.
We started a new space Data Eng/ Data Sci and then it became critical. We've been able to maintain products by building long term platforms/systems that meet current and (anticipated) future needs in clojure. Initially, product and platform/system was all the same team. We've grown a little and now individuals tend to focus more on one or the other, but we're still <15 eng and shipping daily.
We maintain everything we've built to date, and we planned for that going in. Clojure's other superpower is that it is stable AF over long periods of time, handles refactors and testing well.
As for signoff in adopting clojure... our VP is old hat in clojure/lisp. We got lucky.
I've kept my team small on purpose, and we were able to be very effective by adopting clojure. We have sway because we deliver- and that's a different type of leverage than headcount. We don't get thrown every hot potato, instead we're consistently aligned with the critical portfolios.
A small dedicated team with tools such as clojure will outdeliver, and outcompete a larger team who cannot remain agile and require significant overhead to manage.
I'm basically rehashing PG's 'Beating the Averages', but it's been my experience as a dev mgr. http://www.paulgraham.com/avg.html
> unlimited [...] discussion, planning and coordination
turns into evidence once "intent" and an "overt act" thresholds are crossed, for all involved.
[1] https://criminal.findlaw.com/criminal-charges/conspiracy.htm...
Tim Ewald - Clojure: Programming with Hand Tools https://www.youtube.com/watch?v=ShEez0JkOFw
I recommend Eric Topol's books as well, including The Creative Destruction of Medicine and Deep Medicine. Lots of food for thought.
While not open source by any stretch, I see the personal EHR space being ushered along by companies following Apple's lead. Aside from complex patients, I think the generally healthy/mild-chronic person is uninterested in owning/managing their health data unfortunately. Apple is contributing useful tools to understand fundamental determinants of health including cardiovascular, sleep, fitness at massive scale. It just happens to come with your watch and phone, and their vision for health is starting to come into focus. This puts the patient in the position of generating the primary data (sensors, etc.) and sharing it with their care team on their terms (more or less). As telehealth becomes more prominent, I suspect the patient will be required to engage with their data more often as it will be the means of conveying a shared understanding vs. observations recorded in the clinical setting and stashed away in centralized EHRs.
Furthermore, if labs and other diagnostics are available directly to consumers it puts the individual in a position of ownership. I think the default position is whoever generates the data owns it, and determines how easy/hard it is to share with others. If the individual is empowered to generate information about themselves- this will start to swing toward "patient owned." I too look forward to more of this, but it will have to come with more direct to consumer and digital offerings.
One of the coolest examples I've seen of individuals taking ownership in open source med tech is openaps.org . I'm not one, but T1D's are some of the most resourceful and resilient folks around. Good on them for building a community to solve real problems together. Shout out to the #wearenotwaiting crew.
I work for a large hosp. operations company and serve as the Dir. Engineering for our clinical operations group. Hacking Healthcare is required reading for new members of my team. It serves as an excellent introduction (with a healthy amount of critique) to the dynamics in the hc technology ecosystem. Thank you for providing this perspective on the industry and its challenges with tech.
We've been successful developing using open source technology internally. In fact, I take a fairly hard stance on disallowing proprietary healthcare specific "solutions" from working their way into our stack (aside from the EHR itself, it has staying power). We're lucky in that we are positioned as somewhat of a startup within a larger org, and are able to take that approach.
To avoid some of the issues you raise, we generally are working to reduce the surface area of the EHR to become simply the transactional backend which is then mirrored to a larger ecosystem of custom apps. This has the effect of boxing in the regulated entity. We focus on data integration (by spending $$$$ on custom HL7 interfaces, unfortunately not everyone can afford) to get outside of the walled garden. This means we can use the information/data for new and interesting purposes without worrying about the EHR vendor's roadblocks/tolls. More importantly to some people, we don't disrupt the billing cycle that originates from the EHR.
Do you notice any trends where healthcare operations/providers are starting to develop internal technology that integrates with the EHR to compliment vs. replace the core transactional system?
The talk where I realized Rich is a data head too. Introducing his design goals and life experiences with db(s). https://www.infoq.com/presentations/The-Design-of-Datomic
"Clojure for the Brave and True" author, Dan Higginbotham, on key themes and overview. http://www.flyingmachinestudios.com/programming/datomic-for-...
Congrats to the Datomic team and Cognitect. I hope this move to cloud opens Datomic's design principals and ideas to a wider user base.
This clause was in-place and stood out to me as well. I had a chance to ask their legal team about it. The clause is written in legal-ese, which always sounds overbearing.
I asked the question in the positive sense, "what if have some really nice metrics from our use cases, and want to talk about them at a conference?" They simply asked to be consulted and request written permission to share. The intent, like others have noted, is to request (legally: insist) that Cognitect have a chance to review and point out potential implementation issues (good or bad) prior to customers making performance statements about their product.
The clause can/does put a damper on 'notes from the field' reports, which often help when deciding on tech direction.I look for community based reports to reinforce perceptions of a tool (to a degree). Completely agree with OP, do your own performance testing.
One thing I will say is that it would be hard for someone who hasn't invested in learning the inner-workings of Datomic's decoupled architecture to pick apart storage speed vs. transactor speed. For example, storage speed (SQL, DEV, Dynamo, etc.) is not a concern of Datomic, but a key dependency to measurable perf. This may change in the AWS service announced today, and become more uniform on dynamo and S3 "storage resources". https://docs.datomic.com/cloud/whatis/architecture.html#stor...
Datomic is a unique product and there are many ways to make it sing (or blow up) depending on how you use it. We designed data models, streaming processes, and queries with Datomic in mind and have had success. Exactly how much success, I'm not at liberty to say just yet.
One of the persistent issues on my team is the reliance upon a dataframe representation w/ R or python to do this type of aggregation and windowing. Most people will eschew learning the 'advanced' SQL and instead bring data locally to do imperative style munging on it.
This creates a few issues, mainly adding complexity to the analytical stack: - Instead of querying the data and doing ETL/feature engineering in the db- you are moving data around (usually to less powerful machines, such as a laptop) for simple exploration.
- This wastes time and usually results in more dependencies (dplyr for example- no hate Hadley), sometimes even limiting you to single threaded operations. Teradata, for example, is massively parallel and will perform these operations in short order. I've seen Data Scientists wait 6hr for R to do the same thing a SQL query against a prod system returns in 3min.
- Code is not portable. A query can be executed and results retrieved through ODBC, JDBC or native connections. Without these, data engineers are often asked to install R (including libs) on some intermediate machine just to do munging/ETL/feature engineering. If SQL driven, moving from quantitative exploration to operational is quite easy (maybe just a query tune).
All that to say, I'm glad this post is highlighting some of the advanced SQL that I hope more people rely upon. All of these ideas are better articulated in MAD Skills [0]
I also see a key feature being the tracking of performance over multiple sessions. Noahlytics will likely answer: who is shooting the best this week? May impact starting line-ups for coaches, etc.