paphillips
The financial criticism ('napkin math') appears to estimate DynamoDB costs of USD $8 million for 2017 to 2020. Uber revenue for the same period is roughly USD $42.5 billion, thus this cost weighs in at about 0.02%, or 1/50th of one percent. This is a rounding error for a high growth company, and not something that warrants a witch-hunt and firing. It's easy to blow more than $2 million per year on software engineers in pursuit of an alternative high-scalability solution.
I'm also not on board with the 'resume driven development' criticism as the explanation for solution churn. Perhaps that is actually what happened. I wasn't there and don't know, but if that is being asserted I expect to see evidence presented to support it.
The bizarre thing was that the page source at that time had almost nothing except some odd javascript to build a "508 Daily Report" table of Jira tickets or some nonsense.
That view appears to still be up here: https://www.census.gov/main/
I can't really blame someone for a bad deployment given the raging government dumpster fires at the moment.
Oddly, the direct hyperlink to this page doesn't seem to work properly though: https://primer.prooftrading.com/lifecycle-of-order/
The AMF project [1] can be used to parse and transform to/from RAML, OpenAPI, GraphQL, and json schema. Code generation to languages of choice can be bolted on from there.
I'm using this approach to define canonical data models. Subsequent code generation scaffolds internal application integration apis, master data management (MDM) entities, and SQL/OLAP artifacts for ETL / BI purposes.
This approach keeps overall end-to-end data architecture consistent, in sync, and versioned under source control. Additionally, flat types as required by relational systems are re-used and composed into nested complex types more appropriate for apis. Metadata is layered on as needed to refine the models for system-specific needs, for example to add user-facing field groups, descriptions, and formats for BI datasets, sensitivity levels and other data security controls, business rule definitions for MDM, etc.
[0] https://github.com/raml-org/raml-spec/blob/master/versions/r... [1] https://a.ml/docs/
Shout out to Anchor Electronics in Santa Clara, San Mateo Electronics Supply, and of course Jameco, which are still alive.
I'm also lamenting the loss of SF Bay area electronics surplus: Weird Stuff, Halted/HSC, and the latest casualty, Excess Solutions. Is there any place left around here to find used/surplus electronics?
It is intuitive that, as soon as enterprising individuals catch wind of high profits being made somewhere, there will be an inrush of competitors looking to seize their share, which then continues until until some type of equilibrium is reached.
The fundamental breakdown in this type of efficient market mechanism is that it requires a reasonably level playing field: referees and rules. Complex systems without adequate regulation may result in local optima one or a few participants, who achieve regulatory capture, externalize costs, or achieve monopoly, oligopoly, or similar advantage to the disadvantage of all others. Regulation is required to achieve the global optimum for the wider group (i.e. society).
Cancer is an a example of a biological system exhibiting high growth with broken mechanisms of regulation. Similar outcomes can be observed when there is a disruption to a predator population, leading to an explosion of prey species, resulting in an ecosystem that is overrun and exhausted until balance returns.
I recently heard a description about the way this technology will change technical work that resonated: we will become more like the movie director, and less like the actors.
XL Construction is an award-winning Northern California general contractor/builder, consistently voted as one of the best places to work in the Bay Area by the San Francisco Business Times / Silicon Valley Business Journal, and is ranked as the fourth largest GC in the Bay Area as of last month.
We are in the early stages of executing a long-term enterprise-wide data strategy and this role is key.
XL is seeking an individual with 5 years of application integration development experience with Mulesoft or similar, and a commitment to software engineering best practices in a team environment. We run a modern suite of applications so you won't have to fight crusty, antiquated, and temperamental APIs and can better enjoy the satisfaction of getting information flowing.
I am XL's interim data architect and look forward building great things together.
Please review the full job posting [0] and apply there if you are the one. Reply here if you'd like to give me a head's up.
[0] https://www.xlconstruction.com/careers/open-positions/ (under Business Technology > 'Integration Developer')
I had been making the SV surplus store rounds at least once a month - Halted (HSC), Weird Stuff, Excess Solutions, etc (plus the Fry's swap meet). Those nicely furnished my garage workspace with industrial-quality furniture, equipment, and other random bits, but sadly all have closed.
Nowadays I am down to one SV store on my short list that is still around, and they only sell office furniture and electronics workstations - neither of which I can use any more of. I'm hoping these types of stores are a trailing economic indicator and a fresh group will pop up soon.
Any there any other surplus/liquidation stores in the SF Bay Area still around (not the military surplus kind)?
[0] https://portlandrobotics.org/links_surplus.php?link_id=21
1. In terms of software efficiency, engineers may lament the perceived waste, inefficiency, and imperfection in the produced code, but from a business standpoint it is a rational cost/benefit decision. It is useful to view software through the lens of economics. In economics there is a concept that Labor (e.g. a software engineer) and Capital (e.g. servers, infrastructure) are substitutable. Many sub-optimal programs and systems built with reduced labor cost are perfectly usable by substituting more hardware. Optimization only makes sense where there is a clear benefit that exceeds the cost.
Thus, as a contrived or extreme example, would a manager spend $200k in labor to produce a highly optimized program, hand-crafted in assembly, or spend $500 in labor to produce a program in a higher level language such as Java, that does the same thing but uses more compute resources? The spread in cost between those two choices allows one to throw a lot of hardware at the sub-optimal program. Thus it is frequently a better business decision to produce inefficient software and throw more hardware at it. It may make the engineer feel bad, but what they wish to optimize is not aligned with what the business wishes or needs to optimize.
2. In terms of the short 'shelf-life' of software, the same problem infects hardware, consumer electronics, and other products. I've purchased a number of IPads for my family over the years. After a few years and IOS versions, more and more apps stop being compatible, until it becomes effectively useless even though the hardware is the same as when I bought it.
Again let's view this through the lens of economics. A cynic will look at the IPad situation and and think 'What better way to separate me from my money than to force me to buy a new product every few years, solely by software shenanigans?' Of course businesses enjoy selling more product, but they also have cost constraints in order to be viable (ignoring those who are perhaps making 'obscene profits' before competitors take notice, as my econ professor used to shout so passionately).
We might consider as an alternative that it is simply too expensive to maintain many versions of an app, on multiple platforms, with backwards compatibility and security concerns. The business instead is making a rational decision to only support their application on the OS versions and platforms that the majority of their customers are running at any point in time, similar to how a web developer at some point has to stop bothering to ensure their site works in IE 5.0.
None of that reduces my frustration at planned obsolescence but maybe this is just the reality of things.
3. I'm on the fence about the labor-exploitation part: this seems like a different and very complex issue. Some may argue that more hardware manufacturing provides good jobs without extended education or training requirements, while others may argue that those jobs are exploitative because the working conditions are poorly regulated or the position does not pay enough by their standard.
At a macro level, global poverty levels have significantly decreased over the past 30 years [1], so humanity seems to be doing something right. An optimist may say that as regions of the world move out of poverty, the regulatory environment will inevitably follow to reduce abuse, pollution, and safety risks. Time will tell but it requires patience - human systems are slow to change, in contrast to software and hardware.
[1] https://blogs.worldbank.org/opendata/april-2022-global-pover...
Traditional software will continue to be chosen when we want predictable, unbiased, mechanical execution of instructions. There are many areas where this is preferred, and I don't see that changing. Mechanical and later silicon calculation devices are invaluable for their speed, but the greatest benefit is that they are predictable and consistent: they do not make errors unless the design is in error.
AI, machine learning, and other training/learning-based technologies also have many useful and tantalizing applications. For applications such as those that enhance productivity, provide entertainment (e.g. art and music), or autonomously perform tasks where mistakes can be tolerated, these training/learning-based technologies will reap great things.
However, for many applications we don't want a complex device, whose behavior, while it can be ostensibly tested, cannot be completely understood and examined to be provably correct. Or, whose faulty action cannot be definitively reproduced and root-caused after a mishap. Or, whose 'black-box' can be infected or influenced by bad actors in a manner that is undetectable.
I don't ever want to see a radiation dosing machine that is clever, an industrial control process that is expected to be trained to infer its own decisions where injury or life is at stake, nor do I wish to argue with a machine to open my pod bay door.
Alternatively, perhaps legal precedent will just establish the degree to which machines are allowed to make mistakes, and if they make fewer than a human, we will just accept the cost/benefit of injury, loss of life, or evil as 'practical', and move on. 'Actuary Shrugged'?
The most ominous prospect is if humanity fails to evolve past war and conflict faster than this technology's destructive capability. Maybe Fermi will get his answer.
For example:
public void CheckIsTrue(bool value, [CallerArgumentExpression("value")] string? expression = null)
{
if (!value)
{
Debug.WriteLine($"Failed: '{expression}'");
}
}
[1] https://learn.microsoft.com/en-us/dotnet/csharp/language-ref... [2] https://learn.microsoft.com/en-us/dotnet/csharp/language-ref...
An example: I'm using thin events in a master data application integration scenario to send a 'sync this record' type of command message into a queue. The message body does not have the record details, only the basic information to uniquely identify the record. It also doesn't identify the type of change except for a flag to identify deletes. The 'sync' message is generalized to work for all entities and systems, so routing, logging, and other functions preceding the mapping and target operation have no coupling to any system or entity and can expect a fixed message format that will probably never change. Thus versioning isn't a concern.
Choosing team 'thin event' does result in an extra read of the target system, but that is a feature for this scenario and what I want to enforce. I can't assume a target system is in any particular state, and the operation to be performed will be determined from the target system at whatever point in time a message is processed, which could be more than once. If the message ended up in a dead letter queue, it can be reprocessed later without issue. If one production system's data is cloned down to a lower environment, the integrations continue to work even if the source and target environment data is mismatched. No state is stored or depended upon from either system and the design is idempotent (ignoring a target system's business rules that may constrain valid operations over time).
In contrast, other scenarios may benefit from or require a fat event. I've never used event sourcing, but as others mention, if current state can be built from all previous events 'rolled forward' or 'replayed', then each event must be a stand-alone immutable record with all information - thin events cannot be used. Or, if a scenario requires high performance we might need to use a fat event to eliminate the extra read, and then compensate for the other consequences that arise.
However, the statement was only for cases of verbatim copies produced by Copilot. The AFC Wikipedia article states that "Proving copyright infringement requires proving both ownership of the copyright and that copying took place." The 3 detailed tests developed in that case appear to be "expand" the determination of infringement to close potential loopholes where, while there is not a verbatim copy, infringement is still deemed to have occurred because of "substantial similarity". e.g. someone copies a program but changes the variable names.
So where is the line between infringement and not, in cases where there is an exact copy of a code fragment? Can we still use the utilitarian defense or is that only used by the court to exclude portions of the code in the tests for "substantial similarity"?
Specifically within AFC, note the "idea/expression dichotomy" [1] which clearly states:
"copyright law protects an author's expression, but not the idea behind that expression"
Thus, if this tools spits out someone else's code verbatim it is a definite copyright infringement. If it outputs code that is similar but not verbatim then it "could" be an infringement, at your own risk, and to be determined by the courts. Simply expressing the same idea in a different way is not a definite infringement.
Program code is naturally copyright-risky because the keyword/grammar space is constrained. It is far more difficult to accidentally duplicate verbatim the expression of one's ideas in a full language, such as English, than in C. And what of two separate programs (or constituent sub-parts such as functions) that by chance emit the same compiled binary?
Personally, for now I won't use this tool due to the risk of accidental plagiarism, and because it is a black box: I can't examine any lineage or attribution metadata to understand the source(s) of what I would then be incorporating into my own body of work. Of course I doubt I could get that type of traceability information for any other trained ML model I might use, so perhaps I need to re-examine my policies heading into the future.
[1] https://en.wikipedia.org/wiki/Idea%E2%80%93expression_distin...
Specification has a fundamental tension between clarity and completeness. I don't believe it is possible to maximize both. As a specification becomes more specific, rigorous, and 'correct', it diminishes in understandability, due to the limits of our working memory. As a specification diminishes in clarity, it loses appeal with most audiences. See also the rise of JSON in reaction to XML.
This is the key tradeoff with UML. If the ultimate goal is to model a system, with platform-independence, at a level of detail sufficient to enable code generation, the specification and model representation will become complex, and the friction and overhead become too great for a typical corporate internal or external product, thus limiting the specification tool's adoption. The tool is no less valid, and can be useful for high-criticality areas, such as aerospace, or for academic exercises.
One should objectively analyze the tangible and intangible costs of UML against benefits and use something else if it provides a better outcome. Perhaps a 'UML-lite' approach, such as C4 [1] or PlantUML [2], is a better tradeoff on the completeness-clarity continuum for a given scenario.
Another interesting modeling framework is 'Object Process Methodology' (OPM) by Dov Dori [3].
The two aspects of OPM that stand out to me:
1. A focus on the model alone, maintaining understandability, and an effort to keep things simple. Information hiding techniques such as zooming are provided as a means to manage system complexity during the modeling process.
2. Instead of an ultimate promise to allow bidirectional translation from model to code, and code to model, the visual OPM representation has a textual counterpart, 'Object-Process Language' (OPL). With this facility one can translate from model to descriptive text, and vice-versa. Thus, one may read or define a model visually in diagram form, or as a series of sentences using a minimized set of English words that is also machine parseable.
Still we must recognize the tradeoffs: OPM may provide more clarity, but has less completeness. It won't round-trip with a code base, and it lacks modeling capabilities that can descend into an operation/process, e.g. one cannot model control flow concepts such as sequences, branches, loops, etc. Within the UML garden, concrete operation implementation can be specified in OCL, and the newer fUML can, in theory, reach down even further into operation specification.
[1] https://plantuml.com/ [2] https://en.wikipedia.org/wiki/C4_model [3] https://en.wikipedia.org/wiki/Object_Process_Methodology
I left a number of years ago and I'm certain things have evolved since, but the cost/benefit calculus is similar to any make vs buy decision.
In my opinion the primary benefit of an in-house developed ERP system (or any purpose-built system) is that it does only what it needs to do and nothing else. Thus, some types of accidental complexity are reduced, business user cognitive load can be lower, and performance and scalability can potentially be higher. Also of course there is no application licensing cost and hence user count is constrained only by infrastructure and code quality.
Now the difficult part: you spread the cost and risk of development over one customer, and ERP software is not a simple matter to build: essential complexity inherent in modeling business processes in software remains unchanged. You also have challenges finding subject matter experts as most of the folks who understand the internals of ERP systems are already working for one of the commercial vendors and will scoff at your audacity.
To be successful it requires leadership with vision, commitment, and patience; as well as a team of talented analysts and engineers.
Prior to that I was an ERP technical consultant. I've had the opportunity to work with many ERP systems at a deep technical level, on both sides of an implementation: the old system a company was moving off of, and the new system I was implementing.
I'm neither an advocate for, nor detractor of, any particular ERP system. Each offering has things it does well, and each has pain points. Unfortunately a lot of people cargo-cult their particular system of choice and lose objectivity. Software is simply a tool and is far less important than the implementation and execution.
A few SAP aspects that stood out to me from a technical perspective:
1. You are virtually guaranteed to have an existing capability available for any arbitrary business process need
2. The system architecture facilitates efficient data processing. Tables, forms, and reports tend to be narrow slices of an overall business process, and thus can reduce contention and concurrency problems. Batch jobs are utilized for background processing.
3. The user interface is lightweight and very performant over remote connections (e.g. VPN). Similar to older green-screen systems, data entry can be very fast for experienced users and does not rely on the mouse.
4. The API options for enterprise application integration are good
1. The surface area and complexity to match the capability is crushing. Tesla's relatively vanilla SAP environment had 10,000+ reports and T-Codes in it, of which I would guess less than 1% were actively in use.
2. Optimizing the architecture for scalability came at the expense of the user experience. One would often need to perform several tasks in order to answer a single need. For example, I would observe users exporting several reports, each of which provided a narrow slice of the information, and then combine the dumps in Excel to get the answer they sought.
1. It is very difficult to find subject matter experts who can deeply understand what a complex system offers out of the box, and successfully select the optimal capabilities and implement them for business processes. Or, in the case of true functional gaps, can modify and extend the system without creating a mess.
2. It is vital that a company retains the people who have the institutional history of a system and its modifications. Many end up with a revolving door of consultants, each of whom may be an expert in the vanilla product but has no company-specific context.
3. Clarity and completeness are at odds with one another. As you attempt to drive a system or specification toward completeness, clarity and understandability will diminish.
4. The longer an ERP system has been around, the more likely it will accumulate legacy and historical impediments. If a product aspect is working and widely deployed it tends not to be changed or updated to modern patterns or technology. An example in SAP is the cluster table concept, which as I recall from the lore some years ago was a workaround because SAP needed more columns per table than Oracle allowed. Cluster tables do not spark joy in data migration and egress when ABAP is not a feasible option.
5. Also for long-lived ERP systems, from the user experience standpoint, as these systems incrementally evolve over time the standardization and consistency of behavior across modules and application areas tends to fade or diverge. AS I recall with SAP for example, some transaction forms and reports can export to Excel, while many others do not: the feature was not or could not be added as part of the architectural core but was instead bolted on later to limited areas. A young ERP typically builds this in as standard core feature which is available on all forms in the product.The most complex thing I've done with it [1]: a tool (MIT-license) that builds diagrams of the data and addressing pipeline for a DSP processor, and lets one 'scrub through' the assembler code frame by frame and see the values propagate through the blocks.
Also PlantUML [2] uses it for most diagrams.
Getting layout and positioning the way you want can be tricky but is usually achievable with patience and hidden objects.
[1] https://github.com/paphillips/DFB [2] https://plantuml.com/graphviz-dot
The fundamental debt metaphor was about the need for ongoing re-factoring of past code and systems to match today's understanding, and the accumulating cost of failing to have the discipline and taking the time to do so.
Cunningham's specific quotes from the video [0] linked in the article: "...accumulate the learnings we did about the application over time by modifying the program to look as though we knew what we were doing all along, and to look as if it had been easy to do", and "If we fail to make our program align with what we then understood to be the proper way to think about our financial objects then we were going to continually stumble over that disagreement, and that would slow us down".
This problem is fundamental to software systems and how they change over time, and is not about making poor choices, tradeoffs, or generally poor engineering to ship code faster.
A system or feature is initially built based on the requirements that were known at that point in time. Requirements are frequently not static though: that original set of requirements may continue to evolve over time, either as we better understand them after implementation, or when requirements are added, refined, or removed by the stakeholders or organization.
Additionally, we can't view each feature or requirement in isolation over time: only when we see the overall set of features in a system will the important patterns start to emerge that inform how refactoring should occur. We can't know this early on, only over time as the system evolves.
I have witnessed this many times on software projects, both small and large. Requirements and features evolve over time, but the code base was never refactored to take into account the overall understanding as of each point in time. Rather, the changes are applied in isolation ('bolted on?') incrementally. Over time these numerous incremental changes result in a system that is fragile, increasingly difficult to understand and maintain, and generally veers toward spaghetti. It becomes harder and harder to make changes. That is the debt that was never repaid.
MIT-license code is here with screenshots: