The Joy and Power of Understanding
Deeper understanding of the code and software systems we work on, is not only pragmatic and practical but highly enjoyable as well.
The usefulness is quite obvious: understanding gives us control and ownership of the systems and code we are responsible for. What we do not understand, we can neither fix nor change.
Interestingly, that deeper understanding not only allows us to be masters of our tools and not their slaves, but also is simply fun and brings lots of joy. There most likely are solid evolutionary reasons behind it - why is comprehension so psychologically satisfying? It actually makes a ton of sense - behaviors and traits that increase our control over the environment should be accompanied by feeling positive emotions strongly.
But, if it is both joyful and powerful, why are we so often prone to skip the struggle to understand and take shortcuts, accepting copy-pasted/generated solutions and generic answers, not analyzed?
Human Nature
At our core, we are lazy beings, driven to minimize our energy expenditure and maximize returns on its investment.
This laziness can be a great motivator to automate tedious and expensive processes and tasks, but at the same time, it is our inherent flaw when it comes to learning, expanding our knowledge, skills and by extension - control, influence and power (in healthy hierarchies based on competence).
In the context of a need to understand, given how many answers & solutions to similar problems are available out there on the Internet - recently even more than ever before and in the exact form we need, thanks to Large Language Models (LLMs) - it is no wonder why people skip on understanding so often by taking those shortcuts. Unfortunately, there are many pitfalls in accepting something that seems to work but we cannot explain how and why.
After all, why bother with learning the syntax and inner mechanisms of SQL (or any other query language), when we might just tell the LLM what tables are there and the data we want to retrieve. It is way easier to write a messy English prompt and copy-paste the result than to learn how to do it properly ourselves.
And I hear you, some people would say something along the lines:
Why should I trouble myself with constructing SQL by hand if I know it? LLMs can do it much faster for me and I do not lose anything, since I am already perfectly able to write and read this language.
Well, it really depends; you might be capable of reading and understanding it today, since you have done it repeatedly by hand in the past, but this ability will diminish over time. Eventually, we do lose what we do not use - and passive reading is not enough to keep these skills sharp. Sure, you may argue that since there are LLMs (and other reusable solutions), we will never have to write these types of things again and that the skills of searching, prompting, reading and verifying LLMs' output is all we need - it is the future. Well, given how LLMs work under the hood (probability), I would say that it is a highly optimistic and unfounded claim (ignoring LLMs long-term sustainability for now). Regardless, at their best, LLMs and other search engines are force multipliers - but we must have the force first and keep it strong. What is this force exactly? It definitely is not the ability to prompt and read the output. I do not see how we can develop and keep it sharp, if we are generating and copy-pasting solutions (maybe with a bit of refinement) all the time. Fortunately or not, struggle is a necessary component of deeper learning and mastery.
Of course, context is king here. For some skills, areas that we rarely use and do not care about that is totally fine, but our core skills and knowledge regularly used must be kept as strong as possible - otherwise, what is that defines us as software developers and problem solvers? Isn't it exactly our knowledge, experience, judgement and skills? It is impossible to develop and even keep it by just reading the code and solutions of other people and machines - we have to be actively engaged in the building and creation process ourselves.
That is why, in the context of understanding, we must fight against our lazy nature on a regular basis - spending much more energy than is required in order to keep expanding our knowledge, skills and by extension - control, influence and power. We ought to read the docs and sources. Grasp the reasons behind solutions under consideration. Know our tools and understand the tradeoffs they introduce. Engage creatively and design solutions & algorithms ourselves - do not just search/prompt and passively verify, hoping to get some working solution; somehow, someday, maybe.
Short- vs Long-term productivity tradeoff
Practically speaking, does it always make sense to grasp the code and solutions we work on fully?
Of course not - it depends and it is a spectrum.
Throwaway script that automates an operation of low importance and risk? Completely fine to copy-paste/generate. Internal UI/page used by two or three people? Same here; nothing wrong with copy-pasting/generating that one as well.
Code and solutions that we will own, maintain and evolve over months and years? They should be created in a language and technology we know deeply and can understand every line, word, character and/or config option (or at least are on a path for it to be the case). Here, we want to optimize for long-term understanding, maintainability and productivity, not the fastest & largest output at this very moment.
Are there in-between cases? Situations where we might reuse/generate something grasped only partially, requiring us to take more time to understand, fix and modify when such need arises?
If we work on a Minimum Viable Product (MVP), not being sure whether it makes sense/is going to work, or an experimental feature within the existing product - it is reasonable to lower the quality and comprehension standards a bit. After all, we cannot yet tell whether potential results justify the invested effort. We can treat it as taking on a cognitive debt - it allows us to move faster here and now, but we will have to repay more later on (if the product/feature works or fixing and/or modification is needed). But still, we must at least be able to grasp and verify it up to a point where we can confidently say - it works. Nobody is going to use and pay for broken products and half-working features. I would then also ask: is generating something the best way to find out whether there is a need and want? Maybe a better idea would be to get this knowledge first and then build a considered product/feature in the right way? But sometimes, it might be the case that generating, validating and rewriting it from scratch is a reasonable strategy to take.
A similar approach could be taken when it comes to technology stack - if we use a certain programming language, library or a tool just from time to time, it probably does not make much sense to invest time in deeper learning and mastery. Copy-pasting/generating something we understand partially but can verify results of, is sometimes perfectly fine. But again, here lies the tradeoff - by not going through the learning phase and struggle, where it is natural to be slower, we rob ourselves of the possibility of ever mastering given technologies and becoming productive with them. For our core technology stack - languages, libraries, frameworks and tools used regularly - mastery pays off hundreds and thousands of times. Not only are we more independent, since we just know things, but even more importantly - knowledge and skills have a compounding effect. The more we know and are able to do, not only we can build things on our own faster, but also acquire new knowledge and skills at an ever increasing pace. Coming up with new solutions and novel ideas is another side benefit of constantly increasing our competencies. That is why, we must never give up on learning and constantly pushing our cognitive capacities to their limits and beyond.
Output- vs Outcome-driven metrics
Related to the productivity topic touched on above is a problem of definition. How do we even understand, measure and evaluate productivity?
There are two main schools: output- and outcome-driven.
Measuring output is easy:
- How many lines of code were produced?
- How many pull requests (PRs) were opened and merged?
- How many features were implemented?
- How many bugs were discovered and fixed?
- How many tasks were done?
- ...
There are problems with this type of output-driven approach though:
- It is easy to game these metrics: write verbose code, open more but tiny PRs, artificially divide tasks into smaller ones, introduce useless features and so on
- How do we know whether the right things are being measured?
- Is having more PRs necessarily better?
- Is a growing codebase sign of going in the right direction?
- Do we really need this feature? How about taking some unused ones away?
- ...
If we focus more on the outcomes:
- production releases are finally stable - thanks to the newly designed CI/CD process
- code was refactored and simplified - making maintenance and future changes far easier
- integration solution was redesigned - allowing it to both add new partners much faster and saving on compute resources
- more tests were written - spotting a few bugs before they even had a chance to happen and making future changes safer, by preventing regression
- metrics & alerts were added to the system - showing all kinds of potential issues and bugs proactively, allowing to smoothly resolve them
- tedious manual process was automated - saving time as well as removing the possibility of critical errors
- ...
Of course, many spreadsheet-driven managers prefer outputs to outcomes, since it is often harder to measure the latter in numbers alone; they require more context to evaluate correctly.
It is a slightly different discussion, but I would argue that it is wiser to focus more on outcomes and less so on outputs. Long-term productivity and understanding is much more aligned with outcome-driven metrics: not every output makes sense; more is often not better and sometimes, reducing/taking something away is what actually adds value.
Growing Complexity and the importance of Fundamentals
As the Fundamental Theorem of Software Engineering states:
Any problem in computer science can be solved with another level of indirection, except for the problem of too many layers of indirection.
Complexity of modern software development can be truly mind-boggling at times. How do these systems even work and are maintained? There are so many layers, components and factors, ever growing it seems:
- Runtimes & Platforms - browsers, servers: virtual & bare metal, clouds, mobile, desktop, embedded...
- Networks - their various layers, protocols and infrastructure: HTTP, DNS, TLS, TCP, UDP, IP, WebSockets, WebRTC, databases and message brokers/queues usually have their own custom protocols...
- Security, Authentication & Authorization - not all, but definitely most problems here have their roots in network access
- Operating Systems and their peculiarities
- Virtualization & Containerization - and if that is not enough, there are container orchestration platforms of the Kubernetes kind
- Databases - SQL, NoSQL, local (embedded), remote and distributed
- High-Level Programming Languages - Java, Java/TypeScript, Python, C#, PHP, Ruby, Go, Rust, C/C++... - requiring a whole pipeline of compilers, interpreters and transpilers to finally be executed as the binary machine code, the only language computers fundamentally understand
- Libraries, Frameworks & Package Managers - allowing for code reuse, but requiring ongoing maintenance and possibly introducing vulnerabilities
- APIs & External Services - we might buy certain functionalities and delegate responsibility there, but they introduce reliability dependency, often unwanted coupling and possibly legal & reputation issues with data transfers and storage
- Practices & Approaches, endlessly proliferating - CI/CD, Test-driven development (TDD), Behavior-driven development (BDD), GitOps, Infrastructure as Code (IaC), Domain-driven design (DDD), Event-driven architecture (EDA), Event Sourcing, Command Query Responsibility Segregation (CQRS), Server-side rendering (SSR), Client-side rendering (CSR), Clean Architecture, Hexagonal Architecture, Vertical Slice Architecture, Modular Monolith, Microservices, Microfrontends...
- LLMs & AI - when they do help and when they actually hinder development, maintenance and evolution of systems
- ...
What to do about this dizzying Complexity? Is it possible to understand even partially? There are so many layers, components and factors!
Worry not, there are some good news:
- many software systems are greatly over-engineered and can be significantly simplified - we might contribute to this process and they often are not as complex as it seemed at the first glance
- at any given period, we usually specialize and need only to master a certain area, a smaller subset of the software development landscape - of the rest, we should just have an awareness
- if we spend more time on mastering Fundamentals, as well as focusing on recognizing general patterns and underlying principles behind the tools we use - it will give us tremendous leverage, a great shortcut, in learning any new tool, approach or technology
Let's focus on the latter.
What are Fundamentals?
Fundamentals are the most basic, rarely changing rules, constraints and mechanisms underlying tools, libraries, frameworks, protocols and various components used to develop software; as well as the core principles governing computers and computation.
A rather comprehensive list would be:
- Computer Architecture & Hardware - CPU architecture, instructions execution, memory hierarchy, registers, caches and storage devices. What the machine actually is?
- Machine Code, Assembly and High-Level Programming Languages - why assemblers, compilers and interpreters are indispensable? What are the tradeoffs of different language types?
- Operating Systems - what are they and why do we need them? Crucial abstractions: processes, threads, scheduling, locks, synchronization, virtual memory, file systems, inter-process communication (IPC), system calls (syscalls), I/O operations...
- Algorithms, Data Structures & Complexity Analysis - they are taken advantage of and matter in pretty much every piece of software, programming language, library, framework and a tool
- Networks - how computers talk with each other. What can we assume about them? Are they reliable? What about throughput and latency? Why do they come in layers?
- Databases & Data Systems - ACID, transactions & isolation levels, indexes, storage types, relations & data modelling...
- Software Design & Architecture - why modularity matters so much? How to manage dependencies and responsibilities? Why should implementation details be hidden and information encapsulated? Modules, layers, client-server, peer-to-peer, events, coupling and cohesion...
- State & Data Flow - identity, source of truth, conflict resolution, state derivation, cache & memoization, state invalidation, state machines, events & commands, consistency and synchronization...
- Distributed Systems - CAP theorem, replication, delays, partitions, consensus, service discovery, eventual consistency...
- Concurrency & Parallelism - locking, synchronization, what can be divided & conquered (in parallel), race conditions, deadlocks...
- Security, Reliability & Performance, Testing & Verification, Operations, Product Engineering, Economics & Tradeoffs, Human Systems and probably something more...
That is a lot! But let's remember that we do not have to and it most likely is not possible to master all those branches - but we should know a bit about each of them, aiming to become great at the chosen few.
When our focus shifts to these fundamental principles, over time, we develop certain universal intuition, a powerful meta-skill. Understanding deeply how computing & computers work, alone and in clusters, we are able to quickly grasp and foresee what is possible. Then, we are not bogged by the ephemeral details of always-changing tools, libraries, frameworks and approaches. Ironically or expectedly, once we have reached this level, it actually becomes a piece of cake to learn any new shiny object of the current day. That is something worth striving for!
Feel Joy and have Power
As Leonardo da Vinci has said:
The noblest pleasure is the joy of understanding.
And as we have seen, in the context of software development, it gets even better - it is highly practical too, bringing us a lot of influence and making us powerful. The scope of software engineering is tremendously vast, but if we focus on Fundamentals, we can master it.
Let's then have the right focus and always keep pushing beyond our current cognitive capacities - experiencing pure joy regularly and gaining a lot of influence and power in the process!