In the battle of Narva (on the border between Russia and what we now call Estonia) on November 20, 1700, King Carl of Sweden and his 8,000 troops attacked the Russian army, led by Tsar Peter the Great. The tsar had about ten times as many troops at his disposal. Most historians agree that the Swedish attack was irrational, since it was almost certain to fail. Moreover, the Swedes had no strategic reason for attacking; they could not expect to gain very much from victory. However, because of an unexpected blizzard that blinded the Russian army, the Swedes won. The battle was over in less than two hours. The Swedes lost 667 men and the Russians approximately 15,000.

— Martin Peterson. 2009. An Introduction to Decision Theory.

Sweden army didn’t have any good reason to attack a significantly outnumbered Russian army. It didn't predict a blizzard which would significantly up their chances of winning. Yet, it attacked. It is easy to see that their decision to attack wasn't a well informed one; an act of valour maybe, but definitely not a rational one. But despite the irrational decision, the outcome turned out to be the most favourable one for the Sweden army — they won!

So an irrational decision turned out to be right, but certainly, a rational decision has to be right!

Software development is much more than just writing code. There are meetings to be attended, work to be prioritised, documentation to be kept up to date, customers to be spoken with, new teammates to be onboarded, other teams to be collaborated with, dependencies to be managed etc. etc. — aka the “glue work”. The importance of glue work is indispensable, and its impact is significant. This underscores the crucial role that a good manager plays in the team. It is rational for one to keep picking up the glue work from time to time and keep the team moving forward. Counterintuitively, it can be a severe career limiting decision when you’re a junior. In her excellent talk, Tanya Reilly captures this “Glue work is expected when you’re a senior, and risky when you’re not” phenomena brilliantly. When you’re a junior, and you concentrate on glue work more than technical tasks, the appraisers tend to interpret either that you’re less technical or that you’re more managerial material. Either ways, in their eyes you’re less of “an engineer” and more of “a manager”. Your promotion will be impacted because of this. This is great if management was more appealing to you in the first place, otherwise the rational choice of doing glue work has proven to be a wrong one. This is an interesting thing to notice — a choice which is rational for one person (the junior) is an irrational one for the other (the junior’s appraiser).

Rationality

Decisions seem right or wrong after the fact, depending upon whether the outcome is a favourable one or not1. This temporal relation between taking a decision and observing its outcome, makes it impossible to know upfront whether a decision is going to be the right one. But, what can be said about the time when we are about to make the decision? What about the rationality of our decisions? That is pre-hoc analysis.

Most of us have an intuition about what rationality is. If there’s a forecast about 70% chance of rain, it is rational to carry an umbrella with you before you step outside. It would be irrational not to. Whether it actually rained that day and you could use your umbrella to protect yourself is post-hoc. Similarly, if you prefer apple over orange, and orange over banana, it would be irrational to choose a banana when a juicy apple was lying there in your fruit basket2. Generally speaking, we have an aim — to not get wet, to enjoy a fruit, to get a promotion, to be recognised. Rationality will be the act of choosing the decision(s) which puts us in the most optimal path to achieve our aim. This “means to an end” notion of rationality, without surprise, is criticised throughout the literature, for instance if your aim is to get rich and you steal3.

We could also say that rationality is choosing an act (from all possible available acts) which maximises the value we receive from the outcome.

“Writing tests will take us longer, let’s stop for the time being and get the release out first. If we don’t have a release, if we don’t have the users using it, what is even the point of these tests?”. There goes testing out of the window! “Vulnerability checks are time consuming, let’s hold on till the release. If we don’t have a release, if we don’t have the user using it, what is even the point of having a secure software?” There goes security out of the window! “We can’t take on tech-debt right now. If we don’t have a release, if we don’t have the user using it, what is even the point of having a maintainable software?“ And all the tech-debts were surely addressed after the release!?! Quite a few times such decisions turn out to be right4. In most of the other cases, we minimise our optionality for future and end up regressing. “why are we not going any faster? — because there are regression failures”. “Why are there regression failures? — because we don’t have tests!”.

Bounded Rationality.

We know that humans make irrational decisions under bias, and sometimes quite predictably so. We make quite reasonable decisions based on the information we have, but our holistic information is itself limited — either we miss the feedback from distant yet related part of the system, or the feedback itself is delayed, or we deny the information itself because it is dissonant with our mental model.

Most systemic failures happen because of our bounded rationality5. For instance, in Westrum’s bureaucratic and pathological organisations, it is most rational to withhold the information pertaining to failures; because either you will be punished or the department will justify its act and ignore the information (read sweep under the rug). One is still maximising the expected value, it’s just that what might be best for the individual turns out not to be so good for the whole organisation.

This highlights an important (obvious when sighted, yet overlooked most of the times) aspect of decision making — the leverage points6. The reason leverage points become so integral to decision making is because most of our decisions are influenced by culture, and change over time and across cultures. Rationality of decisions at the pivotal leverage points, then, becomes the prime directive. No good amount of policies or practices (read post-mortems, retrospectives, root-cause analysis etc.) is impactful enough to bring about a considerable change until the organisation learns how to embrace the information around failures, unless the organisation treats its mission as the most important thing above everything else, unless the organisation has a mission to begin with!

Closing thoughts

Surely not all decision types are the same, because circumstances are not the same. For some decisions the outcome is quite certain. For instance, if you drop an apple from a tree, it will fall to the ground (how poetic!). In other cases, we know of all possible outcomes and their chances (probability), yet the final outcome is unknown. For e.g. tossing a coin — we know that in an unbiased coin, there is a 50% chance of a toss resulting in a heads and 50% chance of it turning out to be tails, yet the final outcome can only be determined after the toss. These are decisions under risk. Yet, there is another class of decisions where we know all the possible outcomes, but don't know their respective probabilities. For e.g. going to a new restaurant (so no reviews are available) and wanting to try out an exquisite dish — we know that if the chef is really good the dish will be a treat, and if the chef is not, the dish will turn out to be a disaster. But we don't know what are the chances of either of the outcomes. These are decisions under uncertainty. Quite a lot of decisions in our day to day life/work are of the risky and uncertain kind.

The normative techniques of taking rational decisions under risk differ from those under uncertainty. (I wish to cover them in subsequent articles.) Our heuristics and bounded rationality might not make us perfect decision making machines, and one might argue about the importance of studying normative methods of decision theory. To that I will leave the readers with a quote from Peterson (2009):

“Anyone wishing to know what makes a rational decision rational should study normative decision theory. How people actually behave is likely to change over time and across cultures, but a sufficiently general normative theory can be expected to withstand time and cultural differences.”

Naur’s primary motivation for proposing the building of shared mental construct is to give deeper insight into how it influences the modifications in any software — “A prominent reason for proposing the Theory Building View of programming is the desire to establish an insight into programming suitable for supporting a sound understanding of program modifications.”

Naur’s idea of “theory of a program” is influenced by idea of “theory building” and “theorising” from Ryle’s work in his book The Concept of Mind. I’m quite fascinated by the concepts of “theory”, “theory-building”, and “theorising”. I had read the paper earlier, but rereading it after having read “The Concept of Mind” presented it in a different light for me. Naturally, I’ll concentrate more on these meta concepts in this article. To have a better appreciation of Naur’s thoughts in his paper, I’ll also present some parts from Ryle’s book pertaining to the concepts of “theory” and “theorising” (or “theory building”).

Theory and Theory-Building

An intelligent act is different from an unintelligent one. While a task can be done well by just adhering to certain criteria or following a set of rules, like looking both ways before crossing a street, it doesn’t necessarily qualify as an intelligent one. One could precisely follow the intricate recipes of Reynold Poernomo and succeed in recreating his signature desserts, but that alone does not qualify as intelligent cooking.

Ryle analysed that an intelligent performance is more than just being able to do an activity well according to some criteria. A person able to perform an act intelligently would not only be able to apply the criteria, but also be able to identify and rectify errors, iterate and enhance accomplishments, gain insights from his own and others' experiences, and more.

“The well-regulated clock keeps good time and the well-drilled circus seal performs its tricks flawlessly, yet we do not call them ‘intelligent’. We reserve this title for the persons responsible for their performances. To be intelligent is not merely to satisfy criteria, but to apply them; to regulate one’s actions and not merely to be well-regulated. A person’s performance is described as careful or skilful, if in his operations he is ready to detect and correct lapses, to repeat and improve upon successes, to profit from the examples of others and so forth. He applies criteria in performing critically, that is, in trying to get things right.”

— Gilbert Ryle. 1949. The Concept of Mind

An intelligent behaviour transcends following rules of some kind. For if there are rules on how to perform an activity intelligently, there will have to be a set of rules on how to effectively follow the rules, and then another set of rules on how to follow the following of rules and so on. An infinite regression!

“Knowing That” is not the same as “Knowing How”

“What characterises intellectual activity, over and beyond activity that is merely intelligent, is the person’s building and having a theory, where theory is understood as the knowledge a person must have in order not only to do certain things intelligently but also to explain them, to answer queries about them, to argue about them, and so forth. A person who has a theory is prepared to enter into such activities; while building the theory the person is trying to get it.”

— Peter Naur. 1985. Programming as Theory Building

“Knowing that” is the knowledge of or knowing a theory. “Knowing how” is the ability to be able to apply the theory within the given context. “Knowing that” will be knowing Newton’s laws of motion, to be able to state them, to be able to write their equations fluently and to be able to derive them even. “Knowing how” or “having the theory” of Newton’s laws would be to have the ability to identify which category of reality belong to classical mechanics so as to be able to apply the laws of motion effectively. “A person having Newton’s theory of mechanics must thus understand how it applies to the motions of pendulums and the planets, and must be able to recognise similar phenomena in the world, so as to be able to employ the mathematically expressed rules of the theory properly.”

“Newton’s theories were used when correct predictions and retrodictions were made on the basis of them, when machines were designed in accordance with them, when the hope of building perpetual-motion machines was given up, when some other theories were abandoned, or else were codified with his, when books were produced and lectures delivered enabling students to grasp the whole or parts of his theories and, lastly, when some or all of his theory-building techniques were learned from his example and successfully applied in new investigations. To be a Newtonian was not just to say what Newton had said, but also to say and do what Newton would have said and done. ”

— Gilbert Ryle. 1949. The Concept of Mind

Frequently, the mastery of "knowing how" relies upon a foundation of "knowing that." Achieving proficiency in TDD, for instance, necessitates grasping the sequential red, green, and refactor steps. Yet, a mere grasp of these stages might prove insufficient, as true effectiveness in TDD requires an intellectual capacity to comprehend contextual criteria and significance at a higher level than others. It is one thing to be able to write tests and another to reason that we should not test private methods. It is one thing to learn how to mock and another to know when not to and use stubs instead. This transition from factual knowledge ("knowing that") to practical mastery ("knowing how") embodies the process of building a theory for oneself. Once one has his theory solidified, he attains a deeper contextual understanding, enabling him to apply the theory with discernment.

The Theory To Be Built By The Programmer

In terms of Ryle’s notion of theory, the developer has to build a theory about how certain aspects and functions of the real world are to be handled by the program that has to be written. Programming as a whole encompasses many activities, but not limited to, starting with understanding of the real world problem that the program is intended to map and solve, designing, implementation, deployment, maintenance and enhancement of the software.

Programming then, has to be about knowledge that is beyond just knowing how to write idiomatic code or knowing about different patterns of asynchronous communication in a distributed system. It must have the knowledge about the real-world use case, and how to map the business rules effectively in the code. It has to do with reasoning about how to design the system to be able to incorporate future modifications as they come, and more importantly also to be able to rationalise when a future extensibility is wishful thinking.

According to Naur, the knowledge possessed by the programmer by virtue of his/her having the theory of the program transcends any artefacts like the source code, documentation, diagrams etc. in at least the following three areas:

1. The programmer having the theory of the program can explain how the solution relates to the affairs of the world that it helps to handle. By far the largest part of the world aspects and activities will of course lie outside the scope of the program text, being irrelevant in the context. However, the decision that a part of the world is relevant can only be made by someone who understands the whole world. This understanding must be contributed by the programmer.

2. The programmer having the theory can explain decisions like why a certain approach was taken over the other, why a certain class or set of classes were designed in a way and not the other way, why synchronous communication was preferred over asynchronous and much more.

3. The programmer having the theory of the program is able to respond constructively to any demand for a modification of the program so as to support the affairs of the world in a new manner. Designing how a modification is best incorporated into an established program depends on the perception of the similarity of the new demand with the operational facilities already built into the program. The kind of similarity that has to be perceived is one between aspects of the world. It only makes sense to the agent who has knowledge of the world, that is to the programmer, and cannot be reduced to any limited set of criteria or rules, for reasons similar to the ones given above why the justification of the program cannot be thus reduced.

What Theory Is Not, Theorising Is

To get a real appreciation of why the theory of the program, held by the developers, has a primacy over other artefacts, we need to understand how we generally build our theories (or how we theorise) and then later describe our theories through some written or verbal discourse.

The transition from theorising to putting our theories in didactic fashion is a transition from non-linear thinking to a linear output, and is lossy. When we are theorising, we’re going through the process of assuming, hypothesising, criticising, conjecturing, proposing. This is mostly a non-linear process. When we later intend to explain our theories through verbal or written mediums we tend to linearise this vision. Imagine the transition between when you are brainstorming on a white board or making a mind-map — a non-linear representation, and then later when you summarise and put forth the outcome in some sort of documentation — a linear representation. This transition is bound to be lossy. When we whiteboard a systems design, we present our worldview of the problem statement, we put forth our knowledge, run through different scenarios and challenge each other's hypothesis. We can do our very best to capture these transitions, but it can never capture the theorising process in entirety.

Karl Weick argues that theories are a continuum and not a dichotomy. A software has very good reasons to keep undergoing changes. As a result, our worldview, understanding, and theory of it is forever changing too. Documentation only works up to a point and generally lags behind, if not missing completely, in capturing this changing worldview.

Documentations usually are after the fact; they are written after the developers have already built the theories in their minds. In this respect documentation serves as a mnemonic for what you already know, not as a tool for learning as Baldur Bjarnason puts it.

Program Life, Death, and Revival

Naur presents an interesting idea about the lifecycle of a program. Since theory of a program is built and held in the minds of the programmers and cannot be reduced to or expressed conceivably through other materials, the lifecycle of the program is dependent on the team’s changes and churn.

“The death of a program happens when the programmer team possessing its theory is dissolved.” The program may continue to serve its purpose long after the team is dissolved, the actual death of the program only becomes evident when there are changes to be made or the program needs to be extended. The new team, devoid of original theory, might still be able to make modifications but these modifications are bound to be divergent from the original theory, and the resultant modified program different from how the developers having the original theory would have done the changes. (** coughs 𝕏)

Naturally, the program’s life gets extended when new developers come in and get to build their theory of the program by working closely with the developers already having this theory. It is always insufficient for new developers to get onboarded efficiently through just reading documentation and getting familiarised with the source code. It is for this reason that onboarding of a new developer should be a team activity.

Revival of the program is to rebuild the theory of it by the new developer team. Naur suggests that since the original theory cannot be reconstructed through just the source code and documentation, it is better to rewrite the code. — “In preference to program revival, the Theory Building View suggests, the existing program text should be discarded and the new–formed programmer team should be given the opportunity to solve the given problem afresh. Such a procedure is more likely to produce a viable program than program revival, and at no higher, and possibly lower, cost.” This is one idea that I found myself heavily disagreeing with. Rewrite is always a bad idea in my opinion and should be avoided at any cost.

Closing Thoughts

In revisiting the paper after having read Ryle’s book, I had different takes from when I read the paper for the first time. In day to day of modern software development, we take pride in writing clean and idiomatic code, we write our documentation elaborately, we create story maps to layout the real world problem that the program is supposed to handle, we annotate our system design architecture quite well, we often record our design decisions in ADRs, we write tests to ascertain the scenarios our code is supposed to handle, and much more. All these artefacts are secondary to the shared mental construct of the team that is working on the software. They are necessary but not sufficient.

Although I still find myself disagreeing with the idea of a rewrite. Maybe I’ve not experienced the “death of program” as Naur mentions it. Maybe if you have, I’d love to know your thoughts and context in the comments below.

“We’ve come to value experiential learning much more, and we’ve regained a strong pragmatic focus, but I think it would be a shame if we lost sight of some of the deeper things which people have learned over the past 50 years. Rediscovering them would be painful, and (to me) not knowing them would be a shame.”

— Michael Feathers. 10 Papers Every Developer Should Read

“Arguments and Results” is about patterns that can be used to tackle complexity in APIs (or as Noble calls them “protocols of an object“). It primarily focuses on their arguments and return results; hence the title of the paper. The patterns listed in the paper are not exhaustive by any means, but are quite extensively applicable.

Two primary motivations behind these patterns:

1. To make software simple and concise: “Since smaller, simpler programs are generally easier to read and write, the patterns are concerned with the complexity or size of a design, such as the number of messages an object understands”

2. To constrain inevitable complexity from spreading out: Over time, complexity in software is inevitable. But we can constrain the complexity to few places instead of letting it leak to other parts.

In this part 1, I’m going to only focus on “arguments” and in part 2 I’ll cover the patterns related to “results”.

We’re going to look at three patterns to tackle complexity with respect to arguments of a function/object:

1. Argument Objects: Or how to make implicit concepts explicit

2. Selector Objects: Or how to simplify methods that do similar things.

3. Curried Objects: How to simplify extremely complicated APIs by currying and partial application.

Arguments Object

In simple terms, this is about extracting common and related arguments into respective value objects.

A function that accepts large number of arguments is more prone to modifications. Primarily because it already does too many things. It doesn't hurt to let it do one more thing, and then some more. Moreover, “adding an eleventh argument to a message with ten arguments is qualitatively quite different to adding a second argument to a unary message”

"If you have a procedure with 10 parameters, you probably missed some"

— Alan Perlis

By lifting and grouping the parameters into respective objects, we make the purpose of a function more explicit. These groupings (abstraction into specific types) make the reasons for change more explicit and meaningful. For instance, when writing a Paint software, it makes sense to talk in terms of coordinates (both X and Y value) together. So it makes API more readable when we extract them into a Point type:

This makes it easier for the client developers to reason with the APIs and eases their cognitive load during implementation.

I found this pattern very similar to DDD’s “make implicit concepts explicit” and using value objects over primitive obsession.

Selector Object

Quite often an API exposes several functions which perform similar underlying operations. For instance, continuing on our Paint software theme, lets say we have a graphical “View” type which exposes various methods to draw different shapes:

“Protocols (APIs) where many messages perform similar functions are often difficult to learn and to use, especially as the similarity is often not obvious from the protocol's documentation. Because the messages are conceptually closely related, they will often need to be maintained as a group, which will require changing a number of method implementations in servers and many different message sends in clients.”

In a typical Object Oriented world, you would use polymorphism to have different `Shape` types and expose a base class `draw()`function which can take different types of shapes and then draw them. These different instances of shapes then behave like a Selector Object to the base draw function.

Noble mentions that in simpler scenarios even just an enum type would also work fine as a Selector Object. In more complex scenarios you could use multimethods or even double dispatch. I also found Composite pattern to be quite similar to Selector Object concept.

One could also use Selector Object as Flyweights .

Curried Object

This is one of my favourite go to techniques, especially when I’m dealing with external libraries and their APIs. Let’s take a typical example of using node’s fetch API to get some data from an HTTP API. Let’s say I am interested in getting all the posts in my blog and comments from a particular post. A typical code would look like below:

There are a plethora of options available while configuring the request. For the sake of this article, I’ve limited these options to passing of a bearer token for authentication purpose and using `Get` method of fetch. Evidently, each client code interested in calling the `getPosts()` and `getComments()` functions would need to pass the token. It needs to be aware of authentication mechanism. Most of the projects, after first authentication with server, would keep this token in some sort of a middleware. With the above code, each client function would also need to be aware of middleware and invoke some sort of get to obtain the token before passing to the getPosts() and getComments() methods. This adds unnecessary coupling and complicates the client code.

The paper mentions how such arguments are “content” and the client needs to cache them (for e.g. knowing about bearer token before it can call getPosts() method) in order to use the APIs:

“These kinds of arguments increase the complexity of a protocol. The protocol will be difficult to learn, as programmers must work out which arguments must be changed, and which must remain constant. This information is not explicitly represented in the protocol, and often not provided by standard documentation. The protocol will be difficult to use, as clients must cache constant arguments between sends and compute the values of slowly-varying arguments.”

This is where currying and partial functions shine. We could very well use currying and expose partial functions to the client code, making the authentication mechanism “invisible” to them:

Conclusion

• Quite often, the design can be simplified by “finding” additional objects. Strong emphasis on trying to “find” objects that part of the ubiquitous language. This makes sure we’re not introducing phantom concepts in the system.

• These patterns introduce another layer of indirection. While this can simplify understanding of objects/modules in isolation, at times it can interfere with ability to reason with the global design of the system.

• Since with all these patterns, additional objects and their construction are getting introduced, at times it can even introduce space and time complexity. Each pattern is to be employed with caution.

• Use Factory or other creational patterns when construction of these objects (Argument, Selector etc.) gets complicated. Using a creational pattern can greatly simplify the API for client.

This is a fortnightly newsletter in which I share interesting articles, talks, quotes from literature, comic strips 😊 and many such things that I found insightful around software and its makers.

Things this fortnight:

1. Serverless and Cost Duality.

2. What Is ChatGPT Doing… and Why Does It Work?

But first, we need to make this “Re-Org” rap as an official re-org anthem! 🙌

Serverless and Cost Duality.

Serverless often gets a bad rep in the industry. Mostly because if left untapped , it is very potent in drying out your entire capital. We also have commentaries on how teams are able to reduce costs by moving away from serverless.

"As a bootstrapped company, there was no way for us to come up with $72K.

By this time, I was well versed with Chapter 7 and Chapter 11 of Bankruptcy and mentally prepared of what could come next."

— Source

Forrest Brazeal has written about the risks of cloud billing citing horror stories of bleeding money across the three major cloud providers. Brazeal mentions having near real time billing and ability to have hard caps on cloud spend would really help mitigating the horror. Unfortunately, we're not there yet.

There are ways you can codify some controls in though, like limiting the concurrency limits of your lambda functions and being stringent about the memory limits. Yan Cui explains how AWS lambda pricing works and how you can optimise the costs. You can also use AWS Lambda Power Tuning tool which runs in your AWS account and helps you decide the optimum configuration of your lambda.

On the other side, serverless when used for prototyping is actually a huge cost saver. Infrastructure cost is not the only cost a team has to think about. There is development cost and maintenance cost. Every developer also comes at a cost. Add dedicated ops people to the mix and the costs start to rise non-linearly. At early stages of a company, there is also an opportunity cost. Miss the opportunity to land your product at the right time, and you may never recover from it. The ecosystem of serverless and various building blocks makes it easier to build and ship faster.

Subscribe now

Adrian Cockroft has a great short video explaining how serverless first is like building with lego blocks. Building with serverless components is “quick and cheap”. It shortens the feedback loop. If you’re trying to launch a new product, you get to launch sooner and test the waters before your competition does. But then why not start with monolith directly? Do you even need serverless at this time? Monolith first is generally a good advice, but there are a lot of nuances. I plan to cover this aspect soon, but in a different article to keep this one a bit shorter.

If everything goes right, your product will pick up more traffic, and will have more and more users coming in. You’ll start to notice the costs of lambdas are going higher and higher. You’ll start to think about bespoke optimisations. This is a right time to start taking out the concerned serverless components and start replacing them with bespoke services and deployments.

What Is ChatGPT Doing… and Why Does It Work?

ChatGPT, in simple terms, is doing a next word prediction.

The first thing to explain is that what ChatGPT is always fundamentally trying to do is to produce a “reasonable continuation” of whatever text it’s got so far, where by “reasonable” we mean “what one might expect someone to write after seeing what people have written on billions of webpages, etc.

So given an incomplete sentence: “The best thing about AI is its ability to…”, ChatGPT would rank the next set of possible words by their probabilities of occurrence from the training data (lots and lots of human produced text on Internet) and then choose a word to complete the sentence “randomly”. I’ve quoted “randomly” because given the same incomplete sentence, the completion wouldn’t be the same.

So how is it able to complete the sentences in so human-like manner? How is it able to write lengthy essays on a given topic in such seemingly meaningful way?

Stephen Wolfram’s What Is ChatGPT Doing... and Why Does It Work? is one of the best papers I’ve come across which describes the inner workings of ChatGPT from ground up.

Subscribe now

etc.

• Last week I published my summary and annotations of D.L. Parnas paper “On The Criteria to be used in Decomposing System into Modules”. To which Naveen Negi suggested a follow up read: Volatility based decomposition.

• The Value Flywheel Effect, by David Anderson is good read about how value chain mapping and serverless can compliment each other.

That’s a wrap!

Have good days! 🍀

Share

“If programmers sang hymns, some of the most popular would be hymns in praise of modular programming”

— D. L. Parnas, 1972 [1]

This week i read the paper “On The Criteria to be used in Decomposing Systems into Modules” written by D. L. Parnas in 1972. This article is a summary of the paper. I’ve put in my thoughts and annotations too.

What makes this paper a classic is that although published in early 1970s, the ideas presented in it about modularisation are still relevant in the modern development. One could even extend them in designing distributed systems like microservices. The paper is a very good case for good encapsulation, higher cohesion and lesser couplings wrt modules.

Abstract

“This paper discusses modularisation as a mechanism for improving the flexibility and comprehensibility of a system while allowing the shortening of its development time. The effectiveness of a "modularisation” is dependent upon the criteria used in dividing the system into modules.”

The paper’s major motivation is to explore “how to modularise effectively”. It posits that a good modularisation is not just an assembly of a runnable process, but more than that. It posits that modularisation enables independent development & testing. A good modularisation reduces managerial overhead, increases flexibility and comprehensibility (elaborated later) which in turns reduces development time. It is worth pointing out that Parnas defines “module” as “a work assignment unit rather than a subprogram”.

He adds:

“The modularisations are intended to describe the design decisions which must be made before the work on independent modules can begin.”

Now, this is not to be misinterpreted with big upfront design. But instead, having some directional thinking where your team envisions the software to get structured as. DDD and event-storming fall within this paradigm.

The paper begins by quoting Gauthier[2] that modularisation of code enables independent development and testing with little overhead:

"A well-defined segmentation of the project effort ensures system modularity. Each task forms a separate, distinct program module. At implementation time each module and its inputs and outputs are well-defined, there is no confusion in the intended interface with other system modules. At checkout time the integrity of the module is tested independently; there are few scheduling problems in synchronising the completion of several tasks before checkout can begin. Finally, the system is maintained in modular fashion; system errors and deficiencies can be traced to specific system modules, thus limiting the scope of detailed error searching.”

— Gauthier, et. al. Designing Systems Programs [2]

Parnas, while agreeing with the above, points out that little has been mentioned about the criteria to use in dividing a system into modules. Simply dividing a system into multiple modules doesn’t achieve the expected benefits of a good modularisation. So what are the expected benefits?

Expected Benefits of Good Modularisation

The paper posits that a good modularisation results in following three benefits:

1. Managerial: “development time could be shortened because separate groups would work on each module with little need for communication (and little regret afterward that there had not been more communication)”; — think about all the communication overheads, dependencies within and between teams etc when there are poor boundaries between different components of a system.

2. Product Flexibility: “it was hoped that it would be possible to make quite drastic changes or improvements in one module without changing others”; — A whole lot of literature viz. Agile, DevOps, Evolutionary Architecture, Incremental Development, SOLID, DDD etc., talks about this; the ability to increase “changeability” of a software. The greater this ability, the better outcomes for business.

3. Comprehensibility: “it was hoped that the system could be studied a module at a time with the result that the whole system could be better designed because it was better understood.” — This is really about maintainability of a software and ability to reason with it. The lesser cognitive load a developer has while trying to understand the existing system before s/he can start making changes, the better maintainability of a software is.

So, what should be the criteria of modularisation which sets out to achieve above benefits? This is what is explored in the paper.

Subscribe now

At the heart of the paper are two software examples - a KWIC Index Production System and a Markov Algorithm Translator. Parnas uses these two examples and gives different modularisation approaches for them, and then gives reasonings over which modularisation is better. I’ll pick up only the former example for the sake of this summary.

Background on KWIC Index Production System

The KWIC (or Key Word in Context) system was widely employed in libraries to make the text searchable. The system is based on the principle that the title of the document represents its contents. It is believed that the title of the document is one line abstract of the document.

A typical KWIC index system accepts an ordered set of lines, each line is an ordered set of words, and each word is an ordered set of characters. Any line may be "circularly shifted" by repeatedly removing the first word and adding it to the end of the line. The KWIC index system outputs a listing of all circular shifts of all lines in alphabetical order.

Let’s say we have a title of an article/book as “Prevention of diseases of wheat caused by insects”. The first step is to identify a set of keywords to be searchable. Let’s say those are (from the title above): “prevention”, “diseases”, “wheat” and “insects”. The program would accept these keywords along with the title. It interprets the above title then as “Prevention of diseases of wheat caused by insects” (the bolded words are just for illustration purposes. The program may choose alternate ways of recognising the keywords within a given title)

The second step is to circularly shift the sentence so that each significant keyword appears first with the rest of the sentence following it, which the preceding words wrapping at the end (circularly shifting). The outcome of this step would be following list of sentences (note that ‘/’ is used as an identifier of end of original title):

• Prevention of diseases of wheat caused by insects

• diseases of wheat caused by insects/ Prevention of

• wheat caused by insects/ Prevention of diseases of

• insects/ Prevention of diseases of wheat caused by

In the third step, the program would take the above list of sentences and arrange them in alphabetical order. So the output of this step would be:

• diseases of wheat caused by insects/ Prevention of

• insects/ Prevention of diseases of wheat caused by

• Prevention of diseases of wheat caused by insects

• wheat caused by insects/ Prevention of diseases of

With the above context in mind, let’s look at the two modularisation approaches that the paper illustrates:

Modularisation 1

In the first modularisation attempt, following modules are identified:

1. Input: accepts the input lines from a given medium and stores them in memory

2. Circular Shift: Called after the input module. Rather than storing all circular shifts in memory, it prepares an index which gives the address of the first character of each circular shift, and the original index of the line in the array made up by module 1. It leaves its output in memory with words in pairs (original line number, starting address)

3. Alphabetising: This module takes as input the arrays produced by modules 1 and 2. It produces an array in the same format as that produced by module 2. In this case, however, the circular shifts are listed in alphabetical order.

4. Output: Uses the output produced by modules 1 and 3 and outputs the result.

5. Master Control: Orchestrates the above modules.

Modularisation 2

In the second technique, following modules are identified along with their responsibilities:

1. Line Storage: It contains a number of functions defined on a “line”. For e.g. it contains a function to add a character at a specific position, another function to find the k-th character of a particular word in a given line, another function to set a k-th character of a particular word of a given line, and so on.

2. Input: Reads the input from input media and calls Line Storage module to store them internally.

3. Circular Shifter: This module is again defined in terms of functions, similar to module 1. For e.g. it contains function to get characters of a circularly shifted line.

4. Alphabetiser: It contains primarily two functions: one for setup and the other to get the index of the circular shift which comes i-th in the alphabetical ordering.

5. Output: This will give the desired printing of set of lines or circular shifts.

6. Master Control: Similar to its function in modularisation-1

You can begin to see the immediate differences in the two approaches, even though the algorithm of the two programmes remain identical in their “runnable representation”. The biggest differences are that modularisation-2 gets rid of data coupling completely and instead uses functional dependencies between the modules.

Comparison of the Two Modularisations

The paper takes a look at both the modularisations through the lens of the previously stated three benefits of modular programming. Although the two modularisations are almost identical in the way they run, they are significantly different in the way they can be comprehended, maintained and developed.

“I [Parnas] claim that the systems are substantially different even if identical in the runnable representation. This is possible because the runnable representation is used only for running; other representations are used for changing, documenting, understanding, etc. In those other representations the two systems will not be identical.”

Changeability

“There are a number of design decisions which are questionable and likely to change under many circumstances.”

1. Change in input format. In this case, both the modularisations would need a change. But the change in both is confined only to one Input module.

2. The decision to have all lines stored in memory. For larger indices it may prove inefficient and impractical to keep all the lines in memory while processing. This change would result in changes in every module of modularisation-1. In case of modularisation-2, the change is still confined to only Line Storage module.

3. The decision to change the underlying data structure in which the lines are stored. This yet again causes change in all the modules of modularisation-1 but because modularisation-2 abstracts the storage through Line Storage, it is able to confine the change only to one module.

4. The decision to make an index for circular shifts, rather than storing them as such. This change would be confined within the Circular Shifter module in modularisation-2. In modularisation-1 though, apart from Circular Shift, even the Alphabetiser and Output modules would accrue change because of data coupling.

The theme is clear: better modularisation is able to better confine the changes caused due to future design decisions. This makes modularisation-2 more adaptable to change.

Independent Development

Contrasting between the two modularisations, the paper states:

“In the first modularisation the interfaces between the modules are the fairly complex formats and table organizations described above. These represent design decisions which cannot be taken lightly. The table structure and organisation are essential to the efficiency of the various modules and must be designed carefully. The development of those formats will be a major part of the module development and that part must be a joint effort among the several development groups.“

While the overhead of design decisions is paramount in modularisation-1, modularisation-2 results in lesser development overhead:

“In the second modularisation the interfaces are more abstract, they consist primarily in the function names and the numbers and types of the parameters. These are relatively simple decisions and the independent development of modules should begin much earlier.”

Comprehensibility

In modularisation-1, there is little to no possibility of local reasoning. You cannot understand how Output works without understanding what Circular Shift and Alphabetiser do the underlying storage structure, which in turn can’t be reasoned without taking Input into consideration. This increases the Cognitive load on developers, as they can only understand the system as a whole, not through its different parts.

In modularisation-2 however, because of clear responsibilities of modules, and the fact that they encapsulate the functionalities within themselves respectively, it is relatively easier to reason with and hence improve/refactor and test individual modules without having to comprehend the entire system at once.

Subscribe now

The Criteria

Finally, the paper highlights and contrasts the different criteria used in the two modularisation. It states that modularisation-1 is mostly influenced by a “flowchart” of sorts of the programme:

“In the first decomposition the criterion used was make each 'major step' in the processing a module. One might say that to get the first decomposition one makes a flowchart.”

Parnas states that this style of modularisation (modularisation-1) is most common, mainly because that’s how programming is generally taught. My confirmation bias found me agreeing with it completely.

“This is the most common approach to decomposition or modularization. It is an outgrowth of all programmer training which teaches us that we should begin with a rough flowchart and move from there to a detailed implementation.”

He contrasts this with the encapsulation approach that modularisation-2 takes:

“The second decomposition was made using "information hiding" [encapsulation] as a criteria. The modules no longer correspond to steps in the processing. The line storage module, for example, is used in almost every action by the system. Alphabetization may or may not correspond to a phase in the processing according to the method used. Similarly, circular shift might, in some circumstances, not make any table at all but calculate each character as demanded. Every module in the second decomposition is characterised by its knowledge of a design decision which it hides from all others. Its interface or definition was chosen to reveal as little as possible about its inner workings.”

Conclusion

The paper puts forth a strong case of encapsulation as a first class citizen when breaking down the system into modules. It strongly recommends that while it is important to consider the “flow” of the software, it is a poor theme on which the system should be solely be modularised upon.

“We have tried to demonstrate by these examples that it is almost always incorrect to begin the decomposition of a system into modules on the basis of a flowchart.“

It is important to take a Systems Thinking approach and analyse how it affects other teams, think about plausible future enhancements in design with respect to your problem domain, think how work will get hashed out and how to do it with least overheads.

“We propose instead that one begins with a list of difficult design decisions or design decisions which are likely to change. Each module is then designed to hide such a decision from the others. Since, in most cases, design decisions transcend time of execution, modules will not correspond to steps in the processing."

Notes:

• [1]: D. L. Parnas. 1972. On the criteria to be used in decomposing systems into modules. Commun. ACM 15, 12 (Dec. 1972), 1053–1058. https://doi.org/10.1145/361598.361623

• [2]: Gauthier, Richard and Stephen Pinto. 1970. Designing Systems Programs. Prentice-Hall, Inc.

This is a fortnightly newsletter in which I share interesting articles, talks, quotes from literature, comic strips 😊 and many such things that I stumbled upon.

Let’s jump straight in.

Three things this fortnight:

1. The importance of having slack in your plans

2. Many bad takes on Prime Video’s move to monolith from microservices

3. Cryptography: History and Basics

As always some relevant links and recommended reads are present in the notes section at the bottom.

But before all, my favourite comic this fortnight. Courtesy @vincentdnl

The importance of having slack in your plans

Teams often get accustomed to their iteration plannings. So much so that they like to optimise for every bit of capacity that they have. If the team’s velocity is 15 stories points on average, it will definitely load up 15 points. At times it would throw in another 3-5 points as a stretched effort, a concept which is as antithetical to flow as possible.

“A plant [system] in which everyone is working all the time is very inefficient”

— M. Goldratt, The Goal

Although setup as a fiction about a manufacturing plant, the book explains Theory of Constraints better than most others. I particularly like the above quote because it highlights the importance of thinking in systems, as a whole.

Optimising for individuals != Optimising the system

It is intuitive to think that if we optimise for individual resources, the system as a whole will be efficient. If we pull in as much stories as much as we have capacity in this sprint, surely we are efficient, isn’t it? Well, not quite. We’re considering capacity as a sole measure. We’re optimising for one (local) measure.

In reality local optimisation doesn’t always lead to optimisation at a global level. As shown in the above pseudo-graph, optimisations done for local or neighbouring problems might result in improvement of the local, a part of the whole system. It doesn’t necessarily lead to the overall optimisation that the system is capable of achieving. When we optimise only for capacity, other BAU tasks which demand capacity of the team take a hit.

Subscribe now

A software team has many more responsibilities than just writing new code for that prospect shiny feature which is hypothesised to earn that extra revenue. It has to look after the cruft in codebase, optimise the load balancing of resources, monitor the deployments and respond appropriately to any unavailabilities of the services, automate that manual task which takes forever and a lot of cognitive load to execute, as so on. Many of these tasks are unpredictable and yet of quite grave importance. How do you plan for these unpredictabilities? — By adding slack to your plan!

In his article Slack, Martin Fowler talks about how you can introduce slack in your plan:

“A good way to introduce slack into planning is to use it to cope with the inherent uncertainty of planning. A team that averages 20 stories per iteration won't complete exactly that number every iteration. Instead we'll see a range: say from 15 to 22. In this situation the team can plan at their lowest consistent number (15) and treat the additional time as slack.“

— Martin Fowler

Key takeaways on how to utilise the slack time:

• The most obvious thing to do is to tackle additional stories. If the team is done with planned stories, it is good to pull in more work and get more done on an as-possible basis.

• Respond to urgent requests. You may choose to have a dedicated “developer on support” person on rota basis. While a dedicated person attends to urgent requests (from end-user, a dependent team, an external vendor) it lets others concentrate on the feature.

• Pair on a story and get it to done. Experiment pairing if you have a pull-request based workflow. See if pairing made it any better. At times your QA team might be overwhelmed with the number of stories to be tested, try giving a hand.

• Automate that mundane manual tasks which takes a lot of steps to remember, but is mostly repeatable.

• Explore how can the builds be made faster and more reliable. Maybe the test setup of your functional tests is flaky. Maybe the assertions themselves are flaky. Maybe there are some tests which should be there but aren’t.

• Revisit some of the hurdles which are impeding developer productivity, try to solve them.

• Pen down that last important architecture decision that the team took but couldn’t get time to document it.

• Improve upon your API document and write down contract tests which have been pending since that new team started consuming your services.

• Update the dependencies. Make sure no vulnerabilities are present. Better yet invest time in automating the security within your build pipelines.

The above are just some of the examples; you may have others depending on your context. But the key thing to remember is without having slack, these activities can’t be picked up. While some are BAU activities, some cause impediments and others slow down the throughput of the team over time. It is important to prioritise and revisit.

Many bad takes on Prime Video’s move to monolith from microservices

Recently, Amazon Prime Video’s team published Scaling up the Prime Video audio/video monitoring service and reducing costs by 90% and the internet went wild with their takes on it. Quite a lot of them, quite bad!

One of the good ones I liked is by Adrian Cockraft.

“The Prime Video team had followed a path I call Serverless First, where the first try at building something is put together with Step Functions and Lambda calls. They state in the blog that this was quick to build, which is the point. When you are exploring how to construct something, building a prototype in a few days or weeks is a good approach.

The problem is that they called this refactoring a microservice to monolith transition, when it’s clearly a microservice refactoring step and is exactly what I recommend people do in my talks about Serverless First”

— Source

His take on why microservices are earning a bad rep:

“In contrast to commentary along the lines that Amazon got it wrong, the team followed what I consider to be the best practice. The result isn’t a monolith, but there seems to be a popular trigger meme nowadays about microservices being over-sold, and a return to monoliths. There is some truth to that, as I do think microservices were over sold as the answer to everything, and I think this may have arisen from vendors who wanted to sell Kubernetes with a simple marketing message that enterprises needed to modernize by using Kubernetes to do cloud native microservices for everything.”

— Source

Subscribe now

Brief Concept and History of Cryptography

Cybersecurity is a vast and burgeoning field. One of the tenets of this area is privacy. Privacy is all about algorithms and best practices to prevent unauthorised users’ from accessing the data that doesn’t belong to them. The article Cryptography: Brief Concept and History by Ashok Hegde is a recommended read.

That’s a wrap!

Thank you for reading and being part of the community. If you’d like to suggest topics to cover or if you’d like to write for us, do tell us at newsletter@partnest.io.

Have good days! 🍀

In the previous article, we visited why it is important to focus on getting valuable feedback from metrics to focus on improvements and not letting the measures get used as a lever to influence other’s behaviours. In this article, we’ll visit why it is necessary to have a balanced metrics regimen.

Sin #2: Unbalanced Metrics

From the paper:

“The second sin has to do with the need for a balanced metrics regimen. The need for this is fairly readily apparent. If you focus on one aspect of performance (say productivity), other aspects will go down (quality, customer satisfaction, etc.).”

Continuing our wellbeing theme from earlier article, just measuring and achieving 10,000 steps in a day certainly won’t be enough to be qualified as a full-proof step towards your overall wellbeing. What if you ate 2 chocolates on the gloomy Monday? Or went for a couple of craft beers with your friends on Friday to blow off the steam? You would of course need to keep a measure of your diet, your stress levels, sleep habits etc. as a bare minimum to ascertain that you’re headed towards a better living.

Intuitively we already know, and the fact is also backed by research that developer satisfaction is important for productivity.[1] While the team might seem productive, but their satisfaction might have taken a dip. Maybe they’re burnt out because of the extra work load they had to endure to ship that “important” delivery. Maybe this is a pattern that keeps repeating. This results in short-term productivity gains and long-term loss in work-satisfaction and even leads to greater than optimal churn.

Moreover, can we, with certainty, ascertain that just because the team is productive, the overall quality of the product is also high? Or that customer satisfaction is also maintained well? I bet we can’t. We need observability in other areas too. Hence, the importance of having a balanced regime of measures.

Larry proposes a well rounded measure from four key areas:

“I recommend that you launch a metrics feedback effort with at least one measure from each of these four areas:

1) Do it fast [Responsiveness]

2) Do it right [Quality]

3) Do it when expected [Predictability], and

4) Keep doing it. [Employee Satisfaction]”

“These outcomes dimensions form the foundation of the Software Development Performance Index (SDPI), which quantifies insights about development work and provides feedback on how process and technology decisions impact a team’s performance. Of the metrics in each of the quadrants shown above (productivity, responsiveness, quality, predictability, customer satisfaction, and employee satisfaction), the first four can be extracted from ALM tools […]. The fifth and sixth metrics we recommend you gather via lightweight (maybe even single-question) surveys.”

Do It Fast (Responsiveness)

In the current markets, business model lifetimes have only grown shorter. Naturally, the advantage is with the organisations which are able to adapt changing business goals through software. Inevitably, to be able to ship features and enhancements as quickly as possible, per the changing business models, gives a steep advantage and leverage to make the most out of the dynamic markets.

Measures like lead times and deployment frequencies can give an indicative view of how fast the software can be shipped. A Time in State chart shows you how long your tasks stay in each step of your workflow on their journey to completion. It is a mighty chart to have in hand when you want to understand how quickly your tasks are completed and whether there are bottlenecks in your workflow.

Do It Right (Quality)

What is the point of shipping it faster, if that is not what the customer finds useful enough? The quality of software is not just about less buggy features, but also about the quality of UX and data flows. A software that doesn’t live up to the expectations of its users, or worse makes their job even more cumbersome, might as well never had been made in the first place — no matter how quickly the team was able to ship it and how early the business was able to capture the initial markets.

While the quality in terms of rework can be tracked via majority of leading tools, customer satisfaction can easily be captured with surveys like Net Promoter Score. Change failure rate and Time to restore service are great indicators of stability too.

Subscribe now

Do It On Time (Predictability)

Business strategies are future dated. They are seldom siloed to a single business function. It’s highly likely that there is a marketing team working on how to market the product to the target customer, a sales team preparing the demos and sales pitch, a finance team allocating necessary funds for these related activities. All these activities require coordination to effectively mitigate the dependencies. The predictability of being able to ship the software when it was needed to be is of sheer importance.

Although, you need to be cautious about falling for the “estimation is deadline fallacy”. It is important to have accuracy over precision and to incorporate probability when giving forecasts.

Keep Doing It (Employee Satisfaction)

Research shows that developer satisfaction is important for productivity.[1]. It is imperative to build sustainable environments. Not just a balanced work-life, but developers need an environment which fosters learning and knowledge sharing, and gives the team a sense of self-actualisation; and does so repeatedly in a sustainable fashion.

These measures are more qualitative in nature. In fact Spotify uses a pure qualitative measure for its teams — Spotify Squad Health Check Model

In the next article in this series, we’ll discuss Sin #3 — Believing that metrics can replace thinking. We’ll look at why are qualitative measures as important as the quantitative ones, and how well they compliment each other.

Notes:

• [1]: Graziotin D, Wang X, Abrahamsson P. 2014. Happy software developers solve problems better: psychological measurements in empirical software engineering. PeerJ 2:e289 https://doi.org/10.7717/peerj.289

• If you manage broader goals in your organisation it is worth looking into well rounded measures for organisation - The Balanced Scorecard

• What is DORA Metrics and the Research behind it which proves why these metrics work - Accelerate, Measuring DevOps performance.

• DevOps Trends in industry and how your team compares to others - State of Devops Report 2022

My favourite comic of the fortnight:

The “Knowledge is an Investment” metaphor

I enjoy re-reading some of the old classics. Mostly, when I get reminded of a vague idea that I fuzzily remember having read in one of the books. Most of the times such triggers are some or the other conversations i’ve had with interesting people.

I remember I’ve read it somewhere. I find my eyes racing along the bookshelf, making a mental note for the n-th time that I need to clean it. I digress.

I’m feeling quite confident and take out The Goal, by M. Goldratt. Flipping through the pages, hoping that I highlighted it, I find it quite sooner than expected.

It reads…

“Money for knowledge has us [managers in a manufacturing company] stumped for a while. Then we decide it depends, quite simply, upon what the knowledge is used for. If it's knowledge, say, which gives us a new manufacturing process, something that helps turn inventory into throughput, then the knowledge is operational expense. If we intend to sell the knowledge, as in case of a patent or a technology license, then it's inventory. But if the knowledge pertains to a product which UniCo [the company] itself will build, it's like a machine to make money which will depreciate in value as time goes on. And, again, the investment that can be sold is inventory; the depreciation is operational expense.”

— M. Goldratt, The Goal

The Goal posits that the goal of any [for-profit] organisation is to make money. And that all other initiatives are a means of achieving this goal. An organisation does so by increasing the throughput, while simultaneously reducing both its inventory and operational expenses.

In other words, throughput is the money coming in. Inventory is the money currently inside the system. And, operational expense is the money we have to pay out to make throughput happen.

While it is relatively simpler to visualise this in a manufacturing setup (the original premise in which the book is written), for knowledge-work like in software engineering it is relatively complex to define.

However, “knowledge is an investment” as a metaphor is pretty befitting. In [another classic] The Pragmatic Programmer, Andy Hunt and Dave Thomas take the investment metaphor and describe keeping up to date with your knowledge as “Managing a Knowledge Portfolio”. And just like with an investment portfolio, their advise in managing your knowledge portfolio is to:

• Invest regularly. “Even if it’s just a small amount, habit itself is as important as the sums”, they suggest. Try to read more often than you do now. Go attend meetups. Follow some good twitter and linkedIn (*swallows a big lump) accounts for byte size knowledge streams.

• Diversify. Your value grows multifold with more diverse things that you know. Don’t just stop at learning in depths about the tech that you’re working on currently. Explore. Get curious about what other tech is being employed in your project, or in your organisation, or around other companies that you find interesting. Networking and attending good meetups helps a lot in this regard.

• Manage risk. “Technology exists along a spectrum from risky, potentially high-reward to low-risk, low-reward standards. It’s not a good idea to invest all of your money in high-risk stocks that might collapse suddenly, nor should you invest all of it conserva- tively and miss out on possible opportunities. Don’t put all your technical eggs in one basket.”

• Buy low, sell high. “Learning an emerging technology before it becomes popular can be just as hard as finding an undervalued stock, but the payoff can be just as rewarding”. Being an early adopter has a great payoff! But remember to manage risk.

• Review and rebalance. Software engineering as an industry is quite dynamic. Your next project might be very different from your current one. Other companies might have started looking into some other tech. The framework you were proficient in might be at the stage of its retirement now. Always review your knowledge and keep abreast.

A few helpful trends to watch out for in this regard:

• Thoughtwork’s tech-radar

• Gartner’s trends

• State of DevOps reports

The above by no means is an exhaustive list. But it should be enough get started with.

Subscribe now

QWERTY keyboard and Path-Dependence

Path dependence is the dependence of economic outcomes on the path of previous outcomes, rather than simply on current conditions. Many a times, the choices made earlier constrain later events or decisions. In a path dependent process, “history matters”.

Take for example, a town that is built around a factory. Ideally, a factory is located at a distance away from residential areas for various legitimate reasons — primary ones being safety and costs. However, factories are often build first, and then the worker’s homes and amenities are built close by — resulting in a town built around a factory. Bhilai, a city in Chattishgarh, India, is a great example of such a development; The foundation of which was laid in 1955 when the Indian government signed a historic agreement with the Soviet Union to establish a steel plant near the village.

Another example of such path-dependent decision is the QWERTY keyboard. The arrangement of letters on QWERTY keyboard was in fact designed to slow down the typing and to avoid jamming of ribbons in a typewriter. People were trained on a QWERTY keyboard. Their habits became so sticky that despite other more efficient keyboard designs like Dvorak, QWERTY continues to persist dominantly in the market.

Not investing in automated tests is a “False Economy”!

Writing tests (even if you’re not doing TDD; although you should whenever possible) is one of the best investments that you can make for your team. At some point, and quite early IME, not having tests begins to cost much higher than what you saved by not writing them in the first place — aka False Economy!

Quite hilariously, within a few days of posting this tweet, I had to download my account statement from HDFC bank’s app. And what do I find?

Stalemate!

I’m very confident this worked earlier and is buggy only recently. So what changed? — Enhancements without the safety-net of tests!

Quite interestingly, having no-tests leads to further path-dependent decisions. Your manager might decide to get more manual QAs, or maybe the release will sit longer in UAT now — all in turn increasing inventory and reducing throughput — an antithesis of Continuous Delivery!

That’s a wrap! Thank you for reading.

Wishing you great days ahead! 🍀

With the advent of data and data analysis, and an immense amount of inspiration from manufacturing industry alone, software engineering continues to employ a plethora of measures to seek the answer to “how efficiently do we build our softwares”? Agile pledges to continue to uncover better ways of writing software. And with uncovering of “better ways” arises an innate question of “Are we doing better and if yes, how do we quantify it?”

Agile itself has been interpreted in various ways, Larry writes —

“Agile can be perceived in different ways: as a manifesto of values or list of principles, as an emphasis on collaboration, as a repackaging of spiral and iterative development concepts, or as the overarching prioritisation of adapting to change over following a plan. I’ve been around software development long enough to see several of these shifts in perception and I’ve come to think about all such shifts as reactions to environmental changes, similar to what occurs in nature.”

With newer ways of working and emphasis on collaboration and early feedback, a lot of qualitative measurements were drawn in. But qualitative measurements are not effective, alone, in trying to collaborate with business owners. You can't walk upto your CEO and say “We should be pair programming as it yields better quality software as per our team”. This is (usually) not enough. You have to quantify it for them. They know better that just a qualitative feedback is not enough to take critical decisions. You have to bring in “Pair programming reduced the defects in software by 15% (hypothetical figure) last quarter, we should do it more often”. This lead to discovering of quantitative measures, clubbed with qualitative measures, to be able to seek the answer to “Are we really doing better than before?”.

Data analysis is like nuclear fission — use it well and it’s an evergreen source of valuable insights, use it wrong and it will cause havoc. As much as data can help in better decision making, it can also be evil and lead to malignant behaviour in teams and organisations. Larry identified that with data, there needs to be a necessary perspective shift in order to be use data as a boon, not bane.

“I have been working hard to identify the the key perspective shifts necessary to successfully introduce measurement into an Agile development environment, and I’ve coined the phrase, “Seven Deadly Sins of Agile Measurement” to capture this learning.” — Larry Maccherone

Let’s look at the first sin:

Sin #1: Using measurement as a lever to drive someone else’s behaviour

“If feedback emphasis is key to the success of Agile, the key to effective Agile measurement is to think of measurement in terms of feedback, not as the traditional lever to motivate behaviour. Using measurement as a lever often devolves into keeping score, which is where the dark side of measurement starts.” — Larry Maccherone

There is a subtle difference between a feedback and a lever:

“Feedback is something that you use to improve your own performance not to influence other’s behaviours. The difference is in how you use the measure more than the measure itself.”

Let’s take the example of your smart watch; that you wear every day. You need to check upon your daily steps count to make sure you’ve been active enough. You need this feedback to improve your overall wellbeing. You would never cheat on these figures with yourself. If you’ve done lesser steps than you aimed for, you will try to do more tomorrow. If you did more steps, you will reward yourself and ask “Can I do even better?”. Now imagine your company is “celebrating a wellbeing month”. To help spread the awareness and healthy habits, the company decides to reward the employee who does the maximum number of steps over the period of the month. Will you cheat now? Quite a few would — they would tie up the smart watch to their dogs, try ceiling fans or even put them in drier to maximise the steps. Btw these solutions are not made up, people have actually employed them. Anyways, can you see how people’s behaviour got influenced? Even when the intention of the company is absolutely pure and selfless. This is how the same data measurement can either behave as a feedback, or as a lever.

Let’s take the example of an average cycle-time plot of user stories presented in the paper. Every dot represents a user story, the higher the dot is on the plot, the longer it took for the user story to get delivered. The red line denotes the “control line”, meaning anything higher took more than ideal time to get delivered. The red dots represent such outlier user stories.

Larry mentions how such charts can lead to gamifying of the system and hiding the actual data which the team embarked in the first place to seek.

“What’s going to happen the next time you show this chart? There probably will be fewer red dots, but why? I’d like to think it will be because people have deeply analysed their process and made necessary changes, blah-blah-blah ... but what’s more likely is that they will just game the system to make sure they don’t have any red dots. Maybe they’ll split stories artificially instead of where they deliver value. This is bad because it’s wasteful and doesn’t improve the process; but what really hurts is that you’ve now hidden data from yourself. You’re making decisions with a distorted picture of reality.”

Replace the above with any similar metric, say your sprint-burndown chart which has that “ideal burndown line” for that matter — the result is going to be similar.

There are three laws at play now:

Measures tend to be gamed when used for target-setting — Campbell’s Law

The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.

The team did not achieve a better lead-time. Instead they gamed it to achieve the target (to be below the red line). Splitting a story artificially into multiple stories doesn’t deliver more value to the customer. Neither does by increasing story points to game your velocity. These are false positives.

When a measure becomes a target, it ceases to be a good measure — Goodhart’s Law

The cycle-time measure has become a waste now. What the team set out to achieve has gone in vain. Instead they are running after a pseudo target — to not let their stories cross the red line. In doing so, the actual data has been hidden from the team. The team’s opportunity to discover the various constraints in the system is lost. With this lost opportunity, the team stands little chance to improve their processes and channel their efforts productively.

Individuals will modify an aspect of their behaviour in response to their awareness of being observed — Hawthorne Effect.

Upon being held accountable for the outliers in lead-time plot, the individuals will naturally try to protect themselves. They would give more story points to the story. They would resist in trying new things, say to introduce a new testing strategy or discover a better design pattern — because they need to deliver fast. Do you think the team stands a fair chance of a good introspection, let alone improvement?

Subscribe now

Inspire Pull and Resist the Push.

Larry states an alternative:

“As an alternative, I propose this chart:

It’s conceptually very similar to the first chart. The primary difference
is the lack of a red line and red dots. This visualization is designed to allow teams to explore their evidence, enabling learning and improvement. You can hover over a dot and get the details about what happened, which should enable discussion. You can talk about probabilities to help gauge risk. You’ll see that 95% of all stories finish in 28 days; maybe that will help you make a service-level agreement commitment.

So, the heavenly virtue here is to inspire pull and resist the push. Use metrics as feedback to improve yourself: never as a lever to alter someone else’s behavior.”

The above is a Time in State chart which shows you how long your tasks stay in each step of your workflow on their journey to completion. It is a mighty chart to have in hand when you want to understand how quickly your tasks are completed and whether there are bottlenecks in your workflow.

Software development has lots of steps and handshakes in between. Typically, a story would get groomed and would be ready for development. Developers will work on the story and (usually) raise a PR for the changes to get merged. One or two reviewers would review the code and sometimes ask for a bit of changes to be made. After the changes, the story would be tested by the QA team. If found acceptable, the build would be shipped to production, if not it will have to be sent back to development for rework.

Every step is a handover, from one person (or a team) to another. Every handover has a wait time and a processing time — imagine every step as a processor which takes x amount of processing time to work on the item and while it is working, there would be work items that will wait for y units of time in the queue to be eventually picked up for processing. The overall cycle time would be a summation of such x(s) and y(s). Every now and then one or more such handovers become a constraint in your system. It is more important to focus on the constraint, rather than trying to locally optimise (or gaming things).

A good metric helps uncover such constraints. A good metric helps team retrospect and think. A good metric helps supporting or refuting a hypothesis. A good metric helps in making improvements and increases flow efficiency.

In the next article in this series, we’ll discuss Sin #2 — Unbalanced Metrics. We’ll look at why is it important to have a balanced metrics scheme. And why it is imperative to measure other aspects of software engineering — not just productivity, but also customer satisfaction, quality, employee satisfaction etc.

Notes

1. Theory of Constraints - The Goal, Critical Chain, The Unicorn Project

2. Lean Methodology - The Lean Startup

3. Flow Efficiency - Hakan Forss: Flow Thinking

My favourite comic of the fortnight:

People cannot have a fruitful discussion on topics that are part of their identity. — Paul Graham

Paul writes that religion and politics eventually become part of the identity of people. In such cases the discussion ceases to be about the topic at hand and instead becomes about people’s identities themselves. At this point of time, the discussion ceases to be meaningful for either of the participants.

“As a rule, any mention of religion on an online forum degenerates into a religious argument. Why? Why does this happen with religion and not with Javascript or baking or other topics people talk about on forums?

What's different about religion is that people don't feel they need to have any particular expertise to have opinions about it. All they need is strongly held beliefs, and anyone can have those. No thread about Javascript will grow as fast as one about religion, because people feel they have to be over some threshold of expertise to post comments about that. But on religion everyone's an expert.

Then it struck me: this is the problem with politics too.

[…]

I think what religion and politics have in common is that they become part of people's identity, and people can never have a fruitful argument about something that's part of their identity. By definition they're partisan.”

— Keep your identity small, Paul Graham.

Each and every one of us has witnessed or (and) been part of an opinionated discussion which doesn’t seem to head towards any outcome. The more strongly we hold our opinions, the more they embed themselves in our identities. An attack on our opinion seems like an attack on ourselves.

The key to have a meaningful discussion, then, is to keep your identity small.

The most intriguing thing about this theory, if it's right, is that it explains not merely which kinds of discussions to avoid, but how to have better ideas. If people can't think clearly about anything that has become part of their identity, then all other things being equal, the best plan is to let as few things into your identity as possible.

— Keep your identity small, Paul Graham.

Spotify’s Squad Health Check Model

Spotify uses a qualitative and human-centric approach towards improvement and gaining insights into “how are their teams doing?”.

Key Insights:

• Qualitative data on a squad’s health is gathered through face to face discussions; and online-surveys are avoided. The discussions help in gaining meaningful insights and context, which often get missed in online surveys.

• Such a model works only when there is high-trust, curiosity and psychological safety within the environment.

• The data is collected through a lean model with the help of their health-indicators cards called “Awesome Cards”. The cards contain an example of “what good looks like” and an example of “what crappy looks like”. Each card stands for an aspect of work that the team is interested in learning more about, for e.g. “Ease of release”, “Health of codebase”, “Delivering Value” etc.

  

• Results, when put collectively for all squads gives an easy reflection on how the company as a whole is doing.

  

“It shows how 7 different squads in a tribe see their own situation. Color is current state (green = good, yellow = some problems, red = really bad). Arrow is the trend (is this generally improving or getting worse?).

Stare at it for a minute, and you start seeing some interesting things:

Scan each column, and you see some major differences between squads. Squad 4 is happy with just about everything. Squad 2 is having lots of trouble, but most things are improving.

Scan each row, and you see systemic patterns. Every squad is having fun (and it’s even improving)! Motivation is apparently not a problem at all. But the release process is problematic, and the codebase is overall in bad shape. Over time, that will probably reduce the Fun as well.

Scan the overall picture, and you see that just about every arrow is up, only two down arrow in the whole picture. That means the improvement process (the most important process of all) seems to be working.”

— Spotify

• Each team itself is responsible and encouraged to modify the model, supporting collective autonomy within squads and how they choose to learn about their work.

• Health-checks are designed to have no incentives, in order to prevent them from getting gamed.

• Follow ups with teams to gauge the usefulness of the model for them and feed back the insights towards improving the model, thus fostering a virtuous cycle.

Subscribe now

Watch Spotify’s latest talk on Squad Health Checks:

Functional Design Patterns by Scott Wlaschin

This is one of the best talks out there which introduces functional programming paradigms and patterns without getting into [too much] theoretical concepts about Category Theory.

A highly recommended watch for folks trying to learn FP patterns. Equally recommended as a leisure watch for seasoned FP and Category Theory enthusiasts — you’ll appreciate the simplicity of Scott’s explanations.

Do explore his website for more interesting articles and talks.

Thank you for reading. This is a public and free newsletter, do share it with your friends if you liked it. If you have something you’d like to share, do write to us at newsletter@partnest.io.

Share

There are only two hard things in Computer Science: cache invalidation and naming things.

— Phil Karlton

What is the goal of naming things in software and naming them “right”? — To reveal the clear intent. By intent, we mean the name of the variable, or function, or module etc., correctly represents its purpose. A name which reveals its intent clearly reduces cognitive load on developers’ minds.

Naturally, naming is neither a one step process nor an individual’s exercise. It is a team game and as @arlobelshee points out it is an iterative process, with each iteration your naming approaches a supple design (DDD).

The intent, however, is not supposed to be just clear to developers, but to the entire team. A typical team would be cross functional — it would not just have engineers, but also domain experts, marketing, and product folks. The entire team should use one language — the ubiquitous language.

In one of the large scale projects which I was a part of, we were developing a business “proposal calculator” for a large bank. The “proposal calculator” was supposed to run 1000s of calculations and tell whether a proposed corporate deal was beneficial for the bank or not. Long story short, naturally there were a lot of financial terms that were part of daily conversations. PnL, collaterals, amortisation to name a few.

A lot of such terms were used as their acronyms, instead of their full forms, by the bankers (the domain folks) in their own conversations and also when they referred to such calculations to us (the developers). So they would not say “Profit and Loss”, but “PnL”. They would not say “Return on Risk Weighted Assets”, but “RoRWA” (pronounced row-rah).

Early in our design phase we took a conscious decision of going ahead and using the acronyms themselves instead of their long forms. In hindsight it was a good judgement call by the team, especially when all literature around clean code was advising otherwise. The literature would advise to use pronounceable names and only use acronyms when they were widely known — highly debatable and very contextual. Turns out RoRWA wasn’t that widely known amongst developers atleast.

Imagine if we had not — When developers would have had conversations amongst themselves, we would use “Profit and Loss”. But whenever we would be in meetings with the domain folks, we would only hear “PnL”. Imagine the fiasco when “Return on Risk Weighted Assets” would come into such conversations. Moreover, our use-cases which were written primarily by domain people would only contain such acronyms, not their full forms. Not only we would have added unwanted cognitive load upon ourselves, since we would always be playing this ontology game of mapping the acronyms with their full forms, but we would have also made the conversations difficult, because the domain folks were more comfortable with acronyms themselves.

It was difficult to onboard new developers at first. Imagine yourself hearing PnL vs hearing “Profit and Loss”, especially when you do that have acquaintance with financial terms. Your mind can make some sense out of “Profit and Loss” in the very first go — because basic mathematics — but won’t comprehend PnL naturally. This was the case with some common-sensical terms, but “Return on Risk Weighted Assets” would still not make sense. The team came up with a glossary. It helped heaps, not just for the new developers, but also for the ones who had been on the project for considerable amount of time.

We made a conscious effort to keep the glossary as a living document — forever changing and changing as frequently as was the need. We would refer to it whenever we had a doubt, added questions to it and added lots of examples. We also shared this glossary with the bankers and they generously added enough information to it. At times they added more information — just to make us, the financial-rookies understand — than was required to be able to successfully model it in our code. But it eventually gave us the comfort and cognitive ease which we needed, but didn’t know we did.

We are in constant awe of the innovation culture at Google. We all revere the autonomous teams at Netflix. We all get inspired by the result-driven nature of the small (pizza sized) teams that Amazon created. We revere such companies so much that we start attempting to replicate (or at-least in theory) their mannerisms in the hope to be able to replicate the “magic” that such organisations were able to create for themselves.

However, trying to be like a Google, Netflix or an Amazon (or any other organisation for that matter) is trying to trivialise the complex adaptive system that an organisation is. An organisation is, ultimately, made up of many individual components which are hard to replicate and difficult to reason with a reductionist approach.

“Which part of the airplane is responsible for its flight?” : Emergence

To further the reasoning, it is important to recall the concept of Emergence.

When systems as a whole function in ways that we cannot predict by looking at their individual parts — This is emergence.

Emergence occurs when an entity is observed to have properties its parts do not have on their own, properties or behaviours that emerge only when the parts interact in a wider whole. In short, one cannot understand the systems with emergent properties by reducing them to their components.

An airplane can be conceptualised as being composed of various engineered parts. Each individual part is responsible for handling a single function, which it is designed for, and handling it well. But it’s not a single component (or two) that is responsible for the flight itself. It’s the complex interactions of those individual parts, as a whole, orchestrated by a fine pilot that makes it possible for the hundreds of tonnes of machinery to be able to fly. So, can one answer: “Which part of the airplane is responsible for its flight?”.

Emergence is all around us. While hydrogen and oxygen exhibit their own properties, the property of wetness is an emergent property of water (and not exhibited by either hydrogen and oxygen individually) which only emerges when hydrogen and oxygen combine in a specific manner.

Emergence, however, is not alike in all the complex systems. Given enough computational power it is possible to simulate a weak emergence wherein systems exhibit functions identifyable by fixed rules. However, doing so for strong emergence is impossible.

We can construct a computer simulation of the flocking behaviour of a group of birds (weak emergence) but not of the interplay of cells in our brains that creates consciousness (strong emergence)

— an excerpt from “The Great Mental Models”, by Rhiannon Beaubien and Shane Parrish

Building upon, individually, humans exhibit certain characteristics which are unique to themselves. The emergence lens suggests that the cumulative actions of a group of people is bigger and non-intuitive while considering the individual people’s characteristics themselves. These groups of people create a behaviour unique to an organisation, in turn, making every organisation unique in itself.

If it walks like a duck, quacks like a duck, it must be a duck!

There have been ample articles, books and talks by the experts themselves on how they helped grow and lead an organisation to the success it is known for, what policies they formulated, what initiatives they ran. When Google released information about their “20 percent rule” a lot of organisations jumped at implementing them. A similar pattern happened when Google talked about “OKRs”.

Amazon’s case is no stranger to this pattern. When it attributed a lot of its success to the “2 pizza sided teams”, every other organisation, coaches alike wanted to jump and execute it.

Also, how can we forget about the famous “culture deck” from Netflix ?!?

Learning about the framework responsible behind a success story is one thing, being able to replicate it within your environment is another.

In his book The Fifth Discipline, Peter M. Senge writes:

“Practicing a discipline is different from emulating a model. All too often, new management innovations are described in terms of “best practices” of so called leading firms.

I (Peter Senge) believe benchmarking “best practices” can open people’s eyes as to what is possible, but it can also do more harm than good, leading to piecemeal copying and playing catch-up”

A butterfly flaps its wings and a typhoon is caused!

David Levy conceptualises how industries are complex, dynamic, and non-linear systems in his paper:

Firms interact with each other and with other actors in their environment, such as customers, labor, the government and financial institutions. These interactions are strategic in the sense that decisions by one actor take into account anticipated reactions by others, and this reflect a recognition of interdependence.

— David Levy, “Chaos Theory and Strategy: Theory, Application, and Managerial Implications

Explaining the nature of chaotic systems further, David highlights that small disturbances multiple over time and that such systems are extremely sensitive to initial conditions, which makes forecasting very difficult. This is a formal study in chaos theory, often known as butterfly effect, a term closely associated with renowned mathematician and meteorologist Edward Lorenz.

So if the organisations are chaotic systems and chaotic systems are highly sensitive to their starting conditions (the nature of market at the given point of time, the people present, the scale, the developments in competitive landscape etc.), can we, with confidence, ever really be sure how the same policies will fan out in another organisation? Sure, we might be able to get some benefits out of the detailed frameworks laid out by successful organisations. The benefits might not be the same benefits that the original organisation achieved. And sometimes that’s acceptable too I guess.

Fooled by Randomness

What is even worse is when we achieve initial success by piecemeal copying of the “best practices” and convince ourselves that “we have got it right!”. We have to challenge ourselves in ascertaining how much of our initial success was skill and how much was luck.

We tend to give ourselves too much credit for something that might just be an act of luck. The reverse is also true: we beat ourselves in the event of failure, more so because in organisations there is immense pressure on the leaders and managers to lead and “to be right”. We are not that great at comprehending true randomness.

“The human mind is built to identify for each event a definite cause and can, therefore, have a hard time accepting the influence of unrelated or random factors”

— Leonard Mlodinow, The Drunkard’s Walk: How Randomness Rules Our Lives

All hope is lost? Not quite. In fact quite the opposite. Being aware of the complexities and uncertainties (randomness) is a strong base (some would argue the very first step even) required to be able to work around them, to be able to make better decisions to foster your company’s prosperity. If you don’t know about them, that’s a bigger problem. Because you’re just playing luck!

So what can be done? Well, one of the starting approaches is to develop a systemic (Systems Thinking) approach, to be able to see the causality among various parts of a system (more on this in the upcoming blogs), and not just trying to optimise the individual parts.

When you’re looking to get inspired by any of the industry leading “best practices”, try to get into more details. Get rudimentary. This doesn’t mean to over simplify and trivialise things. Try to codify what values would have been required to uphold the vision, what principles would have been required to bridge the values and practices to make the initiative a success. Qualify which pieces are missing in your org/team and what can be done to find/substitute them.

The more the number of teams that get involved, at a given time with such initiatives in an organisation, the better the chances of getting a holistic view and able to realise the interdependencies.

On the closing note, a decision journal goes a long way in learning what worked and what didn’t. Remember that you need to avoid getting lucky!

Organisations: Through the lens of Emergence. was originally published in Geek Culture on Medium, where people are continuing the conversation by highlighting and responding to this story.

Motivation

Being a bit inclined towards functional programming, I wanted to try and build a “type safe” and “exhaustive” Pattern Matching in TypeScript, without using any additional libraries, and only using switch statements and the power of Type Inference and Control Flow Analysis provided by TS.

Preface

So... a switch statement…

While switch statements don’t need much introduction and are pretty ordinary in themselves, the sections in this blog would:

• Introduce the code snippet (sample problem statement) we’ll be working with,

• Shall discuss what are the limitations of switch statements when used in their naive form

• And, then go on later to discuss a few enhancements (by harnessing native TypeScript features only!) that we can make to make our switch statements more Type-Safe and Exhaustive.

Note: The aim of this post is not to suggest the resulting code as a ‘better’ pattern for implementing calculator application or similar problems. It is merely intended for demonstrating the type safety and exhaustiveness that we can achieve while employing switch statements by harnessing only native (without usage of external libraries) features of TypeScript.

Problem Statement

We’ll work with a simple calculator application/function. To begin with, lets assume our calculator does only two operations, viz. Add and Multiply. A simple code for the same would be as follows:

Straight forward, isn’t it. We’ve got two types of Operations (Add and Multiply) and then a simple function using switch statement to perform the appropriate operation.

Now, what happens if we remove either of the case statements, or we add a new Operation type, say Subtract? TS compiler won’t complain in either scenarios because we have a default case as a fallback. We agree returning a 0(zero) from default is a big NO. Not only 0 is a legitimate result from one of the operations, it doesn’t signify that the calculator was not able to perform an unknown operation. We can also return an undefined or a null, or throw Error maybe from our default case. But all of these approaches introduce an extra branch and a possiblity that the client code will have to check for.

We can always write unit tests to make sure either of the above doesn’t go unnoticed.

But what if we can get the compiler to fail itself ?!?!

The ‘default’ case and ‘never’ type

Lets focus on default case. Let’s just log the Operation type that goes unmatched and is caught by the default case.

Notice that TypeScript infers operation inconsole.log statement as never type, as shown below.

This is because of control flow analysis done by TS. Since the operation type can be one of ‘ADD’ and ‘MULTIPLY’, there are naturally no more types left for it to assume in the default case, hence it assumes the never type.

Below are some excerpts from Marius Schulz’s blog on The never Type in TypeScript (highly recommended), and TypeScript docs which are most relevant to us in enhancing our solution further.

never is a subtype of and assignable to every type.

No type is a subtype of or assignable to never (except never itself). Additionally, it is a bottom-type.

In a function expression or arrow function with no return type annotation, if the function has no return statements, or only return statements with expressions of type never, and if the end point of the function is not reachable (as determined by control flow analysis), the inferred return type for the function is never.

Deriving from the above, lets create a function which will signal whether or not our switch case matching is exhaustive.

The function takes in a never type parameter and “never” returns. So in case we try to pass any other type as a parameter, TS compiler would be unhappy and fail. But since the return type is also never, it can be used in conjunction with any other return type within a function.

Stitching it all together

So lets replace the return 0 in our default case in our code with the exhaustiveMatchingGuard function we created in previous section.

Notice that TS compiler is happy with our code because (as elaborated in “The ‘default’ case and ‘never’ type” section) there is no other type left for operation to take on.

Let’s notice what happens if we add another type of Operation:

and Voila!! the compiler is not happy that we have forgotten to match the Subtract case!! It tells us that argument of type string is not assignable to type never , as expected!

And hints us that operation: ‘SUBTRACT’ is the culprit. Once we implement it, we’re back to our happy state!

Closing thoughts

One of the places that this kind of exhaustive matching comes very useful is while using redux and deriving next state based on different action types in reducers.

I hope this article gives some inspiration and inclination towards functional and type system side of TypeScript. I’m listing below some additional resources that you might find useful for futher explorations:

• Type Inference in TypeScript

• Control Flow Analysis in TypeScript

• Marius Schulz’s blog on The never Type in TypeScript

• ts-pattern library

Type Safe and Exhaustive ‘switch’ statements, aka Pattern Matching in TypeScript was originally published in Technogise on Medium, where people are continuing the conversation by highlighting and responding to this story.

Code Reviews have long been and will always be an integral part of the software development lifecycle. They, along with Pull Requests were further popularised through open source community, where contributor(s) would raise a pull request against a change-set they’re wanting to introduce and then the owner(s) of the repository would review and either accept, request changes or reject the PR. Naturally, the change-set is kept from getting merged till the entire process is completed, rendering it blocked from getting shipped.

The pull requests and code reviews have been aggressively adopted in “closed” source projects too. But does the blocking nature of code reviews via PRs really help in the closed source projects? I’ve long proposed PRs and code reviews in teams, more so in new(er) teams. As time progressed, and as I read more (and listened to more talks) and learned from others’ experiences, I started realising the traditional code reviews have more demerits than merits. Let’s try to reason along.

Note: Unless explicitly stated, I’ll be referring to the blocking code reviews in context of code reviews done via a Pull Request (or PR as referenced in rest of the article), where the code awaits (hence blocked) for merging till a code review has been done to satisfaction.

What are the benefits of good code reviews?

Before we jump into analysing the pitfalls with blocking style of code reviews, let’s agree upon why code reviews (whether blocking or not) are important in software engineering. Broadly speaking, benefits of a good code review are as follows:

• It ensures design and implementations are consistent across the project

• Helps check for potential vulnerabilities in the code and logic errors.

• Fosters knowledge sharing through discussions involved between contributors and reviewers.

• Helps validate the developed feature against the intended scope and desired outcome.

So, whats wrong with blocking code reviews?

1. Increases the Lead Time

• Most of the assumed and agreed upon benefits of a code review come at the cost of blocking the code from getting to trunk(or master/main branch) and eventually blocking the developers from picking their next story. This blocking will increase the Lead Time and also make the feedback loop only bigger.

• Not forgetting to mention the plethora of merge conflicts the review will continue to accumulate the longer it “sits in the review lane”.

• The biggest impact on the lead time, although is the context switching and overhead of asynchronous communication. (for both the submitter and the reviewer alike). Dragan Stepanović explains it very beautifully in one of his slides:

2. Blocking code reviews are not Continuously Integrated.

• Just because you’re using a top notch “CI/CD” tool, doesn’t guarantee that your team has either of Continuous Integration or Continuous Delivery.

• Pull Requests used in conjunction with feature branching imply anything but continuous integration. Although few ardent advocates of TBD and CD argue that a code kept “long enough” on local master instead of pushing it to remote “frequently” (at least once a day) is still a local branch, I’m only referring to long lived branches and explicit feature branching here.

“Feature branches are by design intended to hide changes. Continuous Integration is designed to expose changes. The two are mutually exclusive.” @DaveFarley

3. Prone to “Pseudo Rubber Stamping”

• No one wants to stand in the way of a unit of work and block it from happening. The more critical or bigger the change-set to be reviewed is, the more time it is going to take to review it. The more time it will take to review, the more it will block. We’re more eager, in such cases, to just rubber stamp the PR with a “LGTM” (looks good to me) and let it move on.

  It is not that larger PRs either have too much or too little discussion, but rather they have a larger variability in their amount of discussion than that seen in smaller PRs.

• There is an interesting and detailed study done by Jellyfish which is available in their blog, which suggests that amount of discussions (review comments and replies) vary greatly as the size of the PR increases. The variability in the discussion suggests that not all the PRs are getting the due attention that they require. As a result, not all the PRs are getting reviewed as we would like in the first place. This just defeats the very purpose of a code review itself.

• It’s also implausible to expect the individuals (and not the team as a whole) to understand all the aspects of the system themselves. Eventually expecting them to be able to do all critical code reviews with due diligence is a fallacy! This starts becoming more evident as both the code and team grow in size.

4. Dilutes responsibility and ownership

• In teams where there is over reliance on few seniors (maybe a tech lead or an architect) to be able to do critical reviews, not only have bigger bottlenecks, but also keep these seniors busy enough to keep them from picking up other meaningful and important work themselves.

• The moment a code is sent for review, the ownership of the code shifts from the original dev (who wrote the code) to the reviewer. This can be a challenge. It gives a false signal to the devs that their work is “done”. In reality, it’s not done till its shipped and monitored briefly for sanity after deployment. Any other “done” is really just a facade!

5. Impedes effective knowledge sharing amongst the team

It has been long argued and advocated in the community that code reviews help share knowledge within the team. With code reviews, not only the team aligns itself with the consistency in coding and designs, but also discovers and shares new and better techniques to do things. Adding on, I would argue that blocking and async fashioned code reviews are in general antithetical to knowledge sharing.

• With async code reviews, mostly the devs partaking in the code review are aware of the conversations. It’s quite likely that other team members will repeat the same mistakes/pattern unless the entire team becomes aware of the knowledge.

• Knowledge sharing at the time of async code reviews is already too late in most cases! It’s being reactive as opposed to proactive. If the knowledge was shared upfront (and with the entire team), the time spent in reviewing and verifying the consistencies and design could’ve been saved and put better use of.

Can we do things differently?

We all agree without a doubt that code reviews are important. We also reasoned certain pitfalls when it comes to do code reviews in a blocking and async fashion. Is that the end of road or can we do things differently to reduce the pitfalls, and still achieve the same benefits that we set out to achieve in the first place?

1. Pair programming and continuous code review

With pair/mob programming, you get a continuous and instant feedback on the code. The review is happening for every line of code that is being put, in real time.

• Least context switching happens. The pair’s or mob’s single-most and only responsibility at the given time is the code they are working on. Doing synchronous review as they code and reason along is a huge bonus!

• There is no waiting for reviews and then making changes and getting reviewed again. In most cases, the cycle is longer. With pairing or mobbing, every participant is present all the time.

• The Lead Time is immensely reduced.

2. Rotate your pairs

• Effective pair rotation helps in keeping the knowledge silos to a minimum.

• Frequent change in the pair of eyes looking at the functionality in progress can bring in fresh perspective as well as uncover things which previously were hidden.

• Makes software a shared and social responsibility.

3. Host “Show and Tell”

Organise frequent “Show and Tell” in your team.

• Talk about the design vision more often.

• Showcase the new techniques of implementation that you discovered.

• Talk about the desired way of writing code.

• Discuss the debts that your code has accrued over time and how can it be dealt with.

• Equally important, showcase the learnings from code failures and bugs.

So Pair Programming is the silver bullet, you say?

Well, it’s kind of a rhetorical question to ask. If every team and every org could work through pairing and mobbing, we would see it more often than we do now.

There are several cases where Pair Programming or Mobbing might not be effective:

• Lack of proper infrastructure and tooling.

• The teams are more leveraged than they should be. The ratio of experience levels and tenure in the team is a skewed one.

• There are multiple (sometimes even two are more than sufficient) geologically separated teams. The teams are dependendent on each other and neither truly distributed nor autonomous.

• Low to no psychological safety. In a low psychologically safe environment, pairing can be a daunting task. It can come across as being “monitored” or being “watched upon”.

• Teams who do not understand pragmatic pairing. And treat it like dogma!

• Critical code reviews, like pertaining to governance and compliance (often related to security) and only few in the org/teams having the complete knowledge of it.

• Introduces fatigue if not done with agreed upon pairing session timings and breaks.

Closing Thoughts…

The aim of this article is not to propose adopting an absolutist approach. I understand that “it depends”. I understand that one methodology might not be applicable to different engineering teams. I also understand that the same methodology might not be applicable to the same team, all the time.

This article is rather an attempt to highlight the trade-offs which a team inherits when it employs blocking code reviews. These trade-offs might not be justified for every type of code review. Maybe there are few which require more detailed attention than the most. My advise would be to choose carefully. Don’t make it a dogma! Use your judgement wisely!

Why does a code review have to be blocking? was originally published in Geek Culture on Medium, where people are continuing the conversation by highlighting and responding to this story.

Age and Integer (as much as they look like they were made for each other) are quite different types and have different properties and behaviours. Let's try to think out loud. An integer can be added to or subtracted from another integer, would you need to do that with age? Maybe. An integer can be multiplied with or divided by another integer, would you need to do that with age? Probably Not. An integer can be negative, can age ever be negative? Definitely not.

Just with some initial arguments, we see that Integer has started to diverge from being a quintessential representation of Age. This misrepresentation of richer constructs in programming with primitive types is often referred to as a Primitive Obsession

Primitive Obsession

is using primitive data types to represent domain ideas. For example, we use a String to represent a message, an Integer to represent an amount of money, or a Struct/Dictionary/Hash to represent a specific object.
 — https://wiki.c2.com/?PrimitiveObsession

So if Age is not merely an integer representation, then what is it? Let’s explore the idea in more details building on the complexity of representing Age correctly…

NOTE: The code examples listed below use TypeScript. However, the concepts are basic enough to be incorporated in other languages too.

First Cut

Age can be represented in days, months and years, or even a combination of some/all of these.

If we simply used age as an integer variable, how do we get the other devs to know whether the age is in years, months or days? The most naive way I’ve seen this being addressed is through variable naming conventions.

Isn’t it too much work for a dev to keep calculating the values for different representations? Also, we’re way too optimistic that the meaning wouldn’t be lost in translation. Imagine if this age variable was to be passed as parameters to other functions.

The problem with this approach is that it’s too prone to human error and interpretation to be comfortable with. If you’re not already familiar with, read about failure of Mars Climate Orbiter.

So how do we improve further? By building a basic abstraction — a class.

Alright, so we’ve gotten to a good start. Converted a primitive type to a good abstraction with some behaviours/methods to get age in different representations of days, months and years.

We can further improve our code. Remember we talked about the fact that age cannot be negative. But our above code accepts just any number values for years, months and days. So let’s fix that quickly in our constructor:

The above snippet makes sure that our Age object is constructed only with valid and allowed properties.

Final Cut

So far we’ve built upon the basics of a much richer concept called Value Object.

Value Object:
A value object is an object that contains attributes but has no conceptual identity. They should be treated as immutable.

In simpler terms, taking age as an example, the age object doesn’t have any identity of its own. It's generally associated with another entity (an object which can be differentiated from other objects by a unique identification) like a person (every person will have a unique SSN number for e.g.).

The second concept about immutability is what makes value objects even more exciting. If the age changes, the existing object would spit out a new age object itself, leaving the original one untouched. For instance, when we add days/months/years to an age object, it would return a new object with the new attributes, as shown in the code below.

With immutability, your code is more robust against multi-threaded programming because the referential integrity is intact for every object. It might add overhead in certain languages which were not designed keeping immutability as primary constructs, however, the tradeoff at this level is pretty minimal and you’ll almost always benefit in the multi-threaded world.

You may further enrich this model by including the comparison behaviour as well to help compare the two age objects with each other. Combining all the changes that we have made so far, our final Age representation would look like below:

Bonus (update)

Even though I’ve used numbers (years, months and days) to construct the age object (to keep the blog simpler and shorter), effectively age is a difference between two points in time. So it might make more sense to accept a Date/Time object parameter in the constructor to denote the birth date & time(if being used for a person, for e.g.).
Then, the other calculations like getAgeInMonths, getAgeInYears and comparision of ages etc could just be derived from the initial date/time itself.

Thing to watch out for: Remember to take Timezone into consideration along with the date-time object.

Summary

Always explore the richer abstractions aka Value Objects that you can build instead of prematurely obsessing over primitive types. Just like Age, there are several other constructs which are often misrepresented by just using the primitive types. Money is yet another example. Most of the codebases would portray Money as a float or a BigDecimal, completely missing the currency aspect of money.

So next time when you think about a type, conceptualise it using real-world scenarios and following their behaviour in real-world when put to different use cases.

More content at plainenglish.io

Why You Should Stop Representing Age As a Number in Your Code was originally published in JavaScript in Plain English on Medium, where people are continuing the conversation by highlighting and responding to this story.

Some top considerations while contemplating over Monolith vs Microservices are scalability, independent deployability, increased releasability & maintainability. However, thinking only about application without considering how the choice would affect the team, is like looking at only one side of the coin.

Just as a “good” application which is easier to scale, maintain, release and test enables to serve more end-users (by handling more user requests), a “good” team helps is the force behind success of such applications. While a well structured ecosystem of applications/services is capable of shipping features faster and getting them out to users without having them to wait “too long”, in turn reducing Time to Market, the team is really the hero that makes it happen.

Glossary

Let’s begin by collectively visiting some of the terminologies used in this writeup. If you’re already familiar with them, jump straight to Considerations section.

Modular Monolith:

• While referring to a monolith in this article, I’ll be referring to a “single process deployment”, i.e.; a system in which the code is deployed as a single process. There might be multiple instances of this process for scale, resilience etc. But conceptually the entire code is packed and deployed as a single process at a time.

• Generally, we think of modularity only from code perspective. However, having modular code is just half the job done. The underlying persistence must also be made modular. So, when you structure your monolith into modules, also have a decomposed DB. Try hard not to have that foreign key reference to another module’s table even if it seems the most sane thing to do.

• It might seem an overkill in the beginning to have so many DBs, and it makes sense to have this apprehension. But you can even harness the power of schemas that mostly all the DBs now provide and have separate schemas for different modules.

Segregating the code into microservices, still, is relatively easier as compared to separating the data into individual databases (owned by respective microservices) later and dealing with tedious migrations. DB migration is a “hard problem”.

Feature based packaging:

Firstly, if you’re not familiar with software packaging principles, it’s a good place to begin with and understand the underlying concepts behind them. They are as fundamental to writing and structuring code as are SOLID principles.

“Package by layer” is typically what we’re all aware of and most widely used in projects. If say, we’re working on an API service, we’ll try to segregate the packages by layers. So all the code related to routes will go to a route directory, all the code related to services (or adapters) would fall into the infamous services package and then the pattern continues with entities and repositories and so on so forth.

While the most easy to get started with, Package by layer is a maintenance hell, especially as the project grows, more people start contributing and more and more features get developed. Imagine browsing through the entire entities package to find out just the relevant entities that actually require change. Now imagine doing that for all such layers and imagine all the developers and teams doing that, the pain it would cause. In short, discoverability of code is very low.

“Package by Feature”, on the other hand, uses packages to reflect the feature set. It’s an attempt to place all items related to a single feature (and only that feature) into a simple directory/package. This leads to packages with high cohesion and modularity, with minimal to no coupling between the packages. You may still have the layered architecture in the feature package to begin with, as shown below, but a good north star is to adopt a more Domain Driven Design based packaging structure. For, e.g. if “Food delivery App” had “Restaurant Search” & “Delivery” as capabilities, each of them would end up as a feature package containing relevant layers/sub-packages within them as shown below:

Each feature team can individually grow and maintain their feature packages with minimal conflict and bending their minds to search for the right code file to change.

Cross-Functional teams:

• Most conventional organisations have their teams set up in functional silos. A single Marketing team would be responsible for doing all the market evaluations and competitors study for various product lines/features. They’ll relay this to the Product team which is solely responsible to productise it into various features or suite of products. These feature epics would then be passed on Engineering team to execute, through a grandiose planning session. This relaying of information-models from one function to another is so fascinating that I began calling them as Relay-Functional teams, analogous to a relay race, where one runner after finishing his leg, passes the baton to the next one to do his part. In large organisations, the individual functions might comprise of multiple teams within, however, conceptually the information model would still flow from one function (as a whole) to the other.

• The most common model that comes to our mind when we think of cross-functional teams is engineering teams. We think of devops. We think of having Devs, architects and QAs as part of the same team and follow the same methodologies and choreograph the various ceremonies together.

• That's good. Additionally, we should also strive for getting representatives from other business functions (like Product, Marketing, Sales etc) in the team. Imagine if Marketing person knew exactly what was being worked upon and could do the market-fit analysis alongside. Imagine the Sales person giving feedback ahead of the time what will fly with the customers and what won’t.

• It’s difficult to have such a team in the place from the beginning, but it’s completely plausible. Such teams take a bit of sweat, experience and feedback from users to get to the right composition. But the core idea is to keep this at the back of your team’s mind throughout the journey, and not to be “comfortable” with what you have.

Considerations while deciding monolith vs microservices:

Before we dive into the considerations themselves, it’s worth pausing for a moment and understanding what does efficacy of a team really implies. While the ostensible idea of adding more people to the existing teams might seem quite reasonable to help scale the overall efficiency, however there is pretty good evidence in the industry that adding more people doesn’t really mean that the projects are going to be delivered faster. (If you haven’t already, it’s worth reading The Mythical Man-Month which has quintessential essays on this topic.)

Hence, while thinking about scalability, also think about how can existing teams be scaled up with respect to their capabilities and contribution capacity, and not just size.

With the above premise, let’s explore how various considerations play their role in influencing the choice of monolith vs microservices. Moreover, let’s also explore some middle grounds simultaneously.

• A team can be scaled effectively on a well structured and modular monolith in a mono-repo codebase too. You don’t explicitly need a multi-repo Microservices based setup to do that. Been there, done that! The key however, is “well structured modular monolith with feature packaging around key sub-domains / business functionalities” (I know, it’s quite a mouthful).

• Don’t forget to think about the right team composition. It plays a more crucial role in the overall architecture and structuring of the project than you might have thought of, especially in the longer run. (Oversimplified Conway’s Law)

• If essentially you have Relay-Functional teams working in silos, then you’re way better off having a modular monolith backed by a mono-repo. Different code repos maintained by the same team might not be an exciting overhead to have. Some organisations are able to champion this very well. But requires enough maturity in the team. However, the overhead can get overwhelming as the team size increases beyond a threshold (Not quoting a number due to lack of actual research on the threshold. Sometimes wisdom precedes science and statistics)

• However, if there are cross-functional teams, i.e. teams composed of Product, Engineering, Marketing etc and each team looks after individual sub-domains or product (sub-products or features too, if I may), then a solution around Microservices, and code split around well identified code repos (e.g. one repo per Microservice is a popular choice) might work wonders for you. Yet again the maturity of the team matters.

• When the product is in its nascent stages, the domain is just evolving and very often misinterpreted. Without clear bounded contexts, premature modelling into Microservices might do more harm than benefit. I’m sure you’ve heard of horror stories of teams having to merge services and rethink the boundaries. Not a happy place to be at.

• If this is the first time the team is diving into the microservices pool, it’s better to begin with feature teams working on a modular mono-repo codebase. Later split into microservices as the need arises and as the team gains more confidence.

• Feature teams give the utmost confidence to Engineering and other business functions alike. It is relatively more comprehensible to think which feature requires more attention than the other, in turn allowing you to focus on particular teams at a time. Also, the approach helps in devising bespoke solutions for respective teams.

• You may also go via the library/packages route of taking the modules in the monolith out into separate code bases that yield respective packages. Thereafter, combining them together to deploy the monolith as a single process. The repo separation of packages (suggested segregation based around business capabilities or “bounded-contexts”) helps different teams worry only about code that matters to them. But remember to keep the “packaging principles” in mind else unnecessary dependencies might slow things down instead of helping.

• The releasability impedance is also caused by incorrectly identified bounded contexts while using Microservices. These are often termed as “Distributed Monoliths”. It might be easier to recover from a poor structured monolith than poorly structured labyrinth of microservices.

Summary

“It Depends”. Just like most things in software, context is the key. An approach might work beautifully for a team and might not for the other.

When in doubt, try to begin with a monolith. Especially in a startup or when trying to explore the feasibility of a new product. Also stick with monolith if you’re unclear on what problems will Microservice actually solve for you. Use feature based packaging, it will help the team maintain the codebase better as it grows. Put in usage metrics and monitoring. Observe. Usage of the app by the end users is the real truth. See if it makes sense to split features lead by individual cross functional teams yet. Wait for it before going Microservices. You may never require them as well.
If the need occurs, your feature based packaged mono repo can further be decomposed into Microservices much easily as compared to codebases that are structured in a typical 3 tier architectural pattern. Feature packaging helps your team think more about business capabilities (bounded contexts) which are the building blocks of Microservices anyways.

Bonus: Get to know more about Clean Architecture, Hexagonal Architecture and Onion Architecture to take the feature based packaging to the next level by incorporating Domain Driven Design into it.

Monolith vs Microservices : From the team’s perspective was originally published in Geek Culture on Medium, where people are continuing the conversation by highlighting and responding to this story.

Hosted by Technogise
Speakers: Nayab Siddiqui and Gaurav Pasarkar

We compare the sync vs async approach and discuss in detail “the Good” and “Not so Good” things around each pattern and approach. Starting with a brief recap of Monoliths and Microservices, we’ll dive deep into Orchestration vs Choreography styles of designing workflows. Also we’ll explore and discuss different ways of implementing “Distributed Transactions” (like 2 Phase Commit and Saga) along with Distributed Queries using different patterns like replication, CQRS and more.

Sections | Topics

Link to the timeline of each section/topic is as follows:

• Intro to Technogise & TechnoWise

• Agenda

• Recap of Monolith & Microservices

• Orchestration vs Choreography

• Commands vs Events

• Distributed Transactions (2PC | Saga)

• Distributed Queries (Replication | CQRS)

Also, you can watch the full presentation down below:

<a href="https://medium.com/media/461fb791d2fc8b584523694864d56ade/href">https://medium.com/media/461fb791d2fc8b584523694864d56ade/href</a>

Unfurling Asynchronous Microservices was originally published in Technogise on Medium, where people are continuing the conversation by highlighting and responding to this story.

It often seems best to look out for another job to learn more, or better yet to learn more to get that new job. Often, it seems best decision to always hire people to fill the tech void within the company, and seems a waste of efforts to let people learn those new skills. Well, let me show you some of the positives which you’ve been missing out on, so far.

Technology around us is changing day by day. What was the norm yesterday is “un-cool” and outdated today. Newer projects are being written as you read through this blog. Every new project is trying to use the next-gen tech stack (if not the very bleeding edge technologies) in order to keep up, or at least try to. Some of these projects are also inventing new tech as they go, thereby complimenting the tech ecosystem . This trend is an over simplification of the theory of Diffusion of Innovations — by Everett Rogers, and Gartner’s Hype Cycle.

As a result, the demand for developers well versed with such ‘new(er)’ technologies is increasing by the day (if not by the hour in certain cases). The companies are expecting devs to know more (than ever) and the job descriptions are getting heftier in their expectations. In turn, this is creating a serious deficit of talent in the pool, which in turn impedes the recruitment for a company and even worse, makes the projects compromise on the right tech stack that they could/should have used.

So what do we do about this ? Well, encourage learning on the job more than ever!

What’s in it for the companies ?

Investing in people often results in the best returns for both the company and its employees.

• Existing devs already have domain context. Eric Evans explains the benefits of thinking in terms of bounded contexts in a quintessential manner in his book Domain Driven Design. Adding new tech skills in the arsenal beside domain knowledge is a proven winner in solving real problems pertaining to the company/business.

• Companies which empower learning on the job, are often the companies which attract and work with some of the best minds in the industry. People who are able to adopt and apply newer knowledge are often proven to be ahead of the evolution curve and generally layout the future path for others through their thought leadership. Conducive environments for these people is simply a ‘straight flush’ and a win-win for both the devs and companies. Continuous improvement (kaizen) and meaningful innovation is the key to success and sustainability for every tech company (product and consulting alike). People who are curious to learn play the biggest hand in making this a reality.

• It generates a path of learning and career growth within the company. It encourages people to challenge themselves in a healthy fashion and keeps them from falling in the ‘comfort zone’ trap. There is nothing more harmful than a cohort of staff which continues to ‘not’ reflect and think alternatives to what they’ve been used to.

• With the new(er) stack, there are only so many devs experienced in it, and even lesser ‘available’. Everyone is already going after them. And why should these devs leave their current environment which helped them be who they are, in the first place. Imagine you empower devs in your org to achieve that. Not only you don’t have to keep scratching your head figuring out how to ‘lure’ them in, but you’ve also managed to create a sustainable phenomena on your own. Imagine not having to stall projects because of unavailability of devs. Imagine not having to go empty handed in the meetings with your VPs/CXOs and disappoint them by telling them that their brilliant idea won’t see the light because you don’t have people to execute it. Imagine not being afraid in choosing the best stack suited to solve your problem. Imagine NOT BEING CONFINED!

What’s in it for the developers (YOU) ?

• Software engineering is akin to craftsmanship. Like almost all the other arts, its also majorly self-taught. Its imperative that you focus and keep investing in yourself. The day you choose not to learn anymore, is sadly going to be also the day you’ll stop being of much value. Don’t!! Grab a new book, follow a new blog, listen to an exciting keynote, write a ‘Hello World’ in new… BE INSPIRED!

• Stand out from the crowd. Become a thought leader. How frequently do you Google (yes, I’m aware of duckduckgo. And no!! I’m never using bing in a sentence) your answers vs answering others’ queries on the net/in your company? Get ahead!! Experiment, build hobby projects, share your wisdom, write a micro blog, tweet something interesting. The community needs more forerunners like you.

• There are high chances that you’re already surrounded by seasoned devs in your org. Having them play mentors for your learnings is a phenomenal idea. They might not be experienced with the stack that you’re chasing after, but they must have done this for themselves when they were in your shoes (or they’re still doing it). Having such mentors can make a world of difference to how you approach learning a new stack. They can pitch in the pragmatic ways and often warn you of the unknowns that you might not be looking out for.

• You need ‘more’ to get to the next level. You’ll need to own more responsibilities than you do now. You’ll have to guide and mentor people. You’ll have to ‘serve’ more than you do currently. Are you planning on doing all of that without being better than you already are right now ?

‘Learning new’ is not just about learning that new shiny language, or knowing about that new reactive framework in town, or going DevOps crazy (maybe DevSecOps is the new big thing eh! Oh wait, it is!). Nope! it’s NOT JUST that! It can be even finding alternate ways of writing the same piece of code, learning better ways of refactoring without halting your stories further, finding alternate ways of structuring your tests (Does it surprise you if I tell you can write BDD tests without using a BDD framework ;) ), or even just brainstorming with your fellow geeks on what’s not so right in the architecture right now and what can be done in the least disruptive and continuous way.

Don’t be fooled by thinking that the next job is going to be better. Don’t be naive in thinking it’s just this project manager who is always strong arming you into doing more than you signed up for. Don’t be fooled by thinking that getting an expert on that stack is going to solve your existing problem.

Think about what’s the guarantee that the next job is not going to turn up the same? And if it does turn out similar, will you not be on the lookout again? Think about whether ‘that new geek’ that you hired, is not going to be obsolete soon in your environment, because after all people are not learning in your environment.

I’m not suggesting you keep lingering on to your job no matter what. Even if its near impossible to find time to learn, because lets face it, they hired you to DO the job and NOT to “waste your productive time in learning and not being the code monkey to just write off those user stories”. Yup, such floors do exist, unfortunately! But what I am suggesting is to give it a fair chance. I know there are some ‘awesome’ companies which have dedicated 1 day off, 80–20 rule (God knows when engineers will stop using this ratio!) and what not to let people explore and invest in themselves. And yours might be rather more humble. But its worth a try. Not all projects are the same, not all teams are same and not all times within the same project and team are same. Find the right opportunity and time and utilise it to the fullest. Better yet, use that time to fix things in your project or help them look at better solutions. Chances are you’re that catalyst which the company has been waiting upon to realise it needs to motivate people to learn more on that paid time.

On the other hand, there is nothing wrong with hiring people to fill in the deficits of knowledge pool. Its always good to bring in fresh perspective to already existing staff. However, whats wrong is to almost always prioritise this, over enabling people to learn new things. This phenomena is more prevalent in consulting companies which do project specific hiring to suit the dynamic clientele and projects. Please take note that people are going to stay even after the project is over (well, more often than you think :) ). Moroever, newer projects are going to come in as well, which may command a different tech set than the existing ones. If everytime we got out to hire a new batch, we might be jeopardising more than just the values and vision the company stands tall upon.

Don’t get me wrong here. I’m not concluding that a newer batch is not capable of doing so or they don’t come from a background where these values are already practiced and cherished. What I am trying to convey is that with the similar values and vision, every company has its signature way of materialising them in to tangible means, which creates its identity, and it takes time to get to know and adapt this mannerism.

If you have any Pro Tips on how to keep learning, or how to help people learn in your team/company, please do share/comment. If you have any other thoughts or there are things you disagreed within the blog, I’m more than happy to hear you out, discuss and debate :).

Nevertheless, let me leave you with the following thought:

If you have always done it that way, it is probably wrong.
 — Charles F. Kettering