The 3 biases of software development - Number 1: Stratification

Dividing software into layers is a good idea; tackling one layer at a time is not

Recently, I spoke at the Regional Scrum Gathering Stockholm about the 3 biases of software development: Stratification, Chronology, and Modularity.

The event was awesome, as the organizers (Evelyn, Giuseppe, Khurram, and Thomas) and the volunteers did an amazing job putting it together. I left the event truly happy and with a lot of food for thought. That was my motivation to transform my presentation into three posts: one for each bias I described there.

So, what are these biases?

A bias can be defined as "A preference or an inclination, especially one that inhibits impartial judgment". In software development, that could be translated as modeling a software project using concepts that are hardwired in our brain, thus avoiding an alternative approach that could help us create something more flexible.

We do things this way because we've always done them this way. And that's an infinite loop.

To break these biases, we must first understand them, so let's start with the first one: Stratification.

Stratification?

Yep, I came up with these three names, and I'm not really satisfied with them. But, as Phil Karlton said: "There are only two hard things in Computer Science: cache invalidation and naming things".

The stratification bias happens when, after splitting software into layers (let's say: database, backend, and frontend), we immediately start to develop the basemost layer, which is often the database model. Picture this: you start with the database layer, meticulously crafting each table and relationship. Only when that's "done" do you move on to the backend, and heaven forbid anyone mentions the frontend until the API is complete!

This is probably the most famous of the biases, as it's long been coined as BDUF (Big Design Upfront) and was one of the first barriers we had to overcome to develop more flexible software. Sad thing is, it still happens. A lot.

There are two main problems with this approach. Number one: side-effects. A change in a single point of the uppermost layer could generate multiple changes in the lower layers.

Second problem: side-effects of side-effects, or bugs. Now, because you had to change the database, some part of the backend that is barely related to the initial change might be affected and also affect some frontend part that is completely unrelated to the initial change.

If you’ve been paying attention, you noticed that by fixing a bug you could trigger the exact same process again, which takes us to an infinite loop of despair, frustration and customer-angriness.

Am I a victim of this bias? How do I escape it?

There are symptoms. Your customer won't be able to provide valuable feedback because they won't see anything usable, the amount of rework needed whenever you have to change something will be unbearable, and you might feel the urge to discard some nice ideas because of their development costs. Most importantly: your developers will be bored to death.

And just because this was the status quo not long ago, the bored developers back then decided to create something to help them out of this situation. Something that would allow the database to safely evolve over time without generating a lot of technical debt. They called it "Database Migrations".

It's beautiful, isn't it? A technical solution created to solve a technical problem that was hindering the team's capability to adapt to change. I'll come back to this later after the technical example, so if you're already familiar with the concept or if you're not familiar with code, just skip the next section.

Getting our hands dirty (with code)!

Let's take a look at a Database Migration using Elixir and Ecto. Imagine that you are creating a table to store the users of your system. You (and your customer) might have a lot of neat ideas of all the data you can store there: name, email, a short bio, an avatar, their birthday, and the list goes on and on.

If we understand that the database can evolve, we can start with something simpler. Just the username and the email, for example.

defmodule MyApp.Repo.Migrations.CreateUsers do
  use Ecto.Migration

  def change do
    create table(:users) do
      add :username, :string
      add :email, :string
      
      timestamps()
    end
  end
end

Doing this first part will provide us with options for the next iteration. We can actually ask ourselves what is the most important thing to develop: the avatar? the short bio? something entirely different in another part of the system?

Later, if we decide to implement the user avatar and the short bio, we could just create another migration like this one:

defmodule MyApp.Repo.Migrations.AddProfileToUsers do
  use Ecto.Migration

  def change do
    alter table(:users) do
      add :bio, :text
      add :avatar_url, :string
    end
  end
end

This way we can ensure that the customer's database will always be up to date and that nobody will run the same database script twice and break the whole thing.

There are technical benefits, but non-technical as well

Besides the obvious benefit of creating something more flexible, there are some very interesting non-technical benefits as well. For example: because you built a lighter database, whatever the team needs to develop next will take less time just because there will be fewer fields and tables to deal with. This will also reflect on the amount of side-effects (aka bugs) your development process might generate, because the complexity of your software won't grow exponentially like it did before. We could summarize that as: breaking the stratification bias equals saving time.

That time you saved could generate a butterfly effect. The team can now think about what to do with that time. Maybe learn how to write some unit tests, or take some time to experiment with a new technology. Because the answer to this question isn't obvious, they'll have to collaborate to find the best option. Note that everyone will have to collaborate: developers, designers, testers, business people.

This whole database migration concept was created because someone challenged the idea that "We cannot change the database as we please". A technical person, with a technical solution. Because sometimes when a developer faces a challenge like this one and has the proper time and resources to deal with it, an innovation is born.

That's also the case when we break the Chronology and the Modularity biases, but we'll talk about them later. :)

---

Ps: If you want to read further about this subject, you can check Martin Fowler’s amazingly detailed blog post, here: https://martinfowler.com/articles/evodb.html

---

Next Article on this series:

---