Software: Time vs. Quality and those who lie

Developing software is always a race between time and quality, and knowing when to lean one way or the other is part gut and part gamble. Some people will lie to themselves and others and say something can be done immediately, when what they really mean is that a proof of concept can be built quickly. Others will lie saying that something will take weeks that can really be done in less than a day. Those who venture to be honest about their best guess are either chastised for being too ambitious, or for not being ambitious enough. That is to say that perception often overrides reality. It’s simple enough to say that there are tradeoffs and wave our hands in the air, but what if we look at this a bit more critically?

A House of Cards, Smoke and Mirrors

Two engineers are brought in to build a new system of great scale that needs to work well for the foreseeable future. One asks for time to investigate and build, the other boldly claims that she can have it up and running in a week. We’ll call the engineers Wise and Bold respectively. Management is intrigued by Bold’s proposition and eagerly awaits the result.

One week later Bold comes back with a demo that shows amazing capabilities, at least in a controlled scenario. Management is wowed and can’t wait for more. Bold invokes the names of famous well-built software and their capabilities, wearing them as a mantle as if she authored them herself, and makes grandiose promises of the fortune they will bring when Bold’s projects reach fruition. Meanwhile, Wise looks under the hood and finds that the demo is nothing more than a thin shell around existing software. What’s more, the implementation doesn’t address many of the hard requirements of the system. Wise reports this, but management has seen nothing concrete from Wise yet, and so chooses to ignore her for the time being.

As time goes on, Wise begins to build things of her own but with more care. Management is not impressed, but they notice that things tend to work well enough, and that there is some predictability in progress. Bold continues to make haphazard demos that impress, but as time passes the actual project at hand is not being addressed. Eventually it becomes clear that while Bold can build to impress immediately, she cannot make what is required, and cannot fulfill the longer-term promises that her earlier work had made. Management, as if waking from a long slumber, thanks Bold for the work and sends her on her way. Wise continues planning and incrementally building as before, eventually reaching a system that, while not perfect, fits the criteria required and offers guarantees of stability.

Analysis: Wise could have easily built demos to show off existing technologies to impress management, but she chose not to because they had little bearing on the project at hand. This was the right choice, but could have cost her her job if Bold had convinced management that Wise was useless because she produced no immediate output. Real life is not fair or just, and although in this case management realized Bold’s act of smoke and mirrors, just as often they are tricked by fraudsters and cast out the person actually building what they need.

Lesson: Beware fluffy talk and parlor tricks. Stay awake, ask hard questions, and stick to the real goals. Dare to open the front door and discover a house of cards.

Give Me At Least 2 Weeks

A new page needs to be added to a website. Two engineers familiar with the website’s codebase are brought in to estimate how long it will take to implement it. One engineer takes a deep sigh and puts on a troubled look and says “give me at least 2 weeks.” The other, taking a cursory look and thinking a bit, quickly realizes it will more likely take 1–2 days and brings this up. We’ll call these engineers Over and Real respectively. Management perceives that Over is thinking deeply about a real estimate, and decides to go with it. Real sighs, but realizes this is just a job and that there are worse things than lost time, and goes to handle another task.

A few days pass, and Real decides to check on Over’s progress. There’s no code available, but maybe Over is developing locally and hasn’t pushed a branch yet, thinks Real. A week goes by, and Real begins to get concerned, as Over’s lack of progress could reflect badly on her team. Real finally talks to Over to see what’s going on, and asks if there’s any code she can help with. Over is at first reluctant to share, but realizing the deadline is coming up and seeing no other options, shows Real her code.

To Real’s surprise, the page appears to be completely implemented, but there are extra pieces not part of the task which are half-broken. It turns out that Over had finished the task in 2 days, but since the estimate given was 2 weeks, she had continued down several rabbit holes that were only tangentially related to the page, having seized the opportunity to build other things she was curious about. Real has Over come clean to management about this, and Over is reassigned to a less critical project. Afterward, Real talks to other engineers on the team, and finds out that Over was well known for this, but nobody had called her out on it until now. Real cleans up the code to only handle the immediate task in a few hours and deploys the final page.

Analysis: Real could also have made a gross overestimate to give a wide berth in case of unforeseen consequences, but decided to give management a real answer. Often, management does not have a good grasp on how long things will take, and will receive someone who claims something is quick with great cynicism and disregard.

Lesson: It’s tempting to believe people who appear to be contemplating things seriously, but don’t fall for the ruse. Ask for justification for an estimate, and listen to counter arguments from others. It’s not always clear who has the more accurate picture, but it’s easier to see who’s full of hot air by poking the proverbial balloon.

Real Tradeoffs

What if the dream team of Wise and Real were put together and called in to build the first version of a big project in a month? Suppose they bang their heads together and realize that the project done very well would take 3 months. After much discussion, they whittle it down to 1 month, but only just. What kind of tradeoffs could they have made in this hypothetical project?

• Lower testing coverage / edge-case testing

Testing has many purposes: ensuring correctness of existing code, ensuring that discovered bugs don’t regress, and documentation among others. By focusing heavier testing on the most critical features and letting others temporarily fall by the wayside, you can win back time in exchange for well-understood tech debt.

• Hardcoded data

Abstracting things out into clean and separate reusable pieces is great, but the first version of software often only has need for 1 use case. Hardcoding and leaving a note will save time in the short-term, and has the added benefit of stopping people from overthinking solutions from the start.

• Unscaling implementation with flexible design to implement the scaling solution later

An API that is small and simple enough to build and understand quickly, but flexible enough to swap out internal implementations later is a great way to save time. People often spend too much time scaling beyond their needs by using technologies with the best throughput at the expense of setup time and ease of use (premature optimization). Faster implementations are almost always easier to swap in than better designs.

• Negotiation with product owner for feature reduction, or phased development (next deploy would have more features)

At the end of the day, if a product owner dictates a deadline, they must also be ready to drop feature requests as needed, or to break things out into initial launch, post-launch phases and so forth. Oftentimes many things are just feature creep that a product owner will let go of if nudged slightly. This can result in huge reductions of time.

• Simplified infrastructure (minimal redundancy)

Unless you’re planning for Twitter levels of scale in the first year of operation, chances are you don’t need exabyte scale from day one, nor high redundancy. A relational DB with 1 master and 1 slave, or if a DB that requires a quorum is needed, just 3 nodes. Keep it simple. If you’re on a cloud services provider scaling is often trivial, but if you’re not, setting up physical boxes and everything involving them can be a big time sink.

The most important thing to understand about real tradeoffs is that what is being gained and what is being lost is well understood and well documented, with strict plans for being addressed later in order of importance. Building the wrong thing quickly is not a goal. Taking more time to add unnecessary things is not a goal. Building the minimum possible that achieves the goals in as reasonable a time as possible is the goal, and the best way to do this is by being honest and preparing to challenge assumptions of time, difficulty and necessity every day, even at the risk of looking stupid by admitting that your initial take on something was wrong. The world doesn’t like projects taking longer, but liars and charlatans do the world no favors. Real tradeoffs for real wins.