Jezen Thomas

Robert Roskam made a point on social media about the inadequacy of test coverage. > 100% test coverage is not enough. Here’s an example as to why in Python: > > > def division(x, y): > return x / y > > > If you pass in 0 for y, it’ll raise an error. You have to know to test for > the value specifically. Test coverage won’t save you here. With 100% line > coverage or branch coverages, it won’t be found. That’s how 100% test > coverage is not enough. So, if test coverage is not enough, what can we do here? Well, at least a couple of things. ## Surfacing Failure Cases Automatically It wouldn’t be sensible to try testing our division function against every possible input, because the set of possible inputs is effectively unbounded. Fortunately, we don’t have to. We can instead use a property-based testing library like hedgehog to generate inputs to our division function and show us the minimal way to reproduce an error when it occurs. This example is in Haskell, and only accounts for integral numbers. Otherwise, it’s a near enough direct translation of the Python version. import Hedgehog import Hedgehog.Gen qualified as Gen import Hedgehog.Range qualified as Range division x y = x `div` y prop_division_identity = property $ do x <- forAll $ Gen.int Range.linearBounded y <- forAll $ Gen.int Range.linearBounded let q = division x y -- quotient r = x `mod` y -- remainder q * y + r === x We’re relying on the integer division law to test our function. The integer division law says: > When you divide one whole number by another, you get a quotient and a remainder. If you multiply the quotient by the divisor, then add the > remainder, you get back the original number. This law holds for divisor ≠ 0. So, the test generates a range of integers, applies the division function to them, and checks that the result always adheres to the integer division law. Running this in GHCi quickly surfaces the problem. ━━━ Main ━━━ ✗ prop_division_identity failed at div.hs:15:13 after 1 test. shrink path: 1: ┏━━ div.hs ━━━ 10 ┃ prop_division_identity = property $ do 11 ┃ x <- forAll $ Gen.int Range.linearBounded ┃ │ 0 12 ┃ y <- forAll $ Gen.int Range.linearBounded ┃ │ 0 13 ┃ let q = division x y -- quotient 14 ┃ r = x `mod` y -- remainder 15 ┃ q * y + r === x ┃ ^^^^^^^^^^^^^^^ ┃ │ ━━━ Exception (ArithException) ━━━ ┃ │ divide by zero In this case, we didn’t have to explicitly check what happens when the divisor is zero. Furthermore — and perhaps more importantly — writing this test has forced us to think about both the types of the inputs and the range of those inputs. I didn’t want to have to specify the upper and lower bounds for the range of inputs, but not all numeric types have explicit bounds. That led me to choose to constrain my division function to only support integers. We can see that limitation directly in GHCi. λ minBound :: Int -9223372036854775808 λ minBound :: Float <interactive>:49:1: error: [GHC-39999] • No instance for ‘Bounded Float’ arising from a use of ‘minBound’ • In the expression: minBound :: Float In an equation for ‘it’: it = minBound :: Float This also guides us towards thinking about edge cases. If we have to consider division on floating point numbers, then the obvious challenge is going to be in specifying to what degree of precision we expect to see in our results. We would then have to change the assertion in the test so that it accepts a result to a specific acceptable degree of accuracy. Again, the test encourages us to think, which in turn leads to better design. That’s true whether you’re writing a small function like division, or a larger, integrated path through your system. This is why TDD is often thought of as Test Driven Design, rather than just Test Driven Development. ## Making Failure Impossible Now that we’ve surfaced the error, how do we design our function to make this kind of failure impossible? One common approach is to express the failure case in the output. This typically means wrapping the output in a Maybe to represent that the result may not exist. division :: Int -> Int -> Maybe Int division x 0 = Nothing division x y = Just (x `div` y) This works, but it necessarily implies that the caller of this code will have to handle the possibility of failure. This change will then ripple through the rest of the codebase, which is noise that you probably don’t need. Alternatively, you can use the smart constructor pattern to constrain the input and have the changes ripple back towards the boundary of your system, which is usually a better design choice. In the case of our division function, this would mean ensuring at compile time that the function will only accept a non-zero divisor. division :: Int -> NonZero Int -> Int division x (NonZero y) = x `div` y Finally, we could of course return zero in case of failure. Which of these options is the right one is debatable, but the point is that both the tests and the types have driven us to think about the design of our system.

After several years working on Supercede, I’ve had time to reflect on some of the technical choices we made. Some of them worked well, others I’d approach differently if I were starting again. Here’s what I regret, and why. ## Elm Was the Wrong Choice for Our UI When I started working on Supercede, I had the idea that any more complex parts of the UI should be written in Elm, rather than with React or perhaps just managing all DOM interaction manually with plain JavaScript. If you need to write in a Single Page Application (SPA) style, I think Elm is probably the best possible choice. I would absolutely choose Elm if I were writing a greenfield SPA today. It doesn’t matter that Elm doesn’t have typeclasses, and it doesn’t matter that Elm doesn’t have the same kind of churn that the rest of the JavaScript world has come to expect. These are both features. I’m familiar with some of the criticisms levelled at Elm, but chief, this ain’t it. The reasons why I have come to regret Elm are not inherent to Elm itself. The problems are in product design, and what that necessarily implies about the system design. As it turns out, Supercede just doesn’t [currently] need any of the user interface written in SPA style. This is enterprise software for reinsurance professionals. Most of the interactivity should be modelled with server-rendered web forms. Kind of like how web applications were written 20 years ago. If you design your application as a backend which serves some JSON and a SPA which consumes and operates on that data, you have two separate systems. To marshal values as they flow between the boundaries of those two systems, you need to somehow manage the synchronisation of the types of those values on both sides (and that’s true also of dynamically-typed languages), and you need to manage how those values serialise and deserialise, and you probably need to duplicate some of the business logic and parsing/validation logic. You might — as we did at Supercede — introduce a code generation mechanism which runs as part of your build step and enforces that all of these things remain synchronised. But this is more code to compile. More to maintain. More to document. More to constantly be aware of. It doesn’t come for free. Alternatively, you could choose to not have these problems by designing your product in a different way. This sounds like a constraint, and it is. But that’s a good thing! Design is all about constraints. Constraints liberate. If you can’t design within constraints, then you’re just a bad designer. ## Event Sourcing Added Complexity We Didn’t Need We knew from the beginning that auditability would be an important property of our system. Event sourcing — where the current state of the system is derived by replaying a chronological sequence of events — seems an obvious choice for this. The problem is that event sourcing is hard. Sure, it’s not insurmountably hard. You can quickly gain a conceptual intuition for how event sourcing should work when it’s explained to you, and you can readily draw parallels to other systems which work more or less the same way, like distributed version control, or your bank account. But the complexity of designing and maintaining an event sourced system doesn’t come for free. It’s not even cheap. When you want to introduce a change to the system, you need to think much harder about threading that change through each component of the event sourcing system. You need to think harder about when effects happen. Should running a projection also send transactional email? Probably not. Other effects? Hard to say. You can learn the theory. You can learn the skills. But it’s probably too much to take on for a startup when they’re still trying to find product/market fit. Your time is better spent building features that paying customers actually care about. And most customers — especially for a product like Supercede — really care about auditability. But they don’t care whether that auditability is implemented with event sourcing, and not, say, some audit tables. There are cases where event sourcing is absolutely the right approach, and should be coupled with deeper measures of integrity like WORM drives, but those cases are less common than the software conference circuit might have you believe. Much like microservices, I suppose. ## Over-Reliance on PostgreSQL Slowed Development This regret isn’t about PostgreSQL per se. It’s more about tightly coupling to database-specific implementation details, and the consequences of such a coupling. One of the consequences is slower test execution. Most of Supercede’s test code is written in an integrated style using the yesod-test library. Between each test, we create a new application state and reset the database so that individual tests do not affect the execution environment of other tests. Truncations are slow in PostgreSQL, and for some specific design reasons it’s not really feasible to wrap each test in a transaction which is rolled back at the end1. We have other projects which use the same libraries but instead run tests against an in-memory SQLite database, which works great and runs quickly. Depending on features specific to PostgreSQL also means you need to run a real database server in your continuous integration setup. It’s more to install, and you need to run it all in a virtual machine. It’s just more stuff. And stuff is a liability. For most of the more generic database work, I would prefer to abstract away the details of the database with libraries like persistent and esqueleto. There might be parts of the application that necessarily rely on specific database features, but I think those parts should be extracted or in some way isolated, perhaps even so they can be stubbed in the general case. ## Lenses Lenses are brilliant, and for any new non-trivial Elm project, I would likely reach for a lens library like elm-monocle. But that’s because I think Elm projects — at least in my experience — tend towards keeping all state in one foundational data structure which will necessarily become complex as the project grows. Your foundational data structure — your Model — is not some peripheral detail that you interact with only occasionally. Because it’s so central, you work with it constantly and so the mental model for manipulating that data structure with lenses becomes rudimentary, rather than a novel curiosity. Lenses are so worth it if you have this design and the ergonomics constraints it brings. But if you don’t? I don’t think it’s worth it. Lenses in Haskell read quite differently from more conventionally written Haskell code. So much so, that while being ordinary Haskell, you could almost consider them a domain specific language. Working with lenses and maintaining a flow state requires familiarity, which I don’t think you can establish if you only flirt with lenses by lightly dusting it around the periphery of your project. At Supercede, we haven’t gone all-in on lenses. Some engineers have made the choice to lightly dust lenses on the periphery. Those parts now cause visible lines in the skin of programmers’ faces any time that programmer is forced to read and understand what we’ve written. So, I regret our lens adoption. In hindsight, it seems gratuitous. Perhaps we indulged in whether or not some code could be written in terms of lenses, while neglecting to consider whether or not that code should be written that way. ## BEM Naming Conventions Stifled Thoughtful Design It’s tough, balancing Not Invented Here syndrome with first principles thinking. It’s not economically sensible to reinvent every wheel, nor is it sustainable to surrender to the learned helplessness of best practice, which is itself a form of fallacious reasoning. The Block Element Modifier naming convention is one such best practice. I regret allowing it to creep in to our codebase because I think it allowed some developers to try to absolve themselves of the kind of thinking required to come up with meaningful names for things. Communicating intent through clear writing is one of the most important parts of the job. The use of BEM has also made us stop thinking about how the problem of CSS selector scoping could otherwise be solved. The BEM naming convention could certainly be the right choice in some tech stacks, but in our Yesod project we have better tools! We generate unique identifiers for components at runtime, with some components being comprised of smaller components. This of course isn’t exclusive to Yesod — other ecosystems solve this problem differently. But the point is that this problem can be solved differently, and to ignore that for the sake of best practice is folly. ## We Abstracted Too Early, and It Cost Us Haskell is a high level programming language. Almost all of the code we write describes what we wish to achieve, and not specifically how the runtime should achieve it. So, premature optimisation is not really a problem we’ve faced historically. At least, not in terms of algorithmic efficiency. Instead, the premature optimisation that we have at times fallen afoul of is abstracting too early, and abstracting the wrong things. The worst offences here typically look like functions which are highly configurable, but don’t really do anything particularly interesting. For example, extracting a user interface component library, so that all the buttons in your UI look the same. But in some places, the button should be a different colour, so you make that a configuration option. And then in some places, the button should be a different size, so you make that a configuration option. And then in some places, the button… You get the idea. A function that requires a load of configuration in order for it to understand how it’s meant to behave is probably a code smell. Instead of one highly configurable function, we could have written a few different functions. Or, we could have written a few different CSS mixins and composed those together. ## Haskell’s Type System Isn’t a Substitute for Good Tests Haskell’s type system is brilliant. The type system obviates the need to write so many of the more onerous tests that we would have had to write and maintain if we were writing our project in something like Ruby or Python. Tests that assert the shapes of data structures, or that functions are called with the right arguments — that kind of thing. But ensuring that a SQL query doesn’t throw an exception at runtime? We still need tests for that. Or documenting and automatically verifying the expected behaviour of a path through the system? We still need tests for that. There are many more examples to list, but this list isn’t meant to be exhaustive. Despite being able to leverage a tool as powerful as the [Glorious] Glasgow Haskell Compiler, I’m still seeing a huge amount of value in using automated tests to drive out the design of the system. This shouldn’t really be surprising either — types and tests are different things and serve different purposes. In parts of our project where the test suite is well written, I’m seeing that it continues to be well written as my colleagues write more production code. This might come from a kind of social pressure — conscientious developers generally want to leave a project in a better state than they found it. If a test suite is well written, it just feels bad to make a change that leaves the test suite relatively less comprehensive. So, my regret here is in not investing enough in establishing a culture of test-driven design. And I don’t just mean we haven’t written enough tests. There’s an art to writing test code, since its purpose isn’t purely to prevent regressions. If we had a better cultural awareness of how to use automated tests to drive out the design of our system, we’d almost certainly have a better production codebase for it. We do have this culture at least partially established in our team, but it’s harder to add this after the fact. And because the existing test code isn’t always ideal, the path of least resistance isn’t to write ideal test code, and the relative growth of good test code against new production code isn’t self-perpetuating. In other words, it takes more effort now to get to where we want to be with our test code than if we had done this up front. ## Slow CI Kills Productivity Programmers have a kind of adversarial relationship with continuous integration systems. It’s a healthy conflict though — the whole point is to ensure that the changes you make are sound. As you work, you push changes and your CI system runs an assortment of checks against them. You might be otherwise happy with your contribution and all of your automated tests are passing during local development, but then the CI system flags that you neglected to add a migration. Or that the code fails some linting rules, or that while your code works just fine on the compiler version or system architecture that you are running locally, it fails on other compiler versions or system architectures that you also need to support. So, the build on CI is failing. You make another change, push that change, and wait again for CI to pass or fail. Another failure. Make more changes. Push again. Wait. This style of working is of course fine, but if CI is slow, it means your iteration cycle is slow. And a slow iteration cycle easily distracts programmers and kills motivation. Scaling this lack of efficiency up to a team of people means the company isn’t shipping software as fast as it should. In hindsight, I should have invested more effort into keeping tests fast and build times short. The compound productivity gains from this just make economic sense. ## Lessons Learned Looking back, the choices we made were [usually] generally well-motivated. If I were starting Supercede today however, I would take a firmer stand against over-engineering, and I would be more conservative — not just in the number of tools and libraries we depend on, but also in how much of any given language we use. Haskell is a great language, but we don’t need to use all of it. This retrospective has also reaffirmed to me how important test-driven design is as a way to approach software development. I learned all about it a decade ago, forgot it, and then had to relearn — painfully — why TDD matters. Finally, I don’t regret for a minute any of the time I spent learning about logical fallacies. Software engineering as an industry is rampant with fallacious reasoning, and often it’s enough to recognise fallacious reasoning when it’s being employed in order to not fall prey to it. --- 1. Or maybe there is a way to do this, and I just haven’t figured out how yet.↩︎