Context

For a time I worked on a system where the codebase supported multiple applications. I think we had 6 or 8 at thtat time. We tried to reuse ideas across business silos to improve quality. We supported multiple business unis each with their own rules and formulas. We 'app' used a shared shell, security, and backing database. It was the business rules that varied across applications. We supported a very small user base. There might only be one or two users for a given app.

WIth over 90% code coverage, the lead thought he had a quality system.

On the operations side, we had a very short SLA for turning around code fixes. We had 15 minutes to respond and 60 minutes to push a fix.

During a cycle, the analysts were required to turn in their numbers before 15:30 on Friday's. The numbers they defined impacted what the production teams made and delivered to meet customer demand. If an analyst missed the deadline due to a software defect, an angry Vice President would be in contact. Friday's were always a bit tense until the deadline passed.

How the Problem Surfaced

It's a Frida like any other. No one is working on anything "important" because... Friday. A defect would be reported, someone would jump on it. A fix was made including updates to the unit tests. The developer would build the app, show green on the tests, and we'd merge.

Then the build would fail because unit tests in another part of the app suite were failing.

No build. No deploymnet. Queue the angry (E)VP.

The Fix

We realized was the unit tests were not protecting us. They actually prevented us from meeting SLAs.

We had a big ball of mud aka spaghetti code. We had to untangle the mess. Remove unit tests that strayed beyond their scope. Remove side-effect programming patterns and show a little rigor.

Lessons Learned

  • Raw code coverage metrics do not provide a clear view of the quality of a system.
  • A big ball of mud resists tests because responsibilities are not actually separated.
  • The work is not to chase a percentage. The work is to improve confidence in behavior.

What I Watch For Now

  • Tests that require too much arrangement.
  • Code that can only be tested through large workflows.
  • Coverage numbers that rise while design confidence stays low.
  • Teams treating testability as a QA concern instead of an architecture concern.