Shipping quality software in hostile environments

SHIPPING QUALITY SOFTWARE IN HOSTILE ENVIRONMENTS @kll #PyConBLKN 2018.

WHO? Luka Kladaric Chaos Manager @ Sekura Collective recovering web developer of 10+ years architecture, infrastructure & security consultant also a startup founder and remote work evangelist 2 — @kll #PyConBLKN 2018.

HOSTILE ENVIRONMENTS? 3 — @kll #PyConBLKN 2018.

WHAT IS TECH DEBT? 4 — @kll #PyConBLKN 2018.

Tech debt is the implied cost of additional rework caused by choosing an easy solution over a longer and better approach. 5 — @kll #PyConBLKN 2018.

Tech debt is: an API that returns a list of results without pagination 6 — @kll #PyConBLKN 2018.

Tech debt is: fragile code that everything runs through 7 — @kll #PyConBLKN 2018.

Tech debt is: entire systems that have become too complex to change or deprecate 8 — @kll #PyConBLKN 2018.

Tech debt is: parts of the codebase nobody wants to touch 9 — @kll #PyConBLKN 2018.

Tech debt is: broken development tools and processes lack of confidence in the build and deploy process 10 — @kll #PyConBLKN 2018.

Tech debt is: everything the team wishes they could change, but can't afford to 11 — @kll #PyConBLKN 2018.

WHERE DOES IT COME FROM? 12 — @kll #PyConBLKN 2018.

Insufficient up-front definition Tight coupling of components Lack of attention to the foundations Evolution over time 13 — @kll #PyConBLKN 2018.

WHAT'S THE HARM? 14 — @kll #PyConBLKN 2018.

DEVELOPERS ARE JUST SPOILED... 15 — @kll #PyConBLKN 2018.

FALSE. 16 — @kll #PyConBLKN 2018.

Unaddressed tech debt breeds more tech debt 17 — @kll #PyConBLKN 2018.

"We'll get back to that later" "why does X have to be clean, when Y isn't?" 18 — @kll #PyConBLKN 2018.

Productivity over time decreases Deadlines slip 19 — @kll #PyConBLKN 2018.

CASE STUDY 20 — @kll #PyConBLKN 2018.

So I get a call one day. 21 — @kll #PyConBLKN 2018.

Within a few days, my alarms start going off 22 — @kll #PyConBLKN 2018.

Massive monolithic git repo 23 — @kll #PyConBLKN 2018.

No concept of stable 24 — @kll #PyConBLKN 2018.

Hand-crafted build server 25 — @kll #PyConBLKN 2018.

No local dev environments Everyone works directly on production systems 26 — @kll #PyConBLKN 2018.

No db schema migration system or versioning 27 — @kll #PyConBLKN 2018.

Over 1/2 of the servers not deployable from scratch 28 — @kll #PyConBLKN 2018.

Code review tool is self-hosted abandonware 29 — @kll #PyConBLKN 2018.

Outages a daily occurrence 30 — @kll #PyConBLKN 2018.

Everyone focused on shipping features 31 — @kll #PyConBLKN 2018.

How do you even begin to fix this? 32 — @kll #PyConBLKN 2018.

It took over a year and a half. 33 — @kll #PyConBLKN 2018.

HOW? 34 — @kll #PyConBLKN 2018.

build server rebuilt from scratch 35 — @kll #PyConBLKN 2018.

build and deploy jobs defined 100% in code 36 — @kll #PyConBLKN 2018.

monolithic git repository split up into 40 smaller repositories 37 — @kll #PyConBLKN 2018.

all servers rebuilt and redeployed with Ansible 38 — @kll #PyConBLKN 2018.

better code review tool 39 — @kll #PyConBLKN 2018.

most dev work doesn't require VPN any more 40 — @kll #PyConBLKN 2018.

etc. 41 — @kll #PyConBLKN 2018.

JOB WELL DONE! 42 — @kll #PyConBLKN 2018.

The moral of this story is: don't wait for permission to do your job right. 43 — @kll #PyConBLKN 2018.

If you see something broken, fix it 2. If you don't have time to fix it - write it down 3. But do come back to it when you can steal a minute 4. Even if it takes months to make progress 44 — @kll #PyConBLKN 2018.

The team was well aware of how broken things were. If we pushed for it to be a single massive project, it would've never happened. 45 — @kll #PyConBLKN 2018.

EXCEPT... 46 — @kll #PyConBLKN 2018.

That's not how things should be. 47 — @kll #PyConBLKN 2018.

How do we do better? 48 — @kll #PyConBLKN 2018.

"Tech debt" work is difficult to sell 49 — @kll #PyConBLKN 2018.

It's not like paying off your credit card 50 — @kll #PyConBLKN 2018.

It's incredibly difficult to schedule work to address tech debt 51 — @kll #PyConBLKN 2018.

If You Don’t Schedule Time for Maintenance, Your Equipment Will Schedule It for You 52 — @kll #PyConBLKN 2018.

I recently came across an article that changed the way I think about this 53 — @kll #PyConBLKN 2018.

Sprints, marathons and root canals by Gojko Adzic HTTPS://GOJKO.NET/2018/08/30/SPRINTS-MARATHONS-ROOT-CANALS.HTML 54 — @kll #PyConBLKN 2018.

New name: sustainability work 55 — @kll #PyConBLKN 2018.

Budget vs planning 56 — @kll #PyConBLKN 2018.

Helps with morale 57 — @kll #PyConBLKN 2018.

QUESTIONS? @kll #PyConBLKN 2018.

THANK YOU! Luka Kladaric twitter: @kll luka@sekura.io www.sekura.io @kll #PyConBLKN 2018.