Three Questions I Ask Before Turning Repetitive Work Into a System

Not everything that repeats should be automated

At a campus operations meeting last March, an operations manager who had been compiling attendance data by hand every week and sending it to homeroom teachers asked me, “Can’t this be automated?” It is the question I hear most often as a data analyst. Building weekly learning reports for each campus. Cleaning up class attendance data and sending it out. Sharing Playfit members’ attendance trends with their coaches. The more a task repeats, the stronger the urge to turn it into a system, and that urge is natural.

But over the past year, I have watched a hastily built system create more work three separate times. Once, a script collapsed into a patchwork of accumulated exception handling. Once, a report went out every week that nobody opened. And once, wrong numbers were quietly delivered for weeks on end. I will come back to that last one in detail below. My habit of writing down three questions before starting any automation came out of those three experiences.

The first question: does it really repeat in the same form?

Tasks that look identical turn out to differ slightly every time, once you look closely. When I set out to automate the campus weekly report, I laid the past three months of them side by side and compared. The format was the same; the content was not. During exam periods, mock exam results replaced the assignment status section. Newly opened classes had no prior data to compare against. And some weeks, the teacher in charge rewrote the comments by hand before sending.

What I count first, before automating anything, is the ratio of exceptions. Take the last ten repetitions and count how many had exactly the same inputs and outputs; that tells you whether the task is a rule or a judgment call. In those three months of reports, five out of thirteen weeks were exceptions. When exceptions run past two or three in ten, I don’t automate the whole thing. I carve out only the parts that are reliably identical (data aggregation, say) and leave the rest in human hands. That report still works this way today: the system handles the aggregation and the tables, and the teachers still write the comments.

The second question: where does human judgment remain?

The second question is about the system’s boundary. A pattern where a learner’s attendance and assignment submissions drop together, for instance, is something data can catch. On weekly data, eight or nine out of ten such signals line up with actual churn. But what to say to the parent once you have that signal is an entirely different problem. Numbers alone can’t tell you whether something happened at home, whether the child is in a slump, or whether the class placement is wrong, and a single automated alert sent in error can chip away at trust that took years to build.

It wasn’t a design principle of mine that drew this boundary. It was one remark in a meeting. When we first discussed churn-signal alerts, a homeroom teacher said, “Don’t send it straight to the parents. Send it to me first. I’m the one who knows what was going on in that family that week.” I carried that request directly into the design: the system splits at detection on one side, judgment on the other. Gathering the warning signals and surfacing them to the teacher and coach in charge is the system’s job; whether to reach out, and in what words, remains a person’s. The purpose of automation is not to eliminate judgment but to make sure that when the moment for judgment arrives, the person making it is holding better information. This boundary has a number attached to it, too. If the signal’s hit rate falls below seven in ten, we stop the alerts and revisit the criteria first.

The third question: how will you know it’s working?

The third is the question I paid the most to learn. Last fall, after a column name changed in the source sheet, a campus weekly report I had built went out for three weeks with the assignment status fields blank. Delivery succeeded every week, and the error log didn’t have a single line in it. What told me was not any monitoring metric I had defined but a message from a homeroom teacher at one campus: “The assignment column in the report keeps coming up empty. Is it just my class?”

A system’s real life begins not on the day it is built but after it is forgotten. An automated report keeps going out looking perfectly healthy even after the source data’s format changes. Errors make no sound, so unless you decide in advance which metrics to check, you end up shipping wrong numbers for weeks at a time.

Since those three weeks, there are two things I always write down before rollout.

1. The baseline. That is, the time and error rate this task cost before automation. In the weekly report’s case, it took four hours a week when a person built it. Only with that number can you say what the system actually saved.
2. The health metrics, and the person who watches them. Delivery success rate, the share of recipients who actually open it, the number of times it went out with empty data. Set the metrics, and assign someone to check them every week. If the open rate drops below half, we stop sending and redesign the report’s content first.

Measuring the baseline and the metrics also surfaced one uncomfortable number. After automation, build time fell from four hours a week to forty minutes. But when we first measured open rates, roughly half the recipients weren’t opening the report at all. Saving time and making something that gets used turned out to be different problems. A system nobody watches is the same as no system.

Conclusion: a system buys time so people can spend it on judgment

The three questions converge, in the end, on a single test: where does this system let people spend their time instead? Good automation doesn’t make work disappear. It hands the parts that rules can handle over to the machine, so that people spend their time on what only people can do: observing, and judging.

And each system built this way makes the next one easier to build. Organized data becomes the raw material for the next question; saved time becomes the room for the next improvement. This is why I don’t rush automation. When you understand the work well enough to answer the three questions, a system reduces it. Build one before you can answer them, and it adds work, as it did three times for me this past year.