Research Suggests, Never Proven: Rebuilding the Pals on Evidence

There's a study about food waste that I really wanted to put in pantryPal. A team in Norway asked families to plan their leftovers into later dinners, and the families cut their dinner waste by 39 percent. That's the kind of number that sells an app. A meal planner with leftover slots, backed by a randomized trial, cutting waste by almost 40 percent.

Then we read the paper. The 39 percent was real, at seven weeks. But the control group, the families who got no help at all, cut their waste by 22 percent too. A year later, the difference between the two groups was no longer significant. Both had dropped by about the same amount. The waste was self-reported, and the study started with 230 families and ended with 144.

Bar chart: 39 percent less dinner waste for families who planned leftovers versus 22 percent for families with no help, with no significant difference after a year

The study behind the 39 percent. The families who got no help improved too.

So we don't say 39 percent. We don't say it anywhere. And that's a decent summary of what "research-based" has come to mean for us.

Why we're doing this

palStack's apps touch the parts of life where people carry real stress: money, home, food, friendships. A lot of apps in those spaces lean on borrowed authority. "Studies show." "Science-backed." "Save 30 percent." Most of the time, nobody on the team has read the study.

This fall we decided every pal should be built on what the research actually says. Not as marketing, but as the reason a feature exists and the limit on what we're allowed to claim about it. We wrote a research document for each product: one on the psychology of money for finPal, one on friendship and loneliness for clubPal, one on homeownership and the mental load of upkeep for propertyPal, and one on household food waste for pantryPal.

Every claim gets a grade

The most useful thing we built wasn't a feature. It was a grading scale. Every source gets a letter. An A means we read the paper itself and the claim is stated and supported there. A B means we read the abstract or a primary page but didn't see the method. A C means we only know it from a secondary summary, like a press release or a search result. A D means we couldn't find it, or it didn't hold up.

Four cards for grades A to D, with A and B allowed in the app and C and D kept out

Only A and B reach a user, and only as "research suggests."

Only A and B claims are allowed into the product, and when one appears, it says "research suggests." Never "proven." pantryPal goes further: a number can only be quoted from an A source, and the app never says you "saved" money, because we can't measure what you would have spent otherwise. What we can say honestly is that something was "used in time."

The grading forced honesty about our own process too. Some publishers block automated downloads, and we didn't try to get around that. Those papers are marked as blocked until a person gets them. propertyPal's research document was written when the sources couldn't be opened at all, so every figure in it is a search-summary figure, and the whole thing is gated: nothing from it reaches a user until each source is opened, read and graded.

What reading the papers changed

The surprising part is how often reading the source changed the design, and not always in the direction of "do less."

The Norway study is the best example. The finding everyone quotes was weaker than it looked, but there was a second finding hiding in it. Both groups cut their waste, and the authors concluded that simply tracking what you throw away lowered waste for a full year. That made a feature we had planned, a one-tap log of whether an item got used or tossed, much better supported than we'd thought. The log isn't a nice extra next to the intervention. It is the intervention. And because control groups improved in both trials we read, any claim like "waste fell after you started using the app" would prove nothing without a comparison. So we won't make it.

A US trial compared giving people information against helping them make if-then plans, like "if it's Sunday, then I check the fridge before shopping." The plans did better on behavior. But the authors were clear that neither approach measurably reduced food waste in the two weeks they studied. So the little "why this works" card in pantryPal will say that planning changed what people did, and that the study couldn't yet show less waste. That's a less exciting sentence. It's also true.

A famous result about scarcity, the idea that financial stress drains your mental bandwidth, turned out to have failed its authors' own large replication. We still let it inform tone, as a reminder not to pile on people who are already stretched. We don't build anything that depends on it.

Where research shapes the pals now

In clubPal, the loneliness research is the reason for the warmth rules: never guilt, use names, cheer the firsts. People underestimate how much a message from a friend means, so the app makes reaching out a little easier.

In propertyPal, the most transferable finding is about planning prompts, so every maintenance task is getting a concrete next date instead of a vague "soon." The cost of putting off a repair will show up at the moment you're deciding, as a range and never as a scare number. And since there's no trial for home upkeep, we're planning to measure our own completion rates, privately, to find out what actually helps.

In finPal, the research on how kids learn about money shaped an idea we're calling familyPal: a parent sets up a paper account, a practice account with pretend money, for each child, tied to a real family goal. The research says the parent is the main teacher, and that the kids who need the most help often have the parents who feel least sure, so the parent gets short "what to say" prompts too.

What I'd want other builders to take from this

Reading the paper is cheap compared to building the wrong thing. Most of what we found wasn't fraud or bad science. It was the normal gap between what a study measured and what a headline says it measured. That gap is exactly where an app overpromises, and it's the place users eventually stop trusting you.

The next step is wiring this into the products for real. pantryPal gets a single evidence file the app reads from, where every claim carries its approved wording, its grade, a link and a line about its limits. propertyPal's sources get opened and graded one by one. When something we say in an app changes, it'll be because the evidence did.

← Previous post Next post →