Every Shopify agency website says the same four things. Senior team. Conversion-focused. Data-driven. Partners, not vendors. You could swap the logos between any ten agency sites and almost nobody would notice.
That is a problem when you are the one choosing, because the claims are free and the mistake is expensive. A bad agency engagement does not usually fail loudly in week two. It fails quietly over four months, and you find out when the site launches and the numbers do not move.
The way through it is not to read more agency websites. It is to ask questions whose answers cannot be faked by a good marketing page. Below are nine of them, with what a good answer sounds like and what should worry you.
1. Who will actually be doing the work?
Ask for names and ask whether those people are in the room right now.
The most common structure in this industry is that the people who pitch you are not the people who build. Sales closes, an account manager translates, and the work goes to whoever has capacity — often junior, often offshore, often on three other projects. None of that is disclosed and none of it is illegal. It is just how the economics of a large agency work.
A good answer names individuals, and those individuals are on the call. A worrying answer talks about "our team" and "our process" without ever getting specific, or introduces you to a dedicated account manager as though that is a benefit rather than a layer.
Follow up with: if I have a technical question in week three, who answers it, and how long does it take? If the honest answer involves relaying it to someone, you now know the shape of the engagement.
2. Can I see a store you built where the results did not go up?
This is the single most useful question on the list, and almost nobody asks it.
Every agency has case studies. Case studies are, by construction, the projects that worked. What you learn from asking about a project that did not is how they think when things go wrong, whether they can diagnose their own mistakes, and whether they are honest with you before you are paying them.
A good answer is specific and slightly uncomfortable: here is one, here is what we got wrong, here is what we would do differently. A worrying answer is that everything has always worked, or a deflection onto client behaviour without any share of the responsibility.
An agency that cannot name a failure either has not done enough work or is not going to tell you the truth when yours is the project in trouble.
3. How do you measure whether the work succeeded?
The answer separates agencies that do CRO from agencies that use the word.
Look for something falsifiable. A split test, a defined metric, a period, a threshold agreed before the work starts. If success is defined after the fact by whichever number happened to rise, it is not measurement.
A good answer sounds like: we would A/B test where your traffic supports it, we would define the primary metric before starting, and we would tell you if it did not reach significance. A worrying answer is "we look at the analytics" or a promise of a specific percentage lift before anyone has seen your store.
Be especially careful with guaranteed numbers. Nobody can guarantee a conversion lift on a store they have not examined, and anyone offering one is either inexperienced or counting on you not measuring.
4. What does your reporting actually look like — can I see a real one?
Ask for a redacted example. Reporting is where the difference between a partner and a vendor becomes visible, and it is easy to check before you sign.
A good answer produces a document with numbers, a comparison against a baseline, plain statements about what worked and what did not, and a recommendation. A worrying answer is a dashboard screenshot with no interpretation, or a monthly summary of hours worked. Hours are an input. You are buying an outcome.
5. What happens if the test says your redesign lost?
Redesigns lose sometimes. Anyone who has done enough CRO has shipped a variant that underperformed the original. What matters is what happens next.
A good answer describes rolling back, working out why, and treating it as information. A worrying answer treats the possibility as hypothetical, or reveals there is no mechanism to find out — which usually means the redesign will ship regardless and nobody will ever know whether it helped.
6. Who owns the code, the accounts, and the data?
Get this in writing before the project starts, not at the end.
You should own the Shopify store, the theme code, the analytics properties, the ad accounts and any custom app built for you. Some agencies retain ownership of accounts or build on a proprietary framework you cannot take elsewhere, which is a commercial lock-in dressed up as a technical decision.
A good answer is immediate and unambiguous: you own everything, and here is how it transfers. A worrying answer hesitates, or explains why their proprietary layer is better for you.
7. What is your scope-change process?
Every project changes scope. The question is whether that is a conversation or an invoice.
A good answer is a defined process — change is priced and approved before work starts. A worrying answer is hourly billing with no cap, or a fixed price so low that the change orders are clearly where the margin lives.
Ask directly: how often do your projects finish at the quoted price? The number itself matters less than whether they have one.
8. What would you tell me not to do?
A good agency has opinions and will risk the sale to state one.
You are testing whether they will disagree with you before you are paying them — because if they will not, they certainly will not in month three when disagreeing is more expensive.
A good answer contains a real objection to something you said in the call: your budget is aimed at the wrong problem, you do not need a rebuild, the platform you are asking for is a mistake. A worrying answer agrees with everything. Total agreement in a sales call is not alignment, it is a sales technique.
9. Why would we be a bad fit?
Related, but sharper. Ask them to disqualify themselves.
A good answer is specific: we are not the cheapest, we are wrong for pre-launch products, we do not do the volume of output a marketplace brand needs. A worrying answer is that they are a good fit for everyone. Nobody is a good fit for everyone, and an agency that says so has either not thought about it or is willing to take work they cannot do well.
The pattern underneath all nine
Every one of these questions is really the same question: will this agency tell me the truth when it costs them something?
That is the only variable that reliably predicts whether an engagement goes well. Technical skill is common and checkable. Honesty under commercial pressure is rare and is exactly what you need in month three, when the test came back flat and someone has to say so.
A checklist to take into the call
Ask all nine. Write down the answers. Then read them back a day later, away from the enthusiasm of the meeting:
- Did they name the people doing the work?
- Did they describe a project that did not go well?
- Is success defined by something falsifiable?
- Did they show you a real report?
- Is there a mechanism to find out if the work failed?
- Do you own everything, in writing?
- Is scope change priced before it happens?
- Did they disagree with you about anything?
- Did they name a client they would turn down?
If most of those are yes, you are talking to a good agency, whatever their site says. If most are no, the site was the best part of the engagement.
About the author
Manpreet Singh
Manpreet Singh is the founder of Proscube, an ecommerce growth studio. He leads the studio's Shopify and Shopify Plus engineering, headless builds, CRO, and its work on AI engine optimization, and writes its guidance on how to grow a DTC brand without wasting money. He works directly with founders — no account-manager layers between you and the people doing the work — and would rather tell a client not to build something than sell them work they don't need.
More about Manpreet
