How to Run a Technical Reference Check on a Software Vendor

Every vendor hands over three references. All three will say yes. That is the entire problem with the last step of most procurement processes: the reference call has become a ceremony that confirms a decision already made rather than a control that can reverse one. A technical reference check software vendor teams actually learn from looks nothing like a 20-minute courtesy call with a hand-picked advocate.

The stakes are not abstract. Research from McKinsey and the University of Oxford, covering more than 5,400 IT projects, found that large IT projects run 45 percent over budget and 7 percent over time on average, while delivering 56 percent less value than predicted  and software projects carry the highest risk of cost and schedule overruns of any project type. 

Those overruns rarely trace back to a technology choice. They trace back to staffing that changed after the contract was signed, scope that was never pinned down in a statement of work, and code that the next team could not maintain. All three are visible in a past client’s experience months before they are visible in yours.

The gap between a useless reference call and a decisive one comes down to three things: who you talk to, what you ask, and whether you are willing to go around the list you were given. A reference you sourced yourself and pushed past the first polite answer will tell you how many developers were actually subcontracted, who left the project in month three, and what the client had to rebuild after launch.

This guide gives you the full call script, the sourcing tactics for reaching clients the vendor never listed, and the scoring framework to turn scattered call notes into a defensible decision.

What Is a Technical Reference Check Software Vendor Evaluation?

A technical reference check software vendor evaluation is a structured interview with an agency’s former clients that verifies delivery claims  team composition, budget accuracy, code quality, and post-launch support  rather than general satisfaction. It replaces subjective testimonials with specific, checkable facts about how the vendor performed under pressure.

The word “technical” is doing real work in that definition. A standard reference call asks whether the client was happy. A technical one asks what the client’s engineering team inherited, what it cost to maintain, and which of those outcomes the vendor controlled.

The Core Problem: You Are Interviewing the Vendor’s Best Friends

Selection bias is the whole game. When an agency supplies references, it supplies the two or three accounts where everything went right, the relationship survived, and the contact still owes them a favour, the same effect that makes directory reviews and rating badges weaker evidence than buyers assume. That sample is not representative of the vendor’s median project; it is the top decile.

Three structural problems make this worse in agency hiring specifically.

Team substitution is invisible on paper. The senior architects who joined your pitch call are frequently the agency’s most billable people and are already allocated. Industry-standard practice at mid-size firms is to rotate them off within 4–8 weeks of kickoff. A curated reference will not raise this unprompted because, on their project, the senior people stayed.

Subcontracting is rarely disclosed. An agency presenting a 40-person team may run 30–50% of delivery through partner firms or contractors in other markets. That is not automatically a problem, but it changes your risk profile on IP ownership, security clearance, and continuity  and it is almost never on the capability deck.

Rework costs land after the reference is happy. The client who gave a glowing reference three weeks post-launch may be six months into a rebuild by the time you call. Post-launch technical debt typically surfaces at the 4–7 month mark, when a new feature requires touching the original codebase. Reference calls timed at handover, before acceptance criteria have been tested in production, miss this entirely.

The result is predictable. Teams treat the technical reference check software vendor stage as a formality, spend 6–10 weeks on vendor due diligence, run three reference calls in the final week, hear nothing alarming, and sign. Then they discover in month two that the tech lead has been reassigned and two of the five “core team” developers were contracted through a third party.

The 12-Question Reference Check Script for Vendor Evaluation

This section is the operational core. A reference check script vendor teams can run consistently beats improvised conversation for any technical reference check software vendor process, because consistency is what lets you compare three agencies against each other instead of against your own mood on a given afternoon.

Budget 35–45 minutes per call. Anything under 20 minutes has not gone past the surface. Run at least four calls per finalist vendor, and insist that at least one comes from a source the vendor did not provide.

Before the Call: Setup That Determines the Outcome

Send a two-line email, not a calendar invite with an agenda attached. People answer honestly when they have not rehearsed.

Ask for the client’s engineering counterpart, not the project sponsor. The VP who signed the contract will tell you the vendor was collaborative. The engineering manager who reviewed the pull requests will tell you what the code looked like, applying the same technical vetting lens you would use on the agency directly.

State up front that you are not recording and that nothing will be attributed back to the vendor. Then be true to that. Your agency vetting process depends on candour you cannot compel.

The 12 Questions to Ask, In Order

Ask these in sequence. The early questions build rapport; the hard ones land after the reference is already giving you detail.

1. Walk me through the first 30 days  what did their team actually produce in that window? Ramp-up speed is the single best predictor of whether an agency has done this domain before. Vague answers mean a slow start you will pay for.
2. Who from the pitch team stayed on the project, and for how long? This is your bait-and-switch test. Ask for names and months, not impressions.
3. At peak, how many people were on the project, and how many were subcontracted or freelance? Most clients know this number. If they do not, that itself tells you how transparent the staffing was, and whether you are effectively buying a staff augmentation partner under an agency label.
4. What was the original quoted budget and timeline, and what was the final number? Push for figures. A 10–15% variance is normal. A 40%+ variance needs an explanation you find credible.
5. How did they handle the first serious disagreement about scope? You are testing for change orders used as a revenue mechanism versus genuine scope negotiation.
6. When something broke, who did you escalate to, and how fast did you get a response? A named escalation path with a stated response window is a maturity signal. “We messaged the PM on Slack” is not.
7. What did the codebase look like to the next engineer who touched it? Ask specifically about test coverage, CI setup, README quality, and inline documentation.
8. What did you have to fix or rebuild in the six months after launch? This is the highest-value question in the script and the one most people skip.
9. How did they manage credentials, environment access, and security offboarding? Ask whether access was revoked at project close and how long that took.
10. What did the handover look like  and could your internal team run the system without them after 30 days? Weak handover documentation is how agencies engineer dependency.
11. Which of their people would you take back tomorrow, and which would you decline? The named answers here are gold. They tell you exactly who to name in your own contract.
12. Can you point me to someone else who worked with them  ideally on an engagement that did not go smoothly? This question is why the call exists. It is covered in full below.

How to Ask for the Project That Went Badly

Question 12 is the one that changes a technical reference check software vendor exercise from theatre into diligence. Ask the vendor directly: “Give me a reference from a project that went sideways, one where the client was unhappy at some point.”

Roughly half of agencies will refuse or deflect. That refusal is itself information, and it is cleaner information than most of what you will get on the call. The other half will produce someone, and those conversations are consistently the most useful in the entire process.

A vendor that can hand you a client who says “month four was a disaster, here is what they did about it” is demonstrating two things at once: that they have recovered a failing project, and that the relationship survived the failure. Neither is provable from a success story.

Frame it as a strength test, not an accusation. “Every agency has had a project go wrong. I’m more interested in how you recovered than in whether it happened.” That phrasing produces a real name far more often than a demand does.

How to Find Unlisted Past Clients via LinkedIn and App Credits

The references you find yourself are worth more than the ones you are given, because nobody prepared them. Five reliable sourcing routes:

• LinkedIn employee history. Search current and former employees of the agency. Individual contributors describe projects by client name in their experience sections far more often than the company’s marketing does. Former employees are the most candid source available; they have no incentive to protect the relationship.
• LinkedIn recommendations and tagged posts. Client-side managers write recommendations for agency developers by name. Agency posts tagging a client at launch remain visible long after the case study is pulled from the website.
• App store developer credits. On the App Store and Google Play, the developer account, support URL, and privacy policy domain frequently point back to the building agency even when the app is white-labelled. Cross-reference apps sharing a support domain to map an unlisted client roster.
• Code and infrastructure fingerprints. Public GitHub repositories, commit author domains, package registry publishers, and analytics or CDN configurations often survive handover. A site’s footer credit, WHOIS records, and old job postings referencing a specific client project work the same way.
• Conference talks, podcasts, and award submissions. Agencies submit client work to industry awards under the client’s name, then never link it publicly. Award directories are searchable.

The goal is two self-sourced references per finalist in every technical reference check software vendor cycle you run. When you call, be straightforward: you are evaluating the agency and would value 20 minutes. Acceptance rates are lower than with warm introductions, but signal quality is far higher.

Reading the Answers: Signal Versus Noise

Specificity is the marker of truth. A reference who names the tech lead, remembers the sprint where things slipped, and can quote the change-order amount is describing something that happened. A reference who says the team was “great to work with” and “very responsive” is describing a feeling.

Watch for hedged praise. “They were strong on the front end” usually means the backend was a problem. Follow every qualified compliment with “and on the other side of that?”

Case Studies: What This Surfaces in Practice

A Series A fintech, 5-month platform build. The client ran four reference calls on their preferred agency, including two sourced from LinkedIn employee histories. Both self-sourced references independently reported that the named solutions architect had rotated off their projects within six weeks. The client renegotiated to name the architect in the SOW with a minimum 70% allocation clause tied to milestone payments, and the engagement shipped within 8% of the original 5-month timeline.

A logistics company replacing an outsourced mobile app. During a technical reference check software vendor review, one past client disclosed that roughly 60% of the delivery team had been subcontracted to a partner firm across two delivery markets, a detail absent from the capability deck. The client shifted to a different shortlisted agency with an in-house team, avoided an estimated $40,000–$60,000 in projected rework based on the reference’s own post-launch fix costs, and cut onboarding ramp from six weeks to two.

A Scoring Framework for Reference Sources

Not all references carry equal weight. Weight each technical reference check software vendor conversation by how much the vendor controlled the introduction  that single variable predicts bias better than anything else.

Score each finalist across the twelve questions on a simple 1–3 scale, then weight self-sourced calls at double the value of supplied ones. Three vendors that all “sounded good” will separate immediately once the weighting is applied.

Platform-level verification closes the remaining gap. Marketplaces that run layered checks on website ownership, email domains, team details, and review authenticity before a profile goes live  GetProjects among them  remove the basic identity and capacity questions from your call, which is one of the practical arguments for direct-matching models over open bidding.

What Most Teams Get Wrong

The most common failure is not asking bad questions. It is asking good questions of the wrong person.

Procurement teams default to the client-side executive sponsor because that is who the vendor offers. Sponsors evaluate vendors on communication, invoicing, and whether the relationship felt good. They genuinely do not know whether test coverage was 12% or 80%, and they were not the ones who spent two quarters paying down someone else’s shortcuts. Ask the sponsor for their engineering lead’s contact, then call that person.

The second mistake is running references last. By the time you are checking references, you have spent eight weeks, built internal consensus, and told your CEO which agency you prefer. Confirmation bias at that stage is close to total, and because reversing the decision means restarting the process, ambiguous answers get read charitably. Move at least two calls to the shortlist stage, before preference hardens  a technical reference check software vendor process run early can still change the answer.

The third is treating a refusal as neutral. When a vendor cannot produce a single client from a difficult engagement, teams tend to record it as “declined to provide” and move on. In a properly run technical reference check software vendor process, that is a scoring event, not a blank. It means either the agency has no recovered relationships or it is unwilling to expose one.

The fourth is accepting a reference who cannot answer question 8. If a past client does not know what was fixed or rebuilt after launch, they were not close enough to delivery to be useful to you. Ask for someone who was.

Before You Sign, Make the Calls Count

How to call agency references is a skill that compounds: the same twelve questions, run consistently across three finalists, will separate vendors that look identical on paper. Build your technical reference check software vendor script into the procurement checklist, weight self-sourced calls higher than supplied ones, and never sign without asking for the project that went wrong.

If you are shortlisting agencies and want to start from a pool where the basic  company legitimacy, team details, and review authenticity  are already verified, GetProjects connects businesses directly with vetted IT companies at zero commission, with no bidding and no listing fees. Post a project in under two minutes, then spend your diligence time where it matters: on the calls.

Frequently Asked Questions

How many references should you check before hiring a software vendor? 

Four per finalist is the working minimum: two supplied by the vendor, one from a difficult engagement, and at least one you sourced independently. Fewer than three gives you no way to distinguish a pattern from an outlier. Above six, returns drop sharply  by then the same themes repeat, and the marginal call adds little.

What questions should you ask a software vendor’s past clients? 

Focus past client questions software buyers usually skip: team continuity after kickoff, subcontracting percentage, budget variance from the original quote, escalation response times, and what needed rebuilding in the six months after launch. Avoid satisfaction questions entirely; they produce agreeable answers that do not differentiate between vendors.

Can you talk to a vendor’s clients that aren’t on their reference list? 

Yes, and you should. Nothing prevents you from identifying past clients through LinkedIn employee histories, app store developer credits, public repositories, or award directories and contacting them directly. Be transparent about why you are calling. Self-sourced references consistently produce the most accurate picture of median delivery quality.

What are the red flags in a vendor reference call? 

Vagueness about team names and dates, inability to recall budget figures, hedged compliments about one discipline, and any reluctance to discuss post-launch support. On the vendor side, the strongest red flag is a refusal to supply a reference from a project that went badly; it usually indicates there is no recovered relationship to point to.

How do you verify agency claims references cannot confirm? 

To verify agency claims references have no visibility into  certifications, team size, financial stability  use independent channels: company registry filings, verified marketplace profiles, public repositories, and the agency’s own hiring activity. Cross-check the headcount on their website against their LinkedIn employee count; a large gap is worth asking about directly.

How long should a technical reference check software vendor call take?

Plan 35–45 minutes. The first 10 minutes establish rapport and produce polite answers; the useful material arrives after that. Calls capped at 15 minutes rarely move past the surface and are the main reason reference checks get dismissed as a formality.

What if a vendor refuses to provide any references at all? 

Treat it as disqualifying for engagements above roughly $25,000. Below that threshold, a young agency may genuinely have NDA constraints  in which case ask for a technical conversation with the engineers who would be assigned, plus a paid two-week pilot before committing to a full scope.