If you've ever sat in a vendor evaluation meeting for a $1M+ enterprise software project, you know the pattern: three or four vendors present nearly identical decks, each claiming to be the "trusted partner" you need. The case studies look similar. The team photos look similar. The certifications all check the same boxes. The hourly rates are within $30 of each other.
Then 18 months later, one of those vendors has shipped exactly what you needed, two have shipped something workable but late and over budget, and one has produced a system that has to be rebuilt by a different vendor at 2x the original cost.
The differences that determine which vendor you become were visible at the evaluation stage — if you knew what to look for.
This guide is written for the CIO, CTO, VP of Engineering, procurement leader, or CFO who is actually trying to select an enterprise software development partner and avoid the failure modes that consume 75% of large-scale enterprise software projects. It's not a sales pitch. It's the framework experienced enterprise buyers use to separate vendors who will deliver from vendors who will sell.
We'll cover the 14 questions that surface real capability, the answers that should impress you, the answers that should disqualify a vendor, and the procurement red flags that almost always predict project failure.
Why Enterprise Software Vendor Selection Is Harder Than It Looks
Before the questions, the structural problem worth acknowledging:
The most polished vendor in your evaluation is rarely the most capable. The vendor with the best sales process is almost never the vendor with the best engineering process.
This isn't cynicism. It's pattern recognition from hundreds of enterprise software audits. The reasons:
- Sales sophistication and engineering quality are independent variables. A vendor can have a world-class sales motion and mediocre engineering. The reverse is also common — exceptional engineering shops with weaker sales operations.
- Case studies are marketing artifacts, not engineering evidence. Most case studies are written by marketing teams reverse-engineering success stories. They show outcomes, not the engineering decisions that produced those outcomes.
- Reference calls are pre-curated. Vendors only provide references they've already vetted. You're hearing from clients who liked them — not from the clients who fired them.
- RFP responses reward writing skill, not engineering rigor. A well-written RFP response can mask underlying capability gaps. The vendor with the best technical writer often beats the vendor with the best architects.
The 14 questions below are designed to cut through this. Each one is built to surface real capability that's harder to fake than a polished deck.
The 14 Questions That Surface Real Capability
1. "Walk us through a project you delivered that went badly. What happened, what did you do, and what did you change?"
Why it matters: Every vendor has failed projects. The honest ones can describe them in technical detail with self-aware analysis. The dishonest ones say "we've never had a project go badly," or describe a failure that's actually a humble-brag.
Good answer: Specific project details, specific decisions that went wrong, specific changes to internal process that resulted. Names of frameworks, architectural choices, governance failures.
Disqualifying answer: "All our projects succeed." Or a vague answer that blames the client for unclear requirements. Both signal a vendor that doesn't learn from failure — and you don't want to be their next case study.
2. "What does your discovery phase actually look like, and what does it cost?"
Why it matters: Discovery is where enterprise software projects succeed or fail. Vendors that skip or rush discovery are pricing on assumptions that won't survive contact with your real systems. The cost of a proper discovery phase is typically 10–15% of total project budget — and good vendors charge for it because they invest real senior engineering time.
Good answer: A defined 4–8 week discovery process with specific deliverables (architecture documents, integration map, risk register, capacity plan, technical spike outputs). Senior engineers — not just business analysts — involved. A separate, paid statement of work.
Disqualifying answer: "We include discovery in the build cost." Or "We can start coding next week." Both mean discovery is being skipped — and the cost will surface later as change orders.
3. "Who are the senior engineers we'd be working with, and what's their current allocation?"
Why it matters: The "bait and switch" problem is endemic in enterprise software vendor selection. Sales presents senior architects. The actual project gets staffed with junior engineers because the senior people are over-allocated.
Good answer: Named individuals, with LinkedIn profiles, current project allocations, and contractual commitments about who will actually work on your project. Specific percentages of senior engineer time guaranteed.
Disqualifying answer: "We'll assemble the right team once we kick off." Or evasiveness when asked about specific allocations. Both mean the team you'll get isn't the team you saw in the pitch.
4. "What's your engineering process for code quality, and how do you measure it?"
Why it matters: Code quality compounds across the lifetime of an enterprise system. The difference between a vendor with rigorous code review, automated testing, and CI/CD discipline versus one without it is the difference between a system that ages well and one that becomes unmaintainable in three years.
Good answer: Specific practices — pull request reviews with named reviewers, code coverage thresholds, static analysis tools, security scanning, automated testing pyramids. Measurable metrics they can show in their internal dashboards. Mention of CMMI Level 3 or higher process maturity is a strong signal.
Disqualifying answer: "We use modern best practices." Vague answers that don't include specific tools, metrics, or accountability mechanisms.
5. "How do you handle scope changes, and what does your contract say about them?"
Why it matters: Industry data is consistent — 80% of enterprise software projects experience meaningful scope changes mid-flight. The vendor's contract structure determines whether those changes become collaborative adjustments or adversarial change orders.
Good answer: A clearly defined change management process. Pre-defined hourly rates for in-scope vs out-of-scope work. A specific change request workflow with timeboxed evaluations. A willingness to share past change order patterns and their typical magnitude.
Disqualifying answer: "We rarely have scope changes." Or "We'll figure it out as we go." Both signal a vendor that will use scope ambiguity as a revenue lever later.
6. "Show us your security and compliance certifications — and your ongoing audit cadence."
Why it matters: For enterprise software touching regulated data, security posture isn't a feature — it's a baseline requirement. Vendors who can't satisfy your compliance review during evaluation will become a compliance liability after deployment.
Good answer: Specific, current certifications: ISO 27001, ISO 9001:2015, CMMI Level 3 or higher, SOC 2 Type II. Annual audit cadence with named auditors. Compliance documentation available on request, not just marketed.
Disqualifying answer: "We follow industry best practices." Or expired certifications. Or claims about certifications they don't actually hold.
7. "What's your team retention rate, and how do you handle handovers if a key engineer leaves?"
Why it matters: Enterprise software projects span 12–24 months. If your vendor's key engineers leave mid-project, the knowledge transfer cost is brutal. Some vendors have annual engineering turnover above 30% — meaning by the time your project ships, a third of the team that started it has been replaced.
Good answer: Specific retention metrics for senior engineers (industry-leading retention is 85%+ annual). Documented onboarding and offboarding processes. Specific examples of how they've handled key-person transitions on past projects.
Disqualifying answer: Evasiveness about retention rates. Or "We have a deep bench" without specifics. Both usually mean turnover is high.
8. "Walk us through how you'd architect a system that handles [specific complexity from your domain]."
Why it matters: This is the question that separates engineering teams from sales teams. A real engineering partner can whiteboard your problem in real time. A sales team will defer to a follow-up meeting where their architects can prep an answer.
Good answer: Live architectural reasoning with specific trade-off discussion. Real questions about your constraints. Multiple architectural options with stated pros and cons. Honest acknowledgment of what they don't know yet.
Disqualifying answer: A pre-canned diagram that looks the same as the one they'd show any other prospect. Or "Let me get our architect on a follow-up call." The latter is fine for deeper sessions, but they should be able to engage substantively in the first conversation.
9. "What technologies have you actively built production systems with in the last 12 months?"
Why it matters: Enterprise software portfolios on vendor websites often list every technology under the sun. The reality is that most vendors have deep expertise in 5–10 technologies and surface familiarity with another 30 they've put on the website.
Good answer: A specific, narrower list of technologies they've shipped in production within the last year. Honest acknowledgment of technologies they know but haven't recently deployed. Specific named senior engineers tied to specific technology specializations.
Disqualifying answer: "We work with everything." Or a list that includes more technologies than the company could possibly have meaningful experience with given their team size.
10. "How do you handle production incidents, and what's your on-call structure during the warranty period?"
Why it matters: Enterprise software fails. The question is what happens at 2 AM when it does. Vendors with mature operations have specific runbooks, SLAs, and incident response processes. Vendors without them will leave you on your own when the system has its first major outage.
Good answer: Defined SLAs for response time and resolution. Specific on-call rotation structure during warranty. Post-incident review processes with documented learnings. Examples of past incidents and how they were handled.
Disqualifying answer: "We'll be there if anything happens." No specifics. No documented SLA. Both mean you'll be writing the incident response process yourself when production goes down.
11. "What does your handover and documentation process look like at the end of the engagement?"
Why it matters: Vendor lock-in is the most expensive long-term cost in enterprise software. If your vendor's documentation is poor and their handover process is minimal, you're locked into them for years — at whatever rates they choose to charge you for ongoing maintenance.
Good answer: Specific documentation deliverables (architecture decisions records, runbooks, deployment guides, code annotations). A formal knowledge transfer period built into the contract. Clear ownership of all artifacts. A documented process for transitioning to a different vendor or internal team if you choose to.
Disqualifying answer: "We provide standard documentation." Vague answers about handover. Both are signals that the vendor's business model depends on you not being able to leave them easily.
12. "Can we talk to clients you've worked with for 3+ years, including any whose contracts ended?"
Why it matters: Most vendor references are recent successes. The more telling references are long-term relationships and ended relationships. Long-term clients reveal whether the vendor is good to work with over time. Ended relationships reveal how the vendor behaves when business is ending — which is when many vendors become difficult.
Good answer: Willingness to provide both types of references. Specific names and contact details. Permission to ask anything.
Disqualifying answer: Only recent client references. Refusal to provide ended-relationship references. Both signal that long-term satisfaction and graceful exits aren't part of their typical experience.
13. "How do you price the work, and what's your gross margin?"
Why it matters: This question feels invasive, but it surfaces critical information. Vendors operating at unsustainable margins cut corners on engineering quality. Vendors with very high margins are extracting more value than they're delivering. Vendors in the 30–45% gross margin range are typically running healthy, sustainable engineering operations.
Good answer: Willingness to engage with the question. Some directional transparency about cost structure. Confidence that their pricing reflects real engineering investment.
Disqualifying answer: Strong evasion. Pure rate-card answers without context. Either extreme — unsustainably cheap or unjustifiably expensive without a clear quality differentiation.
14. "If we were one of your clients today, what would you tell us we're getting wrong?"
Why it matters: This is the meta-question that reveals everything. Vendors who can immediately surface honest, useful observations about your situation are demonstrating the strategic thinking that defines real partners. Vendors who give vague compliments or generic best-practice advice are demonstrating that they can't add strategic value beyond execution.
Good answer: Specific, sometimes uncomfortable observations about your existing strategy, technology choices, or organizational dynamics. Constructive framing. Honest acknowledgment of where they're uncertain.
Disqualifying answer: "You're doing everything right!" Or generic advice that could apply to anyone. Both are signals of a vendor that's selling, not advising.
The Five Red Flags That Almost Always Predict Project Failure
Beyond the 14 questions, these procurement-stage signals correlate strongly with downstream project failure. Watch for them carefully.
Red Flag #1: The Vendor Quotes Before They Understand
Vendors who provide firm pricing in the first or second meeting — before any meaningful discovery — are pricing on assumptions. The price might be too low (and they'll change-order their way up) or too high (and they're padding for unknowns). Either way, you're not getting an accurate quote. You're getting a sales tactic.
A real enterprise partner says: "We can give you a directional range based on the parameters you've described, but we won't commit to a firm price until we've completed discovery." That's the answer of a partner. The vendor that gives you a firm $850,000 in the first meeting is selling, not estimating.
Red Flag #2: The Sales Team Is Better Than the Technical Team
In your evaluation meetings, watch the dynamic. If the salesperson dominates and the technical lead defers to them on questions about architecture, that's the operating model — sales drives, engineering executes. After the contract is signed, you'll deal with engineers who weren't empowered in the sales process and aren't empowered in delivery either.
Healthy vendors have technical leadership that runs the room when the conversation gets technical. The salesperson sets context, but the architect drives the substantive discussion.
Red Flag #3: They Can't Articulate What Kind of Work They Don't Do
Every serious engineering shop has a focus. They specialize in certain types of work — enterprise SaaS, regulated industries, specific tech stacks — and decline work outside that focus. Vendors who claim to do "all kinds of work for all kinds of clients" either lack focus (and therefore lack depth) or are willing to take work they can't deliver well.
Ask: "What kind of project would you turn down, and why?" A good vendor has a clear answer. A vendor that struggles with this question is a vendor that will struggle with your project if it's outside their actual capability — but they'll take it anyway.
Red Flag #4: The References Sound Coached
Reference calls have a predictable cadence when they're rehearsed. The reference uses the same phrases as the vendor's sales deck. They struggle to recall specific details. They redirect questions about challenges back to outcomes.
Real references talk about challenges openly, mention specific people by name, recall specific decisions, and describe both the good and the difficult parts of the engagement. If a reference call sounds like a testimonial, it probably is one.
Red Flag #5: The Contract Heavily Favors the Vendor on Risk
Read the contract carefully. Specifically look at:
- IP ownership clauses (you should own everything built for you, period)
- Limitation of liability (some limitation is reasonable; capping liability at 1x fees is unreasonable)
- Termination rights (you should be able to terminate for convenience with reasonable notice)
- Documentation deliverables (specifically listed, not left to "industry standard")
- Knowledge transfer obligations (specific, time-bound)
Vendors whose standard contracts heavily favor them on these dimensions are signaling how they'll behave when the relationship is under stress.
The Procurement-Stage Evaluation Framework
Beyond the questions and red flags, here's the structured framework experienced enterprise buyers use to score vendors objectively:
| Evaluation Dimension | Weight | What "Strong" Looks Like |
|---|---|---|
| Engineering depth in your specific tech stack | 20% | 12+ months of recent production work, named senior engineers, specific case studies |
| Industry domain expertise | 15% | Multiple projects in your industry, regulatory familiarity, specific compliance experience |
| Process maturity (certifications, governance) | 15% | ISO 27001, CMMI Level 3+, documented engineering practices |
| Cultural fit and communication | 10% | Time zone alignment, communication cadence, English proficiency, working-style fit |
| Pricing competitiveness | 15% | Within reasonable market range for the quality tier; not the lowest, not the highest |
| Risk profile (financial stability, retention) | 10% | Strong financials, low engineering turnover, established track record |
| Reference quality | 10% | Strong long-term references including ended engagements |
| Contract terms | 5% | Balanced IP, liability, termination, and handover clauses |
The vendor that scores highest on this framework is the vendor most likely to deliver. The vendor that feels most polished in the room is often a different one.
What Strong Vendor Evaluation Actually Looks Like
For enterprise software projects above $500K, the evaluation process itself is a meaningful investment. The math is clear: spending 60–90 days on rigorous vendor evaluation is the highest-ROI activity in the entire project, because the vendor choice determines whether the project succeeds or joins the 75% failure statistic.
A robust evaluation process for enterprise software typically includes:
- Initial RFP with 20–30 specific questions (most enterprise RFPs ask too few, and too many of those questions reward writing skill rather than engineering depth)
- Technical deep-dive sessions (4–6 hours per shortlisted vendor, with your technical leadership engaging directly)
- Reference calls including at least one long-term client and one ended-relationship client
- Architecture exercise — a paid, scoped technical assignment that gives you direct evidence of how the vendor thinks and works
- Pilot engagement for high-stakes projects (a 4–8 week scoped pilot before committing to the full build is a powerful de-risking tool, even at the cost of a slight delay)
- Procurement review of contract terms, financial stability, and risk profile
Vendors who object to this process are the vendors most likely to disappoint you. Vendors who embrace it are demonstrating exactly the kind of rigor you want for the actual project.
The Bottom Line
Enterprise software vendor selection is the single highest-leverage decision in any enterprise software project. Get it right, and the project has a fighting chance even when things go sideways. Get it wrong, and no amount of project management can rescue it.
The 14 questions in this guide are designed to surface real capability that's harder to fake than polished decks. The five red flags are designed to identify the failure patterns that predict project failure before contract signing. The evaluation framework is designed to introduce objectivity into a process that's typically dominated by relationship dynamics.
Use them. Apply them rigorously. Reject the vendors who can't pass them, even when the relationship dynamics make rejection uncomfortable.
The right starting question for any enterprise software vendor evaluation isn't "who has the best price?" or "who has the best deck?"
It's "who is most likely to be a trusted partner three years from now, when this system is the operational backbone of our business?"
That's a conversation worth having before the first proposal arrives.