
Supplier Evaluation Criteria: A Weighted Scoring Framework
Learn how to build a systematic supplier evaluation framework with weighted scoring criteria. Includes templates, scoring rubrics, and practical guidance for technology vendor assessment.

Technology Readiness Levels (TRL) provide a standardized framework for assessing how mature a technology is — from basic research to proven operational capability. Originally conceived at NASA in 1974 by Stan Sadin and later formalized into the 9-level scale by John Mankins in 1995, TRL was adopted by the U.S. Department of Defense in 2001 and codified as an international standard (ISO 16290:2013). Today, TRL assessment has become essential for R&D portfolio management, technology scouting, and investment decisions.
Yet many organizations struggle to apply TRL consistently. The most common problem: technology providers claim TRL 6-7 for technologies that, under rigorous assessment, are actually TRL 3-4. This gap between claimed and actual readiness has derailed countless technology acquisition projects. What separates TRL 4 from TRL 5? How do you assess technologies outside your technical expertise? When should TRL influence decisions?
This guide provides a practical checklist for TRL assessment, with detailed criteria for each level and common pitfalls to avoid.
| TRL | Name | Summary |
|---|---|---|
| 1 | Basic principles | Scientific research is beginning |
| 2 | Technology concept | Practical applications identified |
| 3 | Proof of concept | Analytical and lab studies prove concept |
| 4 | Lab validation | Technology validated in lab environment |
| 5 | Relevant environment validation | Technology validated in relevant environment |
| 6 | Relevant environment demonstration | Prototype demonstrated in relevant environment |
| 7 | Operational environment demonstration | Prototype demonstrated in operational environment |
| 8 | System complete | Technology proven to work in final form |
| 9 | Mission proven | Technology proven through successful operation |
To make TRL assessment concrete, we'll follow a single manufacturing example through the scale: imagine a novel anti-corrosion nanocoating being developed for industrial steel components. At each grouping, we'll show what this technology looks like at that maturity level.
At TRL 1-3, you're answering a fundamental question: does the science hold up? Investment is low (typically small teams, lab budgets), but technical risk is highest. Most technologies never leave this phase — and that's normal. The key evidence at this stage is published research, lab demonstrations, and proof-of-concept results.
Manufacturing example: A university research group publishes a paper showing that a new nanoparticle formulation repels salt spray corrosion on steel coupons in a controlled lab environment. The principle works. But nobody has tested it on real production parts, at industrial scale, or under actual operating conditions.
Definition: Scientific research begins to be translated into applied research and development. Basic principles are observed and reported.
Checklist:
Evidence types:
Assessment questions:
Common at TRL 1:
Definition: Invention begins. Basic principles are observed and practical applications are identified, but no experimental proof exists.
Checklist:
Evidence types:
Assessment questions:
Common at TRL 2:
Definition: Active research and development is initiated. Analytical and laboratory studies demonstrate that the concept is viable.
Checklist:
Evidence types:
Assessment questions:
Common at TRL 3:
This is where costs escalate sharply — often 5-10x the research phase budget — because you're moving from "does the science work?" to "can we engineer a real solution?" Most technology failures happen in this phase, which is why stage-gate decisions are critical here. The key question shifts from scientific feasibility to engineering viability.
Manufacturing example: The nanocoating team has moved from steel coupons to testing on actual stamped automotive body panels. They've built a prototype spray application system and are testing in a climate chamber that simulates 10 years of road salt exposure. The coating works — but adhesion degrades after thermal cycling, application thickness is inconsistent at production line speeds, and the nanoparticle supply chain doesn't exist at scale yet. These are TRL 4-6 problems: the science is proven, but the engineering isn't.
Definition: Basic technological components are integrated to establish that they will work together. Integration is relatively primitive compared to the eventual system.
Checklist:
Evidence types:
Assessment questions:
Common at TRL 4:
Definition: The basic technological components are integrated and tested in a simulated or somewhat realistic environment.
Checklist:
Evidence types:
Assessment questions:
Critical distinction from TRL 4:
Definition: A representative model or prototype system is demonstrated in a relevant environment.
Checklist:
Evidence types:
Assessment questions:
Common at TRL 6:
At TRL 7-9, technical risk drops but investment peaks — production tooling, qualification testing, regulatory compliance, and supply chain buildout can consume 60-80% of total development cost. The question becomes: can we manufacture this reliably, at scale, at target cost? Many technologies that performed brilliantly at TRL 6 stumble here when they encounter real manufacturing variability, supply chain constraints, and customer-site conditions.
Manufacturing example: The nanocoating is now being applied to production parts on an actual automotive paint line during a scheduled production run. The spray system is integrated with the existing paint booth infrastructure. They're testing with real production variability — different steel batches, ambient humidity fluctuations, operator-to-operator variation. The coating meets corrosion specifications, but cycle time is 8 seconds longer than the standard clear coat, creating a throughput bottleneck. That's a TRL 7 problem: the technology works in the real environment, but it's not yet optimized for production economics.
Definition: A full-scale prototype is demonstrated in an operational environment - the real world.
Checklist:
Evidence types:
Assessment questions:
Common at TRL 7:
Definition: The technology has been proven to work in its final form and under expected conditions. Qualification testing is complete.
Checklist:
Evidence types:
Assessment questions:
Common at TRL 8:
Definition: Actual system proven through successful mission operations.
Checklist:
Evidence types:
Assessment questions:
Common at TRL 9:
When evaluating technologies from vendors, startups, or research partners:
Request specific evidence:
Consider context differences:
Verify with references:
Investment decisions:
Timeline expectations:
Risk assessment:
1. Confusing demonstration with validation
Seeing something work once isn't validation — it's an anecdote. A startup demonstrates their machine vision system detecting defects on 50 sample parts and claims TRL 7. But validation requires performance across thousands of parts, multiple defect types, varying lighting conditions, and production-speed throughput. The gap between "demo" and "validated" is where most TRL inflation lives. Always ask: "How many times, under what conditions, and with what success rate?"
2. Ignoring environment differences
This is the single most costly TRL mistake in manufacturing. A sensor technology proven at TRL 7 in a clean-room semiconductor fab may be TRL 4 in a dusty, vibrating metal stamping plant. Temperature, humidity, contamination, vibration, and electromagnetic interference all differ between environments. An industrial IoT vendor claiming TRL 8 for their predictive maintenance system because it works in a climate-controlled data center is not at TRL 8 for your 40-year-old foundry floor.
3. Assuming linear progression
Technologies can — and do — regress. A battery chemistry that reached TRL 6 in small-format cells may drop back to TRL 4 when scaling to large-format production cells, because thermal management problems that don't exist at small scale become critical at large scale. New requirements (regulatory changes, updated customer specifications) can also reset TRL assessment.
4. Over-crediting vendor claims
Vendors naturally present favorable TRL assessments — their sales process depends on it. In our experience, independent assessment typically finds technologies are 1-2 TRL levels lower than vendor claims. This is especially pronounced with startups, where the pressure to show investor-ready maturity creates incentives to conflate lab success (TRL 3-4) with deployment readiness (TRL 6-7). For critical technology acquisition decisions, always conduct independent technical due diligence.
5. Forgetting subsystem variation
Complex systems have multiple subsystems at different TRLs. A new robotic welding cell might have a TRL 9 robot arm, TRL 8 welding power supply, TRL 6 seam-tracking vision system, and TRL 4 adaptive parameter controller. The overall system TRL is limited by the lowest critical subsystem — in this case, TRL 4. Vendors who quote the highest subsystem TRL as the system TRL are being misleading.
When you need a rapid TRL estimate — during a scouting call, at a conference demo, or in a first vendor meeting — use these five questions:
Q1: Has the core concept been proven in any form? → No → TRL 1-2 (concept/theory only) → Yes, in lab → continue to Q2
Q2: Has it been tested outside a controlled lab environment? → No, only lab conditions → TRL 3-4 (lab-validated) → Yes, in simulated or semi-real conditions → continue to Q3
Q3: Has a representative prototype been demonstrated in conditions close to actual use? → No, only simplified or partial conditions → TRL 5 (relevant environment) → Yes, a realistic prototype in realistic conditions → continue to Q4
Q4: Has a full-scale system been tested in actual operational conditions (real factory, real process, real users)? → No, only scaled-down or simulated operations → TRL 6 (demonstrated in relevant environment) → Yes, full-scale in real operations → continue to Q5
Q5: Is it in routine, repeatable production use with documented performance data? → No, only tested or piloted → TRL 7-8 (operational demo / qualified) → Yes, multiple deployments with proven track record → TRL 9 (mission proven)
This gets you to within ±1 TRL level. For critical decisions, follow up with the full checklist below.
Use this worksheet for each technology assessment:
Technology name: _________________
Assessment date: _________________
Assessor: _________________
Claimed TRL: ___ Assessed TRL: ___
Evidence reviewed:
Environment assessment:
Key evidence supporting assessed TRL:
Gaps or concerns:
Recommendations: _________________
Effective TRL assessment requires understanding the technology landscape:
Technology intelligence platforms help R&D teams contextualize TRL assessments against market data, enabling more informed maturity evaluations.
Yes. A technology may be TRL 9 for one application but TRL 5 for another requiring different operating conditions. Always assess TRL relative to your specific requirements.
The same framework applies, but evidence differs. Software demonstrations may be easier, but scalability and integration challenges still exist. Consider computational requirements, data dependencies, and deployment environment.
Lower TRL generally means higher technical risk. However, high-TRL technologies can still carry implementation, integration, and scaling risks. TRL is one risk factor, not a complete risk assessment.
Reassess when significant new evidence emerges, when requirements change, or at project phase gates. For active development, quarterly reassessment is common.
Technical experts who understand both the technology and the assessment framework. For critical decisions, consider independent technical review.
Yes. If testing reveals problems, requirements change, or scale-up fails, TRL can decrease. This is why ongoing assessment matters.
TRL assessment brings discipline to technology maturity evaluation. By using consistent criteria, requiring evidence, and acknowledging uncertainty, you make better decisions about which technologies to pursue, when to adopt external solutions, and how much risk you're accepting.
This checklist provides a starting point. Adapt it to your industry, technology types, and decision context. What matters most is rigorous, evidence-based assessment that informs rather than replaces good judgment.
Start here: Pick one technology you're currently evaluating — an external solution from a vendor, a startup's pitch, or an internal R&D project. Run it through the Quick Assessment above to get a baseline TRL. Then use the detailed checklist for the relevant TRL range to identify evidence gaps. If the claimed TRL and your assessed TRL differ by more than 1 level, that gap deserves a direct conversation with the technology provider about what evidence they can provide.
Wicely's Solution Scouting platform helps R&D teams systematically find and evaluate external technologies, streamlining the scouting process from discovery to shortlist.

Learn how to build a systematic supplier evaluation framework with weighted scoring criteria. Includes templates, scoring rubrics, and practical guidance for technology vendor assessment.

Learn the fundamentals of technology scouting for manufacturing R&D. Discover how to systematically identify, evaluate, and acquire external technologies that accelerate your innovation goals.

Learn proven techniques for creating unbiased vendor shortlists. Covers cognitive biases, structured evaluation methods, and practical approaches to fair technology vendor selection.