How We Test

Open Developer exists to be honest about AI in marketing — including about our own process. This page explains exactly how every score on this site is built, so you can check our work instead of taking it on faith.

How We Research

Every review, hub ranking, and roundup on this site is built from desk research and aggregated public data, not hands-on product trials. That means: vendor documentation, pricing pages, G2/Capterra review patterns, case studies the company itself publishes, and cross-referenced third-party coverage. We do not accept vendor briefings, demos, or free access in exchange for coverage, and nothing on this site is sponsored — see our editorial independence policy.

We chose this method deliberately rather than claiming hands-on testing we don't actually have capacity to run consistently across 100+ tools and agencies. A site that claims "tested" on every page but can't possibly have used every product in depth is making the exact kind of unverifiable claim this site exists to push back on. When a source disagrees with another source — pricing is the most common example — we say so explicitly on the page rather than quietly picking a number and presenting it as settled fact.

The Rating Scale

Every tool and agency gets an overall score out of 10, shown to one decimal place (e.g. 7.2/10). We use a granular decimal scale instead of 5 stars deliberately: a 5-star scale collapses almost everything into 4 or 4.5 stars, which is exactly the credibility problem this site is positioned against. A decimal scale lets close competitors actually look different from each other when they perform differently.

The overall score is never a standalone judgment call — it's a weighted average of six sub-scores, each shown individually on every review page. You can see exactly how a tool's number was built, not just trust that it was built well.

The Six Criteria

Proof of Results

Weight: 30%

The heaviest-weighted criterion, deliberately. This measures the strength and verifiability of a tool or agency's documented outcomes: published case studies, customer logos that hold up to scrutiny, third-party review platform standing (G2, Capterra), and funding/traction signals where relevant. Marketing claims with no backing score low here regardless of how the rest of the product looks.

Output Quality

Weight: 25%

For tools: the quality of what the product actually produces — content, creative, data, recommendations — based on aggregated user reports and documented capability, not our own generated samples. For services: the quality of the deliverables agencies are documented as producing for clients.

Ease of Use

Weight: 15%

How steep the learning curve is, based on consistent patterns across multiple independent reviews — not a single reviewer's opinion, but where multiple sources agree a product's onboarding, UI, or documentation is a genuine friction point (or genuine strength).

Pricing / Value

Weight: 15%

Whether the publicly available pricing represents fair value for what's included, and — importantly — whether that pricing is even clearly and consistently reported. A product with confusing, inconsistently-reported, or bait-tier pricing scores lower here even if the underlying product is strong, because pricing opacity is itself a real cost to a buyer's time.

Customer Support

Weight: 10%

Documented support quality and responsiveness, drawn from review-platform patterns and what's actually included at each pricing tier (versus what's reserved for the highest, often unpriced, enterprise tier).

AI Transparency

Weight: 5%

How openly a company explains how its AI actually works — what data it's trained or run on, how it makes recommendations, and whether it discloses limitations. This is weighted lightly because it's rarely the deciding factor in a buying decision, but we track it because a site about being open on AI in marketing should hold vendors to the same standard we hold ourselves.

Tools vs. Services Rubric

Tools and services share the same six criteria and weights, but the criteria are read differently for each:

Re-Research Cadence

Every live review is re-researched on a quarterly cycle at minimum, and sooner if a tool ships a major version change, pricing update, or funding event that would plausibly move its score. Every review page shows a "Last researched" date so you can see exactly how current the page is without guessing.

Corrections

If you find an error — a stale price, a discontinued feature, a competitor we've mischaracterized — we want to know. Corrections get applied and dated, not quietly buried. This is a young site and we'd rather be told we're wrong than stay wrong.

What We Don't Do

• We don't accept payment, free product access, or affiliate commissions in exchange for placement or score.
• We don't publish a "best of" list and then decide the ranking afterward to match who bought an ad.
• We don't inflate scores to keep vendors happy, and we don't inflate criticism for clicks either.

If any of this ever stops being true, tell us — publicly, if you'd like.