2026 C/C++ Code Review Benchmark

C/C++ Benchmark Results

This 2026 C/C++ benchmark is grounded in 16 real pull requests from 13 major open-source C and C++ projects, each evaluated against a set of human-verified golden comments: the actual issues a thorough code reviewer should catch. Each code review tool is then measured on how well it found those issues and how much irrelevant noise it produced alongside them.

CloudaEye ranks #1 amongst 9 popular code review agents with 70.0% overall F1 score.

Get started Book a demo View OSS benchmark
The Dataset

Thirteen repositories, two languages, sixteen real PRs

The C/C++ benchmark dataset covers real open source pull requests across systems, runtimes, web servers, databases and developer tools.
It complements the broader open source benchmark dataset.

Evaluation Analysis

Benchmark results

View GitHub Project
Golden Comments

Detection by language

Golden comments grouped by C and C++, with inferred detection results from each tool's review comments.

Key Findings — C

CloudAEye Strengths

  • Strong Recall.
  • Caught two independent undefined-behavior bugs from passing a signed char into isalnum and strchr on bytes ≥ 0x80 in the same PR of valkey's cluster config validation.
  • Caught a closed handle returned to the caller — INVALID_HANDLE_VALUE never restored after a 32-iteration retry loop — in libuv's Windows filesystem code.
  • Good at cross-file consistency: caught the same void* type-erasure regression duplicated across both the declaration and the definition of php-src's URI parser.
  • Caught a compile-time logic error where new CLI option parsing was nested inside a native-emit conditional, silently disabling advertised options in MicroPython.

CloudAEye Weaknesses

  • CloudAEye prioritizes logic and memory-safety defects that are directly evidenced in the diff; very narrowly-scoped stylistic issues (a single warning-suppression pragma, one mismatched path literal) are a lower priority.
  • On PRs touching several related call sites, CloudAEye can surface an additional related observation alongside the confirmed issue.

Key Findings — C++

CloudAEye Strengths

  • Strong Recall.
  • Perfect score on carla's road-mesh regression: caught both the continue that skips the loop's only progress step (an infinite loop) and the lane-mark width regression, in the same PR.
  • Detected a data race on an unsynchronized static counter, incremented without a lock while multiple connections initialize concurrently, in dragonfly's disk-backpressure queue.
  • Caught a discarded return value that silently broke a themed color lookup in Windows Terminal's pane-resource resolution.
  • Identified unreachable code immediately after an early return, leaving Tesseract's hOCR document header un-emitted.

CloudAEye Weaknesses

  • On template- and refactor-heavy C++ diffs spanning many call sites, CloudAEye can surface an adjacent, related observation alongside the confirmed issue.
  • Defensive checks in less-traveled branches (e.g. a missing null-check several calls deep) are still a developing area relative to CloudAEye's strength on core logic and memory-safety defects.
Evaluation Pipeline

How the scoring works

Scoring a code review tool is harder than it looks. Tools produce freeform comments that may bundle multiple issues into one, repeat the same finding across a summary and an inline comment, or describe the same bug in completely different words. The benchmark handles this through a four-step automated pipeline.

01. Issue extraction

Each tool's raw review output is passed through an LLM extractor that breaks freeform comments into individual, standalone issue statements, one problem per entry. A comment like "Found two problems: null input isn't handled, and the cache key uses the wrong field" becomes two separate candidates. This prevents a tool from being penalized for bundling findings, and prevents inflated candidate counts from padding scores.

02. Deduplication

Some tools post the same finding twice, once in a PR-level summary and again as an inline comment on the affected line. A second LLM pass groups near-identical candidates before scoring. The rule is strict: two candidates are duplicates only if a single code change would fix both. The same type of bug appearing in two different functions counts as two separate issues. When in doubt, candidates are kept separate.

03. Semantic matching

Each deduplicated candidate is compared against every golden comment in an all-pairs LLM evaluation. The judge determines whether the candidate identifies the same underlying issue as the golden comment, accepting semantic matches regardless of wording. Each match receives a confidence score; when multiple candidates could match the same golden comment, the highest-confidence match wins. A matched candidate's duplicates from Step 2 are also marked matched, so they do not count as false positives.

04. Scoring

Matched candidates count as true positives, unmatched candidates count as false positives, and golden comments with no matching candidate count as false negatives. Those TP, FP, and FN totals produce the Precision, Recall, and F1 scores reported in the leaderboard.

Scoring Mechanism

Precision, Recall, and F1

TP, FP, and FN counts are micro-averaged across the C/C++ evaluation set to produce the final Precision, Recall, and F1 scores reported in the leaderboard.

  • Precision = TP / (TP + FP)

    Of all the issues the tool flagged, what fraction were real?

  • Recall = TP / (TP + FN)

    Of all the golden issues, what fraction did the tool find?

  • F1 = Harmonic mean of Precision and Recall

A tool that posts 100 comments may get high Recall but collapsed Precision. A tool that posts 2 comments may get high Precision but low Recall. F1 rewards the tool that finds the most real issues with the least noise.

Proof View

See the proof

Side-by-side C/C++ benchmark review comments with CloudAEye fixed on the left and a selectable comparison provider on the right.

Why CloudAEye?

CloudAEye delivers code review that works from the very first pull request. Our @cloudaeye /inspect command is built for speed and precision. In this C/C++ benchmark, it achieves the #1 F1 score overall and the strongest precision in the field. Whether your team is shipping new features or hardening security, CloudAEye consistently reduces noise while finding real issues.


For teams that need deeper coverage, @cloudaeye /review goes further. It re evaluates updated code for bugs and logical errors, runs a full security pass aligned with OWASP standards across web applications, LLM and generative AI apps, agentic applications, and MCP servers, executes PR checklists and custom rules, and provides final merge recommendations, all in one pass.


CloudAEye does not just spot issues. @cloudaeye /implement goes a step further and fixes them for you by pushing changes directly to your PR through commit suggestions, so your team can keep moving without context switching.


Beyond identifying issues, CloudAEye enforces the standards your team actually cares about. It automatically tracks docstring and unit test coverage, catches outdated README documentation, and ensures dependency files never fall out of sync. These quality gates run on every PR without any manual effort. If you need something more specific, you can write your own rules in plain English and CloudAEye handles the rest. With more than 75 prebuilt checklist items spanning bugs, security, LLM security, agentic risks, and MCP server vulnerabilities, your entire codebase is covered from first commit to final merge.


Unlike rigid tools, CloudAEye adapts to how your team works. You can toggle only the categories you care about, target specific file patterns or subdirectories, and tailor every review to your repository's standards. From day one inspection to enterprise grade compliance, CloudAEye grows with your team.

Key Takeaways

#1 Code review tool

The results speak for themselves. CloudAEye ranks #1 overall and leads across every major real world filter, including correctness, security, feature PRs, cross file analysis, authentication, and high risk PRs. Customers who demand precision without noise and recall without blind spots consistently achieve the best outcomes with CloudAEye.

A credible open source benchmark you can trust

The benchmark uses public open-source pull requests and golden comments instead of closed, marketing-only examples. Every score is anchored to real reviewable changes.

Reflects how code is written and reviewed in practice

The repository mix spans scheduling, authentication, observability, forums, error tracking, and systems-level infrastructure across Python, JavaScript, TypeScript, Java, Kotlin, Go, C#, Ruby, C, C++, Perl, and PL/pgSQL.

One review system for accuracy, security, and scale

With the best F1 score, OWASP aligned security review standards, and an extensive PR checklist, CloudAEye is the best fit for production code review.

References

References