Test Failure Analysis in CI

Know why the build failed, not just that it did. CloudAEye finds the root cause of test failures in CI, such as GitHub Actions and Jenkins, explains why each test failed, and proposes a fix.

Book a demo All use cases

From red build to root cause.

Automatic root-cause identification of a test failure in CI, such as GitHub Actions and Jenkins.

GitHubbilling-service · GitHub Actions

Failed build

CI / test2 of 418 tests failed 
FAILED tests/test_refunds.py::test_partial_refund_rounding
AssertionError: assert Decimal('10.04') == Decimal('10.05')
FAILED tests/test_refunds.py::test_refund_total
AssertionError: assert Decimal('20.09') == Decimal('20.10')

Test failure analysis

Root causeBoth failures

A recent change to billing/money.py rounds refund amounts with float instead of Decimal, so half-cent values round down.

billing/money.pyProposed fix 
def round_amount(value):
    return Decimal(value).quantize(CENT, rounding=ROUND_HALF_UP)

Add Suggested Changes applies the fix.

Red build.
Clear reason.

A failed run leaves you with a long log and a guess. CloudAEye reads the failure, finds the root cause, and proposes a fix, in the CI you already use.

The root cause, not just the log.

Each failure gets an explanation of why the test failed, next to the error log and the build details.

  • Error logThe failing output
  • Root causeWhy it failed
  • Info and linksThe build and project
See an example

A fix you can apply.

CloudAEye proposes a code change for the failed test, and the build history shows where it failed and passed.

  • FixProposed per failure
  • ApplyAdd Suggested Changes
  • BuildsWhere it failed and passed
Read the docs

In GitHub Checks.

Comment on a pull request to see the analysis under its Checks. If the failure has not been analyzed yet, the comment starts the analysis.

  • Command@cloudaeye /analyze
  • Shown inGitHub Checks
  • Not analyzed yet?Runs it first
See the command

GitHub Actions and Jenkins.

Connect GitHub Actions with a webhook, or install the CloudAEye plugin for Jenkins. CloudAEye then watches the pipeline for test failures.

  • GitHub ActionsWorkflow webhooks
  • JenkinsCloudAEye plugin
  • Individual planFree
Explore Test RCA

Commands

GitHub pull request
@cloudaeye /analyze

Failures in a connected CI are analyzed automatically. The comment shows the analysis under the pull request's Checks.

Connect your CI.

Start on the free Individual plan, or choose a Team or Enterprise plan.

Read the setup guide
  1. Sign up for CloudAEye and choose Test RCA.
  2. Connect your GitHub repositories.With the CloudAEye DevProd GitHub app, or a personal access token.
  3. Connect your CI.For GitHub Actions, choose each repository's test framework and workflow. For Jenkins, install the CloudAEye plugin.
  4. See the root cause and fix.In the CloudAEye console, or under Checks in GitHub.