Open Factor
  • Demo
  • Docs
  • Pricing
  • Changelog

Solutions

Flaky Test Investigator

Review intermittent test failures from the source you supply. The report covers failure conditions, timing or shared-state hypotheses, repeated-run requirements, and repair plans. It compares earlier saved evidence when available and states coverage gaps.

Try the demoGet started

Factories/engineering

Working14-day run
Draft PRs opened

0

each with a before and after test

Bugs reproduced

81%

the rest reported as unconfirmed

Existing issues

0

commented instead of duplicated

Spend

$0.00

of $25.00 cost budget

Draft pull requests0 of 4 opened as drafts
  • #482Fix 500 on empty cart checkoutReproduce IdleFix and test IdleDraft PR Idle
  • #485Handle expired session tokensReproduce IdleFix and test IdleDraft PR Idle
  • #487Fix off-by-one in invoice datesReproduce IdleFix and test IdleDraft PR Idle
  • #490Guard missing user in settingsReproduce IdleFix and test IdleDraft PR Idle
Agent tasksLatest first
TaskStatus
Read Vercel deployment eventsIdle
Reproduce a checkout errorIdle
Review 4 draft pull requestsIdle
Write a failing test for the session bugIdle
Comment the fix on a Linear issueIdle

Run plan

Each factory run reads deployment events, open issues, and code, then opens a tested draft pull request for each bug nobody filed.

  • Read Vercel events and Linear issues
  • Reproduce each bug before fixing it
  • Open each fix as a draft pull request
  • Cost budget $25.00, warn at 70%

Ready for your review

Review 4 draft pull requests

Each one names the bug, reproduction steps, root cause, fix, and test results.

Sample run: Engineering

From a flaky test to a measured patch

A sample run, start to finish. It reruns a suspect test to measure its pass rate, isolates the condition that makes it fail, and returns a patch with its pass rate after the change.

Factories/Engineering run

IdleIllustrative run
Reading
      POST /api/checkoutVercel - 1h ago
      TypeError: Cannot read properties of undefined (reading 'currency')
          Fixlib/pricing.ts
            Regression test
              Draft pull requests
              0
              Unconfirmed bugs
              0
              Draft pull requestyour-org/your-app

              fix(checkout): price a cart that has no region yet

              fix/checkout-missing-region into main

                New Linear issue ENG-431, labeled bug

                Start with your source

                Provide the material and review the scope

                The trial begins with the material below. No mandatory OAuth connection is needed for accessible public sources or material you supply.

                Starting material

                Public repo + test command

                Your scope

                Set the targets, questions, acceptance criteria, and exclusions. The scope controls what the report investigates.

                Choose when to run

                Manual is the default. Choose hourly, daily, or weekly recurrence when repeated observation is useful.

                How the trial runs

                Prepare, measure, isolate, patch, report

                Prepare the suite, measure each suspect test's pass rate, isolate what triggers failure, patch and rerun, then return the ledger.

                Prepare the suite

                Check out the current commit, install dependencies, and confirm the suspect tests run.

                Measure pass rates

                Run each suspect test repeatedly and record its pass rate over N runs.

                Isolate what triggers failure

                Vary ordering, timing, and shared state to find the condition that makes the test fail.

                Patch and rerun

                Write a patch for the trigger and measure the pass rate again over the same number of runs.

                Write the flaky test ledger

                Return each test's before and after pass rate, the failure condition, and the patch for review.

                What you review

                A job-specific report with its limits

                Failure conditions, timing or shared-state hypotheses, repeated-run requirements, and repair plans.

                New evidence makes recurrence useful

                Public sources can be read again. Uploaded files remain snapshots: provide a fresh upload to assess new private facts. Earlier saved evidence provides a comparison baseline, not a live connection to the original system.

                Coverage is stated, not assumed

                Browser interaction, code execution, authenticated access, and unsupported file formats may be unavailable in a run. Missing access or tools appear as coverage gaps, never as a passed test. Proposed fixes and drafts remain in the report.

                Review before taking action

                Returns a report and proposed next steps for your review. External messages, source changes, payments, and submissions are outside this trial.

                Start from a template

                All templates
                • Flaky Test InvestigatorStart with Public repo + test command. Later scheduled cycles revisit accessible sources and saved cases. Coverage depends on the available sources and runtime. Tests and measurements require actual execution evidence; unavailable capabilities remain visible as coverage gaps.
                  See template
                • AI MaintainerSplits the cleanup backlog into small, independent draft pull requests, each with the check that proves it. Merging stays with you.
                  See template
                • AI Support AgentReads deployment events, open issues and code to find bugs nobody filed yet. Each one comes back as a draft pull request with a test that fails before the fix and passes after, linked from Linear.
                  See template

                Questions about this trial

                Inputs, recurrence, evidence, and the boundary of the report.

                Need help defining the source or scope? Contact us

                1
                What do I supply?
                Public repo + test command. Review the scope and criteria before the first run. If source material is inaccessible, the report identifies the gap.
                2
                Do I need to connect an account first?
                No mandatory OAuth connection is required for public sources and supplied material. This does not grant access to private sites or systems; inaccessible evidence remains an explicit gap.
                3
                What changes on later runs?
                A later run can compare current evidence with the previous saved baseline. The first run has no prior change history. Select manual, hourly, daily, or weekly execution; refresh uploaded files when private facts change.
                4
                Does a report prove that every relevant test ran?
                No. The report separates source-based findings, observed execution results, and unverified checks. A missing browser, runner, credential, or usable input is a coverage gap.
                5
                Will the trial apply its recommendations?
                It returns a reviewable report and proposed next steps. It does not send messages, change source systems, submit claims, or make payments.

                Start with the evidence you already have

                Supply the source, review the scope, and get an initial report before choosing a recurring cadence.

                Start this templateContact sales

                Product

                • Demo
                • Pricing
                • Templates
                • App

                Platform

                • Blog
                • Changelog
                • Customers
                • Developers
                • Docs
                • Stats

                Resources

                • Home
                • Book
                • Contact
                • Sales
                • Support

                Company

                • About
                • Brand
                • Terms of Use
                • Privacy Policy
                • Cookie Notice
                • DPA

                Ask AI

                Open Factor
                Platform activity