Devin 101: Automated QA Testing with the Devin API

Picture this: it’s 8 pm, and your team just rolled out a major update. Everything appears smooth—until a crucial user flow breaks during login, leaving you scrambling overnight. Or, imagine this: you spend days manually combing through tests, only to discover that everything was fine. Sound familiar?

This is the exhausting reality for many QA teams: manual testing is time-consuming and repetitive. Meanwhile, automated scripts for QA can be so fragile that even a small tweak renders them defunct. And, research shows that many companies spend around 25% of their IT budgets on QA alone.

At Cognition, we’ve been working to change this with Devin AI, an autonomous and collaborative AI teammate. With access to a browser, terminal, code editor, and the Internet, Devin can do many of the same tasks as human engineers.

In fact, we use Devin to do QA on Devin itself! Inception vibes, anyone?

We built QA-Devin, a testing framework using the Devin API. QA-Devin has streamlined the way we test new features—reducing required time and effort for our human engineers dramatically. Instead of writing and updating automation scripts, our engineers specify tests in natural language. Devin carries them out and reports back directly in Slack!

Whether you’re maintaining a small indie app or scaling a massive platform that serves millions, Devin can help you:

• Run tests faster than manual processes.
• Avoid brittle automation scripts that need constant updating.
• Get actionable results in minutes—not hours or days.
• Scale faster by letting you spin up as many QA testers as needed

The Problem with Manual QA

What Teams are Doing Now

Where Devin Comes In

How our QA Framework Works

You can create a test by specifying steps and pass/fail criteria in natural language using the following JSON format.

{
  "suite_name": "string",
  "base_url": "string",
  "tests": [
    {
      "name": "string",
      "description": "string",
      "base_url": "string (optional)",
      "login_instructions": "string (optional)",
      "steps": [
        {
          "description": "string",
          "expected_result": "string",
          "wait_time": "integer"
        }
      ]
    }
  ]
}

Specify login instructions if the website you are testing requires authentication. Check out our docs to see how to securely give Devin your usernames and passwords.

Getting Started

Wrap-up