How Eight Sleep Uses Devin as a Data Analyst

Building the future of sleep requires us to make data-driven product, operations, and finance decisions. But as a small team, we faced a challenge every scaling company knows well: the flood of random data asks.

Even with dbt, Looker, and Snowflake powering our analytics stack, these requests piled up. Our queue of ad-hoc tasks grew faster than we could ship. When a new report looked suspicious, we had to stop everything, investigate, and reassure stakeholders. It slowed us down.

Our Eight Sleep metrics took a hit.

We needed leverage. So we hired an analyst who never sleeps: Devin AI.

Bringing Devin into the workflow

Instead of forcing people in the team to funnel every question through our small data team, we integrated Cognition’s Devin directly into our workflows:

• Slack as the interface. Anyone can tag Devin in our data channel.

• Access to our stack. Devin understands our dbt repo, queries Snowflake, uses Looker, and even checks data on Amplitude via its web browser.
• Context is king, but needs to be built. We started by just giving Devin our repo and tools (“vibe context”). Then we layered in curated knowledge to help it find the right data. Like training a new employee or intern, we fed it more specific knowledge to support it for each task it took on.

The setup only took a couple of hours from getting access to our account to asking Devin my first question via Slack. From there, Devin began fielding real data requests immediately.

Hero moment: When revenue looked suspicious

A few weeks ago, our brand-new daily sales dashboard lit up with an unusually high revenue number. Theories flew:

• Was the new data model wrong?
• Did Looker mis-aggregate something?
• Was there a real issue with orders?

Instead of a human analyst burning hours to untangle the logic, we asked Devin to investigate. It traced the Looker dashboard back to the underlying queries, ran checks in Snowflake, and quickly found the culprit: an email campaign that performed better than expected.

The revenue was real. The model was fine. And Devin had defused what could have turned into a fire drill.

The impact

Since bringing Devin into our analytics stack, we’ve seen transformative results:

• Faster shipping. We’re shipping 3x as many data features and data investigations each week compared to before (large requests involve: new dbt models, semantic models, dashboards).
• Democratized access. Our queue of random “pull data” asks is near zero. Devin now fields ad-hoc requests.
• Truth under pressure Devin has helped us resolve “this number looks off” moments multiple times, writing the exact queries needed to validate results.

• New voices, increased curiosity. People who never asked our team for data before are now using Devin. It turns out folks might hesitate to ping if they’re just curious, but never feel guilty about asking an AI. Devin is now answering questions for Product, Ops, Finance, Growth, and R&D.
• Organic adoption. As teammates saw Devin in action, more asked to join. Every day a few new folks see the magic in our Slack channel and ask for access. It really feels like internal product market fit.

For our small team, Devin has become an invaluable teammate. It’s helped us work more efficiently and make better decisions.

Building at the frontier of AI analysts

Reading more about how different people approach the AI Analyst problem, it seems there are usually two approaches at opposite extremes.

• Approach A: Provide as much curated context as possible. Highly prescriptive, but labor-intensive. E.g. When a user asks specifically about revenue, you can only use this table.
• Approach B: Give the AI broad access to what a new analyst would get and let it explore (“vibe context?”). Fastest to the “aha moment”/activation, but less accurate. Devin was great at this, just as you’d expect a coding agent to be.

Our experience shows the future lies in between but towards exploration. AI analysts should be able to:

• Grok your stack. Understand dbt, LookML, and dashboards directly.
• Leverage curated heuristics. Analysts can capture how we think about data without having to prescribe every dataset.
• Learn like an employee. Through memory, context injection, and feedback loops, AI analysts should improve just like new hires do.
• Operate with autonomy within clear boundaries. Last, but certainly not least, giving Devin the right permissions is critical (just like your human teammates). By setting clear boundaries on what Devin can query, we make sure it’s powerful where we want it to be, and restricted where compliance and privacy demand it.

Growing Devin from a Jr Analyst to a Sr Analyst

With Devin, you really get what you put into it 100 fold. Two months ago, Devin started as our Jr Analyst, and is close to becoming our Senior Analyst. We are experimenting with:

Augmenting our semantic layer. Context is king. There’s no reason not to improve your documentation and semantic layer. So we will be trialing adding additional useful context into our dbt yaml files using the meta field. This should help improve how Devin approaches using a dataset.

Building a context/feedback loop. We’re experimenting with building an automated way to provide Devin with relevant data models and context from prior sessions to improve its retrieval. We’re also experimenting with crowd sourcing our knowledge creation by building a Slack bot to ask users for feedback on Devin’s answers. These should help with relevance and groundedness.

Experimenting with charting. Sometimes you just want a quick chart, and you don’t really want to create a new Looker object with a visualization. Devin just launched a Metabase MCP server that Devin can use to create charts and screenshot them back to users in Slack. Check out their guide here.

Back to building

Our partnership with Cognition has been instrumental. We’re in constant contact through a shared Slack channel, collaborating on everything from semantic layer design ideas to suggestions for Devin’s roadmap.

Together, we’re pushing toward a future where AI analysts don’t just assist, they own the repetitive work, while assisting humans focusing on the highest-leverage problems.

At Eight Sleep, Devin has already proven the model: democratized data, faster shipping, and sharper decisions.

The next frontier is scaling this across the industry. And we can’t wait to see what the future holds.

PS: It looks like Devin, our AI intern, will indeed be getting a return offer