---
title: "Do you know how your team is using AI?"
description: "How teams can use telemetry to understand AI usage and evals to choose models for the work they actually do."
url: "https://www.cxo.dev/blog/how-your-team-is-using-ai"
last_modified: "2026-11-20T17:00:01Z"
---

# Do you know how your team is using AI?

> How teams can use telemetry to understand AI usage and evals to choose models for the work they actually do.

[View this page on CXO.dev](https://www.cxo.dev/blog/how-your-team-is-using-ai)

Published 2026-11-20T09:00:00-08:00 · Claire Vo + Zach Davis

![A satellite dish casting a long oval shadow across bare ground, viewed from above.](https://assets.cxo.dev/newsletter/2026-11-20-dish-halftone-1200x400-email.png)

*You can listen to or watch this issue on the Agent-Ready podcast. Subscribe on *[*Spotify*](https://open.spotify.com/show/4FiqhJ4RQqZ4LT4D5b21U6)*, *[*YouTube*](https://www.youtube.com/watch?v=PmtTHfNMMT0)*, or *[*Apple Podcasts*](https://podcasts.apple.com/us/podcast/agent-ready-with-cxo-dev/id6834483173)*.*

There are no magic bullets in AI transformation. Despite all the chatter online (including from us sometimes!) about a specific tool, workflow, or skill that people are finding success with, ultimately the individuals that are getting the most out of AI aren’t all following the same blueprint. What they do have in common is a willingness to experiment and rethink how they work.

At the organization level, you need everyone to build baseline skills to meet the moment, and you need a handful of people excited to stay in touch with the latest changes and how to apply them to their work. What you don’t need is every single employee trying to stay on the edge or dozens of conflicting operating model experiments.

### How do you stay on top of what everyone is using and doing, and what’s working, at scale?

Everyone adopting AI as a tool to get work done has this problem, but very few companies are prioritizing solutions. The companies that are happen to also be the ones that have demonstrated an ability to cut through the noise and the cultural friction to institute real changes to the way people work, and ultimately get more leverage out of AI.

So at some point, with every company we work with, we end up having similar conversations to get to the heart of two questions:

### Do you know how people are using AI?

You bought the tools, your inference costs keep going up every month, but do you really understand how people are using all that spend?

Not like vibes and hallway talk. Real visibility. At a minimum, do you have a dashboard that shows things like tool usage, model usage, and cost, in aggregate, broken down by team, and broken down by individual?

But more importantly, and more powerfully, are you capturing session data somewhere, and are you analyzing it? Inevitably this causes some people to raise their eyebrows. Session data could, of course, be used for evil. But also it’s a gold mine of information on workflow experience that can be used to simultaneously improve employee quality of life and pursue business outcomes more efficiently, and if you can’t push past the hand-wringing because culturally your teams don’t trust you to use it for good instead of evil, then you are missing out on a huge opportunity and likely competitive advantage.

We are strongly in favor of collecting full session data and putting controls around who can access it, treating it like you would customer data. Storing full sessions allows you to mine that data in aggregate and in ways you might not have anticipated. But you can also de-identify session data before storing it, or collect only a subset of session data such as certain lifecycle events, errors, and so forth. Better to start collecting what you can today than to wait until systems and controls get approved and built.

You want to know what tool calls are regularly failing, which skills are being invoked and which aren’t, where the friction is and what errors are popping up. You want to know what models are being used and for what, how much any given PR cost to produce, who is using Fable for every task. You want to know, in detail, how people are really using AI so you can clear those roadblocks, learn from and educate your teams, and optimize workflows to get the same or better results while controlling spend.

### Do you know if AI is working?

Session data is powerful, but it’s a lagging indicator. It tells you about historical usage, problems, and opportunities. You also need to be able to regularly evaluate core workflows against different inputs, different models, and possibly even different harnesses. “Evals” is still a scary term for lots of people, but that’s just lack of familiarity talking.

Pick a workflow or workflow step that you want an agent to handle or help with. Chances are you have historical inputs and human-crafted outputs. Give those to an agent and ask it to create an eval. It can be that easy to get started. Claude has built-in skills to help with creating evals. If you want to go deeper, friend of the newsletter Hamel Husain has [amazing resources on AI evals](https://hamel.dev/blog/posts/evals-faq/).

At this point, every business should have a set of evals, however formal or informal, that capture key AI-enabled workflows. There’s really no excuse not to. Once you have at least a minimal set of evals that have been proven to work (this is easier than it sounds), you now have a platform to experiment with a myriad of levers in the name of cost, performance, or capability.

---

Every team is a rich, continual source of data on how work is being done and what acceptable outputs look like. Any org that isn’t taking advantage of those things through telemetry and evals is leaving major opportunities on the table and unnecessarily impeding its ability to adapt to ongoing changes in tooling and technology.

Here’s what we’re reading this week:

---

### [The Atlassian AI SDLC Index](https://www.atlassian.com/blog/ai-sdlc-index)

Scattered insights from Atlassian’s DX product, using a dataset that includes more than 500 engineering orgs. You can’t improve what you don’t measure. Some things that stood out to us:

- Developers report losing 20% of their time to friction, with the most time lost waiting for CI and review.

- Companies that report a higher share of AI-authored code have longer median waits for a first human review.

- Monthly review load rose from 12.8 PRs per active reviewer in January to 22.3 in October.

- Maintenance and bug fixes are among the most common uses of AI, and yet self-reported time spent on innovation has barely budged. So where are the time savings going?

### [The Gamma Model Index](https://research.gamma.app/2026/11/model-index)

Gamma published their data on which models they use in the product and for which tasks. A great example of how evals can help shape and optimize your model usage. They have the volume and infrastructure to A/B test on real users, but as we discussed previously, you can start with historical datasets and “offline” evals.

> Our agent splits every deck into about a dozen jobs, and each one can run on a different model.

They rate Anthropic as best for orchestration, OpenAI for visuals, and don’t seem to love any of them for writing (very relatable).

### [PostHog’s public MCP leaderboard](https://x.com/posthog/status/2112574313434112435)

PostHog shared analysis of production usage of their MCP. They have the basics, like model usage, but also tool calls, error rates, failure categories, and intent. Gathering this level of intelligence for your product seems like a no-brainer. What if you treated your internal developer experience like a product? What data would you gather?

### [Code review at Linear](https://linear.app/data/code-review)

The Linear team published a data-driven look into how code review has changed for their team. It sounds like they still require human approval, but that approval can come from the human PR “author.” That means they’re trusting their team to gauge risk and to own intent, and to bring in other human reviewers only when they deem it necessary.

> From January to June 2026, only around 2.5% of merged PRs were reviewed by the author alone (and these are largely attributable to fixes made under exceptional circumstances). July marked a turning point, when 14% of PRs were merged without a second human review. That share grew steadily over the following months, reaching 53% in October.

They’ve also seen the number of PRs closed without being merged more than quadruple over the past year, and crucially, they consider this a good thing, not a problem to correct.

> Now, engineers can build several alternatives and explore the trade-offs with colleagues before committing to any one. Each implementation, even the ones that are discarded, gives the team something concrete to test and learn from, helping clarify what to build next \[…\]

### [Building an accounting team with Grok Bot](https://x.ai/news/grok-bot-accounting)

The SpaceXSI team on how they’re using Grok Bots in the accounting org. It’s a pretty low-hype, high-signal post for anyone who isn’t sure how to incorporate “always-on” agents into their work. They treat their bots like team members, setting them up for success with onboarding docs, controlling what they access and do with permissions and approval requirements, and going through a training period with them before they’re trusted to work independently. They walk through the specific bots they created, but whether you’re in accounting or not, it’s a pretty interesting insight into what one vision of the near-term future of work looks like in practice.

---

### One thing to try this week

Whether or not you’re ready to hand your accounting tasks over to Grok Bots, there are lessons from how the SpaceXSI team approached it that you can take away to make any task or workflow more agent-ready.

> A Bot will struggle to run a process that people haven’t already figured out how to perform reliably.

This is generally true for AI, and most people skip this step. So this week, pick a high-toil, low-leverage task that you’d like off your plate and write down the info and access a human would need to do it for you. Then give it to an agent instead of a human and see how it goes.

---

Stay curious.

*— Claire + Zach*

## Sitemap

- [Site index](https://www.cxo.dev/llms.txt)
- [Full sitemap](https://www.cxo.dev/sitemap.md)
- [Visiting agent guide](https://www.cxo.dev/AGENTS.md)
