Impact of AI, Product Velocity, and more: Learnings from Engineering Leaders Forum

‍

We held our inaugural Engineering Leaders Forum event (co-sponsored by Amazon Web Services, Kubiya.ai, Turing and Uplevel) on May 9, 2023 in Sunnyvale. Dozens of engineering leaders gathered to engage in a roundtable discussion on a range of topics:

1. How AI transformation impacts engineering teams
2. Management best practices: Measurement, execution and increasing velocity

Given DevZero’s area of focus, I spent a lot of time discussing with other peers two specific topics: measuring and increasing velocity. My colleague, Shani Shoham from Kubiya.ai wrote a great summary about the AI discussion in What do VPs of engineering care about. Before organizations can increase their velocity, they first need to develop a rubric for measuring an engineering organization’s performance. Some of the metrics mentioned included:

• # of PRs per engineer per week
• # of deployments per engineer per week
• # of commits per engineer per day
• # of hours spent in meetings

Quite a large percentage of the participants weren't familiar with the nuances of the DORA and SPACE frameworks. However, various metrics they were considering incorporated aspects from each of these frameworks into their thought process.

Before going into the details of the more in-depth discussions, these are the top 2 metrics that the group agreed on:

1. # of deployments per engineer per week (best)
2. # of PRs per engineer per week (a solid second)

#

Participants engaged in a discussion on "Increasing Velocity"

Measuring velocity #

To understand the reasoning behind these metrics, we need to understand why each of these factors exists in the first place.

Why deployments? #

Deployments are changes being made to a production system. From the manipulability standpoint, one might argue that engineers will start to break down changes into smaller pieces to increase the metric. This means that smaller-sized changes will go into production, which fundamentally makes for a more resilient system.

Why PRs? #

PRs are a good proxy to measure changes being made to a production system, second only to the deployments that actually happen against that system. It’s an indicator of activity and continuous improvement to the codebase, which ultimately results in the product offering.

Why per engineer? #

It’s a hard metric that can be tracked across engineering and finance. Measuring things like stories, epics, etc leaves too much room for error and qualitative implementation. Baselining on the number of engineers is a quantifiable metric.

Why per week? #

There are times when engineers have days that are heavy with meetings. Engineers also go through cycles of productivity and troughs when not as much tangible work is produced. A week seemed like a good middle-ground between per day and per month.

Sentiment around remote teams #

As teams are getting more distributed, managing and collaborating across time zones is getting increasingly challenging. In discussions around return-to-office, most engineering leaders stated the following factors as main reasons for asking team members to return to offices:

• overhead of managing remote teams
• lack of visibility into day-to-day activities
• impact on culture and perception of Zoom meetings feeling too transactional

The participants also noted that during the Covid pandemic, starting about 12-24 months back, the perception was that remote work would be the new norm. Now, Engineering Leaders feel like a hybrid working arrangement will be the most conducive.

Reorganizing teams #

Given the current economic climate, many engineering leaders have already implemented reductions in force, while quite a few were curious about how to enforce one with the least amount of detriment. Understanding the near-term requirements and mapping them to skillsets available within the team was a strategy we heard from quite a few different participants.

In these times, it’s also very important to communicate effectively (and over communicate) with your teams. This approach enables engineers to make micro-decisions that align with the company’s overall objectives.

Introducing AI to engineering teams #

Engineering leaders are talking to their teams to better understand where AI can help with the day-to-day tasks, while staying cognizant of the security and privacy implications. Shani’s post referenced above covers that in more detail.

Participants sharing their insights and lessons on how teams are utilizing AI internally

Conclusion #

Overall, everyone is very aligned that it’s quite difficult to have a single metric to measure the output of engineering teams. But, being able to measure some sort of velocity is important to have a high-functioning team. It’s also important to acknowledge the impact of the current macroeconomic conditions on the workforce and how we can have the most optimally performing teams in these resource-constrained times. Leaders will continue to evaluate various AI tools but the privacy and security implications of introducing those tools continue to be the most significant obstacles so far.

It was refreshing to see the frank discussions among all the engineering leaders when they got a chance to have an in-person event that didn’t feel like a sales/pitching session organized by a vendor. DevZero looks forward to continuing to build and foster a community of engineering leaders so that we can all share our experiences with one another, thus contributing to the advancement of technology.

‍