Engineering Reporting Guide
Vivian Guo | April 6, 2021
R&D is increasingly becoming a bigger line item in OpEx and a key differentiator for companies. However, unlike Finance or GTM updates we have noticed there is not a standardized approach to reporting on engineering updates - and how these tie to overall business outcomes.
Why is this needed?
R&D is increasingly becoming a bigger line item in OpEx and a key differentiator for companies. However, unlike Finance or GTM updates we have noticed there is not a standardized approach to reporting on engineering updates – and how these tie to overall business outcomes.
How to use this guide:
This guide can be used to frame engineering updates for various forums, such as engineering quarterly reviews, annual planning, or Board pre-reading. While the reporting structure and focus will obviously vary based on the stage each company is at, we believe if a company’s R&D budget is over $10M annually, the management team, Board of Directors, and engineering teams will all benefit significantly from consistent quarterly reporting on key engineering metrics.
While these metrics are intended to solicit discussion around key topics like engineering spend, headcount, and efficiency, we also believe that just having quarterly reporting will force engineering teams to be more introspective as they prepare these metrics each quarter.
Although engineering and product development are closely tied, this guide will be focused on engineering specific updates and challenges. Through more structured and consistent reporting, we hope this guide will facilitate thoughtful conversations to build a more productive and happy engineering organization (often your most expensive asset).
In the attached slides, we suggest teams break out engineering updates into 4 key focus areas:
Key Engineering Metrics
Each meeting should start with a summary of the key engineering metrics and OKRs being tracked on a quarterly and annual basis. Starting with this forces teams to think through what are the actual KPIs, focus the conversation around results instead of activities, and serve to level set the following conversation. Some of the key categories we recommend covering include R&D spend, people, engineering efficiency, and code quality, which will be addressed in greater detail in the following sections.
Engineering Allocation
As an engineering organization grows, different types of questions and challenges start to emerge around the investments in time and people your organization is making and the return on those investments that come from the ever increasing engineering team.
It’s critical to have a framework in place that allows the company to talk about and prioritize engineering investments in a way that makes sense for engineering internally – and also is understandable for the rest of the business stakeholders. This framework focuses the conversation on the levers and choices the business truly has, by categorizing engineering allocation into 4 key buckets: New Capabilities, Quality Improvements, Internal Productivity, and Keep the Lights On (KTLO). You can read more about this framework here.
During meetings, we recommend discussing engineering allocation across these buckets for the prior quarter, current quarter, next quarter, and next 2-3 quarters.
Hiring Update
Hiring the right talent at the right time is critical for organizations. For engineering teams, it’s especially important for the board and management to understand that engineering hiring isn’t about hitting a head-count number. It is about identifying the easy hires that can be done at scale and the much harder hires you will really have to devote time to finding.
To ensure the right level of support, we recommend presenting a picture of the 12-month forward engineering org chart and spending some time discussing the open leadership (director and above) positions. Beyond roles, we also believe it’s important to identify the key expertise gaps (e.g., testing, machine learning, etc.) in your organization. As with other function updates, it is great to leave some time in to celebrate recent hiring wins. On the other hand, it’s also important to keep an eye on how much of your recruiting engine is spent on replacing talent (i.e. regretted / non-regretted attrition).
Engineering Efficiency
The last focus area is around developer productivity and key bottlenecks. Just as sales teams measure quotas and ramp time, it is important for the engineering organization to measure developer productivity. While these metrics will vary across companies, metrics that allow management to understand and track the revenue / FTE cost, release time, and other metrics around developer velocity on a quarterly and trended basis will be critical. Presenting a summary of the key development bottlenecks, planned remediation, and technology risks will also allow engineering teams to receive leadership and cross-functional support where needed.
In our opinion, there is not a single or set of metrics that can accurately track and report on developer efficiency across companies. Instead, developer efficiency can be viewed similar to a sales funnel, with key metrics that can be tracked at each stage of the funnel (e.g., writing code, code review, testing, deployment, maintenance). For example, you may want to track the time from requirements to code complete (writing code), % code coverage (testing), and % of roadmap shipped on time (deployment).
However, if there was one table-stakes metric we recommend all organizations start tracking and reporting on, it would be self-reported developer productivity. This metric should be based on engineering surveys that ask key questions such as “Do you have the right tools and processes to do your work?”, “Where are you spending most of your time?” etc. If you do not already have a regular developer survey in place, you can leverage this template as a starting point.
While this can feel like a fuzzy and self-reported benchmark, we believe this is a critical metric that cuts across all aspects of developer productivity. Above all, it is more important to focus on how you trend over time on this metric, rather than your current score. Measuring individual developer productivity will, for better or worse, start altering behavior. For example, if you measure code review turnaround time and share these scores with engineers, over time they may code faster but probably with lower quality. A good way to mitigate this is by looking at aggregate rather than individual scores.
Rather than tracking every single metric, it is most critical to start building in the practice of regularly reporting and improving on a select few metrics over time.
Do not let Perfect be the enemy of Good!
It’s also important to note that this is a problem that will have different solutions across companies and a topic that will only benefit from increased knowledge sharing, so we also welcome any additional considerations or best practices you may have.