What is Change Failure Rate?
Change Failure Rate is one of the four key DORA metrics that measures how frequently changes to production result in degraded service that requires remediation. To calculate this, Uplevel analyzes the sequence of deployments to production in a given repository.
Consider the scenario where a code change was deployed to production, but it contained a defect that was missed during review. This degraded service, so the next day, a hotfix PR was written was written and deployed to production to restore service. In this scenario, the first deployment (containing the defect) would be considered a "failure".
First, Uplevel classifies each deployment to production within a repository into three categories:
These are deployments to production environments that are followed a subsequent deployment that includes work that signifies there was defect remediation. Uplevel estimates this by analyzing the PR branch name and title, as well as linked Jira ticket titles and issue types, looking for the following keywords: bug|incident|security risk|defect|vulnerability|hotfix|coldfix|patch|fix forward|rollback
Uplevel also looks for rollback deployments that could indicate that there was a problem with a previous deployment.
Additionally, Uplevel takes timestamps of work into account to avoid bug/polish work that was started before a code change was live in a production environment.
These are deployments to production that are not followed by repair work. These represent the majority of deployments that are observed by Uplevel.
These are deployments that are observed to only contain repair work. E.g., every PR contained meets the criteria described above OR a deployment is a rollback. Since these deployments aren't planned changes to production, they are removed from the calculation of CFR described below.
In a given time period, Uplevel considers all deployments that are attributed to a group of people either because someone initiated the deployment
or
authored a PR that was included in the deployment. CFR is then calculated using counts (n) of the various deployments:
Fix-only deploys are removed from the denominator of this equation for two reasons:Consider the following sequence of deployments: