Every engineering lead eventually gets asked some version of "how healthy is our codebase?" and "who needs support?". The honest answer is usually a shrug, because the numbers that are easy to collect, such as lines of code, commit counts, and PR throughput, measure activity rather than quality. This post describes a scoring model that measures what reviews actually find, explains the design choices that stop it being gamed, and shows how to read the numbers once you have them.
What a quality score should and should not do
A useful score has four properties:
- It comes from findings, not volume. The input is "what did review discover in this change", never "how big was it".
- It weights by severity. A hard-coded secret and a misspelt variable are not the same.
- It adjusts for size. A 900-line migration should not automatically score worse than a 10-line fix.
- It rolls up cleanly. PR scores become repository scores become an organisation score, and the same findings can be attributed to the person who wrote the code.
And two things it should never do:
- Rank people for performance reviews. A contributor score is a coaching signal, not a leaderboard. The moment it affects compensation, people optimise the score instead of the code.
- Reward doing nothing. If the safest way to keep a high score is to avoid touching hard code, the metric is broken.
The model DiffNova uses
Every pull request starts at 100. Each finding from the review subtracts points:
| Finding | Penalty |
|---|---|
| Critical | 25 points |
| Warning | 10 points |
| Info | 2 points |
| Security flag (on any finding) | 5 additional points |
The total penalty is then softened for larger changes by dividing by a factor that grows with the square root of the changed lines, capped so a huge PR cannot hide problems entirely. A 500-line PR with two warnings scores better than a 20-line PR with the same two warnings, because the same number of issues across more code is a lower defect density.
The score is clamped to 0 to 100, and the PR's score is recorded against the commit that was reviewed and the author of that commit.
Why the square root
Linear scaling (penalty divided by lines) is too generous: it lets a 5,000-line dump with ten critical findings score in the 90s. No scaling is too harsh: it punishes every large refactor. The square root sits between the two. It acknowledges that bigger changes have more surface area without letting size wash out the findings.
Why security gets extra weight
A security finding is not just "one more warning". It is the category most likely to cost real money after merge, and the one most often missed by human reviewers. The additional penalty makes security-heavy PRs visibly worse than quality-heavy ones with the same count, and it means repository scores drift downward when a codebase accumulates security debt, even if each individual PR looked fine.
Rolling up
Repository score is the rolling average of the last 20 PR scores. Twenty is enough to smooth one bad week and small enough that a real improvement shows within a sprint or two.
Contributor score is the average across all of that person's reviewed commits under the installation. Because it is computed from commit authorship rather than from who clicked "Create pull request", it survives squash merges and shared branches.
Organisation score averages across repositories, so a lead can see the overall trend on one line.
Reading the numbers
Colour bands make triage fast:
- 80 and above: healthy. Keep the review mode where it is.
- 65 to 79: watch. Look at the finding categories. A repo that is all Style findings needs a formatter; a repo that is all Security findings needs a design conversation.
- Below 65: act. Switch the repository to review-on-every-commit so regressions are caught per push, and pair someone with the main contributors.
Trend matters more than level. A repository at 72 and rising is in better shape than one at 85 and falling. The chart on each repository page exists for exactly that reason.
Using contributor scores well
The right way to use a contributor score is as a coaching prompt, in private, with the actual findings in hand:
- A new hire with a low score in their first month is normal. Look at which categories dominate and share the two or three patterns to fix.
- An experienced engineer whose score suddenly drops usually has a context problem, such as a new codebase, a rushed deadline, or a rewrite in a language they know less well. Ask.
- Consistently high scorers are your review culture. Ask them to write the team's checklist.
Never publish the scoreboard team-wide as a ranking. DiffNova shows contributor scores only to the organisation's admins on the Growth plan and above, for this reason.
What you need to compute this yourself
If you want to build this in-house you need: structured review findings with severity and category per PR, changed-line counts, commit authorship, and a job that recomputes rolling averages after each review. It is a few days of work on top of an existing review bot, plus the ongoing cost of keeping the finding quality consistent.
Or install DiffNova on a repository and the scores appear after the first reviewed pull request, with the per-PR breakdown, the history chart, and the contributor view built in.