Competitive audits were first introduced by Code Arena as a straightforward model. Projects seeking security scrutiny for their code amass a set pre-determined sum to finance the audit. This process spans a defined timeframe, during which security researchers are invited to report any vulnerabilities they find in the specified code.
Once the audit period concludes, independent evaluators review, eliminate duplicates, and classify the severity of each report, culminating in a competitive audit report.
Compensation for each security researcher is determined by a scoring system that rewards a finding’s severity and uniqueness.
Issues
The model has the potential to replicate goodhart’s law, and code arena certainly did in its early days. In these times it was very profitable to submit loads of unique informational / low severity bugs.
Economics
A competitive audit is a type of auction. An auction that can be modelled in two ways:
- An auction where the item of value is the time of security researchers.
- An auction where the item of value are actual findings.
A discussion on which model is most appropriate is not relevant here, though I would argue that both likely hold similar merit.
Parallel Competitive Audits
As code arena proved the profitability of this model we’ve seen multiple copycats popping up, running competitive audits in parallel to code arena. Furthermore, code arena themselves also run multiple competitive audits in parallel.
Such a scenario provides an interesting optimisation problem for participants in the auditing market.
Buyer
Audit purchasers are looking at an all-pay auction.
There is a pool of auditors that will decide to work on competitive audits while your competition is going on, and this pool is largely stochastic (not really, but we’ll get to that later).
You’re competing with other projects running their competitions at the same time by bidding higher.
Sure you can compete by selecting a “better” platform for competitive audits, but it’s not clear whether there are meaningful differences.
All-pay auctions have the downside that all bidders have to pay the full amount, regardless of the value they’ve received.
As a result, and because all previous bids are open, projects are incentivised always bid the same as or over the latest audit price.
Note
In determining the right price for an audit, the purchaser would implement normalisation relative to the size and complexity of the project.
Size and complexity correlate with the amount of time that a security researcher would reasonably need to find issues in the codebase.
Seller - Security Researcher
There are some potentially interesting economically viable strategies, optimising revenue per time unit.
Simple
A basic strategy has the security researcher pick one competition and allocate all their time there.
In such a scenario, there are two primary factors to consider:
- prize pool size normalised by size and complexity
- behaviour of other security researchers
This, interestingly, is quite similar to Traffic Flow Optimisation. You’ll have a traffic jam (low returns) if everybody takes the fast road ( high price / code ).
Unlike with traffic, security researchers often aren’t all that similar. An important factor is skill and experience. For example, you might opt to avoid competitive audits that you expect a lot of 13373 auditors to participate in.
Complex
A security researcher doesn’t need to just go for one competition.
Unfortunately it’s not possible to formulate a generalised strategy, as the individual competences of security researchers influence their optimal strategy to a great extent.
That said, it should be noted that this model potentially promotes specialisation. A single auditor specialising in such concepts that are niche (but still relevant enough so there are multiple relevant competitive audits), has the opportunity to participate in the different parallel audits to focus on just vulnerabilities in their area of expertise.
An eventual equilibrium will likely form of generalists and specialists.
I assume that the emergence of specialists is positive effects on the payoff for audit purchasers.