During a 12-week rewrite of the Shopify mobile app, the team averaged approximately 434,000 lines of code changed per week, compared to about 39,000 lines per week at the same time last year. While code throughput increased 11-fold, security reviews cannot also increase 11-fold—at least not in the same way as before.
In a Shopify engineering blog post, Michael Hughes described their response strategy. The core insight is that asking a model to find vulnerabilities across an entire application yields a long, vague list; what truly works is narrowing the problem space to a range where the model can answer precisely.
The specific approach has three layers. The first layer is the attack surface map: for each app, they maintain a list of security-sensitive items grouped by surface—deep links, WebViews, stored data, permissions, dependencies, etc. For each surface, they write one or two sentences explaining why it matters. Each item includes a verdict (Fix, Tradeoff, Secure, Fixed, Debug only) and a link to the relevant code.
The second layer involves writing knowledge down as skill files. The starting point is OWASP's Mobile Application Security Verification Standard (MASVS) and Mobile Application Security Testing Guide (MASTG), supplemented with problem categories learned from past incidents and HackerOne reports. Each skill describes what to list, what good looks like, and what to check. These skills are not tied to any single tool: they are used for scheduled reviews, one-off security assessments, and can be loaded by any agent working in the code. When new knowledge is acquired, the skill is updated, and every subsequent run automatically inherits these improvements.
The third layer breaks reviews into small units. Each skill first performs a complete inventory—every deep link route and its parameters, every WebView and its hosted domains, every WebView JavaScript interface method, every Keychain and Keystore entry, and every dependency with its version—and then reviews them one by one, looking only at the code involved in that specific item. For a deep link, this means examining one route, every parameter it reads, and the code that reads each parameter, then asking: What happens if the parameter is malicious? The answer must be specific enough to be verifiable.
The review rhythm runs once every workday: the latest commit's inventory results are compared with the last accepted map; an item counts as 'changed' only if the cited code line changed or the inventory reported a change. Then, the model judges each changed or new item using read-only tools and code only. A second pass reads through change files to find any security-relevant content not covered by the map. Finally, a new map and a change page are built, and a summary is posted to Slack.
Beyond the map, there is external monitoring: agents regularly read security news, checking each announcement against the map and code, notifying the team only when something affects their own app. The same skills are also used for pre-adoption reviews of third-party libraries, generating maps and verdicts for dependencies identical to those applied to their own code.