LecturaInvestigaciónRadarMarco de inversión
Iniciar sesión / Registrarse
← Volver a la lectura
NoticiasMaterial4 min de lecturaEditorial de DisconfirmAI

Standard Bank Group Reports Automated AI Risk Assessment Cut Review Time from Two Weeks to Two Hours

Co-written by Standard Bank Group's CTO and AWS staff, this post details a production pipeline that uses LLM-as-a-judge on Amazon Bedrock to evaluate generative AI use cases against more than 300 controls drawn from international standards and internal policies. The bank reports reducing manual review from two weeks to approximately two hours across six assessments, with a 14% improvement in risk scores after remediation. The system's question factory converts policy documents into executable checks, so a policy update becomes a test change applied automatically to every subsequent assessment.

2026-11-21 04:37 GMT+8

Every new generative AI application adds one more system that a risk committee must approve before it reaches customers or employees. Teams can now build an application in days, but when approval depends on reviewers reading documents and testing the system by hand, governance becomes the slowest step in the pipeline. Standard Bank Group chose to automate the evidence gathering and testing, then feed the results into the approval gates it already runs. Review and approval stay with the risk committee, but the committee now bases each decision on consistent evidence collected the same way across use cases.

The assessment runs entirely in Standard Bank Group's own AWS account. Documents, code, and test results stay in an Amazon S3 bucket. The AWS Well-Architected Tool handles intake, Step Functions orchestrates the pipeline, and a foundation model on Amazon Bedrock evaluates the evidence using a technique called LLM-as-a-judge, where a large language model checks evidence against defined criteria. The bank controls access through IAM, the same mechanism it uses for other AWS workloads. Because Bedrock offers many foundation models through one API, the team can swap the judge model by changing a configuration setting rather than rebuilding the pipeline.

Each assessment moves through four steps. First, a pre-assessment agent running on AWS Fargate scans the use case's code repository and documentation, maps relevant files to roughly 40 risk questions in the Well-Architected Tool lens, and pre-populates answers with evidence links. The delivery team then reviews and corrects. Second, the pipeline examines four kinds of evidence—system documentation, policy documents, source code with the system prompt, and live calls to the application. For live testing, it calls the application through its Lambda function or API, supplies user questions, adds a built-in set of jailbreak attempts and off-topic questions the application should decline, and sends the same prompt in original and reworded forms while an embedding model checks that answers stay consistent in meaning. Third, the Bedrock judge evaluates each piece of evidence against specific controls, returning yes-or-no answers. Standard Bank Group customized the lens with judge questions built from its own standards, bringing the total to more than 300 checks across the eight AWS responsible AI dimensions. Fourth, the pipeline summarizes results as an executive presentation, a mitigation checklist, a technical report, and a CloudWatch dashboard tracking scores over time.

Iniciar sesión / Registrarse
LecturaInvestigaciónRadarMarco de inversión

Each risk receives a likelihood score (the share of required evidence the model could not find), a fixed severity level adjustable by the bank's risk team, and together they produce a rating from Very low to Critical. Each missing piece of evidence maps to a specific mitigation and to the clause of the standard or policy it came from, so a risk officer can trace any finding back to its source when explaining decisions to a regulator. The assessment asks narrow yes-or-no questions worded to match specific controls rather than open-ended opinions, and a fixed formula the risk team can inspect turns those answers into scores. Every evaluation runs multiple times to measure consistency, and inconsistent findings are flagged in the review interface.

The mechanism that most directly addresses governance speed is what the bank calls a question factory. International standards give every use case a common baseline, but risk committees approve against the bank's own policy. The question factory generates extra judge questions from internal policy documents. Because those checks are derived directly from policy language, a policy change becomes a test change: every subsequent assessment applies the updated rule automatically, with minimal manual rework by the risk team. Sri Elaprolu, Director of Frontier AI Science and Engineering at AWS, summarized the shift: when every rule becomes a check that runs against the live system, a policy update stops being a memo and becomes a test.

The bank reports that across six production assessments, the pipeline reduced a traditionally manual two-week review to a comprehensive risk report generated in approximately two hours, and delivered a 14% improvement in use-case risk scores after remediation. These figures are self-reported by Standard Bank Group in a post co-written with AWS; the article does not disclose the total number of use cases assessed, the complexity distribution among the six, or whether the two-week baseline was measured under identical conditions. The assessment produces evidence against controls and does not certify compliance with any regulation—the bank remains responsible for its own regulatory determinations. The system currently runs in production evaluating use cases before they go live, with plans to rerun on a schedule or whenever code or configuration changes, keeping evidence current beyond launch.

Fuentes

  • aws.amazon.comAmazon Web Services↗
Continuar investigando →