Autonomy Is Earned, Not Declared
In June, a submission arrived at Shepherd and proceeded to a priced quote without any human intervention. Our AI engine, Nova, analyzed the broker's email, classified the attachments, and extracted every required field. This process, covering 100% of the inputs for exposure, losses, and pricing, resulted in what is internally coined a "golden quote."
This milestone followed weeks of incremental progress in live quote accuracy, which climbed from 77% to 91% before reaching the 100% mark. The achievement was commemorated by the team, marking the beginning of an effort to make such results repeatable.
While an underwriter still made the final underwriting decision, the entire workflow from the arrival of the email to the preparation of the risk assessment was handled by the machine.
Reaching this level of automation required addressing the complex math of AI accuracy and recognizing that the primary challenges often stem from data quality rather than model performance.
The inherent complexity of commercial construction underwriting stems from the lack of standardized submission protocols. A typical account may encompass a combination of practice and project-specific policies alongside wrap-ups, each necessitating distinct exposure metrics and varied documentation. Information is delivered through a disjointed assembly of loss runs, contractor questionnaires, and narrative emails, often arriving in unpredictable formats and varying states of completion. Unlike personal lines, which achieved automation by enforcing rigid data structures, commercial construction remains resistant to such constraints because variability is a fundamental characteristic of the industry.
What Nova is (and isn't)
Nova serves as the AI engine that parses every incoming submission. By ingesting broker emails and classifying various attachments, the engine generates the structured data points that traditionally require manual entry. This administrative burden represents the primary throughput constraint within commercial insurance; underwriters currently dedicate a significant portion of their workflow to transforming unstructured files into structured data, a process that offers minimal differentiation and scales poorly.
The broader objective is FAU, Shepherd’s initiative for fully autonomous underwriting, with Nova acting as the foundational engine. Within this framework, progress is quantified through shifting levels of machine authority rather than a simple checklist of features. Each subsequent level delegates more preparation to the system, enabling the underwriter to redirect their expertise toward strategic risk assessment, complex negotiations, and strengthening broker relationships.
The organizational framework takes precedence over the underlying tech stack in this context. As an MGA, Shepherd operates within a unified loop, managing everything from the initial submission and underwriting decision to policy issuance and eventual loss performance. Software vendors offering extraction tools to carriers lack this visibility; they see the documentation but remain detached from the final outcomes or the workflows that generated them. Controlling the entire lifecycle allows us to resolve data issues at the source rather than attempting to build complex models to compensate for upstream deficiencies.
Drawing inspiration from the autonomous-driving industry, Shepherd uses a tiered taxonomy of autonomy. This framework defines the shifting role of the underwriter as the system advances through different levels of machine authority:
The Operational Design Domain (ODD) serves as the conceptual boundary that delineates the specific parameters where the system’s autonomy is validated. By rigorously defining this scope, categorized by submission variables such as product class or industry sector, the ODD ensures the system operates only within its proven capabilities. Submissions that transcend these established constraints are seamlessly transitioned to human underwriters. This strategic focus prioritizes the mastery of high-autonomy execution within high-precision "lanes," beginning with excess project submissions, before orchestrating a broader horizontal expansion.
Autonomy is earned through evidence rather than declaration. Each level in the hierarchy has measurable graduation criteria. For the initial lane, graduating to Level 2 requires approximately 99% per-field accuracy and correct end-to-end pricing for two-thirds of quotes. System rollbacks are integrated features used when performance degrades, ensuring that data proves readiness.
When a submission is received, Nova immediately begins classifying files and extracting structured fields. Recent demos showed the system reaching approximately 79% field completion automatically.
While 79% completion is significant, it does not represent a finished system due to the mathematical realities of automation.
The brutal math
A quote requires between 30 and 50 correct fields. Because field-level accuracy compounds multiplicatively, the requirements for total system accuracy are stringent:
With 90% per-field accuracy across 30 fields, the probability of an entirely correct submission is only about 4%. In such a system, only one in 20 submissions would proceed without human intervention, as underwriters must still locate and correct errors in the remaining 96%.
A 99% per-field accuracy rate results in approximately 73% of submissions being fully correct. Each additional decimal point of accuracy significantly increases end-to-end automation levels.
Edge cases will also compound. If each of 40 pricing inputs has a 1-in-50 chance of being an edge case, over half of all submissions will contain at least one unusual element. Consequently, managing edge cases becomes a primary focus of the development process.
This reality dictates that internal objectives be articulated as two distinct metrics. Aiming for 90% precision on 90% of accounts describes a statistical distribution rather than a simple mean. Averages are inherently forgiving, allowing high-performing submissions to mask localized failures; a system might boast a high average while still failing to meet automation thresholds. While means obscure the effects of multiplicative compounding, thresholds expose them.
Furthermore, the utility of data is non-uniform. Treating a set of 40 fields as an undifferentiated aggregate ignores the specific roles certain inputs play in the automation workflow. Critical fields serve as gates for pricing; an error here halts the entire process. Informational fields carry a lower penalty, requiring only post-extraction correction. Consequently, pricing-critical variables are tracked in isolation and subjected to more rigorous accuracy standards.
Evaluation scoring must also account for the nature of the failure. A "miss" occurs either through an incorrect response or a refusal to answer, and each necessitates a unique technical resolution. By calculating field scores as the product of the response rate and the accuracy rate, we maintain visibility into both vectors. This distinction is vital, as these two variables often fluctuate independently or in direct opposition to one should.
The distinction between a demo and a production system lies in the conjunction of fields. While a demo may rely on average accuracy, a production system requires every field in a set to be correct. The strategy focuses on driving the critical fields that gate pricing toward 100% accuracy.
Metrics such as the acceptance of AI suggestions can be misleading; while they may indicate user enthusiasm, they do not always correlate with accuracy. True autonomy is measured by the system's ability to produce fully correct outputs without intervention.
Where the misses actually come from
To identify the root causes of errors, a failure-mode taxonomy was developed to classify every miss in evaluations. The distribution of these failures provided unexpected insights.
Over half of all failures were attributed to input quality, where the required information was missing from the documents. Retrieval issues accounted for only about 10%. Nearly one-third of errors were caused by a lack of priority guidance in cases with multiple data sources. Establishing rules to prioritize certain documents led to a 7-point increase in overall evaluation scores. The improvement was a result of refining system parameters rather than upgrading the underlying model, a consistent trend in our development lifecycle.
The issue of input quality highlights that insurance submissions are often ongoing negotiations rather than static document sets. Data indicates that 82% of submissions involve follow-up threads, and information is frequently shared through phone calls or secure portals. This makes comprehensive information capture a critical ceiling for extraction accuracy.
The dynamic nature of these negotiations implies that completeness remains a shifting objective. Because submissions evolve post-arrival, benchmarks for machine knowledge must be indexed to the documentation available at the moment of the underwriting decision rather than the initial intake. Measuring performance against a retrospectively perfect file unfairly penalizes the system, while evaluating solely against the first arrival artificially inflates accuracy scores.
This realization shifted the development roadmap toward improving information visibility. For instance, including sibling files from related policies on the same account was found to improve submission completeness by 25%.
It's a data problem
A significant technical shift involved moving from a classic RAG pipeline to an agentic approach. By allowing Nova to traverse the entire file system of a submission like an analyst, success rates for complex fields, such as attachment points, increased from 20% to 60%.
To manage the increased processing time of agentic parsing, thinking budgets and step limits were implemented. Summarizing context and skipping irrelevant fields, such as auto-insurance fields for liability quotes, resulted in significant time savings and improved system efficiency.
The primary constraint encountered was cost. Agentic traversal requires a significantly higher expenditure of tokens and latency compared to a standard retrieval pass. Furthermore, this resource consumption is non-uniform; the most complex submissions naturally demand the most extensive exploration. We determined that the resulting precision gains justified the increased costs. This expenditure is precisely why evaluation infrastructure emerged as a primary bottleneck: these are two sides of the same coin.
When the ground truth is wrong
Pursuing 99% accuracy reveals that human-entered data is not always 100% correct or consistent.
Evaluation scores often identify "errors" that are actually instances where Nova is correct or where underwriters disagree. Developing an autonomous system necessitates an audit of underwriting standards; fields Nova struggles to learn often reflect a lack of consensus among the human team.
Scoreboard
The development process includes persistent challenges and areas requiring further discovery.
However, this is not only a technology problem, it is also an underwriting domain knowledge problem. Commercial construction underwriting depends on judgment that experienced underwriters apply without documenting their reasoning, and that judgment is not present in the submission itself. Building Nova requires extracting it from the people who have priced thousands of these deals, which means our underwriters are directly involved in the development loop. That is slower than writing code, and it is the part that is hard to replicate.
Payroll accuracy has remained at approximately 60% because the information is rarely present in submissions, requiring underwriters to make educated guesses based on past deals. Since Nova cannot parse missing data, the solution lies in standardizing how estimates are produced upstream rather than improving model parsing. This is a good example of the point above: the fix requires underwriting alignment on method, not model work.
Evaluation infrastructure has become a primary bottleneck. High accuracy requirements demand extensive testing, leading to increased costs and slower development cycles. Improving the speed and cost-efficiency of evaluations is a current priority.
Where We Go From Here
As field accuracy improves, the speed of the entire system increases. The objective is to reduce time-to-indication to set the market standard even further for insurance provider responsiveness. Providing next-day indications offers a significant competitive advantage in the market.
The current goals focus on achieving 90% accuracy for the majority of submissions and making "golden quotes" a standard, repeatable output. By delegating preparation to machines, underwriters can focus on decision-making, transforming the technology into a reliable and foundational part of the business.
The objective of these advancements is not to diminish the underwriter's authority, but rather to eliminate the administrative burden of data transcription. By automating the extraction of structured data, we allow underwriters to operate at the top of their license. Ultimately, the development of these autonomous capabilities has served to illuminate, with greater precision, the unique and indispensable value of the human specialist.