Building a product with AI capabilities is architecturally different from building a product without them. The non-determinism, latency profile, cost model, and failure modes of AI components do not fit the assumptions that standard web application architecture is built around. Products that treat AI capabilities as a standard service dependency discover this the hard way — through unexpectedly high inference costs, user-facing latency that is difficult to tune, and failure modes that surface slowly through output quality degradation rather than fast through errors.
This article covers the software architecture decisions specific to AI-powered products: how to structure the boundary between AI capabilities and core application logic, how to design APIs that handle AI latency and non-determinism gracefully, how to manage state for multi-turn interactions, and how to build observability that catches quality degradation before users notice it.
Separating AI Capabilities from Core Application Logic
The most important structural decision in an AI product is where the boundary sits between AI capabilities and core application logic. A well-drawn boundary means you can change the underlying model or provider, tune prompts, or switch from an external API to a self-hosted model without touching application logic. A poorly drawn boundary means every model change requires changes across the application.
The AI capability layer should be a service — either a separate microservice or a well-isolated module within a monolith — with a clean interface that abstracts the AI implementation details from the application. The interface expresses the operation in domain terms: `classify_support_ticket(text) -> Category`, `generate_product_description(product) -> str`, `extract_contract_fields(document) -> ContractFields`. The implementation behind this interface handles prompt construction, model invocation, output parsing, and error handling. When the underlying model changes, the implementation changes; the application interface stays the same.
Avoid embedding prompt construction in application code that also handles business logic. Prompts that live in the same functions as business logic create coupling that makes both harder to maintain. The prompt for classifying a support ticket should live with the classification service implementation, not scattered across the request handling code. This separation enables prompt versioning, A/B testing, and evaluation — none of which are possible when the prompt is interleaved with application logic.
API Design for AI Features
AI operations have a latency profile that synchronous REST APIs handle poorly. An LLM call that takes 3–15 seconds to complete holds an HTTP connection for that duration — a problem for clients with short timeouts, load balancers with connection limits, and users who may navigate away before the response arrives. Design AI-powered endpoints with asynchronous patterns: accept the request, return a task ID immediately, and let the client poll for completion or receive a webhook callback.
Streaming responses are the correct model for generative AI features where partial results are useful — text generation, summarisation, reasoning traces. The HTTP streaming pattern (server-sent events or chunked transfer encoding) delivers tokens to the client as they are generated, enabling progressive rendering that makes the latency feel significantly shorter to users even when total generation time is unchanged. Streaming requires infrastructure that supports long-held connections — not all proxy and CDN configurations support it without specific configuration.
Rate limiting for AI API endpoints must account for cost as well as request volume. A traditional rate limiter that allows 1000 requests per minute does not protect against a single request that triggers a chain of expensive model calls. Token-based rate limiting — tracking the total tokens consumed across inference calls, not just request counts — provides better cost protection. Set limits at the level where unexpected usage patterns trigger an alert before they become an invoice shock.
AI API Design Patterns
- Long-running operations: async task + polling or webhook, not synchronous
- Generative features: streaming response for progressive rendering
- Rate limiting: token-based limits, not just request counts
- Caching: semantic cache for equivalent queries, not exact-match only
- Timeout: separate timeout for AI operations from standard API operations
State and Session Management
Multi-turn AI interactions — chatbots, copilots, agentic workflows — require state management that associates a conversation history with a session. The simplest approach stores the full conversation history in the session and sends it with every inference call. This works until histories grow long enough to exceed model context windows or inflate per-call token costs significantly. A session that accumulates 20 turns at 500 tokens per turn will include 10,000 tokens of history with every inference call — a 10× cost multiplier compared to stateless operations.
Conversation summarisation addresses context window pressure by summarising earlier turns when the history exceeds a threshold. The summarised history is shorter than the original, reducing token cost while preserving the essential context. The tradeoff is that summarisation is lossy — specific details from earlier turns may not survive summarisation, which can cause inconsistency in long conversations. For most use cases, this tradeoff is acceptable; for high-stakes use cases where conversation details are consequential, explicit memory stores provide more reliable retention.
Session state should be stored in a persistent, distributed cache (Redis with persistence) rather than in application server memory. Application servers that restart lose in-memory session state — a session tied to a specific server instance breaks under normal deployment operations. Redis-backed session storage survives application server restarts and scales across multiple application server instances without session affinity routing.
Fallback and Graceful Degradation
AI components fail in ways that core application logic typically does not: rate limit errors, context window exceeded errors, model API outages, and quality degradation that does not produce an error but produces a wrong answer. Each failure mode requires a different handling strategy — and all of them should be handled explicitly, not allowed to propagate as unhandled exceptions to the user.
For external LLM API calls, implement retry with exponential backoff for transient errors (rate limits, 503 responses), a fallback to a secondary model or provider for extended outages, and a fallback to a degraded non-AI response path for situations where the AI functionality is supplementary rather than core. A customer support chatbot that falls back to "I can't help with this right now — let me connect you to a human agent" during an LLM API outage is better than one that returns a 500 error.
Timeout handling for AI operations must be explicit and generous. A 30-second timeout that is standard for web APIs is too aggressive for complex AI operations that may legitimately take longer. Set AI operation timeouts based on the P99 latency of the operation, with a user-facing "processing" state that sets expectations for operations that take longer than a few seconds. Users who know an operation is in progress wait more patiently than users who see an apparent hang.
Observability for AI Products
Standard application monitoring — error rates, latency, throughput — does not capture the dimension of AI product quality that matters most: output quality. A system with zero errors and perfect latency can still be producing AI outputs that are wrong, unhelpful, or inconsistent with the product's requirements. Capturing a sample of AI inputs and outputs for ongoing quality evaluation is mandatory for AI products where quality degradation has business impact.
LLM call tracing — logging every inference call with its full prompt, output, model version, token counts, and latency — enables debugging of specific user complaints and systematic quality monitoring. This logging requires thoughtful data handling: prompts and outputs may contain personal information that must be governed according to your privacy policy. Anonymise or filter sensitive content before logging, or build into the consent flow that conversations may be used for quality improvement.
Evaluation on a sample of production traffic — running a judge model or human evaluators over recent outputs — is more representative of real quality than benchmark evaluations on curated test sets. A weekly review of a stratified sample of production outputs, categorised by use case type, catches quality drift that benchmark scores do not reflect. Build this review into the development process as a recurring obligation, not an ad hoc investigation triggered by user complaints.
