Browser automation has a reputation for fragility, and that reputation was earned. Anyone who has maintained a Selenium test suite through a major frontend redesign understands the problem. Selectors that worked yesterday stop working today. Session cookies expire mid-run. Pages that loaded consistently in development time out in production. The question is whether these failure modes are inherent to the category or problems that can be engineered around at the infrastructure level.
The answer depends heavily on what you're automating. Browser-based tests against your own application, where you control both the test and the application, present different reliability challenges than browser automation against a third-party vendor portal you don't control. The latter is the hard case, and it requires infrastructure decisions that most introductory automation guides don't address.
The Failure Modes Are Predictable
Most browser automation failures fall into a small set of categories. Knowing them precisely is the first step toward handling them correctly rather than treating every failure as a unique mystery.
Session expiration is the most common: an authenticated session was established at the start of a long-running workflow, and by the time the workflow reaches a later step, the session cookie has expired. The page now shows a login screen instead of the expected content. Most automation code that doesn't handle this will either throw a timeout error or, worse, continue running against the wrong page state and produce incorrect results silently.
Selector brittleness is the second category: the CSS selector or XPath expression used to locate an element no longer matches because the page layout was updated. This ranges from trivial (a class name was renamed) to significant (the entire DOM structure of a section changed). The challenge here is distinguishing between a selector failure that requires a code update and a transient rendering issue that a retry would resolve.
Network and timing failures are the third category: the page doesn't fully load within the timeout window, a network request fails partway through a page load, or an async operation (such as a server-side report generation) doesn't complete within the expected time frame. These are often intermittent and require different handling than deterministic failures.
Authentication friction is distinct from session expiration: an MFA prompt, a CAPTCHA challenge, or a rate-limiting interstitial appears at a step where the workflow expected clean page content. These require active response, not just retry.
Session Management as First-Class Infrastructure
The most durable approach to session management we've built is to treat the authenticated session as a stateful object with its own lifecycle management, separate from the workflow steps that use it. The session is established, validated, refreshed when needed, and closed cleanly. The workflow steps that interact with portal content call into this session layer rather than managing authentication themselves.
Concretely: before any workflow step that requires portal access, a session health check runs. If the session is still valid, the step proceeds. If the session has expired or is about to expire within a configurable window, re-authentication runs before the step proceeds. The workflow step itself doesn't need to know anything about session state; it relies on the guarantee that the session layer provides.
This matters particularly for long-running workflows. A workflow that extracts data from a portal, processes it, writes results to an internal system, and then updates a status field back in the portal might span 10 to 20 minutes of wall time. A session that was valid at the start is not guaranteed to be valid at the end. Treating session management as a background concern rather than first-class infrastructure produces workflows that work in testing and fail in production.
Selector Strategy for External Portals
For portals you don't control, the selector strategy needs to favor structural stability over precise targeting. Class names are often generated or obfuscated. IDs may be dynamic. The more stable identifiers are typically visible text content, ARIA labels, and position within a known structural context.
Playwright's `getByRole`, `getByText`, and `getByLabel` locators are meaningfully more stable than raw CSS selectors against third-party pages because they target accessible attributes and visible content rather than implementation details of the page's CSS architecture. When the page designer renames a CSS class, the ARIA label or button text usually stays the same.
That said, external portal pages do change, and there is no selector strategy that eliminates the need to update automation when significant page changes occur. The goal is to maximize the time between required updates and to fail loudly when a selector no longer works rather than silently continuing against the wrong page state.
We use strict mode on element resolution, which throws immediately when an expected element doesn't match rather than returning the wrong element. A workflow that fails with a clear "element not found" error is significantly easier to debug and fix than one that silently extracts data from the wrong location.
Retry Logic: Transient vs. Deterministic Failures
Not all failures warrant a retry. Implementing blanket retry logic on browser automation is a common mistake that turns deterministic failures into very slow deterministic failures, and sometimes produces harmful side effects if the failing step involved a write operation.
The distinction that matters is whether the failure is transient (network hiccup, slow page load, race condition in async rendering) or deterministic (element not found because the page structure changed, unexpected authentication challenge that requires human intervention, configuration error). Transient failures are good candidates for retry with backoff. Deterministic failures should fail fast and generate a clear error with enough context to diagnose the cause.
For Playwright specifically, using `waitForSelector` with a configured timeout and handling the timeout as a classified error type, rather than letting it bubble as a generic exception, gives you the control to make this distinction. A timeout on a page that consistently loads in under three seconds is meaningfully different from a timeout on a page that is known to take up to 30 seconds when generating a report. The retry window should reflect the actual timing characteristics of the target system, not a generic default.
Handling Authentication Challenges Mid-Workflow
A workflow that establishes a clean authenticated session at the start will still encounter authentication challenges during execution. Rate-limiting interstitials appear after a series of rapid navigation actions. Step-up authentication prompts appear when accessing higher-sensitivity sections of a portal. CAPTCHAs appear after failed login attempts or after behavioral patterns that trigger security heuristics.
The infrastructure decision here is about detection and escalation rather than automated bypass. When an unexpected authentication challenge appears, the workflow should recognize it, pause execution cleanly, and either resolve it through a configured handler or escalate to an observable error state with full context about where in the workflow the challenge appeared. Running a partial workflow and then failing silently is worse than failing loudly at the first unexpected challenge.
For TOTP-based MFA, the integration is straightforward: the workflow has access to the TOTP secret, generates the current code at the moment it's needed, and enters it. The session established after successful MFA completion should be stored and reused for the duration of the workflow rather than re-authenticating from scratch for each portal interaction.
Observability Is Not Optional
The failure modes described above are not hypotheticals. They happen on real portal interactions at real production frequencies. The difference between an automation that operations teams trust and one that gets abandoned is whether failures are visible, diagnosable, and fixable without deep investigation.
Every workflow run should produce a structured log of: which steps executed, which steps failed, what the page state was at the time of failure (screenshot plus DOM snapshot), and which failure category applies. This is not about debugging convenience; it is about distinguishing between "this workflow needs maintenance because the portal changed" and "this workflow is failing because of a transient network issue."
A workflow that runs silently and produces results is good. A workflow that fails and tells you exactly why is far more valuable than one that fails and forces you to reproduce the failure manually to understand it. The operational cost of the latter is often what drives teams to abandon automation that was worth building in the first place.
The Honest Assessment
Browser automation against third-party portals will require maintenance over time. Portal layouts change, authentication flows are updated, and new security measures are introduced. We are not claiming that a workflow built today will run without any human attention indefinitely. What we are saying is that the failure modes are known, the infrastructure patterns to handle them are established, and workflows built with these patterns in mind will run reliably for months and years rather than days or weeks before requiring intervention. The investment in the infrastructure layer pays back many times over in reduced operational load compared to workflows built without it.
