Developers of agentic artificial intelligence have long promised a future of autonomous systems capable of booking flights, monitoring competitors, and handling procurement cycles without human intervention. Despite the technology largely existing, the infrastructure to make it work reliably at scale remains inadequate. A critical key fact highlighted in recent industry analysis is Gartner's projection that over 40% of agentic AI projects will be canceled before the end of 2027, citing escalating costs, unclear business value, and inadequate risk controls. This statistic underscores a fundamental disconnect between AI agent ambition and real-world performance.
The Web's Resistance to Autonomous Agents
For AI agents to function effectively, they must access websites, interpret responses, and translate that data into actionable insights. This must happen consistently, in real-time, and at scale. However, the current web architecture poses significant barriers. Online platforms that agents would need to query depend on information not being readily available to maintain competitive advantages. They employ personalized results, sponsored placements, and urgency cues to shape user behavior, making it nearly impossible for an agent to objectively compare options.
The Oxylabs Web Openness Index quantifies this challenge by scoring over 120 countries on various aspects of web accessibility. The findings provide key facts: the global average for practical reachability — how well a site responds to standard automated HTTP requests — stands at 83.4 out of 100. However, anti-automation friction measures such as CAPTCHAs, rate limiting, fingerprinting, and bot detection score only 62.8 on average, and structured data interoperability drops further to 60.3. These 20-plus-point differences reveal a structural gap: while sites generally respond to automated requests, restrictions abound and data is often returned in machine-unfriendly formats. Agents that rely on reliable, timely, structured information frequently fall into this gap.
Data-Starved AI Within Organizations
Inside organizations, AI agents face a related but distinct problem: a lack of usable data. Relevant data exists but often remains uncleaned, untagged, or unstructured in ways that AI systems cannot interpret. This problem extends to customer-facing applications built on agentic systems. Without real-time web data — current prices, live inventory, policy updates, market movements — agents are forced to reason based on a frozen version of the world. Latency further compounds the issue: an agent that eventually returns the correct answer is far less useful than one that returns it quickly enough to act upon. In autonomous systems, tolerance for delay is even lower. The constraint is consistent: agents need context they can trust, yet they are not receiving it from their own organizational data or from the web.
Historical Lessons: Infrastructure for Scale
This is not the first time information volume has outpaced processing capacity. The early web held vast knowledge but was unusable in its raw state. Infrastructure built for scale — web crawlers indexing pages, scrapers comparing prices, monitoring systems tracking fraudulent ads — made the difference. A contemporary example is the Debunk.org investigation, a non-profit combating online disinformation, which uncovered a multilingual scam operation targeting former fraud victims. The investigation identified over 50,000 ads, 459 domains, and more than 1,100 related web pages, reaching an estimated 52 million people across Europe. Such coverage requires systematic, automated data collection at scale. Agentic AI demands an even higher level of infrastructure because agents do more with data than any previous application: they need information that is structured, current, complete, and returned fast enough to support real-time action.
The Three Cs of Reliable Agent Infrastructure
Platforms are unlikely to voluntarily open up to frictionless automated access, as doing so means ceding control over discovery, ranking, and customer relationships. While beneficial to consumers, it threatens short-term revenue. Therefore, the infrastructure that makes agentic systems work reliably must be built independently. Three key requirements — the three Cs — stand out: Consistency ensures agents produce reliable behavior; unreliable data sources lead to project cancellations. Currency means real-time access to prices, inventory, availability, and policy; agents reasoning based on stale assumptions create more problems than they solve in commercial contexts. Compliance ensures access built outside fair standards provokes countermeasures; sustainable infrastructure is necessary both technically and in practice.
The web was not designed for agents. Within organizations, the context agents need is often inaccessible or unavailable. These are data quality problems that can be solved, and infrastructure problems that are actively being addressed. The ultimate decision rests with society: whether to welcome AI agents or continue holding them back by failing to provide the necessary foundational infrastructure.