AI is not merely adding more traffic. It is changing who reads, who pays, and who receives the value.

Imagine asking your AI agent to explain a new market, compare three competitors and recommend a strategy before your first meeting of the day.

It reads analyst notes, company blogs and independent research. It returns a crisp answer with caveats and citations. You get what you need without visiting a source.

The information worked. The publishing economics did not.

No advert was viewed. No subscription was considered. No author gained a reader. Perhaps the sources received attribution; perhaps they were merely absorbed into a synthesis.

That is the tension at the centre of an agentic internet: when the reader is a machine and the beneficiary is elsewhere, who pays for the page?

The web was never free

The early web felt like an exchange between people. Someone documented a technical problem, reviewed a product or explained an idea because others did the same. Knowledge created more knowledge. Reputation, curiosity and community were often enough.

Then the web commercialised. We tend to tell that story as a fall from innocence, but commerce also funded the web's expansion.

Advertising paid for attention. Subscriptions turned readers into customers. Retailers funded content that led to sales. Donations supported public-interest work. GitHub can host open source because businesses pay for capabilities around it.

None of this made information free. It moved the bill.

Ethan Zuckerman, who helped create the pop-up advert, later called advertising the internet's "original sin": not the best model, but an easy one for a young web to adopt. It established an economic loop: publishers created, intermediaries helped people discover, and attention or payment returned value.

AI does not break one loop. It introduces three different shifts, each with its own economics.

Shift one: agents become economic actors

GitHub recently said it had moved from a plan to increase capacity tenfold to designing for a future that requires 30 times today's scale.

That sounds like evidence of agents overrunning the internet. It is not.

GitHub is describing a narrower signal: agentic development is driving growth in repositories, pull requests, API usage, automation and large-repository workloads. Machines are doing work, creating artefacts and triggering activity across platforms.

This reflects the future I explored in The Repo of the Future Has No Code in It. If software becomes generated output, machine activity will rise across the platforms producing and operating it.

The commercial question is immediate. Per-seat pricing assumes a relationship between people and consumption. What happens when one employee directs twenty agents, or one agent creates a thousand times more activity than a human?

Do we price the person, the agent, the transaction, the compute or the outcome?

This is not a publishing problem. It is a unit-economics problem for every digital platform built around human-scale behaviour.

That pricing model is not inevitable. Platforms get to choose what they reward, meter and make abundant.

Shift two: crawlers consume without completing the bargain

The second shift concerns machines that collect the raw material of the web.

Cloudflare reports that traffic from a fixed cohort of AI and search crawlers grew by 18% between May 2024 and May 2025. Within that changing mix, some AI crawlers grew far faster.

The distinction matters. A search crawler is not a model-training crawler. A health checker is not an autonomous agent. Cloudflare estimates bots account for roughly 30% of global web traffic, but that activity is not all AI.

It would also be wrong to ignore what has changed.

Traditional search offered an implicit bargain: let us index your work and we will send people back. AI crawling may collect material for training or synthesis without a comparable visit. The publisher pays while another service captures the value.

Publishers can express preferences through robots.txt, but compliance is voluntary. They can block known crawlers, although identities can be spoofed and blocking can also remove useful discovery. Large organisations may negotiate licensing deals. An independent researcher, local newspaper or niche blog has much less leverage.

The problem is not machine consumption itself. It is machine consumption without a clear exchange of permission or value.

There may be an opportunity inside that problem. Previously, charging fractions of a penny for one machine read would have cost more to negotiate and settle than the content was worth. Authenticated crawlers, automated budgets and aggregated settlement could make that microscopic market practical. Could a specialist blogger earn from ten thousand tiny reads without negotiating a global licence? The technology is emerging; whether its value reaches small creators is a choice still to be designed.

Shift three: the human remains, but the click disappears

The third shift is quieter and better evidenced.

The human still asks the question. The AI simply stands between the reader and the source.

In March 2025, Pew Research Center tracked 68,879 Google searches by 900 US adults. With an AI summary, users clicked a traditional result in 8% of visits, against 15% without one. Only 1% clicked a source inside the summary.

This does not prove that agents are replacing human web traffic. It shows something more immediate: AI mediation can nearly halve the likelihood that a human search becomes a visit to a source.

For a user, that may be a better experience. The answer arrives faster. For Google, the user remains inside its product. For the publisher, a piece of content can help answer the question without earning the attention that once funded it.

Agent dominance is not required for the economic bargain to weaken. The click can disappear while the human is still sitting at the keyboard.

It is a design choice whether the answer becomes a dead end or a bridge back to the people and organisations that made it possible.

Insight

AI does not create one bot problem. It creates at least three value-exchange problems: machine-scale platform work, content extraction and answer experiences that remove the visit.

Confidence: High

The ethical fork: commons, enclosure or a two-tier web?

It would be easy to treat this as a pricing problem. It is also a philosophical one.

Knowledge behaves differently from most goods. Sharing an idea does not consume it. Open access lets people inspect claims, challenge power and build on ideas, creating things the original author never imagined. A web limited to information each reader can afford would be poorer, less democratic and, most importantly, less dynamic. The result would be a web with less power to improve socioeconomic outcomes around the world.

Creators also have a legitimate claim. Openness cannot mean that the largest machine operators may copy at industrial scale, wrap the result in a paid product and leave the people producing the knowledge economically invisible.

That tension predates AI. Article 27 of the Universal Declaration of Human Rights puts two ideas side by side: everyone should be able to share in cultural and scientific advancement, while authors' moral and material interests deserve protection.

AI changes the scale and introduces a proxy problem. Is my agent reading a public article meaningfully different from me reading it? What if one commercial assistant reads it for ten million users—or trained on it months earlier?

There is also a risk that we build two versions of the internet.

One might be designed for people: visual, persuasive and funded by advertising. The other might be designed for machines: authenticated, structured and available to agents with a budget. The machine web could improve while the public web becomes a degraded showroom.

If readers increasingly encounter authors through generated summaries, model providers also become editors of work they did not commission and interpreters of ideas they did not create.

Consider this blog. If a post is read by ten thousand agents but no person visits it, has it succeeded, been exploited, or both?

There is no clean answer. Universal access and sustainable creation are both public goods. Any future that preserves one by destroying the other is a design failure.

The fork is not something we merely arrive at. Every access policy, product model and standard helps build one path or the other.

The market is already answering

The response will not be one universal paywall. Several models are emerging.

Response What it changes Strategic tension
Block or reserve rights Crawler controls and machine-readable policies restrict reuse Protects creators but may reduce legitimate discovery
License or charge Paid APIs, direct deals and per-request pricing create an explicit exchange Rewards valuable sources but may favour organisations with scale
Design for agents Structured content and authenticated machine access create a distinct interface Improves utility but may accelerate a two-tier web

Cloudflare's experimental Pay Per Crawl uses HTTP 402 Payment Required to let publishers allow, block or charge authenticated crawlers. It is a private beta from an interested vendor, not a settled standard. Yet agents may eventually carry budgets and negotiate access programmatically.

The grassroots llms.txt proposal gives websites a concise, machine-readable guide to useful material. It could make knowledge easier for agents to use, not harder.

These are not contradictory responses. A creator might make some work freely legible, charge for expensive real-time data and prohibit model training. The old web bundled discovery, reading, copying and reuse into a single public URL. We now get to decide whether—and how—the next web prices and permits them separately.

That is a move toward the metered economy discussed in Right Tool, Right Clock: indirect subsidy gives way to more explicit unit economics, and feedback arrives faster.

Strategy is a choice: map the game, then play it

This future is not something leaders can leave to the infrastructure vendors.

Wardley Mapping offers useful doctrine for acting without pretending the landscape is settled.

First, focus on user need. Is the user the reader, their agent, the model provider, the creator or wider society? "The AI" is not a user need, and those groups want different things.

Second, use a common language, be transparent and challenge assumptions. Map who creates, hosts, consumes and captures the value. This exposes choices that would otherwise hide inside one function's policy.

Third, use appropriate methods. One access model cannot fit public-health guidance, a personal essay, premium research, a live API and a commercial training corpus. Some information should be public, some licensed and some metered.

Finally, think small and design for evolution. Test a machine-readable summary, authenticate one expensive endpoint or license one dataset before applying a universal policy.

Wardley's context-specific gameplay adds a more provocative idea: openness can be a weapon of business strategy.

Open source and open data can accelerate a component's evolution towards commodity. That can expand an ecosystem, remove a competitor's point of differentiation or make a complementary service more valuable. Openness is not only generosity. Applied deliberately, it changes the landscape.

The inverse matters too. Making everything free is not a strategy. A business might open the common substrate that attracts agents while charging for trust, timeliness, specialist judgement or operationally expensive access.

The same discipline applies beyond publishing: a software platform may preserve simple seat pricing for human use while metering machine-scale transactions.

A practical leadership exercise follows:

  1. Map the human and machine users.
  2. Map the value and cost flows.
  3. Decide what should be open, metered or protected.
  4. Run a small experiment and publish what you learn.

The opportunity is larger than defending today's pages from tomorrow's crawlers. We can choose which knowledge becomes a foundation others build upon and where sustainable value should accrue.

What could refute this story?

The alarming version of this argument depends on assumptions that may not hold.

Bot traffic, AI crawling, search summaries and autonomous agents are not one exponential curve. Caching may reduce repeated requests, while agents could send referrals worth more than casual search traffic.

Value can also be negotiated. In 2024, News Corp gave OpenAI permission to display content from its publications and use current and archived material to enhance its products under a multi-year agreement. That model will not suit every creator, but it disproves the idea that AI consumption must return nothing.

There is a more provocative route back to a free web: advertising may adapt. OpenAI has begun testing ads in ChatGPT to support free access, promising clear labels, answer independence, privacy protections and user control. Extrapolating beyond OpenAI's test, an AI could recognise conversational intent and present a useful purchase path at the right moment; less obtrusive than a banner and potentially far more valuable as a referral.

That could restore the subsidy. It also creates a thin ethical line. If an assistant understands a person's needs and vulnerabilities, when does helpful commercial relevance become invisible persuasion? A labelled advert beside an answer is one thing; an answer subtly shaped by advertising value is another. OpenAI says its test preserves answer independence. Keeping that separation inspectable may become one of the defining trust choices of an AI-mediated web.

Human attention may remain valuable precisely because it is scarce. People still seek trusted voices, original experience and community. Identity, attribution, payment and ethical advertising could complete a new bargain rather than end the free internet.

The most plausible future is not uniformly open or closed. It is negotiated, uneven and contested—which means the choices made now matter.

Choose the web, then build it

Return to the executive asking an agent for a strategy.

That answer still depends on someone doing the research, gathering the experience and taking the intellectual risk of publishing an idea. But its future is not preordained by model companies, publishers or infrastructure providers.

A map does not choose the move.

What does your organisation publish that agents can consume? What value returns when they do? Which parts would create more opportunity if they were radically open? Which require permission, attribution or payment to remain worth producing?

There will be no single answer for the whole internet. There can be an intentional answer for your corner of it.

The free internet may not be ending. The bargain that made it look free is being rewritten, and machines now have a seat at the negotiating table.

What kind of internet do you want humans and agents to inherit—and what are you going to build now to make it real?

Sources and further reading

AI-assisted. The observed traffic and referral shifts are sourced above. The prospect of an internet where agents conduct most consumption remains a thought experiment, not a forecast.