Skip to content
Get in touch

Bots and price scraping in an online store

| Article: 3 min read | Tech and security

Bots are programs that visit a site automatically. Some are useful (search crawlers), some are harmful: scraping copies prices and the catalog, credential stuffing tries stolen logins, scalper bots buy up scarce goods, and card testers check card numbers on the payment page.

In plain words

A store is open to everyone, so robots come in freely too. The bad ones are told apart not by the mere fact of automation but by purpose and load. A competitor may collect your prices every day, very fast, another bot guesses passwords in customer accounts, a third fills carts with goods. You see the consequences: a slower site, odd analytics, hosting bills, accounts that were accessed by someone other than their owners.

How much is it

According to the Imperva (Thales) Bad Bot Report 2026, automated traffic exceeds half of web traffic, and about 40% of it is classified as malicious; attacks on APIs and identity systems make up 27% of attacks; AI-driven attacks grew 12.5 times (Thales/Imperva, 2026). The previous report (2025) gave 51% automated traffic based on 2024 data (Imperva, 2025).

A caveat: this is data from a company that sells bot protection, and it measures its own customers, not the whole internet. For a Ukrainian store the share may differ; your server logs show the real picture.

According to the Verizon DBIR 2026 report, AI amplifies about 15% of attack techniques (Verizon).

Types of bots to watch

What they do

What harm they cause

Price and catalog scraping

A competitor sees your prices, content gets copied, load grows

Credential stuffing

Password guessing, account and bonus takeover

Card testing (carding)

Payment attempts, losses on fees and disputes, see payment fraud

Scalping, bulk buying

Scarce goods bought up by bots

Spam in forms and reviews

Polluted database, risk of injections into AI assistants, see AI risks

DDoS

The site is unavailable, see hosting and traffic peaks

The names of the threat classes come from OWASP's Automated Threats to Web Applications project (OWASP); the page links to a separate handbook with definitions.

Useful bots: search engines and AI crawlers

You can't block "everything automated": Googlebot, Bing and others bring buyers. Google states that its crawlers are identified by the user-agent header, the IP address and the reverse DNS name; the header alone should not be trusted because it can be faked (Google). For AI service crawlers and access rules for them, see AI crawlers.

What you can do

  1. robots.txt is not protection. It asks polite crawlers to stay out but "cannot force crawler behavior"; sensitive content needs passwords and other measures (Google).
  2. Request rate limiting per address on search, login and checkout.
  3. Bot checks on "expensive" actions: login, registration, payment, coupon. Not on every catalog page.
  4. CDN or firewall (WAF) level protection. It filters part of the traffic before it reaches the site.
  5. Two-step login and login notifications against credential stuffing.
  6. Log analysis: sharp request spikes, one address or subnet, unusual user-agents.
  7. Prices. You cannot fully hide public prices from scraping; the real answer is pricing policy, not masking: a bot sees the same as a buyer. For B2B you can show prices after login, see B2B online sales.

Common mistakes

  • Blocking by country or user-agent and accidentally cutting off buyers or Googlebot.
  • Treating analytics sessions as people: part of the traffic is bots, and conversion looks lower. See conversion.
  • A hard captcha at every step. It scares off buyers too.
  • Not separating scraping from load: sometimes the problem is weak hosting, not an attack.

FAQ

What is price scraping? Automatic collection of prices and specs from websites, usually by competitors or price comparison services.

Can scraping be banned completely? No. What a visitor can see, a program can see too. You can make mass collection harder and limit request speed.

How do you tell a bot from a human? By behavior: speed, sequence of actions, one address with thousands of requests, missing typical browser traits. One header is not enough (Google).

Does robots.txt help against malicious bots? No, only polite crawlers follow it (Google).

Sources

Want to check this on your store?

E-commerce development

Custom e-commerce solutions and integrations, from improvements to complex systems.

About the service →

Share

Related

  • Hosting, CDN and traffic peaks

    | Article: 4 min read

    Hosting is the server (or cloud) the store runs on. A CDN (content delivery network) is a network of servers around the world that serves part of the site (images, scripts, sometimes pages) from a point closer to the shopper. Together they determine whether…

  • Online store backups

    | Article: 3 min read

    A backup is a copy of the site, the database (products, orders, customers) and files, stored separately, from which the store can be restored after a failure, a mistake or a break-in. A backup only makes sense if restoring from it has been tested.

  • Payment fraud and 3-D Secure

    | Article: 4 min read

    Payment fraud in an online store is an attempt to obtain goods or money using someone else's card data, by deceiving the customer, or through a dishonest payment dispute. 3-D Secure (3DS) is a protocol by which the issuing bank additionally verifies that the…

  • App, PWA or website: what a store should choose

    | Article: 3 min read

    PWA (progressive web app) is a website that can be installed on a phone like an app and can work with a poor connection. A native app is a separate program from the App Store or Google Play. A responsive site is an ordinary site in a browser that adapts to…