Updated for Chrome 136+. This guide uses an isolated, non-default Chrome user-data directory for browser automation. Do not point remote debugging at your everyday Chrome profile or reuse personal cookies and sessions.

You will build a local Amazon product scraper with Python, Selenium, BeautifulSoup and a small Flask API. Use it only for pages you are authorized to access, keep request volume conservative, and review the site's terms and applicable rules before collecting data.

The approach: an isolated Chrome automation profile

Chrome 136 no longer honors remote-debugging switches against the default Chrome data directory. Create a dedicated directory outside the normal Chrome profile and start a separate browser instance:

mkdir C:\selenium-profiles\amazon
"C:\Program Files\Google\Chrome\Application\chrome.exe" --remote-debugging-port=9333 --user-data-dir="C:\selenium-profiles\amazon"

Complete any permitted setup in this isolated profile, then keep it dedicated to the scraper. Never copy credentials, cookies or session files from your personal profile into the automation directory.

Connect Selenium to the isolated browser

The driver connects to the already running isolated instance through debuggerAddress. This keeps the browser visible for debugging, but it does not bypass anti-bot controls, prevent CAPTCHAs or guarantee access.

A simple, maintainable project structure

I'm a big believer in keeping things clean and modular. Here's how I structured this:

src/
├── main.py              # app entry point
├── config.py            # all the boring configuration stuff
├── routes.py            # API endpoints
└── scraper/
    ├── driver_manager.py    # handles chrome connection
    ├── scraper.py           # scraping logic
    └── data_extractor.py    # parses and cleans the data

Reuse one isolated browser session

class DriverManager:
    def __init__(self):
        self.driver = None
        self.wait = None

    def connect(self):
        options = Options()
        options.add_experimental_option("debuggerAddress", f"localhost:{Config.CHROME_DEBUG_PORT}")
        self.driver = webdriver.Chrome(options=options)
        self.wait = WebDriverWait(self.driver, Config.SELENIUM_TIMEOUT)
        return self.driver

This singleton pattern ensures we're reusing the same browser connection. Why? Because starting up a new Chrome instance every time is expensive (both in time and resources), and more importantly, you lose the isolated automation session state.

The WebDriverWait is there for those moments when Amazon's JavaScript takes a hot second to load. Trust me, you need this.

Extract search result data

Here's where we actually grab the data:

def search(self, query):
    response = self._get_response(f"https://www.amazon.com/s?k={query}&ref=cs_503_search")

    results = []
    for listitem_el in response.soup.select('div'):
        product_container_el = listitem_el.select_one(".s-product-image-container")
        if not product_container_el:
            continue

I'm using BeautifulSoup here because, let's face it, it's way more pleasant to work with than XPath or Selenium's built-in element finders. Once the page loads, I grab the HTML and let BeautifulSoup parse it. Simple as that.

Tip: Amazon's search results use a specific structure with div. This is pretty stable across their site variations. I learned this the hard way after my scraper broke twice because I was relying on class names that Amazon kept changing.

Normalize regular and sale prices

Amazon's pricing HTML is... interesting. Sometimes there's a sale price, sometimes there isn't. Sometimes the regular price is crossed out, sometimes it's not even there. Here's how I handle it:

def get_price_from_elements(price1_el, price2_el):
    regular_price = None
    sale_price = None
    current_price = None

    if price1_el and price2_el:
        # both exist = item is on sale
        sale_price = DataExtractor._parse_currency_value(price1_el.text)
        regular_price = DataExtractor._parse_currency_value(price2_el.text)
        current_price = sale_price
    elif price1_el:
        # only one price = regular price
        regular_price = DataExtractor._parse_currency_value(price1_el.text)
        current_price = regular_price

The _parse_currency_value method is where things get spicy:

def _parse_currency_value(s: str):
    # extract currency symbol ($, €, £, etc.)
    currency_match = re.match(r'^[^\s&0-9]+', s)
    currency = currency_match.group(0) if currency_match else None

    # grab all digits and convert to float
    digits = ''.join(re.findall(r'\d+', s))
    if not digits:
        raise ValueError("Invalid price!")

    amount = float(digits) / 100  # convert cents to $s

Why divide by 100? Because "1999" should be $19.99, not $1,999. This handles all sorts of currency formats without breaking a sweat.

Expose the scraper through Flask

I wrapped everything in a simple Flask API because, honestly, who wants to mess with Python imports every time they need to scrape something?

@api.route('/search', methods=['GET'])
def search():
    query = request.args.get('query', '')

    if not query:
        return jsonify({"error": "query required"}), 400

    try:
        driver = driver_manager.get_driver()
        scraper = Scraper(driver)
        result = scraper.search(query)
        return jsonify(result)
    except Exception as e:
        return jsonify({"error": str(e)}), 500

Now you can just:

curl "http://localhost:5000/search?query=mechanical+keyboard"

And get back nice, clean JSON:

{
  "query": "mechanical keyboard",
  "count": 20,
  "results": [
    {
      "id": "item-xyz",
      "title": "Cherry MX Blue Mechanical Gaming Keyboard",
      "url": "https://www.amazon.com/dp/B08...",
      "image_src": "https://m.media-amazon.com/images/...",
      "price": {
        "current_price": {
          "formatted": "$89.99",
          "amount": 89.99,
          "symbol": "$"
        },
        "regular_price": { ... },
        "sale_price": null
      }
    }
  ]
}

The product details

Search results are great, but sometimes you need the full details:

def get_product(self, url):
    response = self._get_response(url)

    product_title_el = response.soup.select_one("#productTitle")
    price1_el = response.soup.select_one('.a-price > .a-offscreen')
    price2_el = response.soup.select_one('.a-price > .a-offscreen')

    if not product_title_el:
        return None

    return {
        "url": url,
        "title": data_extractor.get_product_title(product_title_el.text),
        "price": data_extractor.get_price_from_elements(price1_el, price2_el),
        "images": data_extractor.get_images_from_product(response.soup.select("#altImages .imageThumbnail"))
    }

One cool trick here is the image upscaling. Amazon gives you tiny thumbnails by default, but with a little regex magic:

def get_images_from_product(image_elements):
    images = []
    for image_element in image_elements:
        img = image_element.select_one('img')
        src = img.get('src')
        # replace the size parameter to get full resolution
        new_src = re.sub(r'\.[^/]*?_\.(jpg|jpeg|png|webp)$', r'._AC_SL1500_.\1', src)
        images.append(new_src)
    return images

That regex finds Amazon's size indicator in the URL and replaces it with AC_SL1500 - their code for "give me the big version."

Yeah, I know you’re a hardcoder, but please don’t hardcode anything

I keep all the tunable parameters in one place:

class Config:
    CHROME_DEBUG_PORT = 9333
    API_HOST = '0.0.0.0'
    API_PORT = 5000
    SELENIUM_TIMEOUT = 30
    PAGE_LOAD_DELAY = 5

That PAGE_LOAD_DELAY? Critical. Amazon's pages load in stages, and if you try to parse too early, you'll miss half the data. Five seconds is my sweet spot, but YMMV depending on your internet speed.

Yeah we did it!

Let me break down the advantages of using your own browser:

What this approach helps with

  • A visible browser makes selector and timing failures easier to inspect.
  • A dedicated profile keeps automation state separate from personal browsing data.
  • Reusing one local browser reduces startup overhead for a small prototype.
  • Explicit waits and defensive parsing make missing fields easier to handle.

This setup does not make automation invisible. Amazon can still return a challenge, deny a request or change its markup. Stop the job when access is denied rather than attempting to circumvent the control.

Limitations and suitable use cases

This approach is perfect for:

  • Personal projects
  • Building a prototype
  • Low-volume scraping
  • Understanding how Amazon's frontend works

But it's not great for:

  • High-volume production scraping
  • Running on servers (you need a desktop environment)
  • Parallel requests (one browser = one request at a time)
  • Completely automated, hands-off operation

For production workloads, evaluate a maintained API

If you're running a business that needs reliable, high-volume Amazon data, you probably want something more robust. Managing your own scraping infrastructure gets complicated fast - you need proxies, CAPTCHA solving services, constant maintenance as Amazon changes their HTML...

For production use cases, I'd recommend checking out Amazon Instant Data API from our friends at DataOcean. They handle all the headaches of maintaining scrapers at scale, dealing with rate limits, rotating IPs, and keeping up with Amazon's changes. Sometimes paying for a good API beats maintaining your own infrastructure.

Keep browser, parsing and API responsibilities separate

One thing I want to emphasize is the separation of concerns. Notice how:

  • driver_manager.py only handles browser connections
  • scraper.py only handles page navigation and element location
  • data_extractor.py only handles parsing and cleaning data
  • routes.py only handles HTTP requests

This isn't just me being pedantic. When Amazon changes their HTML (and they will), you only need to update the selectors in scraper.py. When you want to add a new data field, you just extend data_extractor.py. Clean architecture saves your sanity.

Handle missing and changing markup

Amazon's HTML isn't always consistent. Sometimes fields are missing. Sometimes they use different class names for the same thing. That's why I check everything:

if not image_el or not link_el or not h2:
    continue

Better to skip a product than crash the entire scraper because one listing is malformed.

Next steps

Building a scraper is part art, part science. The technical bits are straightforward once you understand them, but the real skill is in making architectural decisions that save you time down the road.

Using an isolated browser through remote debugging keeps a local prototype observable and separate from personal browsing. It's elegant, it is useful for a bounded prototype when its limits are understood.

Is it perfect? No. Will it scale to millions of requests? Also no. But for what it is - a clean, maintainable, easy-to-understand scraper that actually works - I'm pretty happy with it.

Now go forth and scrape responsibly. And seriously, if you need production-scale scraping, check out that DataOcean API or just contact me if your needs are much much than simple API could give you. Your future self will thank you.

Want to build something similar?
The code structure I showed you works for pretty much any website. Just swap out the selectors, adjust the data extraction logic, and you're good to go. The browser-connection approach is universal.

Questions?
Drop them in the comments. I'm always happy to talk scraping strategies, Python architecture, or why BeautifulSoup is superior to XPath (fight me).

Happy scraping! 🚀

👉 You can find the full Amazon scraper code on our GitHub, feel free to check it out https://github.com/letsscrapecom/simple-amazon-scraper