Project
Google SERP API Scraper
A production-ready REST API that scrapes Google search results by programmatically generating valid cookies — bypassing Proof-of-Work challenges and reCAPTCHA Enterprise through residential proxies.
Architecture
Three ECS Fargate services behind an ALB, communicating through Redis as a job queue and cookie pool.
Client
ALB
HTTP :80
ECS Service
API (FastAPI :8000)
Validates key, checks rate limit, enqueues job
Redis
Job Queue + Cookie Pool + Result Cache
ECS Service
Worker (x3)
Consumes jobs, executes search, parses results
ECS Service
Pool Manager
Pre-warms cookies via PoW + reCAPTCHA + proxy
How cookie generation works
Google blocks automated traffic with Proof-of-Work challenges and reCAPTCHA. This system generates valid cookies in 8 steps, each requiring careful fingerprint management.
Route through residential proxy
Sticky session ensures all requests come from the same IP.
Visit google.com
Collect initial cookies with a browser-grade TLS fingerprint (curl_cffi).
Trigger JS challenge
Google serves a Proof-of-Work challenge on suspicious traffic.
Extract solver scripts
Parse the challenge page and pull the solver + orchestrator JS.
Solve PoW with Deno
Execute the solver in Deno with browser shims to compute the answer.
Solve reCAPTCHA Enterprise
Send the challenge to Capsolver API with proxy and data-s token.
Submit to /sorry/index
Post the PoW + CAPTCHA solutions to obtain GOOGLE_ABUSE_EXEMPTION cookie.
Verify and search
Confirm the cookie works with a real search, then return parsed results.
Tech stack
API example
Submit a search query and poll for results. Supports Google, Startpage, and Yahoo with geographic targeting.
Request
POST /v1/search
Host: api.example.com
X-API-Key: sk-demo-key
Content-Type: application/json
{
"query": "best restaurants in Lima",
"pages": 1,
"engine": "google",
"gl": "pe",
"hl": "es"
}Response
{
"job_id": "a3f1c9e2-7b4d-4e8a-b5c6-1d2e3f4a5b6c",
"status": "done",
"engine": "google",
"query": "best restaurants in Lima",
"results": {
"organic": [
{
"position": 1,
"title": "Los 15 mejores restaurantes de Lima - TripAdvisor",
"url": "https://tripadvisor.com/Restaurants-g294316",
"snippet": "Los mejores restaurantes de Lima: consulta opiniones de restaurantes en Lima, Peru y busca por precio, ubicacion y mas."
},
{
"position": 2,
"title": "25 Best Restaurants in Lima, Peru (2025)",
"url": "https://traveltriangle.com/blog/restaurants-in-lima/",
"snippet": "Central, Maido, and Astrid & Gaston lead the list of the best restaurants in Lima..."
},
{
"position": 3,
"title": "Guia gastronomica: restaurantes imperdibles en Lima",
"url": "https://peru.travel/gastronomia/lima-restaurantes",
"snippet": "Lima es la capital gastronomica de Sudamerica. Descubre los restaurantes que debes visitar."
}
],
"total_results": "~12,400,000",
"search_time": "0.48s"
}
}Key features
Anti-bot evasion
Solves Google's PoW and reCAPTCHA challenges programmatically — no headless browsers.
Multi-engine support
Search Google, Startpage, and Yahoo from a single API with consistent output format.
Cookie pool with pre-warming
Background service keeps a pool of valid cookies ready, minimizing latency per search.
Multi-tenant API keys
Rate-limited API keys with tiered plans (free to unlimited), managed via admin endpoints.
Geographic targeting
Set country (gl), language (hl), and location for localized results.
Proxy rotation
Supports Rayobyte, PlainProxies, and Geonode — switchable at runtime via admin API.
Interested in this scraper?
Available as a whitelabel code license, with optional setup and ongoing support.