Performance Degradation & Timeout during Large-Scale Data Extraction (8.8M records) via Hosted Feature Service REST API

ÁdhavanBalakrishnan

Hello Community,
​We are encountering a significant performance bottleneck when performing large-scale data extraction from a Hosted Feature Service (approx. 8.8 million records) via the ArcGIS Online REST API.
​The Issue:
We are using standard pagination (resultOffset and resultRecordCount) with a batch size of 2,000.
​0–2M Records: Requests are stable and performant.
​~2M+ Records: Response times increase significantly.
​3M–3.5M Records: The API begins returning timeouts and "invalid query" errors, despite the query parameters being consistent with previous batches.
​Question:
​Is there a known platform-level "cumulative query constraint" or throttling mechanism for very high-volume extractions in a single session?
​Are there recommended patterns for this scale? (e.g., Would parallelizing queries across different ObjectID ranges or using the Extract Data tool via Geoprocessing be more stable than raw REST pagination?)
​Technical Details:
​Service Type: Hosted Feature Service (ArcGIS Online)
​Auth: OAuth 2.0 / User Token
​Method: GET / POST via /query endpoint

VenkataKondepati

Yes—what you’re seeing is common at this scale. There isn’t an official “2M hard limit,” but very large sequential resultOffset paging gets progressively slower and can start failing (timeouts/“invalid query”) because the service still has to scan/skip more rows as the offset grows, and AGOL also enforces fair-use throttling under sustained high-volume pulls.

What works better (more stable patterns):

Stop using deep offsets. Page by ObjectID instead.

First get OID stats: returnIdsOnly=true (or returnCountOnly=true + orderByFields=OBJECTID).

Then query in chunks like: where=OBJECTID > x AND OBJECTID <= y with orderByFields=OBJECTID and resultRecordCount=2000.

This avoids the “skip N rows” penalty that kills performance after a couple million.

Use asynchronous extract instead of raw query when you can.

Prefer Extract Data / Create Replica (sync/replica workflow) for big exports—those are designed for bulk movement and tend to be more resilient than millions of /query calls.

Parallelize carefully (only if needed).

If you parallelize, split by OID ranges (or time slices) and keep concurrency modest (e.g., 3–8 workers). Too much parallelism will trip throttling faster.

Tune for reliability.

Use POST (not GET) for long params, set timeouts/retries with backoff, and request only needed fields (outFields) + returnGeometry=false unless required.