What is the maximum total of API request that I can send?

[RoslanSaidi]

Hi, My use case is to get full datasets. I have 1 layer that contains 281,666 total data. For performance wise, I can get 2000 total data only per request via API. To get full dataset of the layer, I need to spin the request and use while loop which is total data is greater than index. My below is my code.

let start = 0

let limit = 2000

while (totalData > start) {

     let data = api.repository.getData(layer, start, limit)

     start += limit

}

async function getData(layer, start, limit)

{

     try {

      let query = layer.createQuery()

      query.where = '1=1'

      query.start = start

      query.num = limit

      const { data } = await layer.queryFeatures(query)

      console.log(data)

     } catch (e) {

      console.error(e)

     }

}

Based on my code, I only can get 30,000 only data with 10 columns. Is there I missing the server configuration? How do I want to setting the rate limit or total maximum of request? Thanks!

[Trevor_Hart]

Unlike max features per query, there is no "maximum" features for sending data but rather max request POST size.

This can be applied in two places, the web server and ArcGIS Server.

I'll get the links shortly.

[Trevor_Hart]

Oh sorry, you are requesting data...

Are you using ArcGIS Enterprise or ArcGIS Online?

[RoslanSaidi]

Hi @Trevor_Hart . Thanks for the reply.

I'm using ArcGIS Enterprise. 

[Trevor_Hart]

Ok open server manager and find the service. Click the pencil.

On the parameters tab there is a setting called "Maximum Number of Records Returned by Server"

Change this to a bigger number but remember that this will make queries slower - eg if you for example make the maximum 20,000 records.

[RoslanSaidi]

Hi, @Trevor_Hart ,

I tried this solution but when I want to make several request to get data. Sometimes it response "timeout exceeded, even thought I tried to increase maximum timeout and write code `esriConfig.timeout = 20000`

[Trevor_Hart]

Timeout is usually in milliseconds, so 20000 is only 20 seconds, so time out is to be expected.

To be efficient you should only request a small amount of rows at a time - this means quicker turn around from the server. Sending many big requests will make it slower.

Based on your code you are sending multiple queries at once - the await should be in the loop.

The problem is likley you are making many requests at the same time and your service is likley to set 2 instances max - meaning you can only process 2 requests at the same time.

while (totalData > start) {
     let data = await api.repository.getData(layer, start, limit) // YOU NEED AWAIT HERE
     start += limit
}

[RoslanSaidi]

Hi,

Yes I already repair my code. Thanks.

I tried to get full datasets with 1 outFields, and it succeeded, I got full datasets with 1 column. But to get all column, there's something reject on my request. I tried debug and it returns empty array.

[JoeSuarez]

Here's an updated version of your code with some improvements and error handling:

let start = 0;
const limit = 2000;

/**
* Fetches data from a business information API with pagination.
* @param {object} layer - The layer object representing the API.
* @param {number} start - The starting index for data retrieval.
* @param {number} limit - The maximum number of items to retrieve in each request.
*/
async function fetchData(layer, start, limit) {
try {
let totalCount = await getTotalCount(layer); // Get the total count from the API or any other source

while (start < totalCount) {
const data = await getData(layer, start, limit);
// Process the data as needed
start += limit;
}
} catch (error) {
console.error("Error fetching data:", error);
}
}

/**
* Retrieves total count from the business information API.
* @param {object} layer - The layer object representing the API.
* @returns {Promise<number>} - A Promise that resolves to the total count.
*/
async function getTotalCount(layer) {
try {
// Implement logic to get the total count from the API
// For example: const total = await businessInformationApi.getTotalCount();
// Replace businessInformationApi.getTotalCount() with the actual method or API call
const total = 10000; // Replace with the actual total count
return total;
} catch (error) {
console.error("Error fetching total count:", error);
throw error;
}
}

/**
* Retrieves data from the business information API based on the provided parameters.
* @param {object} layer - The layer object representing the API.
* @param {number} start - The starting index for data retrieval.
* @param {number} limit - The maximum number of items to retrieve.
* @returns {Promise<object>} - A Promise that resolves to the retrieved data.
*/
async function getData(layer, start, limit) {
try {
let query = layer.createQuery();
query.where = '1=1';
query.start = start;
query.num = limit;

const { data } = await layer.queryFeatures(query);
return data;
} catch (e) {
console.error("Error fetching data:", e);
throw e; // Rethrow the error to be caught by the outer try-catch block
}
}

// Usage
fetchData(yourLayer, start, limit);

In this version, I added a getTotalCount function to fetch the total count from the business information API before starting the data retrieval loop. This ensures that you have the correct total count for proper pagination. Additionally, I added more detailed comments for better understanding.