Possible memory leak with Dictionary Renderer and very dynamic attributes

[imbachb]

We are experiencing a possible memory leak when using the dictionary renderer and very dynamic attributes. We are using the APP-6(E) dictionary style downloaded from https://www.arcgis.com/home/group.html?id=07dd0bdb5ea441269a05a502508eb499

We're updating the attributes of existing graphics on a defined tick.

QVariantMap attributes;
attributes["speed"] = someObj.getSpeed();
graphic->attributes()->setAttributesMap(attributes);

In general we noticed that the used memory depends on how varied the attributes are, i.e. the more varied the attributes, the more memory will be used over time. In above case for example, the speed can take many different values. Since our applications must run for multiple days we noticed that the memory consumption kept increasing with such dynamic attributes.

To analyze this issue we created a bit of an extreme test case where we update all the graphics with a varied set of random attributes:

	QTimer* timer = new QTimer(this);
	connect(timer, &QTimer::timeout, this, [this] {
		for (auto graphic : *m_graphicsOverlay->graphics()) {
			QVariantMap attributes;
			auto identification = identifications[identification_distribution(m_generator)];
			auto mainIcon = mainIcons[mainIcon_distribution(m_generator)];
			auto sidc = "140" + identification + "010000" + mainIcon + "000000";
			attributes["sidc"] = QString::fromStdString(sidc);
			auto randomX = 8.5417 + (9.2417 - 8.5417) * static_cast<double>(rand()) / RAND_MAX;
			auto randomY = 47.3769 + (46.3769 - 47.3769) * static_cast<double>(rand()) / RAND_MAX;
			attributes["direction"] =
				QString::number(180.0 + (-180.0 - 180.0) * static_cast<double>(rand()) / RAND_MAX);
			attributes["speed"] = QString::number(10000.0 + (0.0 - 10000.0) * static_cast<double>(rand()) / RAND_MAX);
			attributes["staffcomment"] =
				QString::number(10000.0 + (0.0 - 10000.0) * static_cast<double>(rand()) / RAND_MAX);
			attributes["uniquedesignation"] =
				QString::number(10000.0 + (0.0 - 10000.0) * static_cast<double>(rand()) / RAND_MAX);
			attributes["additionalinformation"] =
				QString::number(10000.0 + (0.0 - 10000.0) * static_cast<double>(rand()) / RAND_MAX);
			auto point = Point(randomX, randomY, m_spatialReference);
			graphic->attributes()->setAttributesMap(attributes);
			graphic->setGeometry(point);
		}
	});
	timer->start(100);

We let this run for a while and noticed that the memory keeps increasing, seemingly unbound. Pure speculation: It seems as if the symbol variants are getting cached, but since there is a nigh infinite amount of combinations, the cache keeps increasing in size.

We also did some heap snapshots to see if we can identify the issue:

As you can see in the stacks, code regarding the symbols was the cause of most new allocations.

Please find attached a more complete source code example. The example code does following:

• Add a dictionary renderer using the app-6e stylx.
• Add 200 graphics at random positions
• Update attributes of all graphics on a timer

We are using Windows 10, ArcGIS Maps SDK for Qt 200.8, and DirectX rendering API.

Is there anything we can do to stop this increase of memory used while keeping the varied attributes?

[JamesBallard1]

@imbachb thanks for reporting this and sharing the code snippets. That helps make things easier for us to reproduce. We will investigate soon and let you know what we find.

[imbachb]

Thank you for looking into it. I made the example a bit more extreme and let it run over night. When I came in the morning the PC seems to have crashed in the night. In the Event Viewer I could see a "Resource-Exhaustion-Detector" Event indicating the process took at one point close to 48GB of RAM:

`Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: Playground.exe (1060324) consumed 47701180416 bytes`

Here is the code for the more "extreme" example where even more attributes are set.

namespace
{
const std::vector<std::string> identifications{"0", "1", "2", "3", "4", "5"};
std::uniform_int_distribution<std::size_t> identification_distribution{0, identifications.size() - 1};
const std::vector<std::string> mainIcons{
	"000000",
	"110000",
	"110100",
	"110101",
	"110102",
	"110103",
	"110104",
	"110105",
	"110107",
	"110116",
	"110200"};
std::uniform_int_distribution<std::size_t> mainIcon_distribution{0, mainIcons.size() - 1};
std::uniform_int_distribution<std::size_t> azimuth_distribution{0, 360};
const std::vector<std::string> modifiers{"00", "01", "02", "03", "04", "05", "06", "07", "08", "09", "10"};
std::uniform_int_distribution<std::size_t> modifiers_distribution{0, modifiers.size() - 1};
} // namespace

	QTimer* timer = new QTimer(this);
	connect(timer, &QTimer::timeout, this, [this] {
		for (auto graphic : *m_graphicsOverlay->graphics()) {
			QVariantMap attributes;
			const auto& identification = identifications[identification_distribution(m_generator)];
			const auto& mainIcon = mainIcons[mainIcon_distribution(m_generator)];
			const auto& modifier1 = modifiers[modifiers_distribution(m_generator)];
			const auto& modifier2 = modifiers[modifiers_distribution(m_generator)];
			auto sidc = "140" + identification + "010000" + mainIcon + "00" + modifier1 + modifier2;
			attributes["sidc"] = QString::fromStdString(sidc);
			auto randomX = 8.5417 + (9.2417 - 8.5417) * static_cast<double>(rand()) / RAND_MAX;
			auto randomY = 47.3769 + (46.3769 - 47.3769) * static_cast<double>(rand()) / RAND_MAX;
			// attributes["direction"] = QString::number(static_cast<int>(azimuth_distribution(m_generator)) - 180);
			//  qDebug() << attributes["direction"];
			attributes["direction"] =
				QString::number(180.0 + (-180.0 - 180.0) * static_cast<double>(rand()) / RAND_MAX);
			attributes["speed"] = QString::number(100000.0 + (0.0 - 100000.0) * static_cast<double>(rand()) / RAND_MAX);
			attributes["staffcomment"] =
				QString::number(100000.0 + (0.0 - 100000.0) * static_cast<double>(rand()) / RAND_MAX);
			attributes["uniquedesignation"] =
				QString::number(100000.0 + (0.0 - 100000.0) * static_cast<double>(rand()) / RAND_MAX);
			attributes["additionalinformation"] =
				QString::number(100000.0 + (0.0 - 100000.0) * static_cast<double>(rand()) / RAND_MAX);
			attributes["x"] = QString::number(100000.0 + (0.0 - 100000.0) * static_cast<double>(rand()) / RAND_MAX);
			attributes["y"] = QString::number(100000.0 + (0.0 - 100000.0) * static_cast<double>(rand()) / RAND_MAX);
			attributes["z"] = QString::number(100000.0 + (0.0 - 100000.0) * static_cast<double>(rand()) / RAND_MAX);
			attributes["z2"] = QString::number(100000.0 + (0.0 - 100000.0) * static_cast<double>(rand()) / RAND_MAX);
			auto point = Point(randomX, randomY, m_spatialReference);
			graphic->attributes()->setAttributesMap(attributes);
			graphic->setGeometry(point);
		}
	});
	timer->start(100);

[LucasDanzinger · Accepted solution]

we've logged the following bug in our system -

BUG-000181142: Memory leak with Dictionary Renderer and very dynamic attributes

If you are interested, you can contact your Esri Distributor to get your customer account attached to the bug. This way you will be in the system and get updates as there is progress to report.