topic Re: Max. text length for InsertCursor field in Developers Questions

Max. text length for InsertCursor field

MarcWouters — Mon, 17 Jul 2017 07:52:22 GMT

I am importing X,Y values from file into a shape, but I need to keep the labels of each point as well in my application.

By lack of a way to keep a list of texts, I create a long string, separated by tabs, to keep the labels and the shape in the same feature in the resulting table.

I use the following code

cursor = arcpy.da.InsertCursor("Pipes", ["Name", "SHAPE@", "PointLabels"])
cursor.insertRow([fileName, polyline, pointLabels])

For some datasets, I get this error

The row contains a bad value. [PointLabels]

I assume that this is caused by exceeding some size limit, but I cannot determine easily which size this could be. (number of points and length of text labels vary quite a lot).

Any idea if there is a limit, and if so, what is this limit ?

Re: Max. text length for InsertCursor field

DanPatterson_Retired — Mon, 17 Jul 2017 08:03:52 GMT

if by 'shape' you mean 'shapefile', then the dbf portion of the shapefile limits a text field to 254 characters

http://desktop.arcgis.com/en/arcmap/latest/manage-data/shapefiles/geoprocessing-considerations-for-shapefile-output.htm

Re: Max. text length for InsertCursor field

MarcWouters — Mon, 17 Jul 2017 08:12:24 GMT

Hi Dan,

I should have made this more clear (due to my short experience with ArcGIS products, I guess).

It's not a 'shapefile' I'm writing to, but a GeodatabaseFeatureTable, and have had no problems with text strings of 200kB.

After posting the question, I re-ran my script with some extra debug print statements, and I could see that it fails on a polygon with 34573 points, and the string with the labels is 230905 bytes long.

Re: Max. text length for InsertCursor field

DanPatterson_Retired — Mon, 17 Jul 2017 08:49:28 GMT

field widths in characters, not bytes is pretty big

http://desktop.arcgis.com/en/arcmap/latest/manage-data/administer-file-gdbs/file-geodatabase-size-and-name-limits.htm

Re: Max. text length for InsertCursor field

MarcWouters — Mon, 17 Jul 2017 09:01:55 GMT

Thanks for pointing out this geodatabase size limits page. Hadn't seen it yet.

Then something else must be wrong.

It's not the content of the text string (backslash or so),it only contains text like 'P1 P2 P3 ... P4357 P4358 ...' and so on

Re: Max. text length for InsertCursor field

XanderBakker — Mon, 17 Jul 2017 11:42:40 GMT

Hi Marc Wouters can you elaborate a little more on what you are hoping to gain with a field that contains 'P1 P2 P3 ... P4357 P4358 ...'? If this represents some kind of relation with other features then you might want to consider using relationshipclasses.

Re: Max. text length for InsertCursor field

MarcWouters — Mon, 17 Jul 2017 11:48:50 GMT

Hi Xander,

In the application (C++/Qt using ArcGIS Runtime 100.0) we're building, we need to be able to identify the label of each individual point of each geometry (pipeline).

I couldn't find a way to store this in a vector/array/list in the file geodatabase.

Relationshipclasses are new to me, no idea if this can help

Re: Max. text length for InsertCursor field

XanderBakker — Mon, 17 Jul 2017 12:15:00 GMT

Some reading on this topic: Benefits of relationship classes—Help | ArcGIS for Desktop I might be useful in this case. You will be able to have your featureclass and a separate table with the "labels" and the relationship class enables the relation between them. You should revise if relationship classes are supported in ArcGIS Runtime 100.0.

Re: Max. text length for InsertCursor field

MarcWouters — Mon, 17 Jul 2017 12:24:45 GMT

Thanks for the hint, but I guess in my case it is not usefull:

this would become a 'many-to-one' relationship : each line/feature in my pipelines table would need to refer to up to 20k labels, how can I link each X,Y point in the shape field to a label in another table ??

Re: Max. text length for InsertCursor field

XanderBakker — Mon, 17 Jul 2017 13:00:59 GMT

When you have a single line that links to 20k labels it is a 'one-to-many' relationship which is quite common. You also mention that you want to link each X, Y point to a label. Are you referring to the vertices of the line? In that case you should manage those as a separate point featureclass. Still not sure what you are trying to obtain. Can you provide an image that explains what you are trying to achieve?

Re: Max. text length for InsertCursor field

MarcWouters — Mon, 17 Jul 2017 13:13:20 GMT

All data MUST be stored in polygons, because we can have several thousands polygons, with several thousands of vertices, and the visualisation would simply be much too slow if we would use one entry per vertex (points table)

Your proposal of having a separate point feature class (in fact duplicating all the data) might work, if we split the visualisation (polygons) and the administration part (points). But it requires a lot of extra linking, I guess

Re: Max. text length for InsertCursor field

JoshuaBixby — Mon, 17 Jul 2017 14:23:23 GMT

Ignoring the data design issues for the time being, which by the way I agree with the other commenters, what is the current field length of the text field you are trying to insert into? I just inserted a 400,000 character string into a file geodatabase text field that was set to 500,000 length, but I get a bad value error if I try to insert a 550,000 character string. In short, make sure your field is specified to handle strings as long as you are trying to insert.

Re: Max. text length for InsertCursor field

MarcWouters — Mon, 17 Jul 2017 14:27:03 GMT

Joshua,

I had just discovered that this was the cause of my original problem.

Thanks for answering to the first question !!

Re: Max. text length for InsertCursor field

MarcWouters — Mon, 17 Jul 2017 14:31:26 GMT

At this moment I'm not going to change the design of the tables, because the C++ application is already using this format, changing everything takes too much time.

So I set my text length to 400k to be on the save side. But many of these text fields are much shorter. This is a huge waste of memory space. Is there a way to set this length dynamically, to store short AND long text strings ?

Re: Max. text length for InsertCursor field

XanderBakker — Mon, 17 Jul 2017 15:34:05 GMT

I guess that if you don't want to change the design of the tables and adapt the code to work with the relationship classes, you will have to accept that there will be memory waste...

Edit: CC vangelo-esristaff who I'm sure can provide some useful insights,,,,

Re: Max. text length for InsertCursor field

DanPatterson_Retired — Mon, 17 Jul 2017 15:36:32 GMT

You can't even modify field length in ArcGIS PRO... unless the table is empty Modify Fields

Re: Max. text length for InsertCursor field

JoshuaBixby — Mon, 17 Jul 2017 16:56:46 GMT

The file geodatabase text field is more like a varchar than char, so you don't really waste any disk space if the max length is set to 100 or 500,000. It is the contents of the field that drive most of the space on disk. That said, there may be performance issues tied to storing lots of short character strings in a field with a max length of 500,000; I don't know the FGDB innards enough to say for sure on performance.