topic Definition expression only applies to some of the data instead of all of it in ArcGIS JavaScript Maps SDK Questions

Definition expression only applies to some of the data instead of all of it

MKF62 — Sat, 11 Dec 2021 06:05:56 GMT

I create a definition expression on a featureLayer through it's relationship with table. The definition expression gets applied to the table first, then the definition expression for the featureLayer is created and applied. The definition expression is pulling records of a certain year in the table, then through it's relationship with the featureLayer I get GUIDs of those that match the pulled records in the table and set the featureLayer's definition expression based on those GUIDs. The problem is that for some years this works and for others it doesn't. I can see in the debugger that the definition expression gets created correctly for the featureLayer, but when I apply it to the featureLayer, nothing displays on the map. If I switch the year to another year, the definition expression gets applied and the appropriate polygons appear for that year. If I switch back to the previous year, again, no polygons are displayed, even though they exist for that year. It's pretty odd behavior and I'm not sure why this would be happening.

Records in the table when the definition expression is set to YearTreated = 2015.

Records in the related feature layer that match these MgmtTractIDs

My code should produce a definition expression like this for application to the featureLayer:

var de = "MgmtTractID = '{...}' OR MgmtTractID = '{...}' OR MgmtTractID = '{...}'‍

..so on and so forth

Apply that definition expression to the feature layer and I get nothing displayed on the map. This definition expression works for 2009, 2010, 2011, 2012, 2013, 2014, 2017, 2018, and 2019. It doesn't work for 2008, 2015, or 2016. I have checked and rechecked that the DE is being created correctly with formatting and that the MgmtTractIDs do indeed show up in the related feature layer. Everything looks fine.

on(dom.byId("data-year"), "change", function () {
    var year = dom.byId("data-year").value;
    //Defintion expressions are already set upon first load of the map, don't need to recalculate
    //if the year hasn't been changed. A change event is fired on load of the map.
    if (year !== currentYear) {
        hbMgmtTableFL.setDefinitionExpression('YearTreated = ' + year);
        hbMgmtTableFL.refresh();
        //The habitat management feature layer needs to be filtered using the related table.
        getTractDefExpression();
    }
});

function getTractDefExpression() {
    var recordIDs = new Query();
    recordIDs.where = "1=1";
    hbMgmtTableFL.queryIds(query, function (OIDs) {
        if (OIDs !== null) {
            //Create relationship query
            var relatedRecordsQuery = new RelationshipQuery();
            relatedRecordsQuery.objectIds = OIDs;
            relatedRecordsQuery.outFields = ["MgmtTractID"];
            relatedRecordsQuery.relationshipId = 0;
            hbMgmtTableFL.queryRelatedFeatures(relatedRecordsQuery, function (fset) {
                //Get all MgmtTractIDs for the current year from the table
                //The table already has it's definition expression set, so it'll
                //only pull the MgmtTractIDs for the year that is set
                var MgmtTractIDsArr = [];
                for (i = 0; i < OIDs.length; i++) {
                    var OID = OIDs[i];
                    var features = fset[OID].features;
                    var mgmtTractId = features[0].attributes.MgmtTractID;
                    if (!(MgmtTractIDsArr.includes(mgmtTractId))) {
                        MgmtTractIDsArr.push(mgmtTractId);
                    };
                }
                //Create the definition expression for the feature layer
                var whereClause;
                for (i = 0; i < MgmtTractIDsArr.length; i++) {
                    if (i == 0) {
                        whereClause = "MgmtTractID = '" + MgmtTractIDsArr[i] + "'";
                    }
                    else {
                        whereClause = whereClause + " OR MgmtTractID = '" + MgmtTractIDsArr[i] + "'";
                    }
                }
                //Set the polygon feature layer's definition expression based on the GUIDs
                debugger;
                hbMgmtTractFL.setDefinitionExpression(whereClause);
                hbMgmtTractFL.refresh();
            });
        }
        else {
            debugger;
            hbMgmtTractFL.setDefinitionExpression('MgmtTractID = 0');
            hbMgmtTractFL.refresh();
        }
    });
};‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Definition expression only applies to some of the data instead of all of it

RobertScheitlin__GISP — Tue, 02 Apr 2019 16:53:05 GMT

Have you tried to console.log the exact SQL that is being applied when it fails and apply that to the layer in Desktop to see if it works? Also I normally do an IN( '...','...') SQL query if I have more than a couple of values instead of using OR.

Re: Definition expression only applies to some of the data instead of all of it

MKF62 — Tue, 02 Apr 2019 17:53:29 GMT

I have tried it in Desktop and it works as expected. Thanks for the tip on the "IN" - I did that and 2015 and 2016 data started showing up. Still doesn't show 2008 data though...

Re: Definition expression only applies to some of the data instead of all of it

RobertScheitlin__GISP — Tue, 02 Apr 2019 18:02:36 GMT

That makes it sounds like there is a problem with the SQL statement then. Sorry it is not going to be easy for me to help you on this.

Re: Definition expression only applies to some of the data instead of all of it

MKF62 — Sat, 11 Dec 2021 06:05:59 GMT

I understand. It's just odd because it generates the statement and when I try it in Desktop it works.

I can't see anything wrong with this:

MgmtTractID IN(
'{B1CE758E-4CA2-469E-9933-8746DFACA3EF}', 
'{A37D1BE3-005A-4D2C-AE76-6B30E98AF981}', 
'{BBA93CA8-08F0-4DF5-9EFE-6F4EDF5664D2}', 
'{75CC656A-BDBC-4BFA-A6A0-1B1BA62DE105}', 
'{88E58225-D2A1-4E35-8E79-9EE5B17A94A3}', 
'{CD2B81CA-1FF3-4930-932C-1E88F1AF1C9D}', 
'{5483E110-E6E2-46BE-98E5-3BE2ED0F60BF}', 
'{BE01BA5D-C604-494B-BCB6-4160AB5474E1}', 
'{0FA279E3-38E1-4BAC-910B-6991066B97D3}', 
'{A6D44511-C057-42D8-8096-B7A84D80BE82}', 
'{9B8F4664-0E2D-452B-AE2D-34768A0BFC0B}', 
'{75E9C08C-3E14-4344-B35A-57CCF41D1497}', 
'{7EEED5DC-3C1E-49DA-AE89-445E35D3BA7A}', 
'{1A52D3FA-FE09-4996-8F0D-4FA6EB68F121}', 
'{CD343F57-A2DF-43D8-8616-872CF8514E16}', 
'{F4E855C9-0A45-4B0E-A5D8-4C02178733B7}', 
'{4814C6C4-C583-4F8E-81FA-122FBA807EFE}', 
'{0ECFC833-928C-4B9C-96F0-F75F31FCD7B9}', 
'{305D7D13-8CEF-4253-A7AC-B418852FA37C}', 
'{3249A913-299E-43FF-82F2-EA76DF8C7FD0}', 
'{22432A46-E938-49DF-B1AA-E52347DA5BAF}', 
'{907B11E8-77E1-4C65-B6B4-A41F0DD9B44E}', 
'{801E7790-5715-44F8-9F26-97BCA0FCF0AD}', 
'{C90E76A7-F195-4B6C-ADF0-50D7BCB8BB9E}', 
'{0A00ACB3-5759-40CF-AD90-739406DFD4C5}', 
'{35D8BDC1-EF20-4185-86B1-3C6DB6F08412}', 
'{95E3803D-8797-47A5-BB68-D69A45711B1C}', 
'{7556D941-DF86-464E-B9F6-1194241479EB}', 
'{DC3B0C72-4FB7-4042-81F4-297F6F254330}'
)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Definition expression only applies to some of the data instead of all of it

RobertScheitlin__GISP — Tue, 02 Apr 2019 18:33:48 GMT

Molly,

Sorry I don't see a problem either.

Re: Definition expression only applies to some of the data instead of all of it

MKF62 — Tue, 02 Apr 2019 18:43:33 GMT

Just noticed in my console that I get 404 Not Found errors when I try to query for 2008 data, don't receive that for any of the other years. I use a proxy to access my data, haven't the slightest why it'd only be one year that it can't query. I'll keep working on it, report back if I figure it out. Thanks for your help.

Re: Definition expression only applies to some of the data instead of all of it

RobertScheitlin__GISP — Tue, 02 Apr 2019 18:46:05 GMT

Can you show me a screenshot of the error?

Re: Definition expression only applies to some of the data instead of all of it

MKF62 — Tue, 02 Apr 2019 18:57:17 GMT

It gives me this 30 times in the console (I think if you click on them you can make them bigger?):

From the network tab:

When I double click the red line in the network tab:

If the "query string is too long" is referring to the MgmtTractID query I make, then that's really weird because 2008 has the least amount of IDs to query. Other years have upwards of 200. This one only has 29.

Edit: Actually, the requested URL here is much longer than the others, so it does have to do with it being quite long. The other successful queries just make a request to the MapServer url:

Re: Definition expression only applies to some of the data instead of all of it

RobertScheitlin__GISP — Tue, 02 Apr 2019 19:04:43 GMT

Molly,

Right click on the url in the console tab and choose open in new tab. Does it provide results or error?

Re: Definition expression only applies to some of the data instead of all of it

MKF62 — Tue, 02 Apr 2019 19:09:18 GMT

That gives me the same detail page as above:

Not sure if you saw my edit to my last post. The request URL for 2008 data is quite different than other years.

Re: Definition expression only applies to some of the data instead of all of it

RobertScheitlin__GISP — Tue, 02 Apr 2019 19:13:30 GMT

Molly,

Are any of the ones that are successful using the proxy?

Re: Definition expression only applies to some of the data instead of all of it

MKF62 — Tue, 02 Apr 2019 19:17:23 GMT

Yes, for comparison, this is what a successful one looks like:

Re: Definition expression only applies to some of the data instead of all of it

RobertScheitlin__GISP — Tue, 02 Apr 2019 19:32:07 GMT

Molly,

So when the method is Post then it succeeds when it is Get it fails, which is logical. When a query string gets to big the proxy should automatically switch to Post instead of Get. In your proxy.config how specific is your serverUrl? Meaning do you have https://....org/.../rest/services/blah/blah/MapService/blah?

Re: Definition expression only applies to some of the data instead of all of it

MKF62 — Sat, 11 Dec 2021 06:06:01 GMT

My serverUrls are not terribly specific (we had a long thread a long time ago where we hashed it out actually ). They look like this:

<!-- Testing environment -->
<ProxyConfig allowedReferers="*"
             mustMatch="true"
             logFile="proxyLog.txt"
             logLevel="Warning">
    <serverUrls>
      <serverUrl url="https://www.xxx.org/xxx/rest/services/HabitatManagement/HabitatManagement/GPServer"
                 username="x"
                 password="x"
                 matchAll="true"/>
      <serverUrl url="https://www.xxx.org/xxx/rest/services/HabitatMonitoring/HabitatData/MapServer"
                 username="x"
                 password="x"
                 matchAll="true" />
      <serverUrl url="https://www.xxx.org/xxx/rest/services/HabitatMonitoring/HabitatClassification/GPServer"
                 username="x"
                 password="x"
                 matchAll="true" />
      <serverUrl url="https://www.xxx.org/xxx/rest/services/PrintHabitatMaps/PrintHabitatMap/GPServer"
                 username="x"
                 password="x"
                 matchAll="true" />
      <serverUrl url="https://www.xxx.org/xxx/rest/services/ShipData/ShipData/GPServer"
                 username="x"
                 password="x"
                 matchAll="true" />
    </serverUrls>
</ProxyConfig>‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Proxy line in JS looks like this:

urlUtils.addProxyRule({
    urlPrefix: "https://www.xxx.org",
    //When switching between testing/development, make sure the change allowedReferers, etc. in proxy.config as well
    //For production environment (publishing)
    //proxyUrl: "/FocalAreas/proxy/proxy.ashx"
    //For testing in development environment
    proxyUrl: "/proxy/proxy.ashx"
});‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Just for the sake of testing, I added more dummy parameters to the end of the 2008 query and after I did that, the method is sent as POST and the data shows up as it should.

//Create the definition expression
var whereClause;
for (i = 0; i < MgmtTractIDsArr.length; i++) {
    if (i == 0 && i == MgmtTractIDsArr.length - 1) {
        whereClause = "MgmtTractID IN ('" + MgmtTractIDsArr[i] + "')";
    }
    else if (i == 0) {
        whereClause = "MgmtTractID IN('" + MgmtTractIDsArr[i] + "', ";
    }
    else if (i !== 0 && i !== MgmtTractIDsArr.length - 1) {
        whereClause = whereClause + "'" + MgmtTractIDsArr[i] + "', ";
    }
    else {
        whereClause = whereClause + "'" + MgmtTractIDsArr[i] + "'";
    }
}
//Set the polygon feature layer's definition expression based on the GUIDs
var de = hbMgmtTableFL.getDefinitionExpression();
if (de == "YearTreated = 2008") {
    whereClause = whereClause + "'{0}, {1}, {2}, {3}, {4}, {5}, {6}, {7}, {8}, {9}, {10}')";
}
else {
    whereClause = whereClause + ')';
}
hbMgmtTractFL.setDefinitionExpression(whereClause);
hbMgmtTractFL.refresh();‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

The 2008 query has 29 records. The 2009 query has 24 and is sent as a GET successfully, so it must be under the character limit.

Re: Definition expression only applies to some of the data instead of all of it

RobertScheitlin__GISP — Wed, 03 Apr 2019 13:09:08 GMT

Molly,

Has your IT Dept set a custom queryString limit on your web server?

Re: Definition expression only applies to some of the data instead of all of it

MKF62 — Wed, 03 Apr 2019 15:28:50 GMT

Nope, it doesn't look like it, but I'm going to talk with someone else and see if he's got any ideas. The query works if it's 28 records or 30 records, but not 29. The number of characters in the whereClause variable for 28 records is 1191 and for 30 records it's 1275, so if nothing else, I can find a workaround where if the number of characters are between those two values, then I can tack on a dummy value to get it where it needs to be. I don't like this solution very much, but I've got nothing left in the arsenal.

Re: Definition expression only applies to some of the data instead of all of it

RobertScheitlin__GISP — Wed, 03 Apr 2019 21:23:19 GMT

Sorry I am out of ideas then