Image Classification Wizard producing completely inaccurate results?

[ClaireSchirle]

I am using ArcGIS Pro 2.6.0. I am trying to use 1m resolution NAIP orthoimagery in conjunction with the Image Classification wizard to classify trees versus non-trees in small towns across Nebraska. I clipped the NAIP raster to the bounds of my town and am using the IR version so that the vegetation stands out from other objects. (see screenshot of example of the raster image I am using).  **Edit: I just updated to 2.8.0 and I'm still having the same issues.

Within the image classification wizard I am using my own classification schema that I created (contains tree, building, road, shadows, and grass types), a spectral detail of 12, a spatial detail of 12, and a minimum segment size of 3 (since I have somewhat course resolution). My Preview_segmented layer looks good when I am zoomed out to ~ 1:2000 scale (see attached) but then gets really wonky when I zoom way in ~1:200 scale. Tree areas which should be red are all of a sudden showing up as grey and buildings are showing up pink like grass. My first question is if this is part of why I am getting bad results from the classifier? I watched a tutorial from ESRI and they zoomed WAYY (and the preview segmented layer didn't show up weird, so I have no idea what is wrong with mine) in when selecting training data, but my preview_segmented layer looks terrible zoomed in...

Because the segmented preview is a terrible representation when zoomed in, I selected my training data while at 1:2000 scale. (see attached for outlines of training data over the segmented preview). The process of selecting train data seems to work well....trees line up well with tree segments, grass with grass segments, etc. I have tried this with many sample numbers from about 50 for each classification category to 200 for each category.

I then go on to train the model using Support vector machine, and check all the segment attributes. I leave Max # sample cases as default 500. 

I then get a good output segmented layer that segments trees, versus grass versus roads well. (see attached of segment layer output) 

I then get my preview classified output and it tells me buildings and roads are trees (which is not true) and that many trees are buildings, i.e. totally inaccurate and unusable. And the preview classified layer also classifies areas differently depending on my scale (see images for zoomed in and zoomed out screenshots of preview classified layer). For example, when I zoom out a bunch of areas that were classified as roads are all of a sudden classified as buildings (area circled in blue in attachment)

I was just wondering if anyone knows what I'm doing wrong because the result I'm getting seems far beyond the expected errors of this tool. It looks like an amazing tool from the tutorials I watched but I'm not sure how to get it to work properly. I have played with training sample sizes, spectral and spatial details and have probably run the tool 50 or 60 times, all with different but still unusable results and I'm starting to get frustrated haha. If anyone has advice I'd be so thankful! Thanks in advance!