this error message means that CityEngine runs out of memory due to a very large number of objects which have been created. GrammarCore is the CityEngine's core element which creates the models.
To prevent this, you need either to create less models inside CityEngine or buy more RAM. In general, I'd recommend to generate as few models as possible in CityEngine. Polygon counts above 2-5 million use a lot of RAM.
You can check the polygon count in the Information Display in the viewport ( hit d,d in the viewport to toggle this view )