is it possible to make a mask in the orthomapping workflow, so that DEM and orthoimages are processed only in a specific AOI of the images?
I'm thinking the Extent Environment setting set to a specific AOI will do the trick! You set Environment Settings on the Analysis core tab, in GP Tools, in ModelBuilder and/or in GP tools in ModelBuilder. Here's a listing of all the environment settings one can use - here.