Train deep learning model won't quit

[MarkSchweder]

My train deep learning model won't quit after 2 and-a-half days. Which is a day longer than it said it would take to complete the task. Is it possible to stop the program and recover what it has done so far? Should I wait for it to finish? Should I cancel and start again? Something else?

Thanks, Mark

[DanPatterson]

Did you try it with a smaller dataset to confirm the process?

Details on the input data type, location, extents and size would be useful as would anything about the destination parameters.

[MarkSchweder]

Yes. its big data and that's the point. I'm trying to find out how big the data can be. It's been cut in half once and it looks like another time is necessary.

In any event, is it possible to stop the program and recover what it has done so far?

Thanks

[RTPL_AU]

@MarkSchweder 
Is it using the correct GPU?

[MarkSchweder]

yes