The JavaScript section of the docs touches on this.
Scroll down and you can see the options for the Modal events in the tables.
You need to import the event bus and then you can emit the modal event with a modal id to open and close the modals programmatically.
import { bus } from 'calcite-web`;
bus.emit('modal:open', {id: modalId});
bus.emit('modal:close');