The security models of both AWS and PostgreSQL might be interfering here, as well as the operating system. The default is "no connectivity" so you need to explicitly grant communication from the IP address of the host where ArcGIS Server is running. If in doubt, try installing a PostgreSQL client like psql on the Server box, and try connecting from that. If psql can't connect, then the networking to PG is still not correct, and it's an AWS admin task.
Note that PostGIS is the spatial extension to PostgreSQL (implementing geometry and geography types), so it's not PostGIS you connect to, but PostgreSQL itself (in this case, the PG RDS instance endpoint). The AWS admin needs to make a route from the ArcGIS Server EC2 host to the RDS endpoint. If the Server host is on-premise, you'll have a suboptimal solution (due to the high latency), but it still requires an explicit grant of access to the Server IP (you may have firewall issues from the Server node to the RDS as well).
Good luck!
- V
