how to re-establish the connection to ArcGIS Server manager or rest pages ?
is there a way to reset the configuration from HTTPS only to HTTPs and HTTP ?
is there a way to default back these security changes in ArcGIS Server ?
Esri support services should be able to help you get your site back to allowing a connection on 6080 (http). It's best to test with both http and https enabled before switching to https only to verify that the SSL certificate will work with ArcGIS Server.
I would search through all the ArcGIS Server configuration files for the one which stores this setting and try updating that.
Another alternative is to clean ArcGIS Server off the machine and reinstall it. By clean, I mean that you uninstall it and manually remove any files related to it from the file system. Just uninstalling the software is typically insufficient.
If you have backups then another alternative is to roll the server back to a point before you made these changes.
Of course, as pointed out by Randall, support services is another option for you.
I believe that the HTTP Only, HTTP and HTTPS, and HTTPS Only settings for the protocol under Security --> Config in AGS Manager pertain to the web adaptor, so altering the value as you mentioned should not cause the issue especially since you said that no web adaptor had been configured and your URL requests use a port number. Based on the info you provided, I would suspect the importing of another security certificate to be the culprit for the time being. It sounds like SSL was working just fine for you initially. A few things to try and/or consider first...
Did you import a CA-signed certificate to replace the default self-signed certificate in AGS? If so, did you use the importRootorImmediate option or the importExistingServerCertificate option? If it's a CA-signed certificate, take a look at the certification path properties for that certificate and verify that all of the certs in the path (i.e., the root cert, any intermediary certs, and the cert itself) exist in the Trusted Root Certification Authority on the AGS machine. You'll know if all of them are trusted when you view the certification path properties depending on whether or not a red X appears next to any of them.
Alternatively, if you need to "clear out" any changes you made regarding the security certificates to get things back to normal per se, you MIGHT be able to try the following:
1. On the server, go to C:\arcgisserver\config-store\machines and open the <servername>.json file (assumes AGS was installed at C:\arcgisserver) using Notepad or Notepad++. Make a backup of the file in the same directory, first.
2. In that file, find the section with "webServerCertificateAlias" and change the value after the colon to be exactly selfsignedcertificate surrounded by double quotes just as you see with the current value. If it's already set to this value, then more than likely you did not try to import a CA-signed certificate into AGS and this most likely isn't your issue.
3. Save the file and then re-start the ArcGIS for Server Windows service.
4. After about a minute following restart of the service in #3 above, try re-requesting the URLs you mentioned to see if the behavior is different. If so, then refer back to the second paragraph of my reply and make sure that, if using a CA-signed certificate, you follow the proper steps to ensure it is trusted on the AGS machine along with any of its intermediary and root certs. If not, then revert back to the backup file created in #1 above and restart the AGS service again since this probably isn't the correct issue and associated fix.
In the config store under the security folder, you have to make changes in the json file. Stop the Arcgis server service from the windows services panel and set the ssl enabled= false in this json configuration file..