As can be seen at http://gis.stackexchange.com/questions/236262/renaming-arcgis-pro-project-aprx-file-geodatabase-and-toolbox I created an ArcGIS Pro project named TestProject in a new folder and after doing considerable work on it I decided that it warranted a better name like ProperProject. In the TestProject folder by default there is TestProject.aprx, TestProject.gdb and TestProject.tbx.
Renaming the TestProject folder to ProperProject leaves the TestProject.* files and file geodatabase with their original name, and there seems to be no easy way to rename TestProject.aprx to ProperProject.aprx that automatically renames its default file geodatabase and toolbox to ProperProject.gdb and ProperProject.tbx.
I'm happy to leave the project folder rename to Windows, but I think ArcGIS Pro should make renaming a *.aprx also rename its default file geodatabase and toolbox.