The basic steps to create a Java custom step are:
1) Create a java class that implements IJTXCustomStepSafe
2) Compile the class and package it into a JAR file. The sample Ant buildfile that comes with the samples should help with this. The key thing to note here is that the JAR manifest must contain a class-path entry that includes the relative path from the <ArcGIS Install>\java\lib\ext directory to the Workflow Manager JAR file (located in <WMX install>\java).
3) Copy the resulting JAR to <ArcGIS Install>\java\lib\ext
4) Open ArcGIS Workflow Manager Administrator
You should now be able to create a new step type that uses your custom java step object.
Please let me know if you would like some more information about any of these steps, or if you are running into an issue when trying to run these.