Most caves require a permit. In National Parks, such as the Leeuwin-Naturaliste, this would be from the Department of Environment and Conservation.
If you are really interested in caving you should contact the caving club or go on a trip with a commercial operator.