Transverse single-spin asymmetry of charged hadrons at forward and backward rapidity in polarized $p$+$p$, $p$+Al, and $p$+Au collisions at $\sqrt{s_{NN}}=200$ GeV}