GeoServer does not support RBG+Alpha with JPEG compression.
Since your alpha channel is basically just a binary mask, use gdal_translate to convert it to a RGB + mask instead, that combination should be better supported.
Something like this should work :
gdal_translate -of COG -b 1 -b 2 -b 3 -mask 4 -co COMPRESS=JPEG input.tiff output.tif
The COG output format ensures you have a GeoTIFF with tiles and overviews out of the box, the first three bands are selected for color, the 4th turned into a binary mask, and the JPEG compression is preserved. Look up the docs for more parameters to tweak the output (e.g., JPEG quality and the like).
Hope this helps
Regards,