I guess that the issue has something to do with a binary nodata mask that the source images seem to have.
Thus the source images have three 8-bit bands for R, G, and B, and an addtional 1-bit band for the nodata. The binary mask is actually supported by GeoServer, see White tiles for geotiff (jpeg rgb + alpha, tiles, pyramid) - #2 by aaime-geosolutions, but maybe the SLD layer is introducing something unexpected for the code.
-Jukka Rahkonen-