You might try the GRASS program i.zc, which is user "tunable" edge
detection filter. It uses FFT processes so you need to keep the size
of your region relatively small so you wont run out of memory and to
improve the running time. Good luck with your coastlines.
Jim Hinthorne
GIS Lab
Central Washington Univ.