First, a fast way to get a hand-drawn raster is using r.digit,
where you are allowed to assign categories as you draw. So draw
the box and assign it category 1. Everything outside that box
is automatically assigned category 0 (no data - or outside my mask
in this case).
Second, when you use a r.mask you choose the categories you
want masked, using the raster file you created in r.digit.
If you only want data within category 1, you mark that category
in r.mask, and all data outside that area will be hidden.
Or vice-versa, you mark category 0 in r.mask, and all data within
the category 1 area will be hidden.