As far as a toga goes, I think it would depend on the style and how much it covers. With a toga, you essentially have everything from your "toga party" bed sheet, to more ornate ones.
Something that's short and only covers half the chest probably wouldn't be allowed, but a longer, more covering and ornate one may be. Such as a toga picta.