The Micro Trace Buffer records the history of the pc and can be used to debug hard faults even when the stack trace is useless.
Everything works fine without USB being plugged in but faults (I think) when USB is plugged in. This is switched away from the USB code from the bootloader onto the USB code thats generated by Atmel Studio using the high level classes from ASF.