On a Pandaboard using U-Boot, you need VBAR to re-point to the vector table in the kernel.
See https://developer.arm.com/documentation/ddi0388/i/system-control/register-summary/security-extensions-registers?lang=en
Currently this register is only available on Armv8-R, but it should also be available on Armv7-A (or, on Armv7-A with Security Extensions). Probably best to add this change to #141 because that already re-organises a bunch of registers to make them available on Armv7-A.