It was missing some ulport swaps that caused isakmp quick mode as responder to fail under certain cases.