As explained in the comments above the rmmdir, the rmmdir should not
fail if it is not empty. As I understand, the directories should only be
removed if they are empty.

Without this change, doc() and lintian() fails and break the build.