abuild size differs depending on file system

abuild uses du -sk to get the size of the package. This command includes the size of directories. The size of directories depends on the file system though. Eg. on ext4 the size is 4K and on a tmpfs it depends on the content. The same probably effects special files and symlinks too.

I suggest only counting the size of normal files. Something like this could be used instead: find . -type f -exec du -csb -- {} + 2>/dev/null | tail -n 1 | cut -f1 (wasn't good)

This would be as far as I can tell consistent with the apk3 behavior then.

Handles

• hardlinks
• spaces in path
  • newlines in path
• when no files exist
• files without read permission
• files up to 2^31-1 bytes (fat32 limit)
• multiple filesystems (inodes per filesystem)
• directories without read/execute permission
  • alternatively errors

Current best

{ echo 0; find . -type f -exec stat -c '%d:%i %s' -- {} + |
awk '!x[$1]++ {print $2}'; } | paste -sd+ | bc | tr -d '\\\n'

or

{ find . -type f -exec stat -c '%d:%i %s' -- {} + |
awk '!x[$1]++ {print $2}' | tr '\n' +; echo 0; } | bc | tr -d '\\\n'

mentioned in merge request aports!63169 (merged)

Worth mentioning: this affects reproducible builds. Depending on the underlying filesystem the size in .PKGINFO can vary substantially.

changed the description

mentioned in merge request aports!64954 (merged)

changed the description

changed the description

makepkg (arch linux) is also using a custom script for the file size: dirsize.sh

There seems to be a lot of things to get wrong.

So this is a working script (handles hardlinks, ok speed, robust math):

Edit: flawed!

find . -type f -exec stat -c '%s
%n' -- {} + | awk '{getline p} !x[$0]++ {print p}' |
xargs cat -- | wc -c

The speed bottleneck is wc.

• TODO handle files without read access

This works for me:

find . -type f | xargs stat -c %s | awk '{ total += $1 }; END { print total }'

This fails to handle a lot of edge cases. I have added the 'current best' version to the issue description and used it in !292 (merged).

changed the description

changed the description

Currently no size calculating script I have seen is able to handle directories without read or execute permission.

Do we accept if abuild suffers from the same issue?

In what kind of situation abuild cannot read/access the package files? How did it even write them?

It is pretty unrealistic but here a small example (as regular user):

mkdir dir
touch dir/file
chmod 0 dir
find dir -type f

This results in find: dir: Permission denied. As root find dir -type f works though. The reason is that real root has the CAP_DAC_READ_SEARCH capability that bypasses the permissions but fakeroot doesn't grant capabilities.

I think I'm missing something here. How does abuild copy dir into the final package if it can't read it?

abuild doesn't move/copy into pkgdir directly. The contents are created in the package step.

But tar would also fail with such unusual directory permissions so we can safely say that this is never the case.

changed the description

changed the description