main/u-boot: fix ramdisk issues on Hifive Unmatched

This issue was reproduced independently by @clandmeter and myself. We are both using this patch, it has been submitted to the u-boot ML and others have agreed that this is a good change [1].