There is really no reason not to make them faster, as right now we have to either just wait or punch them out one by one with our hand which just feels like chore to me. The decay speed should be increased, not too fast, but at least faster than it is currently. I believe this won't really mess with balance of the game either.
p.s. This brings up another suggestion: there are no real tools for leaves. We do have shears, but they have silk touch built into it, making them useless in this case. I'd suggest that axes should take the job, but that's for another post.