You say that as if it is a small, fiddling detail we can ignore in most cases.

I did not mean it that way.

I just don't see how increasing entropy by itself would be sufficient to make it any harder to predict the future from the present than to predict the past from the present. If anything, I'd think it would be easier to predict the future than to figure out the past: what will this piece of matter look like as it wears away? vs. how were these particles arranged before they wore down? Or... these two containers are about the same temperature right now... what temperature did they start at?