and in any case, the point of the Yom Kippur fast isn't to give something up (at least in my understanding) it's more of a "You should be praying all day, not eating" It's not like lent.

The proper thing to say for Yom Kippur is "May you have an easy fast." the point isn't to make it as difficult as possible...