Should try to fulfill my fantasy of having sex with an older man?

Go for it. You only live once. I'm sure there will be plenty of older guys who would love to fulfill your fantasy