If you wish to permanently close or delete your account, you can do so by closing your WordPress.com account. Your data on Gravatar will be deleted, so make sure this is what you want.

There is an option to disable your Gravatar account if that is more suitable.