You can uninstall PGroonga by the following SQL:
DROP EXTENSION pgroonga CASCADE;
If you're using PostgreSQL 9.5 or earlier, you need to also run the following SQL:
DELETE FROM pg_catalog.pg_am WHERE amname = 'pgroonga';