Recently I find a new user is created on my 3.6.0 site and put into "administrator" group, but the "Allow User Registration" option is set to "NO", though the "enabled" and "Activated" status are both unchecked.
I upgraded Joomla to the latest 3.6.4, then I tried to delete the user, but I was lead to a 404 page saying the page isn't working.
I know Joomla's user registration system is often used to inject malicious files so I turn it off. How did this hacker manage to get by it? And does this mean he some how has compromised my super user account?
There are many possible ways that the hacker has broken into your web,
I recommend you see these documents:
As to your question I would bet that the hacker could somehow upload a file to your website with a script that creates the user directly into the database.
With knowledge of Joomla tables and function it is relatively simple to do.
The main reason for 3.6.4 update was exactly what happened to you!
Security Bulletin  - Core - Account Creation states
"Inadequate checks allows for users to register on a site when registration has been disabled."
Then, additional vulnerability which was fixed by 3.6.4 was  - Core - Elevated Privileges "Incorrect use of unfiltered data allows for users to register on a site with elevated privileges."
So, it is crucial to update your Joomlas to 3.6.4 ASAP!