The reason this feature is there is that many extensions play poorly with or should not have access at all to specific sites. For example, Grammarly has no business accessing a corporate billing system and IT might want to disable access. In other cases an extension slows a site massively and a user might want to say, "not that site" to an extension. This could be the start of a big end-user improvement, especially if it can be managed by users and installers.