Hi,
It is not designed for that and I don’t really support that. But I won’t hold you back if you want this.
You could install a plugin like User Role Editor. Then enable the capability gwolle_gb_upload_files for the user roles that you want to give permission.
You really do want monitoring of what is being uploaded.