Some additional notes.
When a chain of commands don’t work, check the output of each command to see if everything is as expected. Then you can reslize yourself you used the wrong number as @meyay already pointed out.
The error message is also descriptive if you know “library” is the default namespace (in other words: organization or owner) of the official Docker images on Docker Hub, which is used when the owner is not set. So when the error messages complains about repository name, the command handles the input as repository name. Since you had a docker rmi command, that reveals you passed the image size to the docker rmi command instead of the image name.
The second error message is trickier and happened because you didn’t use code blocks for the quoted first command, only for the error message, so the forum changed the quotation marks to characters used in printed documents, not in a code. So be careful with copy-pasting 
By the way, if you don’t want to accidentally delete an image called for example "yourregistry/nextcloud-helper`, which has nothing to do with the nextcloud-releases, you can also make the command more specific and simpler at the same time:
docker images -q 'ghcr.io/nextcloud-releases/*' | xargs -- docker rmi
The first part:
docker images -q 'ghcr.io/nextcloud-releases/*'
Lists the short image ID of all images in the ghcr.io registry under the nextcloud-releases namespace. The second part after the pipe character takes all returned image IDs and executes docker rmi IMAGEID, where IMAGEID is one image ID from the list of image IDs returned by the first part.
You could be this specific with grep as well, but that would just make the command more complex not simpler with regular expressions.
update:
Or you could also use labels and delete the images created by a specific vendor if the image is labelled properly
docker images -q --filter "label=org.opencontainers.image.vendor=Nextcloud"
So deleting would be:
docker images -q --filter "label=org.opencontainers.image.vendor=Nextcloud" | xargs -- docker rmi
But it only works as long as you don’t have multiple tags for the same image ID, so you might want to use one of these commands for listing the images:
docker images --filter "label=org.opencontainers.image.vendor=Nextcloud" --format '{{ .Repository }}:{{ .Tag }}'
or
docker images 'ghcr.io/nextcloud-releases/*' --format '{{ .Repository }}:{{ .Tag }}'
Then you would pass image tags to the docker rmi command and it would not complain about trying to delete an image by ID when multiple image tags refer to it.