Return the right error value when dup[23]() newfd argument is too large

Jack Lin reports that the error return from dup3() for the RLIMIT_NOFILE
case changed incorrectly after 3.6.

The culprit is commit f33ff9927f42 ("take rlimit check to callers of
expand_files()") which when it moved the "return -EMFILE" out to the
caller, didn't notice that the dup3() had special code to turn the
EMFILE return into EBADF.

The replace_fd() helper that got added later then inherited the bug too.

Reported-by: Jack Lin <linliangjie@huawei.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
[ Noted more bugs, wrote proper changelog, fixed up typos - Linus ]
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

diff --git a/fs/file.c b/fs/file.c
index d3b5fa80b71b76a973919a6b9cfa2ebb34096adf..708d997a77485989d53583084a3e1b99354fb407 100644 (file)
--- a/fs/file.c
+++ b/fs/file.c
@@ -900,7 +900,7 @@ int replace_fd(unsigned fd, struct file *file, unsigned flags)
                return __close_fd(files, fd);
 
        if (fd >= rlimit(RLIMIT_NOFILE))
-               return -EMFILE;
+               return -EBADF;
 
        spin_lock(&files->file_lock);
        err = expand_files(files, fd);
@@ -926,7 +926,7 @@ SYSCALL_DEFINE3(dup3, unsigned int, oldfd, unsigned int, newfd, int, flags)
                return -EINVAL;
 
        if (newfd >= rlimit(RLIMIT_NOFILE))
-               return -EMFILE;
+               return -EBADF;
 
        spin_lock(&files->file_lock);
        err = expand_files(files, newfd);