[NETLINK]: Fix two socket hashing bugs.

[NETLINK]: Fix two socket hashing bugs.

1) netlink_release() should only decrement the hash entry
   count if the socket was actually hashed.

   This was causing hash->entries to underflow, which
   resulting in all kinds of troubles.

   On 64-bit systems, this would cause the following
   conditional to erroneously trigger:

	err = -ENOMEM;
	if (BITS_PER_LONG > 32 && unlikely(hash->entries >= UINT_MAX))
		goto err;

2) netlink_autobind() needs to propagate the error return from
   netlink_insert().  Otherwise, callers will not see the error
   as they should and thus try to operate on a socket with a zero pid,
   which is very bad.

   However, it should not propagate -EBUSY.  If two threads race
   to autobind the socket, that is fine.  This is consistent with the
   autobind behavior in other protocols.

   So bug #1 above, combined with this one, resulted in hangs
   on netlink_sendmsg() calls to the rtnetlink socket.  We'd try
   to do the user sendmsg() with the socket's pid set to zero,
   later we do a socket lookup using that pid (via the value we
   stashed away in NETLINK_CB(skb).pid), but that won't give us the
   user socket, it will give us the rtnetlink socket.  So when we
   try to wake up the receive queue, we dive back into rtnetlink_rcv()
   which tries to recursively take the rtnetlink semaphore.

Thanks to Jakub Jelink for providing backtraces.  Also, thanks to
Herbert Xu for supplying debugging patches to help track this down,
and also finding a mistake in an earlier version of this fix.

Signed-off-by: David S. Miller <davem@davemloft.net>