[PATCH 3/3] staging: comedi: serial2002: use proper errno

[Hartley Sweeten]

On Monday, August 17, 2015 4:52 AM, Ian Abbott wrote:
> On 15/08/15 01:31, H Hartley Sweeten wrote:
>> checkpatch.pl reports:
>> WARNING: ENOSYS means 'invalid syscall nr' and nothing else
>>
>> Change the error code to -ENOIOCTLCMD/
>>
>> Signed-off-by: H Hartley Sweeten <hsweeten at visionengravers.com>
>> Cc: Ian Abbott <abbotti at mev.co.uk>
>> Cc: Greg Kroah-Hartman <gregkh at linuxfoundation.org>
>> ---
>>   drivers/staging/comedi/drivers/serial2002.c | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/drivers/staging/comedi/drivers/serial2002.c b/drivers/staging/comedi/drivers/serial2002.c
>> index c6f9855..60d7d73 100644
>> --- a/drivers/staging/comedi/drivers/serial2002.c
>> +++ b/drivers/staging/comedi/drivers/serial2002.c
>> @@ -101,7 +101,7 @@ static long serial2002_tty_ioctl(struct file *f, unsigned op,
>>   	if (f->f_op->unlocked_ioctl)
>>   		return f->f_op->unlocked_ioctl(f, op, param);
>>
>> -	return -ENOSYS;
>> +	return -ENOIOCTLCMD;
>>   }
> 
> I think ENOIOCTLCMD means something like "try some other ioctl handler" 
> and should get converted to ENOTTY by the code higher up the return path 
> if there is no other ioctl handler.  Still, it might be safer to return 
> -ENOTTY directly here.  Not that it really matters, as the return value 
> from this function is currently ignored.

There are not a lot of f_op->unlocked_ioctl() calls in the kernel. Looking them
over it appears that either -ENOIOCTLCMD or -ENOTTY is returned when the
callback is not setup.

But, in fs/ioctl.c there is this sequence:

	int error = -ENOTTY;

	if (!filp->f_op->unlocked_ioctl)
		goto out;

	error = filp->f_op->unlocked_ioctl(filp, cmd, arg);
	if (error == -ENOIOCTLCMD)
		error = -ENOTTY;
out:
	return error;

So it "appears" that -ENOTTY would be a better errno.

Also, I just noticed that comedi_compat32.c also returns -ENOTTY from
translated_ioctl() if the unlocked_ioctl callback is not setup.

I'll redo this patch and repost.

Thanks,
Hartley