[BUG] staging: android: ashmem: Deadlock during ashmem_shrink

[Shankar Brahadeeswaran]

Hi Robert,
Thanks for the feedback.

> This doesn't look insurmountable. It isn't necessary AFAICT to hold
> ashmem_mutex across shmem_file_setup.

I thought it may not be a good idea to do so and hence thought its
difficult to fix.
Dropping the lock in-between mmap may not be any issue if the user land code
follows a given sequence. But assuming that the following sequence of event
happens, it would lead to other races.

Process P1                                          Process P2
--------------                                           --------------
Creates ashmem region                       .....

Shares the fd to P2 via binder             Gets the fd

Does an mmap                                     Does an mmap

Releases the mutex before                  Procees with ashmem_mmap
since mutex is
shmem_file_setup and sleeps             available, checks for a
asma->file, still NUL
within shmem_file_setup                      so this also calls
shmem_file_setup.

The expected behavior is, one of them does the shmem_file_setup, puts
it in asma->file
The other process would just do get_file. With the original code
(without dropping the
mutex in-between) this would have been the behavior.
So IMHO dropping the lock in between could lead to other race conditions.

Also, there are other places in the code where ashmem_mutex is held and memory
allocation functions are called, ex:- range_alloc, calls kmem_cache_zalloc

Since ashmem_shrink holds the ashmem_mutex, any where from ashmem driver
if  a memory allocation function is called with the ashmem_mutex held
&&
if there is a low memory condition that leads to shrinkers being called
we'll hit the deadlock.

I'm trying to see if the ashmem_shrink should really hold the ashmem_mutex,
but looks like its necessary.

Warm Regards,
Shankar

On Tue, Apr 23, 2013 at 10:54 PM, Robert Love <rlove at google.com> wrote:
> On Tue, Apr 23, 2013 at 12:20 PM, Shankar Brahadeeswaran
> <shankoo77 at gmail.com> wrote:
>
>> I'm unable to think of a straight forward way to fix this. If you have
>> any suggestions please provide the same.
>> If we are unable to solve this too with minor mods, as suggested by
>> Dan we have to re-look at the locking in this driver.
>
> This doesn't look insurmountable. It isn't necessary AFAICT to hold
> ashmem_mutex across shmem_file_setup.
>
> Patch attached (untested).
>
>            Robert