Ticket #1 (assigned defect)

Opened 2 years ago

Last modified 2 years ago

lockfile for ShelveSessionStore ?

Reported by: jcigar@ulb.ac.be Assigned to: asaddi (accepted)
Priority: trivial Milestone:
Component: flup.middleware Version:
Keywords: Cc:

Description

Hello Alan,

I was just wondering why you didn't use a .lock file for the ShelveSessionStore? (as you did with DiskSessionStore?) to resolve the concurrently accesses ?

For example I did a small FileLock? class with a decorator :

class FileLock(object):

        
    def __init__(self, lock_file):
        self._lock_file = lock_file


    def acquireLock(self, timeout=LOCK_TIMEOUT):

        for i in xrange(1, timeout*10):
            try:
                os.open(self._lock_file, os.O_WRONLY | os.O_CREAT | os.O_EXCL,
                        0660)
                return True
            except:
                time.sleep(0.1)
                continue

        try:
            self.isStale() and self.releaseLock()
        except:
            pass

        return False

    
    def isStale(self):
        try:
            return os.stat(self._lock_file).st_mtime < time.time() - 3600
        except OSError:
            return False


    def releaseLock(self):
        os.path.isfile(self._lock_file) and os.unlink(self._lock_file)


    def withLock(self, fct):
        if self.acquireLock():
            try:
                return fct()
            finally:
                self.releaseLock()
        else:
            raise IOError('Cannot acquire lock')


def with_lock(fct):
    return lambda self: self.lock.withLock(lambda: fct(self))


and you can do something like :

class ShelveSession(BaseSession):
    
    def __init__(...):
        super(ShelveSession, self).__init__(...)
      
        lock_file = os.path.join(os.path.dirname(self.session_file), '.lock')
        self.lock = FileLock(lock_file)
        
        # more stuff here

    @with_lock
    def save(self):
        f = shelve.open(self.session_file, 'c')
        f[self.sid] = self.data
        f.close()


    @with_lock
    def load(self):
        f = shelve.open(self.session_file, 'r')
        self.data = f.get(self.sid, None)
        f.close()
    

    @with_lock
    def delete(self):
        f = shelve.open(self.session_file, 'w')

        try:
            del f[self.sid]
        except KeyError:
            pass
        
        f.close()
        self.clear()

Any reason why to not use a lockfile with a shelve ?

Thanks !

Attachments

locking-shelves.diff (4.0 kB) - added by asaddi on 12/17/06 12:14:40.
LockingShelveSessionStore? patch. Complete, but with issues (see comment)

Change History

11/26/06 17:55:40 changed by asaddi

  • status changed from new to assigned.

I think what dissuaded me from doing file locking with shelves was this note in the shelve module docs:

The shelve module does not support concurrent read/write access to shelved objects. (Multiple simultaneous read accesses are safe.) When a program has a shelf open for writing, no other program should have it open for reading or writing. Unix file locking can be used to solve this, but this differs across Unix versions and requires knowledge about the database implementation used.

However, I see within your implementation that you open and close the shelve within each method. I'm curious if you've tried this out? The only thing I'm not so sure about is the cost of opening a shelf with every session store operation.

I'll look into implementing your session store into a new shelve-based store. (LockingShelveSessionStore?)

Thanks!

11/27/06 02:07:08 changed by jcigar@ulb.ac.be

indeed, the shelve module doesn't support concurrent read/write .. and yes, it's why I open/close the shelve within each method :). I haven't tested it "deeply" (I mean with 50 concurrent accesses), but I don't think that the cost of opening/closing a shelf with every session store operation is expensive (it takes less than 0.1ms for the little of test which I made).

For example with:

from session import ShelveSession
for i in xrange(1, 100):
    a=ShelveSession()
    a['foo'] = 'foovalue'
    a['bar'] = 'barvalue'
    a['lol'] = dict(name='julien', age=25)
    a.save()

$> time ./test.py

real 0m0.588s user 0m0.433s sys 0m0.131s

So I think it's quite fast :)

11/27/06 02:14:13 changed by jcigar@ulb.ac.be

err, s/0.1ms/0.1s

12/17/06 12:13:37 changed by asaddi

I tried incorporating this today, and for the most part, I was successful. But I discovered a flaw: it's still not really multi-process safe. Since the locking is only at the shelve level, it's possible for 2 or more different processes to check out (and then save) the same session. Last write wins.

DiskSessionStore gets around this by locking each session individually (but unfortunately, it makes no distinction between read-only and read-write locks).

Maybe this isn't really a problem for most web developers? I'm not sure. Anyway, I'm attaching the current state of the patch.

12/17/06 12:14:40 changed by asaddi

  • attachment locking-shelves.diff added.

LockingShelveSessionStore? patch. Complete, but with issues (see comment)

12/18/06 14:39:44 changed by anonymous

Hello again :)

You're right, it's not totally multi-process safe, but what I wanted to resolve are the concurrent accesses to the shelve file himself. In fact I think you'll have the same problem with every session engine (as the operations are non-atomic), even with a database as backend (or you have to lock the session during the whole request, which is not .. optimal :)), so I don't think it's a problem (..?)

Also, a better code for acquireLock could be:

    def acquireLock(self, timeout=LOCK_TIMEOUT):
        deadline = time.time() + timeout

        while time.time() < deadline:
            try:
                os.open(self._lock_file, os.O_WRONLY | os.O_CREAT | os.O_EXCL,
                        0660)
                return True
            except (OSError, e):
                if e[0] != errno.EEXIST: 
                    raise
            
            time.sleep(0.1)

        try:
            if self.isStale():
                self.releaseLock()
        except:
            pass

        return False

oh, and it's Julien and not Julian :-)

Regards

12/18/06 14:57:06 changed by jcigar@ulb.ac.be

woops ... it's "except OSError, e:" in place of "except (OSError, e):" of course ...