Ykhjuy

On 19 Aug 2013, Randal L. Schwartz posted this shell script, which was intended to ensure, on Linux, "that only one instance of [the] script is running, without race conditions or having to clean up lock files":

#!/bin/sh

# randal_l_schwartz_001.sh

(

    if ! flock -n -x 0

    then

        echo "$$ cannot get flock"

        exit 0

    fi

    echo "$$ start"

    sleep 10 # for testing.  put the real task here

    echo "$$ end"

) < $0

It seems to work as advertised:

$ ./randal_l_schwartz_001.sh & ./randal_l_schwartz_001.sh

[1] 11863

11863 start

11864 cannot get flock

$ 11863 end



[1]+  Done                    ./randal_l_schwartz_001.sh

$

Here is what I do understand:

• The script redirects (<) a copy of its own contents (i.e. from $0) to the STDIN (i.e. file descriptor 0) of a subshell.


• Within the subshell, the script attempts to get a non-blocking, exclusive lock (flock -n -x) on file descriptor 0.
  
  
  
  
  • If that attempt fails, the subshell exits (and so does the main script, as there is nothing else for it to do).
  
  
  • If the attempt instead succeeds, the subshell runs the desired task.
  
  
  
  

Here are my questions:

• Why does the script need to redirect, to a file descriptor inherited by the subshell, a copy of its own contents rather than, say, the contents of some other file? (I tried redirecting from a different file and re-running as above, and the execution order changed: the non-backgrounded task gained the lock before the background one. So, maybe using the file's own contents avoids race conditions; but how?)


• Why does the script need to redirect, to a file descriptor inherited by the subshell, a copy of a file's contents, anyway?


• Why does holding an exclusive lock on file descriptor 0 in one shell prevent a copy of the same script, running in a different shell, from getting an exclusive lock on file descriptor 0? Don't shells have their own, separate copies of the standard file descriptors (0, 1, and 2, i.e. STDIN, STDOUT, and STDERR)?

Why does the script need to redirect, to a file descriptor inherited by the subshell, a copy of its own contents rather than, say, the contents of some other file?

You could use any file, as long as all copies of the script use the same one.
Using $0 just ties the lock to the script itself: If you copy the script and modify it for some other use, you don't need to come up with a new name for the lock file. This is convenient.

If the script is called through a symlink, the lock is on the actual file, and not the link.

_{(Of course, if some process runs the script and gives it a made up value as the zeroth argument instead of the actual path, then this breaks. But that's rarely done.)}

(I tried using a different file and re-running as above, and the execution order changed)

Are you sure that was because of the file used, and not just random variation? As with a pipeline, there's really no way to be sure in what order the commands get to run in cmd1 & cmd. It's mostly up to the OS scheduler. I get random variation on my system.

Why does the script need to redirect, to a file descriptor inherited by the subshell, a copy of a file's contents, anyway?

It looks like that's so that the shell itself holds a copy of the file description holding the lock, instead of just the flock utility holding it. A lock made with flock(2) is released when the file descriptors having it are closed.

flock has two modes, either to take a lock based on a file name, and run an external command (in which case flock holds the required open file descriptor), or to take a file descriptor from the outside, so an outside process is responsible for holding it.

Note that the contents of the file are not relevant here, and there are no copies made. The redirection to the subshell doesn't copy any data around in itself, it just opens a handle to the file.

Why does holding an exclusive lock on file descriptor 0 in one shell prevent a copy of the same script, running in a different shell, from getting an exclusive lock on file descriptor 0? Don't shells have their own, separate copies of the standard file descriptors (0, 1, and 2, i.e. STDIN, STDOUT, and STDERR)?

Yes, but the lock is on the file, not the file descriptor. Only one opened instance of the file can hold the lock at a time.

I think you should be able to do the same without the subshell, by using exec to open a handle to the lock file:

$ cat lock.sh

#!/bin/sh



exec 9< "$0"



if ! flock -n -x 9; then

    echo "$$/$1 cannot get flock" 

    exit 0

fi



echo "$$/$1 got the lock"

sleep 2

echo "$$/$1 exit"



$ ./lock.sh bg & ./lock.sh fg ; wait; echo

[1] 11362

11363/fg got the lock

11362/bg cannot get flock

11363/fg exit

[1]+  Done                    ./lock.sh bg

The file used for locking is unimportant, the script uses $0 because that's a file that is known to exist.

The order in which the locks are obtained will be more or less random, depending on how fast your machine is able to start the two tasks.

You may use any file descriptor, not necessarily 0. The lock is held on the file opened to the file descriptor, not the descriptor itself.

( flock -x 9 || exit 1

  echo 'Locking for 5 secs'; sleep 5; echo 'Done' ) 9>/tmp/lock &

A file lock is attached to a file, through a file description. At a high level, the sequence of operations in one instance of the script is:

1. Open the file to which the lock is attached (“the lock file”).


2. Take a lock on the lock file.


3. Do stuff.


4. Close the lock file. This releases the lock that is attached to the file description created by opening a file.

Holding the lock prevents another copy of the same script for running because that's what locks do. As long as an exclusive lock on a file exists somewhere on the system, it's impossible to create a second instance of the same lock, even through a different file description.

Opening a file creates a file description. This is a kernel object that doesn't have much direct visibility in programming interfaces. You access a file description indirectly via file descriptors, but normally you think of it as accessing the file (reading or writing its content or metadata). A lock is one of the attributes that are a property to the file description rather than a file or to a descriptor.

At the beginning, when a file is opened, the file description has a single file descriptor, but more descriptors can be created either by creating another descriptor (the dup family of system calls) or by forking a subprocess (after which both the parent and the child have access to the same file description). A file descriptor can be closed explicitly or when the process that it's in dies. When the last file descriptor attached to a file is closed, the file description is closed.

Here's how the sequence of operations above affects the file description.

1. The redirection <$0 opens the script file in the subshell, creating a file description. At this point there is a single file descriptor attached to the description: descriptor number 0 in the subshell.


2. The subshell invokes flock and waits for it to exit. While flock is running, there are two descriptors attached to the description: number 0 in the subshell and number 0 in the flock process. When flock takes the lock, that sets a property of the file description. If another file description already has a lock on the file, flock cannot take the lock, since it's an exclusive lock.


3. The subshell does stuff. Since it still has an open file descriptor on the description with the lock, that description keeps existing, and it keeps its lock since nobody ever removes the lock.


4. The subshell dies at the closing parenthesis. This closes the last file descriptor on the file description that has the lock, so the lock disappears at this point.

The reason the script uses a redirection from $0 is that redirection is the only way to open a file in the shell, and keeping a redirection active is the only way to keep a file descriptor open. The subshell never reads from its standard input, it just needs to keep it open. In a language that gives direct access to open and close call, you could use

fd = open($0)

flock(fd, LOCK_EX)

do stuff

close(fd)

You can actually get the same sequence of operations in the shell if you do the redirection with the exec builtin.

exec <$0

flock -n -x 0

# do stuff

exec <&-

The script could use a different file descriptor if it wanted to keep accessing the original standard input.

exec 3<$0

flock -n -x 0

# do stuff

exec 3<&-

or with a subshell:

(

  flock -n -x 3

  # do stuff

) 3<$0

The lock doesn't have to be on the script file. It could be on any file that can be opened for reading (so it has to exist, it has to be a file type that can be read such as a regular file or a named pipe but not a directory, and the script process must have the permission to read it). The script file has the advantage that it's guaranteed to be present and readable (except in the edge case where it was deleted externally between the time the script was invoked and the time the script gets to the <$0 redirection).

As long as flock succeeds, and the script is on a filesystem where locks aren't buggy (some network filesystems such as NFS may be buggy), I don't see how using a different lock file could allow a race condition. I suspect a manipulation error on your part.