Yeah, sounds like a Solaris tmpfs kernel bug.

Please provide a sample program to demonstrate it.

This is a tmpfs bug. But it seems like checking whether something is a dir and rmdir(ing) it instead of unlink is a good practice. There are a lot of filesystems out there, and POSIX spec clearly says that it is not necessarily supported everywhere. There is already work my team is doing to address this in Illumos, but it will still likely be a problem in Solaris, and in earlier versions which are still in active use, but are not really maintained, like OpenSolaris, and OpenIndiana.

As for sample program, I am assuming you would like that to be a playground link?

Thanks for considering this!

This is a tmpfs bug. But it seems like checking whether something is a dir and rmdir(ing) it instead of unlink is a good practice.

Glad we helped find a bug! :) From memory, this seems like about the 9th kernel bug Go has accidentally brought to light in various operating systems.

As for sample program, I am assuming you would like that to be a playground link?

Or a gist. Whatever you'd like. Something I can click and read online, even if it doesn't run.

Bugs exposed in indirect ways like these are always awesome, and certainly appreciated. :)

Here's that link: https://play.golang.org/p/u8fVnADfsN Program is naive, but should communicate intent. Thanks again!

Solaris does not support using unlink() on directories on ZFS or tmpfs, Solaris 10 and newer (at least) will fail with EPERM if you attempt this either using a ZFS or tmpfs system, so the tmpfs bug appears to be specific to Illumos.

Using the sample program, we can see this if we truss it on any version of Solaris 10+:

/1: unlink("/tmp/tdir") Err#1 EPERM
/1: lwp_unpark(4) = 0
/4: lwp_park(0x00000000, 0) = 0
/1: rmdir("/tmp/tdir") Err#17 EEXIST
...
/1: unlink("/tmp/tdir/testfile.txt") = 0
...
/1: unlink("/tmp/tdir") Err#1 EPERM
/1: lwp_unpark(4) = 0
/4: lwp_park(0x00000000, 0) = 0
/1: rmdir("/tmp/tdir") = 0
/1: _exit(0)

On Solaris, go should always use rmdir() instead to not only avoid the unnecessary overhead of trying and failing repeatedly, but the other undesirable semantics if it is allowed:

   If the path  argument  is  a  directory  and  the  filesystem  supports
   unlink()  and unlinkat() on directories, the directory is unlinked from
   its parent with **no cleanup being performed**.  In UFS,  the  disconnected
   directory  will  be  found the next time the filesystem is checked with
   fsck(1M). The unlink() and unlinkat() functions will  not  fail  simply
   because  a directory is not empty. The user with appropriate privileges
   can orphan a non-empty directory without generating an error message.

Note the "no cleanup" bit above.

In short, Go should not generally be using unlink() to remove directories, at least on Solaris, since it's unlikely to ever work as expected.

OpenSolaris-based derivatives may behave differently of course.

The source explains why Go does what it does:

func Remove(name string) error {
        // System call interface forces us to know                                                                                        
        // whether name is a file or directory.                                                                                           
        // Try both: it is cheaper on average than                                                                                        
        // doing a Stat plus the right one.                                                                                               
        e := syscall.Unlink(name)
        if e == nil {
                return nil
        }
        e1 := syscall.Rmdir(name)
        if e1 == nil {
                return nil
        }

        // Both failed: figure out which error to return.                                                                                 
        // OS X and Linux differ on whether unlink(dir)                                                                                   
        // returns EISDIR, so can't use that. However,                                                                                    
        // both agree that rmdir(file) returns ENOTDIR,                                                                                   
        // so we can use that to decide which error is real.                                                                              
        // Rmdir might also return ENOTDIR if given a bad                                                                                 
        // file path, like /etc/passwd/foo, but in that case,                                                                             
        // both errors will be ENOTDIR, so it's okay to                                                                                   
        // use the error from unlink.                                                                                                     
        if e1 != syscall.ENOTDIR {
                e = e1
        }
        return &PathError{"remove", name, e}

As long as Solaris returns an error on Unlink of a directory and doesn't like catch on fire or something, I think we're good here.