[MLUG] IHOO: The International House of /opt (long)

[EMAIL:PROTECTED]

Well, the chicken-scratch I coded up yesterday seems to be working really
well, so I figure I should share it with whoever is interested.

After my post a few days ago on pervasive /opt usage and the subsequent
replies, I realized that only one strategy of environment modification
was required. I had previously thought that for some environment structures
(eg PATH, LD_LIBRARY_PATH) I could get away with adding lots of directories
to the variables (/opt/gcc/bin:/opt/jdk/bin etc), but with others I would
have to replicate a directory structure, symlinking to all the relevant
files in /opt (my example being info).

However, I later learned that I could probably add additional info dirs
using an INFODIR variable (this isn't well documented). Since this is
the only thing I thought of that would have required lots of symlinking,
it should now be possible to seamlessly integrate /opt packages by
scanning for the relevant directories and adding them to environment
variables. On the other hand, it is just as feasible to replicate
a /usr-style directory structure - with bin/, lib/, include/ etc - 
and add just those directories to whatever variables you want.

There are pros and cons to either approach.

- Adding multiple dirs to your environment variables is probably
a lighter disk load when constructing the variables, because all you
really have to do is scan for all dirs in /opt. But when using symlinks,
you have to do a 2-ply scan down to whatever files you need to symlink,
then symlink those files. These could easily number in the hundreds or
thousands.

On the other hand, the symlink technique requires vasly fewer disk
accesses to actually find files/run programs, because the search
paths are shorter (you're only adding one dir to PATH instead of
dozens).

- If you do symlinking, you vastly reduce the complexity of compiling
programs. Instead of having to explicitly add tons of directories
to your include and library paths when compiling programs
(eg -I/opt/gtkmm/include -I/opt/gnomemm/include -I/opt/libsigc++/include
-L/opt/gtkmm/lib -L/opt/gnomemm/lib -L/opt/libsigc++/lib - don't laugh,
I had to do this a couple of days ago), you only have to add one or two
(-I.opt/include -L.opt/lib or something).

- Multidir variable generation is tougher to implement dynamically.
Say you've installed a new package and you want to start using it.
Your program would have to strip out all references to /opt in
PATH, LD_LIBRARY_PATH, INFODIR, MANPATH, etc., then repopulate it.
In contrast, to regenerate a symlink system you just delete a bunch
of files. Slower, yeah, but probably easier too.

- The symlink approach requires a more complex conflict resolution
system (to choose between two files with the same name in two
different paths), but once it's implemented it is arguably
more flexible. You can choose on the fly which version of a program
you want to export as a symlink, etc.

If it's not obvious already my utility chooses the symlink approach.

The util, attached, will populate a directory structure in the .opt
directory with symlinks to relevant files in /opt. That is, .opt/bin/
contains symlinks to /opt/*/bin/*, .opt/lib contains symlinks to
/opt/*/lib/*, etc. To use this, just run it from your home directory
and add $HOME/.opt/bin to your PATH, $HOME/.opt/lib to
LD_LIBRARY_PATH, and `man -w`:$HOME/.opt/man to MANPATH (because
MANPATH overrides man.conf), etc.

It's written in and requires Python 2.1, although the only reason
it needs that new a version is because of my fetish with nested
scopes. Python programmers would have no trouble adapting it for
Python 1.5.2 or what not.

It works fine for me for every program I've run in the last 18 hours
or so. It also handles all relevant man pages, libraries, etc just
fine. I don't know how well info works though. I haven't exercised the
version choosing support (in my case, running gcc 2.95.3 instead of
3.0) but there's a place you can do that without much fuss.

Feel free to ask me questions about it, but unless you know Python
I wouldn't even bother using it. At this point you need to
hand-modify the code to change the location of the interpreter,
which directories to scan, etc. Also in its current configuration
it will remove files it thinks are in the way with extreme
prejudice. And it's not terribly robust, so it will most likely
crash if there's a root-only dir in /opt, if there's a package
in /opt with a name the util can't parse correctly, if it's a
full moon outside, etc. Consider yourself warned.

-- 
Richard  D. Tollerton II
EMAIL:PROTECTED
#!/opt/Python-2.1.1/bin/python
#
# IHOO: International House of /opt
# A utility to facilitate pervasive /opt usage for virtually all packages
#
# Version 0.1
# Rich Tollerton, EMAIL:PROTECTED
#
# This program is in the PUBLIC DOMAIN and comes with NO WARRANTY
# either explicit or implied.
#
# Usage: ihoo.py
#
# Description:
#
# Most of the file is just functions which could in the future be
# factored out into a module. The last few lines of the file contain
# most of the code which does any work.
#
# The basic idea behind the processing is, for any <dir>,
# symlink /opt/*/<dir>/* to a corresponding .opt/*/<dir>.
# The user then adds .opt to PATH, LD_LIBRARY_PATH, etc.
# There are quite a few complications to this technique (need to
# destroy/recreate symlinks every invocation, proper man page support
# requires a generalized directory search facility) but the code
# does its level best.
#
# Current limitations:
#
# - Packages under opt need to be in a specific format, like
# gtkmm-1.2.5. Version numbers must always follow package names with
# a dash (although the version is optional). Nonnumeric chars after
# the version are ignored. Nonnumeric chars in the version are NOT
# ALLOWED and could possibly crash the program. What follows the
# first dash in the name must ALWAYS be a version number, of the form
# \n+(\.\n+)*.
# - Conflict resolution between files that would symlink to the same
# location works well, but it will flake out when the files come
# from two seperate packages. (Otherwise it will choose the highest
# version of the package.) Custom rules are probably required for this
# (and there's a location in the code to place them).
# - It's pretty slow. A full rebuild takes about 2-3 seconds on a
# 850Mhz P3 with a 7200rpm hard disk and around a dozen packages.
# Figure that it will require about as much wall time as ldconfig.
# - Not generally robust, although it's not likely to delete user files.


from __future__ import nested_scopes
import string
import os
import os.path
import re

def subfiles(dirlist):
  "Returns a list of files under each directory in input list"
  return  reduce(lambda list, d: list+d,  # return a flattened list..
    map(lambda list, pre:     # composed of the files
    map(lambda s: pre+s, list),   # in each directory prepended
    map(os.listdir, dirlist), dirlist), [])# with those directories

def prependstr(list, ap):
  "Prepents a string to every string in list"
  return map(lambda s: ap + s, list)

def appendstr(list, ap):
  "Appends a string to every string in list"
  return map(lambda s: s + ap, list)

def defaultresolve(f1, f2):
  "Chooses between two files to include in a symlink dictionary."
  return f1

def gendict(base, dir, resolver):
  "Generates a dictionary of files and paths suitable for symlink manipulation."
  if base[-1] != '/':
    base = base + '/'
  #
  # Generate all files underneath base/dir/*, store in matchfiles
  #
  # first figure out how many dirs are actually in dir. In some
  # odd cases (mostly man/man1 etc) it's actually 2 dirs
  numdirs = len(filter(None, string.split(dir, '/')))
  subdirs = appendstr(filter(os.path.isdir, subfiles([base])), '/')
  # keep descending the directory structure until we have all the
  # directories we need
  while numdirs > 0:
    subdirs = appendstr(filter(os.path.isdir, subfiles(subdirs)), '/')
    numdirs = numdirs - 1
  # cull directories that don't match dir
  matchdirs = appendstr( \
    filter(lambda d: re.search(dir + '\/$', d), subdirs), '/')
  matchfiles = subfiles(matchdirs)
  #
  # Generate the symlink dictionary
  #
  dict = {}
  for file in matchfiles:
    base = os.path.basename(file)
    if dict.has_key(base):
      # If there's a conflict, use the resolver to resolve the conflict
      dict[base] = resolver(dict[base], file, dir)
    else:
      dict[base] = file
  return dict

def createlinks(home, dir, dict):
  "Instantiates home/dir/dict.key -> dict.value."
  outputdir = home + dir + '/'
  for file in dict.keys():
    try:
      os.symlink(dict[file], outputdir + file)
    except OSError, (errno, perror):
      print "Symlink of " + dict[file] + " => " + outputdir + file + \
        " failed: " + perror
    
def gendirs(home, removep, *dirs):
  "Clears and creates directories to be populated by symlinks later."
  if home[-1] != '/':
    home = home + '/'
  for dir in dirs:
    if dir[-1] != '/':
      dir = dir + '/'
    path = home + dir
    if os.path.exists(path):
      if os.path.isdir(path):
        if removep:
          map(os.remove, prependstr(os.listdir(path), path))
        else:
          print "Would remove files: " + prependstr(os.listdir(path), path)
      else:
        if removep:
          os.remove(path)
        else:
          print "Would remove file: " + path
    else:
      os.makedirs(path)

def buildver(str):
  "Splits a string into an integer tuple representing a version number."
  dotsplit = string.split(str, '.')
  return map(int, dotsplit)

def extractpkg(file, dir):
  "Returns a (name, version) tuple representing a package name and version."
  i1 = string.find(file, dir) - 1
  i2 = string.rfind(file, '/', 0, i1) + 1
  fullname = file[i2:i1]
  # split by dashes, but ignore everything past the second split
  dashsplit = string.split(fullname, '-')
  if len(dashsplit) >= 2:
    return (dashsplit[0], buildver(dashsplit[1]))
  elif len(dashsplit) == 1:
    return (dashsplit[0], [])
  else:
    print "ERROR: couldn't extract pkg/ver for file " + file
    return ("(ERROR)", [])


def resolve(f1, f2, dir):
  "The default resolver for two files that would map to the same symlink under dir."
  # Extract out a package name and a version
  p1, v1 = extractpkg(f1, dir)
  p2, v2 = extractpkg(f2, dir)
  #
  # (user-defined rules would go here)
  #
  # If they're from two different packages, choose one arbitrarily
  if p1 != p2:
    return f1		# choose arbitrarily
  # Otherwise choose the one with a higher version number
  else:
    if v1 > v2:
      return f1
    else:
      return f2

OPTDIR = '/opt/'
HOMEDIR = '.opt/'

DIRS = ('bin', 'include', 'info', 'lib', 'man/man1', 'man/man2', 'man/man3',
  'man/man4', 'man/man5', 'man/man6', 'man/man7', 'man/man8', 'man/man9')

map(lambda d: gendirs(HOMEDIR, 1, d), DIRS)
dicts = map(lambda d: gendict(OPTDIR, d, resolve), DIRS)
map(lambda d, dict: createlinks(HOMEDIR, d, dict), DIRS, dicts)