
I did some serious bug fixing and minor hacking to get pmake 2.1 to
run as reliably as possible in the computing environment at ICSI
(60+ Sparc machines, some old Sun3's, a NeXT, and a small cluster of 
DECstations).  Recently there has also been extensive work porting to
HP Snakes running HP-UX and SGI Iris.

Platforms pmake/customs is known to run on (not necessarily tested for
the most current version):

	Sun3, SunOS 3.5 and 4.1.1
	Sun4, SunOS 4.1.1, 4.1.3, 4.1.4, 5.3, 5.4, 5.5, 5.6, 5.7
	i386-PC, SunOS 5.5.1, 5.6
	DECstation, Ultrix 4.0 and 4.3
	NeXT, Mach Release 1.0 and 2.1
	HP9000/700 (Snake), HP-UX 8.05, 8.07, and 9.01
	SGI Iris, IRIX 4.0.1, 4.0.5F, 4.0.5H, 5.1.1, 5.2, 5.3, 6.2
	DEC Alpha, OSF/1 V1.3, V2.0
	MIPS M2000, RISC/os 4.52 (pmake only)
	BSDI BSD/386 1.1
	i686-PC, Redhat Linux 2.0.32, 2.2.10

Major additions and changes include

o	In a typical NFS environment using an automounter, path names
	will not be identical all over the net, but one can set things up
	so everybody can access the same file using a `standard' path.
	customslib.c now has a function Customs_NormPath() that
	normalizes a pathname prior to exportation.  This will have to
	be customized to the local environment.

o	The availability index computation was changed to be more realistic.
	Rather than combining all criteria into a magic number, we are only
	interested in the load and the relative speed of the machine as
	long as all other criteria are within their allowed ranges.
	The index is simply the load that is `left' (max - current),
	times the relative speed (say MIPS), divided by the number of
	currently imported jobs.

o	The os-dependent code was extended to support the new machines.
	It is now more robust in case some of the information can not be
	obtained (e.g., when debugging as non-root).
	Importquota shows only the criteria actually available to the system.

o	Several problems with byte-ordering surfaced as I ported the system
	to the DEC environment.  These are believed to be all fixed now.

o	Code cleaned up in several places. Avoid conflicts with system
	header files and library functions. Exit with proper status.
	Make sure event handler functions get all the arguments they are
	supposed to. Customs and its clients (except pmake) now use
	xmalloc() instead of malloc() to avoid bad things in case
	virtual memory runs out.  After all, customs runs as root so we better
	make sure there are no nasty surprises.

o	Pmake exit handling (lock file removal) didn't work on non-Sun
	systems.

o	A bad bug in the RPC system caused timeouts on long messages sent via
	TCP.

o	A work-around that really shouldn't be necessary to avoid losing stderr
	output from remote jobs on non-Sun systems.

o	Logd didn't work at all because of an incompatible data structure for
	the information given out by the MCAInfo RPC.

o	Reginfo got a nicer output format.

o	Makefiles were improved (but are still dependent on the compilation
	and installation environment).  Compiler and options are now selected
	to be `best' for each machine.

o	Gratuitous incompatibilties with standard makes were removed
	(see patchlevel 12 below).

o	The notion of host architecture has been generalized to arbitrary
	"host attribute" strings and can be used to describe host resources.
	Both export and pmake support restricting the choice of remote host
	using combinations of attributes.  For more details see the man pages,
	and the etc/ATTRIBUTES file for an overview.

Details of all these changes can be found below.

Andreas Stolcke
stolcke@icsi.berkeley.edu
7/31/91


*** Patchlevel 4 ***

o	customs.conf script to generate startup flags automatically.
	Very site and machine specific, though.

o	Guard against process table overflow on target host using a new
	availability parameter `minimum no. of free process slots'.

o	Propagate process priorities and resource limits to exported job.

o	Implement CPU time limits for imported jobs.

o	Fixed SIGINT and SIGQUIT passing to remote jobs.

o	Allow remote killing of all imported jobs (cctrl -kill).

o	Check passwd entry before allowing job imports.

A. Stolcke
2/18/92


*** Patchlevel 6 ***

o	-nice options for customs and importquota to run all imported jobs
	at lower priority.

o	The pmake.1 man page is now generated with Sprite man macros inserted
	automatically.

o	-idle and -cpu flags now allow time specs greater than 60 minutes.

o	New -help, -quiet and -default options for importquota.

o	Made malloc() used by liblst more secure (like xmalloc()).

o	Found and fixed one tricky source of core dumps in pmake.

o	Fixed kernel symbol names so os-bsd.c works on HP/UX.

o	Pmake support for Common Archive Format (e.g., HP/UX) added
	(see AR_COMMON in unix/config.h).

o	Pmake interprets negative -L arguments as preference for local
	execution.  Can be used to force a certain number of local jobs.

o	Fixed handling of SIGTTIN/SIGTTOU in export.  

o	Support active jobs eviction added (-evict flag).

o	Improved tracking of job status (stopped, signalled, continued)
	in pmake, including remote jobs.

o	Fixed a bug with bogus host handling in customs that could cause
	reginfo to core dump.

o	Fake a non-zero load value if load cannot be obtained from OS.
	Results in better spread of allocated hosts.

o	Can now deliver uncatchable signals (SIGCONT, SIGKILL) to remote
	jobs without killing customs child daemon (so we get status
	change reported back).

o	Allow for tighter security by restricting critical RPCs to 
	reserved ports (see remarks in SECURITY file).

A. Stolcke, 4/11/92


*** Patchlevel 7 ***

o	Added Customs_Port() to customslib for gnumake.

o	Use caller address to authenticate a Import_Kill.  This allows
	export in slave mode to send signals.

o	Make Customs_NormPath() a noop if ORIG_CWD is defined (so
	can be called regardless of local ORIG_CWD setting).

o	Include patch for GNU make 3.62 Customs support.  Also link
	export statically on Suns to speed things up a bit.

o	Use OPTIMIZE macro to customize CFLAGS in Makefiles (see config.mk).

o	Avoid problems with desynchronized clocks by transmitting elapsed
	job run time instead of start time to cctrl -jobs.

o	Fixed some inaccuracies and omissions in the INSTALL instructions.

A. Stolcke, 4/21/92


*** Patchlevel 8 ***

o	All customs error logging done via xlog(). Allows routing to
	file or syslog(3).

A. Stolcke, 7/1/92


*** Patchlevel 9 ***

NOTE: The first four fixes below address various problems that have caused
seemingly random failures, hanging, and core dumps of PMake under
typically heavy load.  Taken together, they have significantly increased
the robustness of Pmake in my experience.

o	PMake: Fixed some bugs with tracking of remote jobs: queue corruption
	could lead to core dumps, "exits from unknown jobs".
	Also added new queue debugging mode (-d q) to help with future trouble.

o	Customs: Doublecheck import timeouts on the server side to prevent
	duplicate job creation and other bad things.  Cf. DOUBLECHECK_TIMEOUT
	in config.h.
	NOTE: This new feature uses an incompatible change in the import
	protocol.  Both customs and client programs have to be reinstalled
	and restarted.

o	PMake: Check that pipe() doesn't run out of file descriptors.
	Can result in major screwup in RPC module when handling remote
	streams!

o	Fix fileno() macro to prevent lossage with file numbers > 127.
	Cf. redefinition at the bottom of unix/config.h.

o	PMake: Fixed a bug with suid-ness that would initialize the
	directory cache before reverting to user's uid.
    
o	Customs: Cleaned up some of the syslogging related to imports.
	More info on security breaches. Log common, unavoidable errors
	as `warnings'.

o	Customs: For ps(1)'s benefit, set process title of child daemon to
	job id.

o	Customs: Detect and warn about reginfo exceeding max RPC size.

o	RPC: Check socket arguments for sanity.

A. Stolcke, 7/23/92


*** Patchlevel 10 ***

o	Customs: log evictions.

o	Customs: kill orphans in process group of imported job.  Also
	counteracts deliberate evasion of the eviction mechanism.

o	Reginfo: allow hostname specification other than localhost, so
	other customs networks can be monitored remotely.

o	PMake: allow runtime control over rechecking strategy (new -R
	option).  Be smart and recheck only locally created targets by
	default.

o	PMake: enable -x option (used to be ignored, even in Make mode
	where it is useful).

o	PMake: prevent endless loops when -L 0 is specified and nothing
	can be exported.

A. Stolcke, 7/30/92


*** Patchlevel 11 ***

o	RPC: make sure Rpc_Return() byte swaps its own data, not the
	caller's.  This could mess up customs on certain RPC calls
	from machines with different byte order.

o	Customs: catch and log termination signals.

o	Cctrl: new -port options to select non-default port/service.

A. Stolcke, 7/31/92


*** Patchlevel 12 ***

o	Reginfo: less redundant, columnized output.

o	Customs: link with -n to free up binary on diskless clients.

o	Need -DSYSV in hpux CFLAGS in makefile.

o	Customs: Now has full support for preserving resource limits in HP-UX.
	(Not very useful unless you have a shell that supports them too.)

o	Logd: man page added courtesy of <chicken@npac.syr.edu>.

o	PMake: Fixed glitch in HP-UX setuid() use that prevented uid
	switching from working when installed suid root.

o	PMake (system mk file): Use /bin/sh5 on ultrix system, where
	/bin/sh error control is broken.
	
The following are small compatibility fixes so as to make PMake
more like UNIX make and/or gnumake in cases where no conflict with
PMake enhancements ensues.

o	PMake: Ignore all dot targets when searching for a default target.
	A dot target is one starting with `.' and not containing a `/'.
	This provides compatibility with other makes, especially when it
	comes to these the GNU makefiles with .PHONY at the top ...
	Previously only dot targets with a built-in special semantics
	were treated this way.

o	PMake: Strip initial ./ path components for purposes of target
	identification.

o	PMake: Let empty command disable transformation rule application
	to target (like in standard make and gnumake).

o	PMake: Don't automatically consider an archive file out-of-data
	just because if has no table of contents.

o	PMake: Accept options, variable assignments and targets on command
	line in any order.

A. Stolcke, 11/3/92


*** Patchlevel 13 ***

o	Rpc: Send 'Access denied' error on non-local calls and report
	procedure number called.  Allow non-local replies to be able to
	get the errors.

o	Rpc: Added workaround for systems where broadcasts don't reach the
	local host (like apparently HP-UX 8.x Snakes).  Also fixed a memory
	allocation bug triggered by broadcasts on multiple networks.

o	Customs: Use undocumented pstat() call to get SETPROCTITLE working
	with HP-UX.

o	Customs: Avoid spurious 'downs' by setting the down event to one
	second after the next availability sampling.

o	Customs/Reginfo: Display the value of criterion that causes
	host to be unavailable, as well as down time.  Also rearranged the
	order in which criteria are check to that most informative ones
	are `prefered' in justifying unavailability.

o	Customs: Modified the handling of cctrl -elect so that the target
	host will become new master with high probability.

o	On Ultrix systems use replacement for non-informative gethostid().

o	Removed more ANSI nits so compiles cleanly with cc -Aa on HP-UX
	(except for wait() incompatibilities).

A. Stolcke, 11/16/92


*** Patchlevel 14 ***

o	PMake: Added failure reason (errno) to various error messages.

o	PMake: Added convenience for root invocations: avoid export attempts
	and ignore lock file creation failure.  Also don't create lock
	files in no-execute mode (-n).

o	PMake: Make rules for archive members behave in a more standard way
	by fixing the shorthands $% and $@ (for $(.MEMBER) and $(.ARCHIVE),
	respectively).

o	PMake: Now handles computed variable assignments ($(VAR) = foo) and
	substitutions ($($(VAR))).

o	PMake: Implemented the .CURDIR and MAKEOBJDIR variables found in
	4.4BSD make (a derivative of pmake).  Should be able to get reasonable
	BSD compatibility by defining SPECIAL_CHAR to '.'.

o	PMake: Force update on archive members to ensure library gets rebuilt
	after updating the external member file.  Otherwise the MAKELIB
	rule will fail to do its thing under RECHECK operation.

o	PMake: Catch and warn about cycles in suffix dependencies.
	Used to leave pmake in infinite loops.

o	Customs: Fixed a potential bug in election protocol.  Also modified
	handling of multiple master detection so as to make one of the
	existing master likely to stay in charge.  This greatly reduces
	the broadcast storm formerly observed after rejoining partitioned
	networks, and speeds up resettling of the customs network.

o	Customs: Fixed a bug in the allocation procedure that could produce
	inaccurate records of availability on the master.  The typical symptom
	was over-allocating jobs when the cluster runs out of available hosts.

o	Customs/Cctrl: List export permits issued in -jobs list (since these
	also count towards the jobs total).

o	Makefile: removed some unwanted files from distribution

o	gnumake: Some minor additions to the patch to make it work.
	Also added an sed script that will convert some of the gnumake
	directives (includes, conditionals) to pmake syntax.

A. Stolcke, 1/22/93

*** Patchlevel 15 ***

o	gnumake: Included patch for GNU make-3.63.

This patchlevel also incorporates all the differences found between the
original pmake release of Sep 19, 1990 (on which these versions are based)
and the newer release of May 6, 1992 from ginger.berkeley.edu.

o	Customs: Only daemons started with the -master flag are eligible for
	master duties.  This is mainly a workaround for a poorly understood
	bug in the election protocol that fails to determined a master if
	a larger number of hosts are involved.  It is also convenient to
	prevent certain (e.g., less powerful) machines from being recruited
	as masters.

o	Customs: A bug fix that should prevent customs restarts to fail
	because a socket address is still in use.

o	Customs: Under SunOS 4.x, include reserved pages in swap usage.

o	Rpc: Bug fix for reentrant Rpc_Wait calls.

o	PMake: Miscellaneous bug fixes.

o	Prefix: As before, this part of the distribution is included unchanged.

A. Stolcke, 1/22/93

*** Patchlevel 16 ***

Mainly lots of fixes to the Makefile parser.

o	PMake: Some fixes to parsing for compatibility with 4.4BSD make.
	Now warns about bad #include syntax, supports .OPTIONAL as a synonym
	for .DONTCARE, and should work correctly when SPECIAL_CHAR != '#'.

o	PMake: The # character for directives is now settable at run-time
	(-Z option). SPECIAL_CHAR provides the default.

o	PMake: Lines of whitespace (that don't start with a tab) are skipped
	just like empty lines, instead of triggering a 'needs a leading tab'
	warning and generating an empty command.

o	PMake: Variable assignment that start with a tab and don't occur
	in a dependency context are accepted with a warning (instead of 
	generating a 'unattached command' fatal error).

o	PMake: Improved error detection in variable modifier parsing.
	In particular, no more infinite loops on missing right paren.

o	PMake: -i and .IGNORE now also work for jobs killed by signals
	(used to work in compat mode only).

o	PMake: Messages and diagnostics cleaned up and made more consistent.
	Error messages and the ones starting with '***' are now written to
	stderr rather than stdout.  So is debugging output (except for -p
	output).

o	PMake: Added support for the SHELL variable as an alternate way of
	specifying the shell for command execution.  SHELL settings are kept
	consistent with .SHELL targets and vice-versa.
	
o	PMake: SHELL or .SHELL now also affects both compatibility mode command
	execution and != assignments.  #undef'ing SHELL or .SHELL: without
	dependencies reverts to default shell.

o	PMake: Fixed two bugs in := assignments: right-hand side used to ignore
	command line variables and left undefined variables unexpanded.

o	PMake: Archives without table of contents revisited: only consider
	them out-of-date if they depend on a member.

o	PMake: Include version info and exportation document string in
	pmake -h output.

o	tests: updated README file and included two new test for environment
	variables and SHELL.

o	PMake: man page updated (and some previous omissions fixed).

o	Reginfo: truncate hostnames to fit column format.

o	Makefiles: the -C flag is no longer required on the command line
	to install pmake. 

A. Stolcke, 1/29/93

*** Patchlevel 17 ***

o	PMake: Various fixes to -t, -q, -n, -s option processing.
	Execute -q on .MAKE targets recursively in PMake mode.
	Execute -n, -t recursively with commands that expand the MAKE
	variable in compat mode.

o	PMake: Fix to deal with NeXT's sorted ranlib tables.

o	PMake: Fixed command line parsing bug that could lead to infinite
	loops on machines where char is unsigned.

o	lib/mk: Cleaned up the way a system specific makefile is included.
	Tries to use the output of /bin/machine or similar to determine
	host envronment.  Also eliminated the pmake define; use .PMAKE instead
	to #ifdef whether a Makefile is processed by pmake.

o	Customs: Added ifdefs for SGI port.

o	Customs/Reginfo/Export: Added support for general host attributes
	that may be set for customs as run-time, and which can be requested
	for exported jobs.

o	Cctrl: New -net option allows restricting broadcasts to a single
	customs network.
	
o	PMake: Support host attributes with .EXPORT= source targets.
	The sources of .EXPORT are used to specify globally required
	attributes.

o	PMake: Added .RESTART attribute to help in the control of long-
	running, continuable computations.  Also automatically restart
	customs jobs that were evicted from their hosts (terminated with
	SIGXCPU).

o	Tests: Included sample for use of .RESTART and .EXPORT=, see
	test/count and test/count.mk.

o	Customs: Don't restart while jobs are running, return "system error"
	to caller.

o	Export/Customs: -force flag lets you export dispite availability.
	Use judiciously.

o	Customs: Fixed a bug whereby the host allocation routine could go into
	an infinite loop if allocations where denied by the importing host.

o	Reginfo/Customs: Use TCP for CUSTOMS_INFO call (has large return data).

NOTE:	I changed the optimization for gcc (version 2.3.3 at ICSI) from -O2
	to -O.  It would screw up the inlining of JobCanRetry in job.c.

A. Stolcke, 2/11/93

*** Patchlevel 18 ***

o	Customs/Cctrl: Added simple statistics collection for imported jobs
	(total number, run times, etc.).  Accessed with cctrl -stats.

o	Cctrl: Added printing of summaries (number of hosts responding etc.)
	for broadcasts (-all, -net).

o	Customs: Normalize signal number for network transmission to allow
	interoperation between BSD and SYSV systems.  Also fixed the byte
	swapping of exit statuses that was broken in one of the early patches.

o	Customs: Bug fixes to SGI/mips load average reading, and
	memory mis-management in getusershell().

o	Customs: Added /etc/utmp reading for better idle time support on
	system without /dev/kbd or equivalent.

o	Customs: Warn about illegal default availability criteria (compiled
	in or defined on command line).

o	Customs/Rpc: Use own extended Rpc_IsLocal() to obtain local inet
	address (instead of get_myaddress()).  Allows error checking and
	guarantees that Local(&localAddr) is actually true.

o	Cctrl: Accept symbolic signal names as arguments to -kill.

o	PMake: Fixed a bug in the cycle printing routine that could lead to
	endless loops.

o	PMake: Fixed suid handling for IRIX.

o	system.mk: Added rule for .c -> .i (preprocessor output).  Useful for
	debugging macros expansions etc.

o	Customslib: Don't normalize a local cwd that point to /tmp, /usr/tmp
	or /var/tmp.  Just use the remote /tmp.  Rationale: /tmp is only locally
	accessible anyway, so we might just be testing, for which it is
	convenient.

o	Removed some nits to make Sun's acc happy.

o	Documentation: closed some gaps in INSTALL and customs/README.
	Also included FAQ, a set of questions & answers compiled by
	gary@chinook.atd.ucar.EDU in the course of his attempts to get
	customs running.

NOTE: If customs aborts soon after startup with the message "No local network
address found", the problem may be with you compiler and how it expands
certain ioctl constants.  For example, gcc needs to have fixincludes run
before it can do this correctly.

A. Stolcke, 4/15/93


*** Patchlevel 19 ***

o	PMake: Extension of VPATH semantics: even non-source targets are
	found there.  This seems to be what Sun, NeXT, and GNU Make are
	all doing, and recent GNU distributions seem to rely on it.

o	PMake: Fixed processing of System V-style 'F' and 'D' modified
	variables, so they work with PMake-style modifiers and '=' substitution.

o	PMake: Fixed bug whereby non-existent libs without dependencies
	were considered up-to-date.

o	PMake: Avoid running out of file descriptors when jobs are deferred.
	Also, try to resume deferred jobs as soon as any other job finishes.

o	PMake: Fix to handling of USELOCAL attribute, so it works when
	jobs as restarted.

o	PMake: Plugged file descriptor leak opened by exporting in -B mode.

o	PMake: Fixed the mechanism that forces a parent to be remade when a
	FORCE-type dummy child exists.  Previously this could fail under
	rare circumstances.

o	PMake: Make sure jobs that are evicted or terminate under .RESTART
	are restarted immediately, without waiting for the next job to
	finish.

o	PMake: Fixed documentation that wrongly claimed that $* expands to
	filenames with directory removed.  (It shouldn't, first, to be
	compatible with standard make, and second, since the :T modifier
	can be used to achieve that effect).

o	Reginfo: Added total count for available, but restricted hosts.

o	Reginfo: New -hosts flag to print just hostnames.

o	Customs: Also check /dev/mouse for idle time on Suns (patch thanks to
	Teemu Torma <tot@Niksula.hut.fi>).

o	Customs: Added note about Customs_NormPath configuration to config.h.

o	Customs/Reginfo: New -server flag to tie availability to server
	being up.

o	customs/libcustoms.a: This new library contains all objects needed
	for linking third-party customs client programs (such as gnumake).
	make install copies the library and required header files to system
	directories.

o	gnumake: Included patches for GNU make-3.64, 3.66, and 3.68
	(they have their releases coming out fast, don't they :-).
	NOTE: On Suns and others, make-3.68 configure fails to include
	HAVE_SETREUID in config.h, which is needed for suid-ness to work!

o	Added tags files and makefile targets everywhere.

o	Moved miscellaneous files to the new etc/ subdirectory.

o	Minor tweaks for HP-UX 9.01 port.

A. Stolcke 8/16/93

*** Patchlevel 20 ***

o	Rpc: Rpc_UdpCreate() always tries binding to reserved udp port.
	Use bindresvport() to avoid back-to-back use of same port by 
	different invocations.

o	Rpc/Customs: Replaced the static LOCAL_NETS access control list with
	a dynamic interface to allow and deny host or network access to RPC
	system. See new -allow and -deny customs options.  A default access
	list suitable for most cases is obtained automatically by
	inspecting the local network interfaces.  (This also means that
	customs/config.h will rarely need any configuration now since
	LOCAL_NETS was often the only thing requiring editing.)

o	Customs: Added provisions for correcting load average upward
	when starting jobs, and downward when jobs finish.
	Should prevent double allocations and encourage repeated allocation
	after a machine becomes free.

o	Customs: By popular demand, allow attribute matching including
	negation and disjunction.  Boolean combinations can be specified
	in disjunctive normal form:  A string starting with ``!'' means
	the following attribute name must *not* be present.  The attribute
	``OR'' separates disjuncts.  See etc/ATTRIBUTES for examples and
	some caveats.

o	Customs: Avoid availability index 0 values by requiring that load
	be strictly lower than max.

o	Customs: Added OS module for SunOS 5.x. 

o	Customs: SIGHUP triggers restart (like cctrl -restart) instead of
	aborting.

o	Customs: Authenticate the CUSTOMS_PGRP child-to-parent RPC call
	to prevent forgeries.

o	Customs: The default MIN_IMPORTS value is now -1.  This means that
	importquota -jobs -1 can be used to disable customs imports to a
	machine. (Note that -jobs 0 would simply remove the number of jobs as
	an availability criterion.)

o	Customs: Don't send EOF on tcp socket when process is just stopped,
	not terminated.

o	Customs: Added a generic os-default module that should allow
	rudimentary support for generic UNIX platforms. It can serve as
	a template in writing platform-specific versions.

o	Customs/Customslib: Make sure that inet addresses returned by
	various RPCs are valid for the caller, which can be a problem
	with multi-homed machines.  A distinguished return value tells
	the caller to use as a result whatever address it used to reach
	the callee.
	Note: This make new customs agents and client progams incompatible
	with old ones, so everything has to be restarted.

o	Customs: Delay reply to a CUSTOMS_IMPORT call until after the
	environment for the user process has been fully set up.  This
	allows errors to be returned to the caller (instead of simply
	causing termination of the job).  In particular, this avoids
	imports hanging indefinitely on the chdir() call.

o	PMake: Be more careful with removing interrupted targets. Only do it
	if file is not a directory and was modified since the make started.

o	PMake: Identify remote host in banner lines printed ahead of remote
	output (in pipe mode).

o	PMake: Make it easier to identify shells spawned in pmake mode by
	including `(pmake)' in the zeroth argument.  Can be disabled by
	#undefing MARK_SHELL_ARG0 in unix/config.h.

o	PMake: Archives in several formats (BSD ranlib'ed, Common archive,
	MIPS ranlib'ed) are now identified at runtime and (hopefully)
	handled correctly on all platforms, except for possible ranlib
	incompatibilities due to byte order and word size differences.
	Symbol table lookup still doesn't work on HP Snake systems, because,
	as all of that system, it's a funny mess: it uses an incomplete common
	archive format, with an undocumented symbol table format.
	
o	PMake: Fixed handling of '-' and '@' command modifiers so that
	they may appear interspersed with whitespace.

o	PMake: Handle System V makefile includes (enabled by SYSVINCLUDE
	in unix/config.h).  This is an improved version of a pmake
	enhancement by Christos Zoulas (christos@deshaw.com).

o	PMake: Under certain circumstances a file specified with a relative
	path would be found relative to VPATH even though it was accessible
	from the current directory.  The current dir now always takes
	precedence.

o	PMake: Explicit dependencies override implicit ones even in -M
	mode. This makes -M behavior consistent with native Pmake and 
	all other makes I have access to.

o	PMake: Ensure makefiles read and included are not directories.
	Print debugging info about files read with -dp.

o	PMake: Fixed != variable assignment to accept arbitrary amounts of
	output from shell command.

o	PMake: The special .NULL and .EXPORT targets now have a meaning
	when given without sources.  .NULL cancels the use of implicit
	null suffixes, whereas .EXPORT resets global export flags and
	attributes.

o	PMake: Fixed various inconsistencies and bugs with the setting of
	the .PREFIX ($*) variable.  It now always includes the full path
	of the target being created, consistent with other makes (but contrary
	to the PMake tutorial). Note that the filename part can always be
	gotten using the :T modifier. There also was a genuine bug which led
	to the stripping of the .NULL suffix even if it wasn't textually part
	of the target's name.

o	PMake: Use the full path to the archive in $(.ARCHIVE).  This is
	consistent with other dynamic variables and fixes various problems:
	libfoo.a(member) and -lfoo(member) now work as expected when
	the library is somewhere on the search path.
	Delay the archive symbol lookup until dependency checking phase,
	so path searches etc. can be used, and don't abort if the symbol is
	not found: simply complain about not knowing how to make the
	archive member.

o	PMake/Export: To increase reliability with long-running jobs,
	remote processes are now checked on at regular intervals to detect
	crashed machines etc (cf. REMOTE_CHECK_INTERVAL in customs/config.h).
	Also, jobs that are detected as lost this way are restarted as
	if evicted.

o	PMake/Export: Wait for EOF on remote output before terminating.
	Otherwise, depending on a race between the tcp stdio connection and
	the udp CUSTOMS_EXIT RPC the last bit of output could be lost.
	Also eliminated use of FIONREAD ioctl, which happens to hang
	Solaris 2.3.

o	PMake: Retry to export jobs that couldn't be exported the first
	time, at regular intervals (EXPORT_RETRY in customs/config.h).
	The previous strategy would only retry exports when some other
	job exits, which can waste a lot of time if jobs are long-running.
	(This generalizes a previous solution for jobs that could *only*
	be run remotely.)

o	PMake: Don't give up when running out of file descriptors.
	Lower the number of parallel jobs and continue.

o	PMake: Fixed bug whereby remote jobs could be restarted locally,
	which currently leads to their output being lost due to the way
	file descriptors are managed.

o	PMake: Fixed handling of a race condition between sending remote
	signals and registering remote exits, manifested as core dumps,
	waiting for non-existing jobs, and what not.

o	Customslib: When normalizing path names for export, leave NFS
	filesystems alone if they don't have the automounter prefix
	(suggested by pomeranz@aqm.com (Hal Pomeranz)).  The default heuristic 
	used by Customs_NormPath() thus becomes:

	1) if path is in /tmp, /usr/tmp, or /var/tmp, leave it alone
	2) if /tmp_mnt/..., remove /tmp_mnt/ from path
	3) if /n/... or an NFS mount, do nothing
	4) add /n/machinename/ and return

	The /tmp_mnt/ and /n/ prefixes can be configured by defining
	TMPMNT and NETMNT, respectively.

o	Customslib: Added convenience functions: Customs_Hostname() and
	Customs_ErrorMessage().  pmake and export changed to use these
	and give more consistent error diagnostics.

o	Customslib: Split client stubs over separate source files to get
	finer granularity during linking.  Also allows the server to use
	routines from the library.

o	Rpc: Changed the fallback broadcast address from INADDR_ANY to
	INADDR_BROADCAST.  (This only affects network interfaces who aren't
	able to report their broadcast address.)

o	Rpc: Added a special error code RPC_NOREPLY, to disable replies
	and acks on a given RPC.  Useful to transfer responsibility for a
	reply to a child process.

o	Logd: Works again -- fixed info buffer parsing.

o	Makefiles: Create all installation directories if necessary.
	Handle build with MAKEOBJDIR set to put objects in target-specific
	subdirectories.  The top-level Makefile sets MAKEOBJDIR to
	default values based on the system type.

o	libsprite: On second build, only include functions not in
	system libc (except for getopt, which is often broken).

o	etc/customs.conf: Added support for HP Snake (9000/700 series).

o	Customs, Export, PMake: Fixed to reinstall signal handlers on SYSV
	systems, so now working correctly on multiple suspend/continue etc.
	Cannot believe I had overlooked this until now!

o	Reginfo: Added -rate and -arch options to provide more selective
	listings.

o	Lots of small tweaks for SunOS 5.3 compile. Try to use POSIX
	interfaces where possible, and check available features by
	#ifdef'ing on the symbols belonging to an interface.

o	Lots of major tweaks for OSF/Alpha port. All data shipped over the
	net is now defined in terms of 'int16' and 'int32' types, defined
	in rpc.h.  Also fixed a bug in WayBill swapping that caused the
	the import server to abort whenever one tried to `export pwd' between
	different endian machines.

o	Included our customs-aware version of xlock in the distribution.

o	BSD/386 port.  This includes handling of variable-length
	struct sockaddr in rpc.c, enabled by -DHAVE_SA_LEN.
	Known problem: /bin/sh does weird input buffering, defeating the
	error and echo control scheme of pmake in native mode (not -M).
	Causes occasional "set" commands to be echoed.

o	Revised configuration (and INSTALL directions) slighly so that
	the system include directory can be edited in the top-level
	makefile (and config.mk) and will propagate to the compile.
	This means that unix/config.h should rarely have to be changed
	at all.

o	Revised PMake tutorial (doc/tutorial.psc) to reflect ICSI these
	CHANGES.

A. Stolcke 6/23/94


*** Patchlevel 21 ***

o	customs/config.h: put MAX_DATA_SIZE override here and increase to
	16K bytes.

o	Rpc: Handle incoming RPCs that exceed MAX_DATA_SIZE with RPC_TOOBIG.

o	Rpc: Handle RPCs of any size.  By original design, return data is
	still limited to caller's preallocated buffer, though.

o	Rpc: Event created by Rpc_EventOnce is now freed automatically after
	being triggered. (Fixes minor memory leak in customs.)

o	Customs: some tweaks to the election/mca code that will hopefully
	avoid recurring "registering with new master: No such procedure"
	messages.

o	Customs: check clock synchrony between master and slave if
	DOUBLECHECK_TIMEOUT is in effect.  This avoids trying to export to
	hosts whose clock is badly off and prevents timeout of the actual
	job import call.  Reginfo will show the condition, thus helping
	to detect the troubled host (always assuming the master system clock
	is suitably synchronized).

o	Customs: handle certain out-of-memory cases with error returns
	instead of aborting the daemon.

o	Customs: workaround for "Address in use" startup error when trying
	to allocate customs service ports. See WAIT_FOR_FREE_ADDRESS in
	customs/config.h.
	
o	customslib: use CUSTOMS_NOCWD error code to indicate failed
	getcwd() or path normalization. Don't print error messages in
	Customs_NormPath, set errno instead.

o	customslib: guard against waybill size overflow due to large
	command and/or environment.

o	customslib: better check whether the current directory is NFS mounted
	(for path normalization purposes).

o	PMake: Fixed handling of the out-of-filedescriptors condition:
	lower paralellism to total number of jobs, waiting or running.

o	PMake: Plugged numerous memory leaks and arranged for per-target
	memory to be deallocated when target is done.  Avoids pmake growing
	during excution phase.

o	PMake: Allow exported jobs to be restarted on local host.

o	PMake: More fixes to file descriptor handling for restartable jobs.

o	PMake: .EXPORT= attributes on specific targets no longer implies
	.EXPORT for that target (i.e., may be run locally if host satisfies
	attributes).

o	PMake: More efficient and robust error and echo control in -B mode.

o	PMake: -M mode tolerates dependency cycles, by ignoring the source
	closing the cycle.

o	PMake: Now uses SIGHLD to be notified of child status changes
	instead of polling.  Dramatically reduces latency for short-lived
	local jobs.

o	PMake: Fixed archive symbol table lookup on Alphas (apparently
	use 32bit integer format for backwards compatibility).

o	PMake: Fixed interrupt handling to react promptly during
	dependency processing phase in no-execute modes (-q, -n, -t).

o	PMake: Workaround to let suid-ness of pmake binary be effective
	with executable pmake scripts (so exporting will work). 
	Works only on systems that stick the full path in argv[0] on
	interpreter invocation (including Suns, but not Ultrix, IRIX, NeXT).

o	SunOS5.x optimizations: use vfork() while it's still there, 'cause
	it's a lot faster.  Use statically linked /sbin/sh for SHELL 'cause
	it's a lot quicker to start up.

A. Stolcke 9/27/94


*** Patchlevel 22 ***

o	export: avoid static memory free when -attr is used.

o	export, pmake: make sure remote exit status can be relayed
	even if job is running on non-local network.

o	RPC: change interface to Rpc_Allow/Rpc_Deny, added Rpc_IsAllowed.

o	customs, cctrl: added remote access list manipulation RPCs.

A. Stolcke 10/1/94


*** Patchlevel 23 ***

o	PMake: avoid rare hanging, either due to missing wait() on children
	or blocking on pipe reads.

o	Customs, Export, PMake: fixed problem with unlimited resource values
	transmited between machines of different wordsizes. Added support
	for RLIMIT_VMEM (virtual memory limit).

o	Customs: New compile-time confuration option to allow any user to
	simulate evictions (ALLOW_EVICT_BY_USER).  etc/cevict is a sample
	wrapper script using this "feature."

o	Customs: Revamped the idle time criterion to be more general and
	configurable at run-time.  Check the -idlecrit option in customs
	and importquota.

o	Customs: Fixed various byte swapping glitches.

o	Reginfo: Added additional flags for host selection: -avail, -up, -down.
	Fixed column alignment for large INDEX values.

NOTE: Protocol changes -- requires recompiling and restarting all daemons.

A. Stolcke 11/24/94

*** Patchlevel 24 ***

o	Customs: Fixed bug with rlimit mapping that caused FSIZE limit to
	override CPU limit.

o	Customs: Added -memory limit on imported jobs. Rewrote descriptions
	of various import policies in man page.

o	Customs: When exporting from a machine withouth file descriptor
	resource limit, set the limit value to a finite value (1024).
	An infinite value may break things at the other end
	(e.g., /bin/csh in SunOS5.3 goes into an quasi-infinite loop trying
	to close all descriptor up to RLIM_INFINITY).

o	Customs: Update a machine's hostname when it changes and the
	machine re-registers with the master.

o	Customs: Evict jobs before daemon aborts.  There is no eviction
	notice or grace period in this case, but at least pmake will restart
	them.

A. Stolcke 1/10/95


*** Patchlevel 25 ***

o	lib/mk/sys-iris.mk: pmake -DSGI_IRIX4 generates binaries using
	the /usr/irix4 compatibility tools.

o	etc/gnumake: latest patch fixes problem whereby subsequent commands
	in a multi-command target would be exported even though no local
	command was running.

o	export: removed a redundant getservbyname() lookup that would often
	lead to NIS lossage under load.  Identify error messages as coming
	from export.

o	RPC: Fixed typo that had effectively disabled the signal protection
	around event handling code. (Amazing that this apparently isn't
	necessary in practice.)

o	RPC: Use system-defined default FD_SETSIZE, so RPC module can make use
	of the larger ranges typically available on newer systems.  Made fd_set
	handling more efficient by keeping track of the largest file descriptor
	in use.

o	RPC: Added a workaround for broken select() in IRIX 5.2. The symptom
	of this was that export would go into a tight loop instead of waiting
	for input. [The underlying bug is select() sets fd_set bits not
	requested by the caller.]

o	RPC: Added workaround for broken recvmsg() in SunOS5.4. The symptom
	were timeouts when the amount of information shipped in the
	CUSTOMS_IMPORT call exceeded approx. 2900 bytes.  [The underlying
	bug is that recvmsg() on TCP streams discards the data from a message
	not fitting in the buffer passed to it.  The workaround is to use
	read() on those streams instead.]

o	RPC: Plugged a potential file descriptor leak triggered by TCP RPC
	timeouts. [This was never an issue until the above problem came up.]

o	Customs: use local IP number for random seed computation, eliminating
	need for non-portable gethostid().

o	Customs: Use ulimit() systems call for file size and descriptor
	limits on systems without getrlimit().

o	Customs: Avoid core dump on cctrl -abort.

o	Customs, Importquota: Added support for SGI's non-degrading priorities.
	Check -npri flags.  Also allows customs to be run under npri(1), since
	priority will be reset for user jobs.  Using -npri, -cpu, -memory
	on systems that don't support them now generates diagnostics.

o	Customs: Changed rating formula to give multiprocessors same value
	as uniprocessors as long as they both have at least one CPU free.

o	Reginfo: Added -master flag.

o	Cctrl: Give preference to hostnames over network names when
	resolving non-numeric addresses.

o	PMake: Make sure the saved uid/gid also gets reset prior to exec-ing
	a command.  Failure to do so causes bad lossage with /bin/csh scripts
	in SunOS5.x.

o	Customs, Export, PMake: Added another export option to request
	exclusive use of a machine, i.e., prevents multiple jobs on a
	single host.  This is achieved with `export -exclusive' or
	`.EXPORT: EXCLUSIVE' (in pmake).

o	Cleaned up signal handling to use either BSD or POSIX interface (but
	not SYSV or a mixture of the above).  Eliminates potential race
	conditions arising from old SYSV signal.

o	xautolock: Included modified version of this X11 idletime watcher.
	The program has been enhanced to update the access time of files
	in a manner similar to Sun's /dev/kbd or /dev/mouse.  This in turn
	can be used to get accurate idle time values for customs purposes.

A. Stolcke 2/18/95


*** Patchlevel 26 ***

o	RPC: Detect bogus message headers by checking magic numbers.

o	RPC: Prevent freeing of RPC cache entries while a handler procedure
	is running.  This manifested itself as 'No reply sent' error messages
	and eventual pmake core dumps under heavy network load/delay
	conditions.

o	PMake: Prevent failure when pmake binary is both setuid and setgid
	(although setuid is all one should ever want).
    
o	PMake: Generalized the code that searches for libraries specified 
	with -l .  Now uses all the suffixes declared with .LIBS to 
	locate the libraries.  For example, SVR4 platforms have libraries
	ending in either .a or .so .

o	PMake: No longer silently ignores out-of-date or missing libraries
	given as dependencies.  Only -l libraries are ignored if their
	full paths cannot be found.

o	PMake: Locate system makefiles only on system include path,
	not in general makefile search directories.  Also allow MAKESYSPATH
	environment variable to override compiled-in system include path
	(suggested by someone at DEC-OSF).

o	PMake: Fixed exit code in compatibility mode to conform to
	standard (same a PMake mode): 1 for out-of-date, 2 for error
	or signal, and 0 if all is well.

o	PMake: various little bugs fixed that were pointed out by DEC-OSF
	folks, including a possible buffer overrun in handling long archive
	member specifications.

o	PMake: Eliminated POSIX compile-time option.  The changes in the
	parser triggered by it are obsolete, and miscellaneous other
	effects are integrated in the default behavior.  Most notably,
	pmake now interprets and sets both PMAKE and MAKEFLAGS environment
	variables.  MAKEFLAGS is set to contain only the standard subset of
	flags understood by most make programs.  This should allow
	interoperability with other make versions, such as when 'make' is
	called directly in a makefile.

o	xautolock: Upgraded to patchlevel 10.

A. Stolcke 3/10/95

*** Patchlevel 27 ***

o	PMake: Fixed dynamic variable expansion so dependencies of
	suffix rules can use them correctly.

o	Customs: Prevent lossage when calling signal handlers before
	initialization.

o	RPC: Renamed int32, int16, etc. to Rpc_Long, Rpc_Short, ....
	Avoids conflict with vendor typedefs and is more consistent
	with general module naming conventions.

o	etc/gnumake: added patch for 3.73 code.

A. Stolcke 4/22/95

*** Patchlevel 28 ***

o	Man pages: documented customs -bias flag.

o	lsf: (new) Simple pmake port to the LSF Load Sharing Facility.

o	xautolock: fixed memory corruption error that would stop updates
	from working in an apparently random fashion (or worse).

o	etc/gnumake: added patch for 3.74 code.

o	etc/customs.conf: various additions.

A. Stolcke 8/20/95

*** Patchlevel 29 ***

o	RPC: fixed bad format string that could lead to core dump in
	rpc-debugging mode.

o	Customs: avoid sending OOB message on process exit: seems 
	unnecessary and avoids 10 second client delay on Solaris hosts.

A. Stolcke 11/17/95

*** Patchlevel 30 ***

o	More general fixes for 64 bit systems.

o	Customs: IRIX64 support.

o	Customs: Portable swap usage computation using swapctl(2).

A. Stolcke 12/31/95


*** Patchlevel 31 ***

o	PMake: Report location of unclosed conditionals correctly.

o	Customs: Fixed -version message.

o	etc/customs.conf: Misc. changes to support new UltraSPARC systems.

o	etc/newvers.sh: Deal with multiple domain specs (use last one).

A. Stolcke 4/18/96

*** Patchlevel 32 ***

o	Customs: Fixed minor problem in the way rlim_max was propagated.

o	Customs: Fixed free swap computation for SunOS5.x.

o	Customs: Optionally allow jobs from local clients (-localjobs 1).
	This is convenient to place all processes spawned by export or pmake
	under customs control.

o	Customs: Added rexport(1) script for simple-to-use parallel job
	exportation.

o	Customs: Prepending of NETMNT prefix can be disabled in config.h
	by leaving NETMNT undefined.  This seems to be the right thing for
	many sites.

o	PMake: Added rule for C++ compiles.

o	etc/customs.conf: adapted to sysinfo 3.1 output format.

o	etc/gnumake: added patch for 3.75 code.

o	Found major bug in IRIX5.3:  Processes executing suid root loose their
	resource limits, and thus export/pmake cannot propagate the user's 
	resource limits when installed suid root as required for priviledged
	port authentication.  This is fixed in IRIX6.2 by setting the 
	kernel variable reset_limits_on_exec = 0.

A. Stolcke, 8/30/96

*** Patchlevel 33 ***

o	Ported to Solaris-PC (i86pc).

o	Customs: use os-bsd module for Solaris since swap space code stopped
	working in SunOS5.6.

o	Ported to Linux PC (i686).

o	RPC: Fixed double-freeing bug in Rpc_Reset().  (Linux was first system
	to expose this problem by core dumping in ImportProcess().)

A. Stolcke, 4/4/98

*** Patchlevel 34 ***

o	Customs: Handle master hosts with multiple network interfaces and
	routes.

o	Customs: Fixed nlist access in IRIX5/IRIX6.

o	Customs: Fixed bug in Linux OS module: -jobs parameter was being
	ignored.

o	Customs: Updated SunOS5.x OS module to work again.

o	Customs: Check that master can lookup own hostname.

o	rexport: Always use /bin/sh for command execution to avoid portability
	problems across different systems.

o	PMake: Workaround for missing _cleanup() function.

o	PMake: Fixed Y2K problem in debugging output.

o	RPC: Workaround for Linux bug (recvfrom() on unconnected TCP sockets
	returns EINVAL instead of ENOTCONN).

o	xlog: Linux 2.2 port.

A. Stolcke
8/25/99

*** Patchlevel 35 ***

o	PMake: Added support for long names in GNU binutils archives.

o	PMake: Fixed archive symbol lookup on little-endian SYSV platforms.

o	Customs: Paths containing automount points are unchanged in
	normalization (assuming that automounter configuration is uniform
	across hosts).

o	Customs: Made handling of EXPORT_EXCLUSIVE more reliable by keeping
	state in affected customs agent, instead of in the master. Also, 
	there are now a maximum number of 'exclusive' jobs per machine
	(by default equal to 1, mimicking old behavior), set by
	customs -exclusives switch.

o	Various fixes for regression tests on Linux.

o	rexport: Try to max out file descriptor limit before starting pmake.
	Fixed handling of shell variable expansions in commands.

o	Miscellaneous bug fixes.

A. Stolcke
05/16/2001

