NNMASTER(8) UNIX System V(Release 6.4) NNMASTER(8)
NAME
nnmaster - nn database manager
SYNOPSIS
nnmaster -I [lmit]
nnmaster -w
nnmaster -v
nnmaster -l [ "lock message" ]
nnmaster [ options ] [ groups ]
nnmaster -F [ options ] [ groups ]
DESCRIPTION
nnmaster is the daemon which is responsible for building and maintaining
the database used by the nn(1) news reader.
Normally, nnmaster is started when the system enters multi-user mode, and
runs until system shutdown. To facilitate this, you should place the
following call in /etc/rc (or similar) to invoke the nnmaster daemon:
$master/nnmaster -l -r -C
where $master is the MASTER_DIRECTORY defined during configuration of nn.
When nnmaster is started as specified above, it will first unlock the
database in case it was locked (-l), perform a thorough consistency check
on the database (-C).
Then, every 10 minutes (-r), it will look at the time-stamp of the news
active file to see whether new articles have arrived on the system (or
whether articles have been expired).
If the active file has been modified, nnmaster will collect the header
information from the new articles and enter them into the database (or
remove the headers of the expired articles from the database).
If it detects that some articles have been expired, it will automatically
remove the header information of the expired articles from the database.
ARTICLE COLLECTION OPTIONS
Normally, nnmaster will collect all available news groups defined in the
news active file. The set of collected groups can be controlled via the
argument line. Groups can be either included or excluded:
A group name, e.g. comp, will cause the group and all its subgroups to be
collected. Individual groups, e.g. news.software.nn, can also be
specified
A group name preceded by an exclamation mark, e.g. !talk.politics, will
cause the group and all its subgroups to be ignored.
An empty argument, i.e. "", will cause all groups that are not ignored to
be collected. For example, to collect everything but rec and misc, use
the following command:
nnmaster -r !rec !misc ""
If the empty argument had been omitted, nothing would be collected, since
the presence of any groups arguments causes nnmaster to ignore all groups
which are not explicitly mentioned.
10/89 Page 1
NNMASTER(8) UNIX System V(Release 6.4) NNMASTER(8)
Example 1: The following commands can be executed by cron to collect
different sets of groups at different intervals or under different
conditions:
nnmaster -B -O14 rec misc sci -LBO -u
nnmaster !rec !misc !sci "" -u
Example 2: The group arguments are used in the given sequence, e.g. to
leave out comp.sys, but still collect comp.sys.ibm.pc, use the command:
nnmaster -r comp.sys.ibm.pc !comp.sys ""
The use of the -u option in the first example is essential, since each of
the commands will update the active file time stamp which will prevent
the other command from detecting new articles that have arrived.
Using this method to keep specific groups out of the database must be
used with great caution; just a single invocation of nnmaster without any
arguments will collect all the otherwise ignored groups!
COLLECTION OF ARTICLES
The following options control how nnmaster performs the collection of new
articles.
-r [ min ]
Daemon mode. The nnmaster will put itself in the background
(unless -f is also specified), and will checks for arrival of new
articles and expired articles every min minutes (and update the
database accordingly). If min is omitted, the default is to check
every 10 minutes.
Without the -r option, the nnmaster will just perform a single
collection of new articles (if any) and then exit. This can be
used to have the nnmaster started by cron(8) at regular intervals
instead of having it as a daemon which sleeps between checking for
new articles. Since the nnmaster is a bit expensive to start up
(it has to read a few files), it is up to you to decide which mode
is best on your system. (I have also heard that it works to call
nnmaster without -r from inews(1). I cannot recommend this unless
you receive batched news; invoking nnmaster for every received
article sounds too expensive to me.)
-h [ sec ]
Hold collection of new articles until next scan if new articles
have arrived withing the last sec [60] seconds. This is useful to
prevent nnmaster from competing for disk i/o with an rnews process
which is unbatching incoming news, or a running expire process. It
will have the side effect of limiting the number of C: entries in
the log, since collection of incoming batches will typically be
done in larger thunks.
-f Run nnmaster in foreground in daemon mode (see -r). Useful if
nnmaster is invoked from inittab. (Notice that if you use a
respawn entry in inittab, you will not be able to stop nnmaster
using the -k option, since init will immediately start another
master.)
Page 2 10/89
NNMASTER(8) UNIX System V(Release 6.4) NNMASTER(8)
-C Perform a consistency check on the database on start-up, and
rebuild corrupted database files. This operation can be quite
time-consuming since it reads through all the database files.
-b Normally, articles without a proper news header (no Newsgroups:
line) are ignored. Specifying the -b option causes these `bad'
articles to be included in the database (normally with no sender or
subject).
-B Remove `bad' articles. Sometimes, articles without a header ends
up in the news spool directory. These articles have no article id,
and therefore, they will never be expired by expire(8). This
option will allow the nnmaster to silently remove these articles (a
`B' entry will occur in the log file).
-O days
Ignore articles which are older than the given number of days.
This may help keep old 'stray' articles out of the database. If
the -B options is also specified, the old articles will be removed
from the news spool directories. Old ignored or removed articles
will be reported with an `O' entry in the log file. This option
can be disable for individual groups by the O flag in the GROUPS
file (see below).
-R N Specifies how the auto-recollect operation is performed on the
groups having this option set in the GROUPS file (see below). Four
methods are available (default is method 1):
1: Run expire on the group when new articles arrive.
2: Run expire on the group on every scan.
3: Recollect all articles when new articles arrive.
4: Recollect all articles on every scan.
-M mode
Normally, nnmaster will send a message via mail to the news
administrator (OWNER) when an error or potential problems
(primarily nntp related) occur. This can be restricted to only
fatal errors (nnmaster terminated) if mode is 1, and disabled
completely if mode is 0.
-Q Normally, nnmaster will print a message on the system console or in
the syslog if a fatal error happens. This option will prevent
this, so only a type 'E' entry is written to the Log file.
DATABASE EXPIRATION
Since articles does not stay forever in the news system, the database
must be cleaned up regularly to remove the information stored for expired
articles. Expiration of the database is normally scheduled using the
nnadmin(1M) command executed by cron at a suitable time when expire on
the news articles has completed. The following command will send a
message to the nnmaster and cause it to initiate expire on all news
groups:
nnadmin =EYW
10/89 Page 3
NNMASTER(8) UNIX System V(Release 6.4) NNMASTER(8)
Selective expiration of individual groups can be done from nnadmin
(interactive mode). It can also be done by invoking nnmaster with the -F
option. For example, the following command will run expire on all groups
except the `rec' groups:
nnmaster -F -k !rec ""
The -k option is required to terminate the currently running master since
two daemons cannot be running at the same time. Thus to run expire (on
all groups) in this way from cron, the following commands must be used:
nnmaster -Fk "" ; nnmaster -r ...
It is also possible to have nnmaster detect expiration automatically (see
-e). This requires that the min field in the active file is updated by
the news expire (this is not the default case when Cnews is used).
However, this is not always a safe indication since the first article may
not have been expired, while a lot of other articles have been deleted.
There are several strategies available in the nnmaster to do this clean-
up, each having their strengths and weaknesses.
Method 1 (default): Rebuilds the database from the existing database
information by comparing the current database contents with the contents
of the news group directories, eliminating entries whose file no longer
exists. This method is guaranteed to eliminate all expired articles from
the database, and it is reasonably fast because it only has to read the
directories, not each article file.
If news is accessed remotely via nntp, the list of existing articles
cannot efficiently be fetched by reading a local directory. Instead an
XHDR request is sent to the nntp server to get a list of articles.
Method 2: Eliminates only the expired articles before the first article
in the group. This is very fast since only the active file and the
database itself is accessed, but it will obviously leave some expired
articles in the database. This method requires that the min field in the
active file is updated by expire.
Method 3: Discard the current database information and recollects all
articles. This is obviously very time consuming, and it is therefore not
recommended, especially not with nntp.
The options related to database expiration are:
-E N Select expire method N. (If N is omitted, the default method is
used).
-e [N]
Automatically run expire in the database on groups where the min
number in the active file has increased by N (1 if omitted)
articles. This is disabled by default (since the min field is
often unreliable).
Page 4 10/89
NNMASTER(8) UNIX System V(Release 6.4) NNMASTER(8)
-F Run expire once and exit. If a list of groups is specified on the
command line, the matched groups (see the rules above) will be
marked for expiration. If no groups are specified, only the groups
already scheduled for expire will be expired. Consequently, to
expire all groups, a blank argument "" (matching all groups) must
be specified.
DATABASE LOCKING
The database can be locked centrally, which will normally disallow all
access to the database, and even block nnmaster from being (accidentally)
started. When a lock is set on the database, all currently running
clients will terminate the next time they try to access the database.
Setting a lock on the database can thus also be used to force running
clients to terminate.
The following options set and clear locks on the database:
-l message
Locks the database with the given message. The message will be
displayed every time a program tries to access the database.
-l Unlock the database if it was locked.
-i Ignore a possible lock and continue. This can be used to have
nnmaster operate on a database which is blocked from normal user
access.
Since only one nnmaster can operate on the database at any one time, a
running nnmaster daemon must be stopped before a lock can be set on the
database. If neither -f nor -r is specified with the -l option (in both
forms), nnmaster will terminate after setting or clearing the lock.
DATABASE INITIALIZATION
The following options are used to initialize and update the central
database files:
-I [limit]
Initialize database. This option will erase an existing database,
and create an empty database containing entries for the currently
known groups. nnmaster will offer you to use an existing GROUPS
file when initializing the database.
The optional limit can be used to put a limit on the number of
articles that will be collected in each group during the first
collection of articles following the database initialization. This
is useful on systems where the 'min' field in the active file is
unreliable or not maintained (Cnews doesn't) to limit the work done
to do the initial collection of news after the initialization of
the database. If news is accessed remotely from an NNTP server,
this is even more important! If limit is omitted, or is zero,
nnmaster will trust the min field and collect all articles in the
range min..last.
10/89 Page 5
NNMASTER(8) UNIX System V(Release 6.4) NNMASTER(8)
-G Reread the GROUPS file. This option is used to instruct nnmaster
to parse the GROUPS file after it has been edited. See the section
on the GROUPS file below.
MISCELLANEOUS OPTIONS
The following options controls various details of the nnmaster's
behaviour:
-D [ N ]
Run nnmaster in "debug mode". If N is omitted, or equals 1 or 3,
this will produce a compact but still very informative trace of the
collection or expire process directly on the terminal. This is
most useful when doing the first collection of articles after
initializing the database with -I. If N is 2 or 3, a trace of the
NNTP traffic is written to a file nnmaster.log in the TMP
directory. This option disables -r.
-H Identifies the host which nnmaster is running on as the nntp-server
for its clients, i.e. that it can access the news spool directory
locally without using NNTP. Normally, nnmaster will detect this by
itself by comparing the host name to the contents of the
nntp_server file, so this option should really be superfluous.
-y retries
In some networked environment, opening an article (shared from
another machine via NFS) may fail for no obvious reason. Using
this option, it is possible to cause nnmaster to perform retries
attempts to open an article before marking the article as non-
existing in the database.
-L types
Exclude the specified entry types from the log file. This is
normally used to exclude the 'C'ollecting and e'X'pire entries
(-LCXO).
-t Trace the collection process. This will place a lot of information
into the log file (T: entries).
-u Normally, nnmaster will just compare the time-stamp on the active
file with a time-stamp saved in the database to see if new articles
have arrived. The -u option forces the nnmaster to read the active
file on start-up to see if new articles have arrived.
-v Print the release and version identification for nnmaster, and
exit.
-w Wakeup the real nnmaster. Send a signal to the nnmaster daemon to
have it check for new articles immediately.
-k Kill the already running nnmaster daemon before proceeding with the
operation specified by the other options (or terminate if no other
operation is requested).
Page 6 10/89
NNMASTER(8) UNIX System V(Release 6.4) NNMASTER(8)
THE GROUPS FILE
The primary purpose of the GROUPS file is to store the names of the news
groups represented in the database. Each line in the file corresponds to
an entry in the (binary) MASTER file, and the sequence of the lines in
the GROUPS file must never be changed unless the database is
reinitialized afterwards.
However, the contents of the lines in the GROUPS file can be edited to
control how the nnmaster should handle each individual group.
The format of each line is:
news.group.name [ timestamp ] [ flags ]
The news group name is the name of the group, and must not be changed in
any way. If the group is no longer in the news active file, and
consequently the group does no longer exist, group name can be replaced
by a `@' character which will instruct nnmaster to ignore this entry
without having to rebuild the entire database.
The optional time stamp indicates when the line was added to the GROUPS
file and is used by nn to detect new groups. When the GROUPS file is
built initially from the active file, the time stamps are omitted which
simply means that they are "old".
One or more of the following flags can be added to the GROUPS line to
control nnmaster's handling of the group:
D Causes nnmaster to treat all articles in the group as digests, even
when they don't initially look like digests. Articles which are
found not to be digests after all, are still not digested.
N Instructs nnmaster to never digest any articles in the group.
O Disables the -O option for this group, i.e. all existing articles
will be included in the database (and they will not be removed if
the -B option is specified). This flag should be set on groups
which you never expire, or have a very long expire time!
R Causes nnmaster to recollect all available articles in the group
whenever a new article is received. This is said to be useful is
some high-traffic clarinet groups with many cancelled articles.
>file Instructs nnmaster to append all new articles to the specified
file. This makes it possible to keep specific groups out of the
way of expire. The archive file can be access directly from the nn
client using the goto-group command. The file name must be a full
path name to a file in an existing, writeable directory.
@ Instructs nnmaster to completely ignore this group - this is
equivalent to setting the group name to `@' as described above.
10/89 Page 7
NNMASTER(8) UNIX System V(Release 6.4) NNMASTER(8)
! or X
Causes nnmaster to ignore the group and not collect the group's
articles in the database.
Comments (starting with `#' and continuing to the end of line) and empty
lines are allow in the GROUPS file, but it is strongly recommended to
keep the changes to the GROUPS file as small as possible, because of the
implicit correspondence with the binary MASTER file.
It is not recommended to edit the GROUPS file while nnmaster is running
because it may add new groups to the file. After editing the GROUPS
file, the command
nnmaster -G
must be run before restarting the nnmaster to parse and verify the new
GROUPS file.
NNTP SUPPORT
The nnmaster can access the news articles from a local news spool
directory as well as from an NNTP server. When compiled with NNTP
enabled, nnmaster will compare the name of the NNTP server and the name
of the local host; if they are identical, nnmaster will bypass NNTP and
access the articles directly.
When it has to access the news articles via NNTP, it cannot time-stamp
the active file, so instead it transfers the entire active file from the
NNTP server and compares it with a local copy of the last active file
fetched from the NNTP server. This is not very expensive in terms of
cpu-cycles, disk-load, or net-bandwidth, but to stay on friendly terms
with the NNTP server administrator, you should probably not use shorter
update intervals than the standard 10 minutes.
Setting a much higher update interval than the standard 10 minutes is not
really recommended either, since an update normally implies fetching a
burst of news articles from the NNTP server, so setting the interval too
long may imply that the load on the NNTP server will be more un-even.
In expire method 1, the use of XHDR just to get a list of existing
articles in a group is definitely a waste of resources on the nntp server
(but still lower than using method 3). Before using the XHDR request,
nnmaster will send a non-standard "LISTGROUP" request; if the nntp server
supports this request, it should return an OK_HEAD status followed by an
(unordered) list of article numbers (one per line) terminated by a `.'
line. The nntp servers supporting this request will be much less loaded
during expire.
The -O option does not work with NNTP. The -B option will only work with
NNTP if the nnmaster is running on the NNTP server.
FILES
The $db, $master, and $news names used below are synonyms for the
DB_DIRECTORY, MASTER_DIRECTORY, and NEWS_LIB_DIRECTORY defined during
configuration.
Page 8 10/89
NNMASTER(8) UNIX System V(Release 6.4) NNMASTER(8)
$db/MASTER Database master index
$db/GROUPS News group names and flags in MASTER file order
$db/DATA/nnn.[dx] Database files for group number nnn
.../.nn[dx] Database files if located in the group directories
$master/GATE Message channel from nnadmin to nnmaster
$master/MPID The process id of the nnmaster daemon.
$Log The log file (the location is configuration
dependent)
$news/active Existing articles and groups
/usr/lib/nntp_serverContains the name of the NNTP server.
The MASTER file contains a record for each news group, occurring in the
same sequence as the group names in the GROUPS file. The sequence also
defines the group numbers used to identify the files in the database's
DATA directory.
The GATE file will be created by nnadmin when needed, and removed by
nnmaster when it has read it. Therefore, to send a message to the
nnmaster requires that you are allowed to write in the $master directory.
The contents of the Log file are described in the nnadmin manual.
SEE ALSO
nn(1), nncheck(1), nngrep(1), nntidy(1)
nnadmin(1M), nnspew(8), nnusage(1M)
AUTHOR
Kim F. Storm, Texas Instruments A/S, Denmark
E-mail: storm@texas.dk
10/89 Page 9