=========
 qtop
=========

------------------------------------------------------
Monitoring tool for Torque & PBS-based cluster systems
------------------------------------------------------

:Author: Fotis Georgatos <fotis at cern.ch>
:organization: LCG/EGEE
:Address: Where no particle has gone before
:Date:   2011-11-24
:Copyright: GPLv2
:Version: 3001
:Manual section: 1
:Manual group: monitoring of PBS/torque status

.. TODO: authors and author with name <email>

SYNOPSIS
========

``qtop`` ``-h``

``qtop`` [-h] [-a] [-b BDII] [-c OFF|ON|AUTO] [-d] [-f COLORMAPFILE ] [-i PBS] [-j JOBSUFFIX_OVERRIDE] [-l LRMS_OVERRIDE] [-s SOURCEDIR] [-n PBSNODES_A] [-p PBSPATH] [-q QSTAT] [-r REPLACE_CHARS] [-s SOURCEDIR] [-u GETUSERS_COMMAND] [-x SUMMARY|NODES|ACCOUNTS] [-w]


DESCRIPTION
===========

qtop is a nifty command-line tool for monitoring PBS systems, esp. torque.
It tries to fit as much information as possible in a screen's real estate,
by joining together the output of pbsnodes -a, qstat & qstat -q.

Output is divided in 3 sections, reporting: a) SUMMARY b) NODES c) ACCOUNTS 
Each user gets mapped to a unique letter, according to number of jobs in
qstat.
Symbol 0 is always the user w. most R+Q+other jobs, 1 is next in # of jobs,
etc.

qtop suppresses color mode automatically, when needed, eg. try: watch -d qtop
Very large sites should try qtop|less -RS and navigate around with arrow keys.
It is very configurable, you can either modify qtop.conf or, try qtop -h.

OPTIONS
=======

-a      Try WN name remapping   This is used in situations where node names are not a pure arithmetic sequence (eg. rocks clusters)
-b      Local BDII hostname     Useful on cluster with grid m/w, to compare values among LRMS and Information System
-c      OFF|ON|AUTO colormode   Select use of ANSI-colored output. AUTO is the default and should work in most cases (sensitive upon input tty)
-d      Debug information       Use twice for extra output; mainly intended for usage by developer/debugging purposes
-f      Set COLORMAPFILE        Describe location of file with the definitions of color output, used for generating ANSI sequences
-i      Type of LRMS            One of TORQUE|PBS|GE|LL|LSF|SLURM|CONDOR (future-proof, currently only the first two are implemented)
-j      Set JOBSUFFIX_OVERRIDE  Use this if autodetection fails. eg.  "jobsuffix.subdomain.example.com"
-l      Set LRMS_OVERRIDE       Use this if autodetection fails. eg. "mylrms"
-n      Set PBSNODES_A command  eg. '$PBSPATH/pbsnodes -a'
-q      Set QSTAT command       eg. '$PBSPATH/qstat'
-p      Set PBSPATH             Default is "/usr/bin". Some systems may have the respective pbs commands in PBSPATH='/usr/local/bin'
-s      Set SOURCEDIR           This is useful for offline testing. Directory should contain the stdout of pbsnodes -a , qstat -q , qstat
-o      Set vertical separator  Put vertical bar every WN_COLON nodes.  Experimental option, output may be garbled in certain cases
-r      Character sequence      Define a sequence of replacement symbols for the individual user accounts. Experimental option
-t      Set alternative TMPDIR. qtop creates there a subdirectory and multiple temporary files underneath
-u      Cmd to get DN mappings  Useful on grid CEs. By default, '/opt/edg/sbin/edg-mkgridpool --list' should output the pool account mappings
-x      Exclude section         Exclude in output one or more of SUMMARY|NODES|ACCOUNTS sections
-w      Autotune, for first run Create a ~/.qtop directory and wget a default colormap file (will overwrite the existing one, so use with care)

For all other options see ``-h``.


EXAMPLES
===========

:qtop:
     try a single run of the utility and check its output:

:watch -d qtop:
     run it with watch; extremely useful for LRMS dynamic observation

:qtop|less -RS:
     this is for sites with too many nodes; use arrow keys for navigation

:qtop -c OFF:
     suppress color use in output (useful if colors appear corrupted); you can also fix this via ~/.qtop/qtop.conf

:qtop -a:
     try to renumber the names of WNs (useful if result is fishy, renumbering uses range wn[101-999])

:qtop -d:
     send this for debugging, if for whatever reason output seems strange; read paragraph on Report issues, too.

.. comments : lorem ipsum lorem ipsum
   lorem ipsum lorem ipsum

Introduction
------------

qtop (pronounced queue-top) is a tool written in order to summarize the state
of an LRMS (Local Resource Management System), 
along with some related information relevant on grid (and not grid) clusters.
At present it supports the PBS family and SGE family is on the works. 
The tool is not intended to be a Swiss-army knife, it rather adopts the unix
philosophy: do one thing and do it well.

So, qtop tries to optimize on as little use of screen real estate as possible.
It will report information in three sections:

Accounting summary

 * Number of Nodes (available/total)
 * Number of Cores (available/total)
 * Number of Jobs (running+queued)
 * Names of Queues, with running/queued jobs in each one; symbol * appears, when queue is blocked (ie. qstop'd OR qdisable'd)

Nodes occupancy

 * Node ID (virtual or real nodename); Please read it vertically. This may be a REMAPed virtual node number.
 * Node state, first letter of (job-exclusive, busy, offline, down, ...). Symbol '-' corresponds to the "free" node state.
 * Job allocation table in a CPU-Node matrix layout; This is how your LRMS does the resource planning.

User account information

 * "id" which is a symbol representing a single unix account
 * User's jobs in Running state vs Total (this includes Completed state)
 * unix account name
 * Grid certificate DN: this is useful on grid clusters; it shows who owns the so-called pool account at the present moment.  NOTE: This feature can only run under root privileges, due to dependency on command edg-mkgridpool --list . There can be workarounds, contact the developer for details on how to do this in your case. If you don't have an LHC/grid cluster, please prefer to run qtop as a non-root user, as normal.

Why qtop is useful
------------------

Here are some scenarios in which this tool has served faithfully:

* Overview of the LRMS state: nodes' status, queues' status, dynamic behavior of the system (eg. using watch -d)
* Test other schedulers or policies; eg. on maui/moab: CPULOAD FASTEST FIRSTAVAILABLE MAXBALANCE MINRESOURCE PRIORITY
* Detect black-holes and identify status of nodes with interesting behavior (eg.  too much swapping with certain user's jobs)
* Security incidents, whereby a user account/DN is supplied and we must quickly find out where the relevant jobs run.
* To combine with ansi2html script and provide a front-end webpage with cluster occupancy details (an example is included in the package) 

Report issues, if any
---------------------

Please report the results of any command listed under "How to use qtop", along with your expectations, in plain English language foremost.

Also, if possible, please do the following and attach the resulting files ::

  qstat -q > qstat_q.txt
  qstat > qstat.txt
  pbsnodes -a > pbsnodes_a.txt
  tar -zcvf my_qtop_source_tarball.tar.gz pbsnodes_a.txt  qstat_q.txt  qstat.txt

If you place them in a single directory and run ./qtop -s mydir/ you should be able to reproduce the exact same effect as the one to be debugged. Put all these in a tarball (tar.gz) or append them in an email to author.

A very common cause for trouble is non-uniform worker node names. David O'Callaghan from Trinity College set an example on how to manage such cases::

  # Because we have WNs with different domains (.cs.tcd.ie, .grid.cs.tcd.ie) I
  # had to redefine PBSNODES_A in qtop.conf
  function pbsnodesagrid()
  {
    pbsnodes -a | grep -A7 wn[0-9]*.grid.cs.tcd.ie;
  }
  PBSNODES_A=pbsnodesagrid
  # I had to make a function definition, as  the command did not work if I put
  # it all on one line.


Epilogue
--------

If you wish to modify the behavior of the script, there are a few variables
possible to tune, which you could easily find inside /etc/qtop.conf, eg.
PBSPATH=/usr/local/bin; (or, qtop -p /usr/local/bin) Once you decide what
should be the defaults, save this file or tune the command line parameters.

NOTA BENE: The tool is really at a prototype state. It is not meant to be
perfect for everybody (nor do I believe will ever that be possible, given the
vast heterogeneity of cluster setups). Enjoy.

FILES
=====
Main files found in the original tarball::

  qtop		Executable shell script
  qtop.conf	Configuration file; env. variables being sourced at startup
  qtop.colormap	File with ANSI derived color codes, used only in color mode
  qtop.man	reSTructured text, source of manpage
  qtop.1.gz	manpage in consumable format

SEE ALSO
========

`qtop <http://cern.ch/fotis/QTOP/>`__

``torque``

``torque-client``

BUGS
====

* Numerous mapping problems with non-uniform Worker Node names
* Spurious bugs with small clusters (<10 nodes); should be fine with more
* qtop.colormap MUST have complete list of matching regex expressions,
  otherwise output appears corrupted (if so, just switch off color mode)
* SPACING & WN_COLON options are fully experimental and often incomplete
* running wget without asking the user may be annoying; to be supressed

