Picus Pickings

Elevate Runas using C#

2011-03-24T21:55:00.003+10:00

Following on from my last post, I thought I'd share the equivalent for a non-PowerShell situation. In this case, I have a piece of C# code that when compiled into an executable allows elevation of an administrative account to full admin status.

I'm sure that this has been done a number of ways, specifically, a more polished product is available in the PowerToys collection, I believe (See http://technet.microsoft.com/en-us/magazine/2008.06.elevation.aspx). This has been provided for some knowledge sharing and as a reminder to me, hopefully someone finds it useful.

Once compiled, I use elevate.exe as follows (assuming Elevate.exe is in c:\UserData\)...

To run a pre-elevated command prompt, create a shortcut with a command line similar to:

C:\Windows\System32\runas.exe /u:mydomain\adminuser "c:\userdata\elevate.exe ""%SystemRoot%\system32\cmd.exe"""

Note the double quoting for correct passing of strings with spaces through two levels.

References:
Adapted from PowerShell example at:
http://stackoverflow.com/questions/1566969/showing-the-uac-prompt-in-powershell-if-the-action-requires-elevation

C# Source:

using System;
using System.Collections.Generic;
using System.Text;
using System.Security.Principal;
using System.Diagnostics;

namespace Elevate
{
    class Program
    {
        static void Main(string[] args)
        {
            if (args.Length > 0)
            {
                WindowsIdentity windowsID = WindowsIdentity.GetCurrent();
                WindowsPrincipal principal = new WindowsPrincipal(windowsID);

                if (principal.IsInRole(WindowsBuiltInRole.Administrator))
                {
                    Console.WriteLine("No elevation required, already running as an Admin!");
                }

                string psiArgs = "";
                for (int i = 1; i < args.Length; i++)
                {
                    psiArgs = psiArgs + " " + args[i];
                }

                // Process.GetCurrentProcess().ProcessName
                ProcessStartInfo psi = new ProcessStartInfo();

                psi.FileName = args[0];
                psi.Arguments = psiArgs;
                psi.Verb = "runas";

                try
                {
                    Process proc = Process.Start(psi);
                }
                catch (System.Exception e)
                {
                    Console.WriteLine(e.Message);
                }
            }
            else
            {
                Console.WriteLine("No command-line provided - nothing to elevate!");
                Console.WriteLine(Environment.NewLine + "Usage:" + Environment.NewLine);
                Console.WriteLine("\t" + Process.GetCurrentProcess().ProcessName + " D:\\Path\\to\\CommandToRun.exe");
            }
        }
    }
}

Elevating Administrative Rights using PowerShell

2011-03-24T21:23:00.007+10:00

I was a fan of RunAs.exe to achieve a form of compliance to the 'Principal of Least Privilege' under Windows XP. I worked on clerical things, browsing the internet etc, with a non-privileged account and I had my trusty privileged command prompt waiting for that inevitable call to arms. Since moving to Windows 7 recently (yes, I skipped the whole Vista thing), I was missing the ability to do all those little things that are denied from the command-line (try running handle.exe for instance).

If you use PowerShell and don't want to be denied those little things that separate the administrator from the normal folk, read on.

Oh, and I won't be getting into philosophical discussions about the right and wrong of an administrator doing the business of administering with privileges enabled - each to his own, I say.

Anyway, to Launch an interactive PowerShell session with a fully enabled Administrative context:
• Add the code below to the profile of the administrative user (e.g. mydomain\adminuser)
• Start the powershell session using "RunAs.exe /u:mydomain\adminuser "


# Profile Code Starts here
#
function Check-IsAdmin()
{
#reference: http://www.interact-sw.co.uk/iangblog/2007/02/09/pshdetectelevation
#reference: http://www.leastprivilege.com/AdminTitleBarForPowerShell.aspx

  # Get the Current User's Security Token
  $windowsID=[System.Security.Principal.WindowsIdentity]::GetCurrent()
  $principal=new-object System.Security.Principal.WindowsPrincipal($windowsID)
  
  # Check if it has the Administrator access enabled
  return [bool]$principal.IsInRole([System.Security.Principal.WindowsBuiltInRole]::Administrator)

}

function Invoke-Admin() 
{
#reference: http://stackoverflow.com/questions/1566969/showing-the-uac-prompt-in-powershell-if-the-action-requires-elevation

    param ( [string]$program = $(throw "Please specify a program" ),
            [string]$argumentString = "",
            [switch]$waitForExit )

    $psi = new-object "Diagnostics.ProcessStartInfo"
    $psi.FileName = $program 
    $psi.Arguments = $argumentString
    $psi.Verb = "runas"
 
    $proc = [Diagnostics.Process]::Start($psi)
    if ( $waitForExit ) {
        $proc.WaitForExit()
    }
}

# Activate Elevated privilege if not already done
if ( (check-isadmin).Equals($false) )
{
 Invoke-Admin $(Get-Command powershell.exe).Path
 $host.SetShouldExit(0)
}

PowerShell's [adsisearcher] FindAll Exposed

2011-03-08T21:58:00.009+10:00

Background
I've been dabbling with PowerShell for some time but haven’t really got my head into it - mainly because I haven't been working in an environment where I can get the most advantage from it. I've decided to bite the bullet and make PowerShell my new best friend. The first thing I needed to do was to search Active Directory - there's not much point doing things by halves and only working on my local workstation.

I was initially a little puzzled by the various blogs that I read on the subject, most of which only give a passing treatment of the use of the [adsisearcher] accelerator and the associated .NET Class, particularly when it comes to what is retrieved. Most discussions that I've read imply that the searcher returns only a Path attribute, requiring a second call to retrieve the content of the properties that are teasingly referred to when you do a pipe to Get-Member.

Getting Active
So, what's the real story, how can a person dive in, grab all that goodness from Active Directory and move on to the rest of the day in the quickest possible way? Well, I'm not sure If this is the actual quickest way but it works pretty well for me…

In case you're not familiar with PowerShell's [adsisearcher], we should get a few things straight before we get to the example.

First, what is [adsisearcher] - simply put, it's a shortcut to accessing a .NET library class (System.DirectoryServices.DirectorySearcher) and is referred to as an accelerator.

Accelerating
To see why accelerators are handy and to get a list of all the accelerators that are defined in your PowerShell session, type:

[System.Type]::GetType("System.Management.Automation.TypeAccelerators")::Get

Now try entering the following (all on one line):

[System.Type]::GetType("System.Management.Automation.TypeAccelerators")::Add("accelerators", [type]::GetType("System.Management.Automation.TypeAccelerators"))

What we just did was add a new accelerator to give us a shortcut to the Accelerators function call. Now, to get a list of accelerators, type:

[accelerators]::Get

And to add a new accelerator, try:

[accelerators]::Add("psping","System.Net.NetworkInformation.Ping")

Isn't that much easier? OK, yes, a little off-track, but good to know!

Searching...
Next, lets look at what information you need to provide to get the results that you're after. The main things that you need to know to get 'what you want to know' are:

A query string to define what objects from the Active Directory you want to find.
A list of attributes from the Active Directory objects that you want see.
In a multi-domain environment, you may also want to specifically choose which domain to search as well.

How about we use an example and assume we want to get a list of computer names from the Active Directory? To achieve this we need to feed an LDAP query to [adsisearcher] that describes what we want - something like this is a good starting point:

objectClass=Computer

I say starting point because, as far as a query string goes, it's a little broad for an Enterprise scale environment. Do we really want 'all the computers' in the AD? Personally, I'm a server infrastructure type of guy, so what I really want is all the servers in the domain, so I might use:

(&(objectClass=Computer)(operatingSystem=Windows Server*))

The query string shown above will return all AD objects that are 'Computer' objects that have a 'Windows Server' Operating System installed.

So to start using the [adsisearcher] accelerator you could use:

$searcher = [adsisearcher]"(&(objectClass=Computer)(operatingSystem=Windows Server*))"

Searching Efficiently!
So now we get down to the pointy end of the business - it's not good enough to just pull back everything there is to know about the computers in the domain, we need to get specific. By specifying the particular attributes that you want to know about, you save yourself some processing and network bandwidth . Also, other tutorials advocate the use of the 'Path' attribute to re-query the Active Directory to get a reference to an AD object using something like:

$searcher.FindAll() ForEach { ([adsi]$_.path).cn }

The above is not recommended as it is performing a query to find all the objects that match the specified query string (i.e. the filter) and then for each object that it found, perform another lookup to retrieve the cn (i.e. the canonical or common name) of the object . So for a Windows Domain with 1000 servers, perform 1001 directory lookups - ouch!

A better way to achieve the above is to use the 'PropertiesToLoad' attribute of the [adsisearcher] class. Simply provide a list of attributes that you want included in the results of the search. Either a single value (e.g. cn) or , if required, a comma separated string can be used. The results of the FindAll() method will be a collection of objects with Path and Properties attributes. The Properties attribute of the result collection is itself a collection that corresponds to the values of the AD Object properties.

Putting it all together, we get:

# Query the AD for a list of computers running a Windows Server OS
$searcher = [adsisearcher]"(&(objectClass=Computer)(operatingSystem=Windows Server*))"

#Add cn to the default ADsPath property in the result set
$searcher.PropertiesToLoad.Add("cn")

# Loop through the result collection and display the cn property
$searcher.FindAll() ForEach { $_.Properties["cn"] }

The most important point about the above is that only one query is performed and all required results can be returned in one efficient AD access.

Multiple Domains
Coming from a multi-domain environment, I just have to mention the 'SearchRoot' property of the [adsisearcher]. By default, the logged-on user's domain and credentials are used to connect to the Active Directory. To change set focus to the 'mydomain.local' domain before invoking the FindAll method, do something like:

$searcher.SearchRoot=[adsi]"LDAP://dc=mydomain,dc=local"

LDAP Syntax
For more information on the LDAP syntax used by [adsisearcher], see the 'Understanding LDAP' white paper that gives a comprehensive description of all the elements.

References:

The Scripting Guys - practical examples of Searching active Directory.
Under the Stairs - Type Accelerators, handy cmdlet function.

Slashed tmp and killed the cluster

2010-06-01T21:42:00.009+10:00

A call for help

I was asked today to help with diagnosis of a Cluster Resource Group on a Solaris 10 host after the Oracle service shutdown unexpectedly.

Quick as a flash I checked /var/adm/messages to find that the Resource Group went off-line after the monitor process detected an Oracle fault condition as described by this log entry:

Fault monitor detected error DBMS_ERROR: 4030 DEFAULT Action=RESTART

Intuitively, I guessed that DBMS_ERROR 4030 must be the Oracle error 'ORA-04030' which appears to be correct. A quick google and I have the answer from techtarget.com:

How to avoid the ORA-04030 error? It is out of process memory when trying to allocate 2457618 bytes. Please tell me what system parameters need to be changed.
The ORA-04030 error is telling you that an Oracle process has requested memory from the operating system but the OS does not have any memory to give. Possible solutions include:
Adding more physical memory (RAM) to the database server.
Increasing the temp space on your database server and/or verifying that your temp disk is not full.
Decreasing your database footprint so that it uses less memory. You may want to decrease the SHARED_POOL_SIZE and DB_CACHE_SIZE initialization parameters.

Thanks, but no thanks...

The above is the answer I returned to my colleague, who was less than impressed as it didn't satisfy the question of what actually happened. He went on to point out that there were other errors related to /tmp being full and "wasn't that somehow related?".

Ok he has a point, maybe there's more to this - so back to the drawing board. Putting it all together, I think I've nutted the problem out (my sincere apologies if this is elementary)...

The cluster consists of a collection of zones/containers with Oracle instances running inside. Originally, the implementor chose to have a separate set of static containers on each cluster node with resources failing between them. Each of the containers has a /tmp mounted on the tmpfs filesystem which is of course mapped to swap. To many large files in /tmp actually causes swap starvation and we soon found that this has an incredibly fast and negative reaction on the stability of the cluster.

Almost there

To correct the issue, we've added the 'size=2048' option to the vfstab entry for the swap and dynamically resized the /tmp. So here it is - my point at last - I found some very tricky notes on how to resize /tmp on the tmpfs filesystem dynamically without a reboot even.

This is very likely unsupported, and I don't know what it might kill but in our situation, there is no way we are going to get an outage to reboot all of the systems to consistently implement this limit. It turns out that this has caused failures in the past that no-one could quite put a finger on.

See Ilap's tech: Grow /tmp Filesystem (TMPFS) on-line under Solaris 10 for the original low-down of how this all works. If you're game (please test it on your own before doing anything rash on a production environment), here's a script to ease setting a size limit for /tmp within a zone (run from the global zone):

#!/bin/ksh
# Define the required quota as the hex value derived from:
# (No. of GB x 1024 x 1024 / 8)
#
TMP_QUOTA="0x40000"
TMP_UNLIMITED="0xffffffffffffffff"

function getMaxAddress {
echo "$1::print vfs_t vfs_data ::print -ta struct tmount tm_anonmax" mdb -k awk '{ print $1 }'
}

IFS="
"
set -A mountPtrs "`echo "::fsinfo" mdb -k grep "root/tmp" grep "tmpfs" awk '{ print $1 }'`"

for mountPtr in ${mountPtrs}; do
maxAddress=$(getMaxAddress ${mountPtr})

# Uncomment the following to set the value
#echo "${maxAddress}/Z ${TMP_QUOTA}" mdb -kw

# Uncomment the following to read back the value
#echo "${maxAddress}/J" mdb -k
done

Oracle Startup Scripts

2010-05-27T08:56:00.013+10:00

Please note that this post has no original work of my own - I am posting this because I found it to be a useful example of a set of KSH scripts that save a lot of fiddly work starting Oracle instances. The scripts are used to automatically start Oracle instances on Solaris 9 and HP-UX and were originally created by a colleague.

Script 1 - Start them all...

This script is the controlling script that parses the oratab file to locate all instances that are 'enabled'. Enabled means that there is a 'Y' in the last column.

NOTE:

Both of the scripts shown here require that the oratab file is correctly formatted and maintained, the listener is called LSNR_%SID% and the owner of the Oracle binaries is the database owner or 'service account'.

#!/bin/ksh
#
#  Start/Restart all instances.
#
ORATAB=/var/opt/oracle/oratab

if [[ ! -f $ORATAB ]]; then
echo "Oracle not configured on this host."
else

# Produce list SIDs - exclude SIDs where autostart is 'N'
ORA_SIDS=`grep -v ^\# ${ORATAB} | awk -F':' ' $3 !~ /N/ {printf("%s ",$1);}'`

# Start each SID 1 at a time
  for SID in ${ORA_SIDS}; do
    # Determine ORACLE_HOME
    ORACLE_HOME=`grep "^${SID}:" ${ORATAB} | awk -F':' '{printf("%s",$2)}'`
    export ORACLE_HOME
    if [[ -d ${ORACLE_HOME} ]]; then
        ORACLE_OWNER=`ls -l ${ORACLE_HOME}/bin/oracle | awk '{print $3}'`
        export  ORACLE_OWNER

        # Is this instance still running
        NUM_PROCS=`ps -ef | grep -v grep | grep -i ora_pmon_${SID}$ | wc -l`
        if [[ ${NUM_PROCS} -ne 0 ]]; then
           echo "`date` ERROR: Need to manually shutdown $SID"
        else
           # Start SID
           ORACLE_SID=$SID
           export ORACLE_SID
           su $ORACLE_OWNER -c "`dirname $0`/ora_start_sid $SID"
        fi
    fi
  done
fi
#

Script 2 - Start an instance

This script is called by the above to start an individual named oracle instance - it can also be used separately to do the same.

#!/bin/ksh
#
#       Start any Oracle Instance.
#
#       Parameter 1 (opt) oracle_sid
#                 2 (opt) startup in progress flag
#

ORATAB=/var/opt/oracle/oratab

# Use the current sid if not specifically provided
if [[ $1 != "" ]]; then
ORACLE_SID=$1
export ORACLE_SID
fi

# Set required Oracle Environment variables
ORACLE_HOME=`grep "^${ORACLE_SID}:" ${ORATAB} | awk -F':' '{printf("%s",$2)}'`
export ORACLE_HOME
ORACLE_OWNER=`ls -l $ORACLE_HOME/bin/sqlplus | awk '{print $3}'`
export ORACLE_OWNER
ORACLE_BASE=`echo $ORACLE_HOME | cut -d "/" -f 1-3`
export ORACLE_BASE

# Assume if an instance specific version of the listener.ora exists it's the right one.
if [[ -f ${ORACLE_HOME}/network/admin/listener.ora ]]; then
TNS_ADMIN=${ORACLE_HOME}/network/admin
export TNS_ADMIN
fi

# if we are root - call self and run as correct user
if [[ "`id -u`" -eq 0 ]]; then
su $ORACLE_OWNER -c "$0 $1 $2"
exit
fi

# set defaults
umask 2
JOB_ST=0

# Set up flag and use as temp file
if [[ $2 != "" ]]; then
TMP=$2
touch $2
else
TMP=/tmp/oradata.$$
fi

# put the correct Oracle_home first in the path just in case.
PATH=${ORACLE_HOME}/bin:${PATH}

# Determine the correct connection method (svrmgrl does not exist in Oracle 9)
if [[ -f $ORACLE_HOME/bin/svrmgrl ]]; then
SQLDBA=$ORACLE_HOME/bin/svrmgrl
CONNECTSTR="connect internal"
else
SQLDBA="$ORACLE_HOME/bin/sqlplus /nolog"
CONNECTSTR="connect / as sysdba"
fi

echo "`date` Starting Oracle Instance ${ORACLE_SID}"
$SQLDBA <<> $TMP
$CONNECTSTR
startup
exit;
_EOF_

# Check logs
grep  'ORA-' $TMP > /dev/null
if [[ $? -eq 0 ]]; then
echo "ERROR-Starting ${ORACLE_SID}."
cat $TMP
JOB_ST=1
else
echo "`date` Oracle Database ${ORACLE_SID} started successfully"
$ORACLE_HOME/bin/lsnrctl start LSNR_${ORACLE_SID}
fi

# Start dbconsole (Oracle10.x)
ls -d $ORACLE_HOME | grep -E '10.1|10.2' > /dev/null
if [[ $? -eq 0 ]]; then
echo "Starting Oracle EM dbconsole"
$ORACLE_HOME/bin/emctl resetTZ agent
$ORACLE_HOME/bin/emctl start dbconsole
fi

#       Cleanup and exit
rm -f $TMP
exit $JOB_ST
#

Replicated Hitachi SAN Storage for Microsoft Fail-over Clusters

2010-05-23T18:37:00.010+10:00

Introduction

This article describes the method used to implement a stretch cluster, built on top of a Microsoft Windows 2003 Server connected to an HDS SAN.

The term 'Stretch Cluster' describes the situation where the storage behind the Cluster Disk Resource(s) is replicated to a SAN at a separate site. By separating the cluster nodes between sites and replicating the back-end storage to a SAN Array local to each node, a higher level of availability can be achieved.

The method used to replicate storage is very much dependent on the tools provided by the storage vendor and usually come with some associated licensing or consultation costs.

History

A few years back, our company looked at the prospect of modifying our Microsoft Cluster design to include 'Stretch Clusters' to provide high-availability for the file and print servers. Initially, the applications was deemed to be unimportant and not requiring the extra costs associated with the Tier 1 storage and Truecopy licenses required.

With the explosion of storage requirements for file and print and other clustered applications within the organisation, the situation was recently reviewed. I had seen a colleague implement the concept on HP-UX and also personally configured the same thing for a Solaris cluster, so thought it wouldn't be too difficult. I was right and wrong at the same time...

The HP-UX implementation was built using custom scripts, where Solaris has built-in support for HDS and EMC. Both Operating Systems have there challenges and Windows is no exception.

Connecting the Dots

There are a number of ways to achieve the functionality we required, but for flexibility and ease of implementation, a solution based on a custom 'Generic Script' resource was devised.

I'm assuming that the reader is familiar with most of the building blocks required to build a cluster with replicated, SAN attached storage. Typically, connecting to the secondary copy of the replicated storage is a manual process that is included in a backup or some recovery procedure. What a stretch cluster provides is automatic fail-over to the replicated storage, and reversal of the replication stream to make the LUNs writable at the active end after fail-over.

Essentially, the steps include:

Configure a basic cluster with shared storage on a single SAN Array (not covered here)
Add replicated LUNS using the usual truecopy configuration for your HDS SAN (also, not covered here)
Write a Generic Script Resource handler to manage the interactions with the SAN
Add the Generic Script Resource to the same Resource Group as one or 'Physical Disk' resources backed by replicated SAN storage.
Make the 'Physical Disk' resource(s) with a dependency on the Generic Script resource.

Writing a VBScript Based Resource Handler

The Microsoft notes on creating a Script Resource are included in the References section below and I don't want to go over the same content. What I would like to share is the main pitfalls that snagged my first attempt at this and some tips to manage them.

Pitfall #1 - Over-confidence
I decided it was simple before I began and launched into a full-featured prototype that I tested as a stand-alone script. The idea was that I would make it work and then just plug it in - hey presto! Trust me on this - start simple and work your way up.

Pitfall #2 - The 'WScript' Object
Having written a lot of VBScript over the years, I am very comfortable with WScript.Sleep, WScript.CreateObject, WScript.Echo etc... The WScript Object does not exist.

NOTE: This is not to say that you can't use 'set x = CreateObject("WScript.Shell")', because you can - instantiating other objects is a whole other thing.

Pitfall #3 - The KISS principal
This is along the lines of Pitfall #1 with a more specific target - keep things simple in regard to global declarations at the top of the Script. Using constructs like:

Dim A : A = "Hello World"

are allowed in VBScript if you want to save some space, but it doesn't seem to work under the Generic Script Resource parser.

If your find that you can't even create the script resource in the Cluster Administration utility and are getting an error #80020009, you've either got syntax errors or are using something 'tricky' like the above.

Pitfall #4 - Logging
Use the cluster log effectively and watch it closely when attempting to create or change the state of your shiny new resource script. The Resource.LogInformation in-built function allows you to write directly to the cluster log as defined in your cluster configuration.

NOTE: By default the cluster log entries are written to C:\Windows\Cluster\cluster.log.

The End Result

The end result of all my labour comes in the form of a Windows Script Component to encapsulate the HDS horcm CCI utiliy logic (this is the command-line interface into HDS Storage) and a 'Generic Script Resource' handler script.

Most of the horcm logic is ported from the HP-UX solution that has been running for years in a production environment, so the concept is well tested.

None of the supplied scripts have any warranty or implied fit for purpose - feel free to use them as examples of how similar tasks might be done in your environment.

Download wcsTCutil components from here.

References

Who's doing what to whom

2010-02-27T10:25:00.003+10:00

While 'playing' around with the Resource Pools facility of Solaris 10, I restarted the pools and pools/dynamic services to see the effect. Not surprisingly, all the zones that had an associated resource pool were reset to use the default pool. Not such a bad thing in the context of a lightly loaded test machine - a heavily used production system may have a different impact.

To reset the pools to the original configuration is easy enough using the poolbind command:

poolbind -p Zone_pool -i zoneid myzone

What I didn't see straight away was how to answer the question 'Which resource pool(i.e. which processor set) is associated with each zone at the moment?'. The answer can be found using the command:

ps -eZ -o zone,pset | sort -u

The above command lists the zone and pset (processor set) fields for each process and then removes duplicates.

Alarm, Alarm!!! - Solaris Swap is exhausted - Part 2

2010-01-24T23:04:00.009+10:00

The Test
In Part 1 of this topic, I stated that I did some testing. The testing was to confirm the state of swap on a Solaris host and validate the use of specific SNMP values to correctly monitor when swap was exhausted.

Read on to see how this was done and why it's good to do your own testing and see that documentation is not always the last word.

The environment
All of the hosts used are virtual hosts running on VMWare Server which may be limiting in some load testing scenarios. In this case it's good enough and quite easy to exhaust available memory.

The monitoring host is a CentOS 5.4 installation with net-snmp, net-snmp-libs and net-snmp-utils installed. This could be any operating system running Net-SNMP, I used CentOS as that's where I am installing Nagios and it has the watch utility installed.

The monitored host has OpenSolaris snv_111b installed.

Testing
Start two separate sessions on the monitored OpenSolaris host and one on the CentOS host to remotely observe the SNMP results.

On the CentOS host run:
watch snmpwalk -v2c -c public sol .1.3.6.1.4.1.2021.4

On one of the OpenSolaris hosts, run:
vmstat -S 5 2000

Now, to consume the memory on the OpenSolaris host, I use variations of:
stress --vm 1 --vm-bytes 640M -v

As I only have 512MB allocated to my virtual machine, 640M is adequate to stress the host. Increase the vm or vm-bytes values to start more processes or use more memory for each process respectively. If you find that the command fails, you're probably using too much memory or creating too many processes.

Results
Watching the various screens you should see the following things happening:

memTotalFree.0 will track vmstat's swap
memAvailReal.0 will track vmstat's free value.
When vmstat's swap and memTotalFree drop below the memMinimumSwap.0 threshold, the flag memSwapError.0 is set to 1 and the memSwapErrorMsg.0 states that the system is running out of swap space.

According to the documentation for vmstat and Net-SNMP:

swap is available swap space (Kbytes)
free is the size of the free list (Kbytes)
memTotalFree is
The total amount of memory free or available for use on this host. This value typically covers both real memory and swap space or virtual memory.
.
memAvailReal is
The amount of real/physical memory currently unused
or available.

These numbers don't appear to line up for me, maybe they do on different combinations of versions or platforms. Feel free to comment if you have some revelations.

References

Stress Testing Tool called 'stress'
CentOS is a derivative of Red Hat Linux.
OpenSolaris is a Sun Microsystems OpenSource Project.
Net-SNMP ucdavis MIB reference

Alarm, Alarm!!! - Solaris Swap is exhausted - Part 1

2010-01-24T22:19:00.011+10:00

The Problem
I recently witnessed a Solaris server die a painful death by memory starvation, which was credited to an un-tuned Oracle 11g instance having it's way.

The obvious questions were asked after the event, including - 'How do we stop this happening again?'. One suggestion was to monitor all the Solaris hosts to ensure that, in the event that swap is running low, we will at least know about it before the system falls in a heap (no pun intended).

So, how do we do this?

The approach
Nagios is the tool of choice in this case, which implies SNMP Monitoring by default (for this particular organisation). Some testing led me to the NET-SNMP UCDavis MIB values:

memSwapError (.1.3.6.1.4.1.2021.4.100.0)
memSwapErrorMsg (.1.3.6.1.4.1.2021.4.101.0)
memAvailSwap (.1.3.6.1.4.1.2021.4.4.0)
memMinimumSwap (.1.3.6.1.4.1.2021.4.12.0)

According to the Net-SNMP notes, memSwapError and memSwapErrorMsg are triggered when 'memAvailSwap is less than the desired minimum (specified by memMinimumSwap)'. In Part 2 of this topic, I'll show how to check that this is actually what happens on your configuration.

Unexpected Fruit
The main point of this blog is to highlight where/how to set memMinimumSwap which eluded me for a while. At first it appeared that this value is a hard-coded value as evidenced by a sampling of a number of different hosts. The value returned is always 16000 which turns out to be the default.

Reading throught the Net-SNMP source code, you find that a more meaningful value can be set in snmpd.conf using the swap directive to supply a number of kilobytes. e.g. To change from the 16MB default to 32MB, use:

swap 32000

Now I can use Nagios and the check_snmp plug-in to monitor the value returned by memSwapError - if it's 0, there are no problems; if it returns 1, the system has fallen below the defined threshold and is in some trouble.

References:

Nagios
Net-SNMP MIB Reference
Solaris 10 snmpd.conf - /etc/sma/snmp/snmpd.conf

Booting a CentOS network install from a USB Stick

2009-12-06T16:57:00.009+10:00

I have recently been playing with CentOS 5.4 with a view to test out some Bare Metal recovery solutions. Rather than cut a CD, I thought I'd try booting from a USB stick which, after a bit of fiddling turned out to be not too hard. There are a number of descriptions of how to do this on the net, each with there own twist.

The purpose I had in mind is a direct replacement for the netinstall CD provided with the CentOS distribution CD set. I intended to use the USB stick to allow me to easily start a kickstart process.

Preparation
To start, you need access to the netinstall cd. Ok, so this sounds a bit counter intuitive, if I've got the CD why not just use that - well, I have the CD in ISO form and play around with VMWare a bit, so, the ISO is just mounted as a Virtual CD. You can also do this using Microsoft's, Virtual CD -ROM software.

You also need to have a working copy of syslinux installed - I tend to use the syslinux provided with the CentOS distribution also.

To install the syslinux tools use the command:

# yum install syslinux

Of course, you also need a sacrificial USB stick (see Warning below!). Using syslinux, we are going to update the boot block and add some files. In theory, this should'nt adversely affect the utility of the USB stick for other purposes, but just in case, backup any files that are already there.

Method
To make a bootable USB stick for use with a kickstart installation, do the following:

Mount the USB stick on the computer with syslinux installed.

NOTE:
CentOS automatically mounts the USB stick under /media - the mount point changes depending on the label applied to the USB stick with the default being /media/disk.

Update the boot block using: syslinux /dev/sdb1

NOTE:
Be sure to substitute the appropriate device name for your installation i.e. /dev/sdb1 was derived by executing 'df' and using the device name associated with the mount point '/media/disk'.

Copy files from the netinstall CD (mounted as /media/CentOS) to the USB stick using:
# cp /media/CentOS/syslinux/* /media/disk
Rename isolinux.cfg to syslinux.cfg using:
# mv /media/disk/isolinux.cfg /media/disk/syslinux.cfg
Unmount the USB stick using:
# umount /media/disk

Congratulations, you now have a bootable USB stick that is a direct replacement for the netinstall CD provided with the CentOS distribution.

To boot the installation, configure the target computer to boot from the USB stick using the boot settings in the BIOS.

Extra information

The above works great as a starting point for your own custom kickstart boot disk. I tweak it a little to save myself a bit of messing around at the boot prompt by editing the syslinux.cfg file as follows:

Change the default boot option to ks i.e. replace 'default linux' with 'default ks'.
Change the timeout to a shorter period by changing 'timeout 600' to 'timeout 100'.
Update the arguments for the ks boot option to include the path to the kickstart configuration file e.g. replace 'append ks initrd=initrd.img' with 'append ks=ftp://path_to_ks_file/ks.cfg init=initrd.img'

WARNING:
I trashed my USB stick configuration a couple of times (yes, it took more than one time for me to realise), because I had the line 'clearpart all' in my ks.cfg. Be sure to specifically list the drives that are OK to re-partition using something like 'clearpart --drives=sda,sdb --initlabel' which clears only the first two SATA drives.

NOTE:
If you did get caught like I did, you can use the Panasonic SDFormatter to reformat the USB stick to reset the whole process and start again.

References

Cheap as chips backups

2008-07-25T15:39:00.004+10:00

I was playing backups on the week-end and found that the combination of Robocopy and Volume Shadown Copies is great for a no budget situation.

The process is outlined here: Egghead Cafe Tutorial

To make it work, you need robocopy (Windows Resource Kit Tools), the VSHADOW.EXE utility found in the Volume ShadowCopy Service SDK and last but not least, a utility called dosdev.exe that connects a shadow copy snapshot to a drive letter to allow easy access to the files.

Possible uses: Replicating Virtual Machine drives, at home backup ????

Linux and the health of your disk

2008-05-26T21:29:00.003+10:00

The CentOS NAS has been singing along for over a month now with no performance problems that I'm aware of. The trick to this is that I've only the word of the people using the system that it's working as expected - so what metrics are important when checking the performance of the disk on the NAS.

Because the system is an iSCSI based solution, the system connect to the NAS and use the iSCIS LUNs like local disk. This presents a couple of different ways to monitor disk performance which are both necessary to get a complete picture of how things are going.

Disk IO on the NAS Server
To check the local performance, I have employed the help of the System and Server Status pages of the Webmin Management UI. A simple script (being a windows guy doesn't help here) was written to return an error if the Disk IO is above a pre-determined threshold.

I don't like to re-invent things so I checked what Google has discovered from other peoples ponderings and came up with a few interesting bits:

Which led to the following script:


#!/usr/bin/ksh
# SCRIPT:       /usr/local/bin/check_iostat
# VERSION:      1.0
# CREATED:      26/05/08
# LAST UPDATED: 26/05/08
# AUTHOR:       Adrian Wood
# DECRIPTION:   Check the iowait value returned by the iostat command.  A high value for iowait indicates
#               that the disk io subsystem is spending too much time waiting to service requests
#               i.e. the system is io bound.# PARAMS:       None
#
# Inspired by examples at: http://www.weborial.com/how-to/iowait.shtml
#
# CHANGES:

IOSTAT="/usr/bin/iostat -c"     #The iostat command to run
THRESHOLD=40                    #The threshold that is considered dangerous (in whole numbers)

# function to extract the fourth token of the last line of the
get_iowait() {
   $IOSTAT  grep -v avg-cpu  grep -v Linux  grep "."  {
        while read line; do
                iowait=(` echo $line  cut -d ' ' -f4 `)
        done }
}

#Get the current iowait value from iostatget_iowait
#if the value returned is greater than the defined threshold, flag an alarm.
if [ $iowait -gt $THRESHOLD ] ; then
        echo WARNING: The iowait value of $iowait has exceeded the defined threshold of $IOWAIT_THRESHOLD.
        echo Some investigation into the performance of the disk i/o subsystem is required to
        echo determine if this condition should and/or can be corrected.
        exit 1
else
        exit 0
fi

To use the above, copy the text into a script on your CentOS server with Webmin installed, make it executable using the chmod +x command and schedule it to run using the Webmin System and Server Status interface.

Notes on the System and Server Status menu of webmin can be found at: http://doxfer.com/Webmin/SystemAndServerStatus

Disk IO from an iSCSI Client Perspective

I'm assuming a windows system for this section of the discussion as a Linux system would be treated the same as above. The readily available method for monitoring the disk performance on a Windows System is to crank up Perfmon. For real-time analysis, that's Start - Run - Perfmon ; or for collecting information for later, try the Performance Logs and Alerts snap-in of the Computer Management console (Start - Run - compmgmt.msc).

The important counters to check include:

Avg. Disk sec/Write and Avg. Disk sec /Read
Both of these counters are found under Physical and Logical Disk counters categories. A value of greater than 25ms over a significant period means that the disk is running slow.
Disk Transfers/sec
If this value falls below 80 I/O's per second when the above thresholds are exceeded, there may be too many LUN's using the same disk on the NAS.

For more information on thresholds and counters for Windows Monitoring, see the article at http://technet.microsoft.com/en-us/library/cc296652.aspx describing the PAL tool.

LDAP Authentication for Tomcat 4.x Administration

2008-05-15T13:17:00.008+10:00

I occasionally have a need to restart an application running in a number of Tomcat environments that I inherited. The default installation (for version 4.1 at least) doesn't seem have an active administrative account and is tied to a local users .xml file when it does.

Tomcat can be configured to authenticate against the local Active Directory which is much easier for global administration and for removing the need to remember yet another password. Once configured, AD Authentication can be used for both application and administration access.

The following steps are what I use to configure the authentication for the Tomcat installations that I manage:

Enabling LDAP Authentication

Download ldap.jar to the server\lib directory
Add the following section to the server.xml file:

Connecting the application

Before the AD Authentication can be used, the application needs to be configured to refer to the appropriate group. This is achieved via the web.xml file that is present in the WEB-INF directory of the deployed application. For the built-in manager and admin applications it seems to be fine to directly edit these, for other applications, it may be best to refer to your deployment process to ensure that it won't get over-written.

Locate the web.xml file (eg. \server\webapps\admin\WEB-INF)
Find the ... tags and replace the role name with the name of the AD group to be used to control access. There should be two of these, one in a security-role section and one in a auth-constraint section.

Important points

After all of the configuration steps are completed, a restart of Tomcat is required to activate the changes.
All of the bits that are site-specific are in bold eg. userPassword is a litteral string that does not change.
For the above to work, the account that will be used must be in the OU as defined by the userPattern parameter and the account must be a direct member of the group defined in the role-name parameter of the application.
The name of the group and service account needs to be the Pre-Windows 2000 name of the group or account.
The account referred to as service account only needs the privilege to query the AD, no changes are made - don't give this account extra privilege as the password is in plain text in the Tomcat configuration file.

References

The following references were used to compile the working configuration:

What type of Computer have you got?

2008-04-21T15:22:00.010+10:00

I was asked today how would I identify the type of computer a script or batch file is running on. Of course, we're talking about a Windows platform and if you do a search on Google you'll find that WMI will give you the answer but most references talk about how to do it using VBScript (a couple of these references are listed below). When trying to do this from a batch file on Windows for XP and above, you can use WMIC.exe to do the same thing, as follows:

@echo off
for /f "delims={}" %%i in ('"wmic systemenclosure get chassistypes | find
"{""') do set chassisType=%%i

If %chassisType% geq 9 if %chassisType% leq 10 goto Laptop
If %chassisType% geq 3 if %chassisType% leq 4 goto Desktop

::Not specifically looking for this type.
Echo Chassis Type is: %chassisType%
goto end

:Laptop
Echo Running on a Laptop
goto end

:Desktop
Echo Running on a Desktop
goto end

:end

The values for the Chassis Type are:
1 - Other
2 - Unknown
3 - Desktop
4 - Low Profile Desktop
5 - Pizza Box
6 - Mini Tower
7 - Tower
8 - Portable
9 - Laptop
10 - Notebook
11 - Hand Held
12 - Docking Station
13 - All in One
14 - Sub Notebook
15 - Space-Saving
16 - Lunch Box
17 - Main System Chassis
18 - Expansion Chassis
19 - Sub Chassis
20 - Bus Expansion Chassis
21 - Peripheral Chassis
22 - Storage Chassis
23 - Rack Mount Chassis
24 - Sealed-Case PC

For examples on how it's done with VBScript, see the references below.

References:
Microsoft Windows 2000 Scripting Guide

Hey, Scripting Guy Article

Windows 2008 Resources

2008-04-19T10:33:00.004+10:00

While looking into Windows 2008, I have come across a few handy resources which I hope to add to this list for handy access:

eBooks

Zipping through your files

2008-04-17T21:45:00.003+10:00

I came across the situation yesterday where I have a bunch of directories full of files, being added to constantly and the owner of the directory insists that all the files are required. This is not such a problem as the files aren't particularly large. The application producing the files is having lots of issues with performance on the disk and finding a file has become a huge problem as the directories typically contain anywhere from 50 to 1,000,000+ files.

It immediately occured to me that the delete command has a /s parameter that would be handy, if only the auditing were turned off - it was a pleasant day dream.

So, the current thought is that I just need to clean things up a little and I've posted the basis for the clean-up exercise below. A script that uses the info-zip command line utility zip.exe to run through the directory, adding anything older than the current interval specified. It's not the speediest process for the number of files in question but it works well enough and can be scheduled to run daily which will keep things trim.

'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
' Description: Zip files into separate daily, weekly or monthly zip files
'
' Requirements: VBScript and zip.exe from http://www.info-zip.org/
'
' Usage: cscript //nologo ZipArchive.vbs [/d/w/m] /p:directoryToArchive [/a:directoryForZips] [/v]
'

'Set the locale to get a predictable Date format
setlocale "en-au"

'Global Settings
Const PATH_UTIL = "C:\Utilities"
Const CMD_ZIP = "%UTILPATH%\zip.exe -D -m %ZIPFILE% %FILETOZIP%"

Const INTERVAL_DAILY = 1
Const INTERVAL_WEEKLY = 2
Const INTERVAL_MONTHLY = 3

'FileSystem and Shell objects
Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objShell = CreateObject("Wscript.Shell")

'Defaults
strTargetPath = "C:\Temp"
intInterval = INTERVAL_DAILY

'Process any arguments provided
set objArgs = WScript.Arguments

'The source path
If objArgs.Named.Exists("p") Then strTargetPath = objArgs.Named("p")

'The destination for storing zips.
If objArgs.Named.Exists("a") Then
strArchivePath = objArgs.Named("a")
Else
'If nothing provided, assume an Archive directory directly under
'the source directory.
strArchivePath = objFSO.GetFolder(strTargetPath) & "\Archive"
End If 'objArgs.Named.Exists("a")

'The interval for grouping of files into a single zip - if more than
'1 interval is specified, the most specific is used.
If objArgs.Named.Exists("m") Then intInterval = INTERVAL_MONTHLY
If objArgs.Named.Exists("w") Then intInterval = INTERVAL_WEEKLY
If objArgs.Named.Exists("d") Then intInterval = INTERVAL_DAILY

bVerbose = objArgs.Named.Exists("v")
set objArgs = Nothing

'Determine which date is the oldest to keep
Select Case intInterval
Case INTERVAL_DAILY : dteRetainFrom = CDate(Day(Now) & "/" & Month(Now) & "/" & Year(Now))
Case INTERVAL_WEEKLY : dteRetainFrom = CDate(DateAdd("d", Now, 1-DatePart("w", Now)))
Case INTERVAL_MONTHLY : dteRetainFrom = CDate("01/" &
Month(Now) & "/" & Year(Now))
End Select 'intInterval

'Check Archive Directory exists
If Not objFSO.FolderExists(strArchivePath) Then
WScript.Echo "Archive Directory missing: " & strArchivePath
WScript.Quit
End If 'Not objFSO

'Dump anything older than the first of the month into an appropriate
'zip file in the Archive Directory.

If bVerbose Then WScript.Echo "Setting current directory to: " & strTargetPath
objShell.CurrentDirectory = strTargetPath
intCount = 0

'Loop through the files adding to the appropriate zip file as you go.
For Each objFile in objFSO.GetFolder(objShell.CurrentDirectory).Files
dteCreated = objFile.DateCreated
intCount = intCount + 1
If DateDiff("d", dteRetainFrom, dteCreated) < 0 Then
Select Case intInterval
Case INTERVAL_DAILY : strZipDate = Year(dteCreated) & Right("00" & Month(dteCreated), 2) & Right("00" & Day(dteCreated), 2)
Case INTERVAL_WEEKLY : strZipDate = Year(dteCreated) & "_" & Right("00" & DatePart("ww", dteCreated), 2)
Case INTERVAL_MONTHLY : strZipDate = Year(dteCreated) & Right("00" & Month(dteCreated), 2)
End Select 'intInterval
strZIPFile = strArchivePath & "\Archive_" & strZipDate & ".zip"
strCmd = Replace(CMD_ZIP, "%ZIPFILE%", strZIPFile)
strCmd = Replace(strCmd, "%UTILPATH%", objShell.ExpandEnvironmentStrings(PATH_UTIL))
strCmd = Replace(strCmd, "%FILETOZIP%", objFile.Name)

If bVerbose Then
WScript.Echo intCount & ". Archiving " & objFile.Name & " to " & strZIPFile
Else
'Display a count of files as the are processed.
WScript.StdOut.Write vbCR & intCount
Wscript.sleep 200
End If

objShell.Run strCmd, 0, true

End If 'DateDiff
Next 'objFile

A CentOS OS Installation

2008-04-17T20:13:00.003+10:00

The build I used for my CentOS based NAS is pretty simple when all the kinks are worked out, I thought publishing the detail may help someone else get through a little more quickly.

Hardware Platform
The following system specification was used to build and test these installation notes:

System: ASUS RS260-E4/RX8
CPU: 2 x Dual Core 3.00GHz Intel(R) Xeon(TM) CPU's
Memory: 4Gb
Disks: 8 x 500Gb SAS
RAID: MegaRAID 8300XLP
Network: 2 x 1Gb Adapters

The choice of ASUS was not mine - I am more used to the tier 1 providers (HP, Compaq, IBM) and found the hardware a little harder to work with, specifically trying to find drivers for this and the Windows 2005 Virtual Server.

OS Installation
None of the following is very technical or challenging, but it is the way I did it:

Boot from the CentOS distribution CD (No. 1 of 7)
At the boot: prompt, press the Enter key to initiate a graphical mode installation
If this is the first time the CD has been used, press Enter to test the media:

If you chose to test the media, press Enter to initiate the Test and wait for the test to complete.

If errors are displayed, the CD should be replaced as the installation may not complete successfully, otherwise, press Enter to eject the CD.

If the CD is out at this point, push it back in and continue with the installation
Tab across to Skip or Continue and press Enter.
At the Welcome Screen for the GUI Mode Installation, click Next
Select English (English) as the language to use during installation.
Select the U.S. English Keyboard option and click Next

NOTE:If this is a re-installation, then carefully read the options presented and choose the most appropriate option. These instructions were not intended for use with an Upgrade process.
On the partitioning screen, select Create custom layout. and select the disk drives to be used for this installation (if there’s only 1 it will be preselected) and click Next
From the Disk Setup screen:

Click New and set the values to be:
- Mount point: /boot
- File System Type: ext3
- Allowable Drives: sda
- Size: 100Mb
- Fixed Size

Click New and set the values to be:
- File System Type: swap
- Allowable Drives: sda
- Size: 2048Mb
- Fixed Size

Click New and set the values to be:
- Mount point: /
- File System Type: ext3
- Allowable Drives: sda
- Size: 4096Mb
- Fixed Size

Calculate: [Remaining Disk Capacity - 512MB]

Click New and set the values to be:
- File System Type: physical volume (LVM)
- Allowable Drives: sda
- Size: [value calculated above]
- Fixed Size
Double-click the Free partition of sda and set the Mount point to /cluster
Double-click each of the remaining Free partitions and change the File System Type to be physical volume (LVM)
Click LVM:
- Set the Volume Group Name to: vg0
- Ensure all Physical Volumes are selected.
- Click OK
Ensure the option for The GRUB boot loader will be install on ... is selected, click Next
From the Network Configuration screen:

Select eth0 and click Edit
- De-select Use dynamic IP configuration (DHCP)
- De-select Enable IPv6 support
- Ensure Activate on Boot is selected
- Enter the IP Address (Address) and Subnet mask (Netmask) for the Storage Subnet
- Click OK to save changes

Select eth1 and click Edit
- De-select Use dynamic IP configuration (DHCP)
- De-select Enable IPv6 support
- Ensure Activate on Boot is selected
- Enter the IP Address and Subnet mask for the Heartbeat Subnet
- Click OK to save changes
Select the Set the hostname manually option and enter the hostname
Click Next, then locate Australia/Brisbane on the world map and confirm that the location is updated from America/New_York after clicking on the map.
Click Next and enter the Root password twice
Click Next and wait for the required software to be catalogued ready for installation.
De-select Desktop - Gnome and select Server - GUI
Click Next to begin installation of software.
After the software is installed, remove the CD and click Reboot.

Openfiler put on the 'one day maybe' list

2008-04-15T20:25:00.000+10:00

So it's a couple of months later and I still haven't posted any observations on Openfiler. The main reason being that I've had to shelve the project and start from scratch. As a result, I didn't have a whole lot of time to be reflecting on my success.

So what went wrong...

For me, the main gotcha was the update mechanism. The basic concept for Openfiler is great - an appliance built from OpenSource components and presented as a complete product. However, the choice of conary for the update manager has me confused. The linux base seems to be of a Redhat flavour, the project started out with CentOS as the base distribution but it doesn't readily support update via RPM.

I'm not a linux guru which only becomes a problem when you need to get down and dirty with the inner workings of linux. This became necessary when I tried to use my shiny new NAS appliance with a Microsoft Cluster. Of course, the first version of the iSCSI Enterprise Target (aka IETD) that is reported to work with Microsoft Clustering is 0.4.15 - the version included with the released Openfiler is (was) of course, 0.4.14. The obvious path was to follow the update process to install the new and required version of IETD. To cut a long story short, the update didn't work, failing to update the kernel module associated with IETD. I tried so many things and ended up with a barely functioning iSCSI Target installation.

In the end, I went back to first principles and looked at what I was actually trying to achieve - an OpenSource implementation of an iSCSI NAS. I now have a working implementation of iSCSI Enterprise Target running on a CentOS platform serving a 64-Bit Windows 2003 Cluster running Windows Virtual Server.

Maybe one day I'll need to revisit the Openfiler solution, but for this exercise I found the 'roll your own' solution to be much less troublesome.

iSCSI for mere mortals

2007-12-19T12:25:00.000+10:00

iSCSI has been around for a while and like most things of this type it's an expensive exercise to set up a good working product. However, thanks to some helpful people willing to share the fruits of their labour there are a number of options available for the price of the hardware and your time.

One of these products is Openfiler - a NAS Server (Network Attached Storage) based on rPath Linux and providing iSCSI, Samba, NFS and Web Services in a ready made package. Obviously, I've been focussing on the iSCSI aspect due to some needs I've been trying to address around providing a clustered storage solution.

So far, so good - I now have two servers configured and running as a clustered solution and I'm working through tuning the performance to achieve speeds equivalent to direct attached storage .

I am hoping to post some observations about the current (2.2) and soon to be released upgrade (2.3) versions of Openfiler.