Fetching data from Hitachi Command Suite with HiCommandCLI, PowerShell and Python

hcs_ps_hosts2ldevs

You may say it’s not worth writing a post on the subject, just use the Command Suite CLI, Luke and that’s all! But sometimes HCS CLI behavior is not so obvious. Telling the truth it may surprise you.

For example, when tryig to fetch information about particular host, we need to submit a “GetHost” command. Also if we want LDEVs information in that output, we have to specify additional “subtarget=LogicalUnit” parameter:

HiCommandCLI GetHost "http://hcs8.server.name:2001/service" "hostname=test" "subtarget=LogicalUnit" -u user -p password

And it will show a long output similar to the example from Hitachi Command Suite CLI Reference Guide (MK-90HC176-19, p. 4-366/4-367):

RESPONSE:
 An instance of Host
 objectID=HOST.39
 name=test
 hostID=39
 capacityInKB=8,097,280
 hostType=3
 managedBy=4
 osType=VMware
 statusOfDBUpdating=0
 virtualizationServerVersion=VMware ESX 4.0.0 build-171294
 virtualizationServerManagerName=manager01
 virtualizationServerManagerIpAddress=10.197.150.27
 List of 1 WWN elements:
   An instance of WWN
     WWN=10.00.00.00.C9.6F.EB.D6
 List of 6 Lu elements:
   An instance of LogicalUnit
     objectID=LU.R600.10037.1008
     devNum=1,008
     emulation=OPEN-V
     devCount=1
     devType=
     capacityInKB=2,000,000
     numberOfLBAs=4,000,000
     path=true
     commandDevice=false
     commandDeviceEx=0
     commandDeviceSecurity=false
     deviceGroupDefinition=false
     chassis=1
     arrayGroup=0
     raidType=RAID5(3D+1P)
     currentPortController=-1
     defaultPortController=-1
     isComposite=0
     trueCopyVolumeType=Simplex
     shadowImageVolumeType=Simplex
     quickShadowVolumeType=Simplex
     universalReplicatorVolumeType=Simplex
     globalActiveDeviceVolumeType=Simplex
     sysVolFlag=0
     externalVolume=0
     differentialManagement=false
     quickShadowPoolID=-1
     universalReplicatorPoolID=-1
     dpType=-1
     consumedCapacityInKB=2,000,000
     dpPoolID=-1
     threshold=-1
     tcaPoolID=-1
     dpPoolVolControlFlag=-1
     managementAreaPoolID=-1

     ... and so on ...

If you think, it looks like a piece of crumpled XML with markup tags omitted, you will be absolutely right! HCS CLI communicates with HCS server using XML format.

So, the question is: “May we see plain XML output in HCS CLI”? And the anwser is: “Yes, sure!” Set -f option to specify format:

hcs_cli_opts

But not for “GetHost” command! When you try it HCS CLI returns the nice error:

KAIC90602-E The option "-f" and the command "GetHost" cannot be specified at the same time.

And it is very weird because of an XML nature of the HCS API protocol!

The same error is for creating a csv table! So we have to do all the thing manually to parse CLI output while preparing our simple storage consumption report.

For me, the easiest way is to write a script and connect with the HCS server directly (skipping an excessive Java HiCommandCLI interlayer) to send commands and receive responses in XML. And here is my example on how to do it in Pyton:

#!/usr/bin/python3

# -----------------------------------------------------------------------------
# "THE BEER-WARE LICENSE" (Revision 42):
# zmey20000@yahoo.com wrote this file. As long as you retain this notice you
# can do whatever you want with this stuff. If we meet some day, and you think
# this stuff is worth it, you can buy me a beer in return Mikhail Zakharov
# -----------------------------------------------------------------------------

# Hitachi Command Suite XML API usage example in Pyton.
 
import base64
import http.client
import xml.etree.ElementTree as ET
 
# ----------------------------------------------------------------------------
hcs_server = 'hcs8.server.name'
login = 'system'
password = 'manager'
command = 'StorageArray'
option = 'all'
 
# ----------------------------------------------------------------------------
b64lp = base64.b64encode((login + ':' + password).encode('ascii'))
 
body = """<?xml version="1.0" encoding="UTF-8"?>
<HiCommandServerMessage>
    <APIInfo version="7.6" />
    <Request>
       <StorageManager> 
            <Get target=""" + '"' + command + '" ' + """option=""" + '"' + option + '" ' + """>
                <StorageArray />
            </Get>
        </StorageManager>
    </Request>
</HiCommandServerMessage>"""
 
headers = {'Content-Type': 'text/xml',
           'User-Agent': 'Deck Eight:One Step:0',
           'Authorization': 'Basic ' + b64lp.decode('utf-8')}
 
httpc = http.client.HTTPConnection(hcs_server, 2001, timeout=30)
httpc.request('POST', '/service/ServerAdmin', body, headers)
resp = httpc.getresponse()
#print(resp.status, resp.reason)
 
data = resp.read()
data = data.decode('utf-8')
 
root = ET.fromstring(data)
for array in root.iter('StorageArray'):
    name = array.get('name')
    hardwareRevision = array.get('hardwareRevision')
    cacheInMB = array.get('cacheInMB')
    totalFreeSpaceInKB = array.get('totalFreeSpaceInKB')
    allocatedCapacityInKB = array.get('allocatedCapacityInKB')
    capacityInKB = array.get('capacityInKB')
    numberOfControllers = array.get('numberOfControllers')
    controllerVersion = array.get('controllerVersion')
    productName = array.get('productName')
    arrayType = array.get('arrayType')
    serialNumber = array.get('serialNumber')
 
    print(name, serialNumber, arrayType, productName, totalFreeSpaceInKB, allocatedCapacityInKB, capacityInKB, controllerVersion, numberOfControllers, cacheInMB, hardwareRevision)
 
httpc.close()

You may prefer Pastebin link to fetch the script: http://pastebin.com/iPdh0yyV

But sometimes we are bound to Windows and its native components. I have spend a day, lerning PowerShell, so here is my script which gets information about all Hosts and their LDEVs registered within the given HCS server:

# -----------------------------------------------------------------------------
# "THE BEER-WARE LICENSE" (Revision 42):
# zmey20000@yahoo.com wrote this file. As long as you retain this notice you
# can do whatever you want with this stuff. If we meet some day, and you think
# this stuff is worth it, you can buy me a beer in return Mikhail Zakharov
# -----------------------------------------------------------------------------

#
# Dump and correlate Hitachi Command Suite Hosts and LDEVs information
#

#
# v1.0	2016.11.30	Initial release
# v1.1	2016.12.16	Storage serial and model/type added

# Usage:
# hcs_hosts2ldevs -hcs_host hcs8.server.name -hcs_user user -hcs_pass password -hcs_cli X:\HCS\CLI\HiCommandCLI.bat -out_csv X:\path\to\your.csv

# Defaul values ---------------------------------------------------------------
param (
    [string]$hcs_host = "hcs8.server.name",
    [string]$hcs_user = "system",
    [string]$hcs_pass = "manager",
    [string]$hcs_cli = "C:\HCS_CLI\HiCommandCLI.bat",
    [string]$out_csv = "HCS-Hosts2LDEVs.csv"
)

# -----------------------------------------------------------------------------
$hcs_url="http://" + $hcs_host + ":2001/service"

Write-Host "Querying HCS data. You have time for a cup of coffee"

$flist = & $hcs_cli $hcs_url GetHost "subtarget=LogicalUnit" -u $hcs_user -p $hcs_pass |
    where {$_ -cmatch "name=|capacityInKB=|osType=|instance|WWN=|displayName=|emulation=|consumedCapacityInKB|commandDevice|arrayGroupName|raidType|externalVolume|dpType|dpPoolID|objectID"}

Write-Host "Processing HCS data. Keep calm and enjoy your drink"
$i = 0
Add-Content $out_csv "Hostname,Host capacity (MB),OS,WWN,LDEV,LDEV capacity (MB),Storage System SN,Storage System Type,LDEV Used (MB),Emulation,Array Group,RAID level,Command Device,External,DP Type,DP Pool ID"
foreach ($ln in $flist) {

    # Show some progress indication
    $i += 1
    if ($i % 10000 -eq 0) {
        Write-Host Row $i : $flist.Length processed
    }    	

    # Split every line to fetch 'variable' and 'value' parts
    $hash=$ln.Trim().Split('=')

    # Fetch Host/WWN/LDEV data and combine everything together to prepare normal table format
    switch ($hash[0]) {
        "An instance of Host" {
            if ($LUN -ne "") {
                $LUNs += $LUN + $SS + $ST + $LUsed + $Emul + $RG + $RGLvl + $CMDDev + $Ext + $DPType + $DPPool
            }

            foreach ($wc in $WWNs) {
                foreach ($lc in $LUNs) {
                    if ("$Hst$OS$wc$lc" -ne "") {
                        Add-Content $out_csv $Hst$OS$wc$lc
                    }
                }
            }

            # Clean variables for the next host
            $Hst = ""
            $LUN = ""
            $OS = ","
            $Emul = ",";

            $WWNs = @()
            $LUNs = @()

            # We are at Host Level: 0
            $l = 0
            break
        }

        "name" {
            $Hst += $hash[1]
            break
        }

        "capacityInKB" {
            # Capacity can be found on Level 0 and Level 2
            switch ($l) {
                0 {$Hst += "," + $hash[1].Replace(".", "")/1024; break}
                2 {$LUN += "," + $hash[1].Replace(".", "")/1024; break}
            }
        }

        "osType" {
            $OS = "," + $hash[1]
            break
        }

        "An instance of WWN" {
            # Go down to WWN Level: 1
            $l = 1
            break
        }

        "WWN" {
            $WWNs += "," + $hash[1].Replace(".", ":").ToLower()
            break
        }

        "An instance of LogicalUnit" {
            if ($LUN -ne "") {
                $LUNs += $LUN + $SS + $ST + $LUsed + $Emul + $RG + $RGLvl + $CMDDev + $Ext + $DPType + $DPPool
                $LUN = ""; $SS = ""; $ST = ""; $Emul = ","; $LUsed = ","; $CMDDev = ","; $RG = ","
                $RGLvl = ","; $Ext = ","; $DPType = ","; $DPPool = ","
            }

            # We are finally at LDEV Level 2 deep
            $l = 2
            break
        }

        "displayName" {
            $LUN += "," + $hash[1]
            break
        }
		"objectID" {
			if ($l -eq 2) {
			# Storage serial
				$SS = "," + $hash[1].Split('.')[2]
			# Storage type/model
				$ST = "," + $hash[1].Split('.')[1]
			}
			break
		}

        "emulation" {
            $Emul = "," + $hash[1]
            break
        }

        "commandDevice" {
            switch ($hash[1]) {
                "false" {$CMDDev = "," + "0"}
                "true" {$CMDDev = "," + "1"}
            }
            #$CMDDev = "," + $hash[1]
            break
        }

        "arrayGroupName" {
            $RG = "," + $hash[1]
            break
        }

        "raidType" {
            $RGLvl = "," + $hash[1]
            break
        }

        "consumedCapacityInKB" {
            $LUsed = "," + $hash[1].Replace(".", "")/1024
            break
        }

        "externalVolume" {
            $Ext = "," + $hash[1]
            break;
        }

        "dpType" {
            $DPType = "," + $hash[1]
            break
        }

        "dpPoolID" {
            $DPPool = "," + $hash[1]
            break
        }
    }
}

# Must process final line as it was not created by the main loop

if ($LUN -ne "") {
    $LUNs += $LUN + $SS + $ST + $LUsed + $Emul + $RG + $RGLvl + $CMDDev + $Ext + $DPType + $DPPool
}

foreach ($wc in $WWNs) {
    foreach ($lc in $LUNs) {
        Add-Content $out_csv $Hst$OS$wc$lc
    }
}

Write-Host "Done."

You may prefer Pastebin or GitHub links to fetch the script.

As a result it creates an csv file.

Just run the command below:

hcs_hosts2ldevs -hcs_host hcs8.server.name -hcs_user user -hcs_pass password -hcs_cli X:\HCS\CLI\HiCommandCLI.bat -out_csv X:\path\to\your.csv

to get the table shown in the picture at the beginning of the post. Hope everything works for you, because if you didn’t add Hosts into Hitachi Command Suite you have to scan for host-groups on every port of all the storage systems registered in the HCS server. But this is a completely different story. Good luck!

UPD 2017.06.01. Here are hcs_hosts2ldevs.ps1 script additions to support TrueCopy, ShadowImage and GlobalActiveDevice fields by <Josef> mailb0x(at)centrum.cz. Everything looks pretty nice, though I didn’t test it:

# Defaul values ---------------------------------------------------------------
param (
    [string]$hcs_host = "",
    [string]$hcs_user = "",
    [string]$hcs_pass = "",
    [string]$hcs_cli = "C:\tmp\hdvm\HiCommandCLI.bat",
    [string]$out_csv = "C:\tmp\hdvm\h2ldev.csv"
)

# -----------------------------------------------------------------------------
$hcs_url="http://" + $hcs_host + ":2001/service"
 
Write-Host "Querying HCS data. You have time for a cup of coffee"
 
$flist = & $hcs_cli $hcs_url GetHost "subtarget=LogicalUnit" -u $hcs_user -p $hcs_pass | 
    where {$_ -cmatch "name=|capacityInKB=|osType=|instance|WWN=|displayName=|consumedCapacityInKB|commandDevice|trueCopyVolumeType|shadowImageVolumeType|globalActiveDeviceVolumeType|raidType|dpPoolID|objectID"}
 
Write-Host "Processing HCS data. Keep calm and enjoy your drink"
$i = 0
Add-Content $out_csv "Hostname,Host capacity (MB),OS,WWN,LDEV,LDEV capacity (MB),Storage System SN,Storage System Type,LDEV Used (MB),Command Device,TC,SI,GAD,DP Pool ID"
foreach ($ln in $flist) {
 
    # Show some progress indication
    $i += 1
    if ($i % 10000 -eq 0) {
        Write-Host Row $i : $flist.Length processed
    }       
     
    # Split every line to fetch 'variable' and 'value' parts
    $hash=$ln.Trim().Split('=')
 
    # Fetch Host/WWN/LDEV data and combine everything together to prepare normal table format
    switch ($hash[0]) {
        "An instance of Host" {
            if ($LUN -ne "") {
                $LUNs += $LUN + $SS + $ST + $LUsed + $CMDDev +$TC +$SI + $GAD + $DPPool
            }
            
            foreach ($wc in $WWNs) {
                foreach ($lc in $LUNs) {
                    if ("$Hst$OS$wc$lc" -ne "") {
                        Add-Content $out_csv $Hst$OS$wc$lc
                    }
                }
            }
 
            # Clean variables for the next host
            $Hst = ""
            $LUN = ""
            $OS = ","

             
            $WWNs = @()
            $LUNs = @()
             
            # We are at Host Level: 0
            $l = 0
            break
        }
         
        "name" {
            $Hst += $hash[1]
            break
        }
 
        "capacityInKB" {
            # Capacity can be found on Level 0 and Level 2
            switch ($l) {
                0 {$Hst += "," + $hash[1].Replace(".", "")/1024; break}
                2 {$LUN += "," + $hash[1].Replace(".", "")/1024; break}
            }               
        }
 
        "osType" {
            $OS = "," + $hash[1]
            break
        }
         
        "An instance of WWN" {
            # Go down to WWN Level: 1
            $l = 1
            break
        }
 
        "WWN" {
            $WWNs += "," + $hash[1].Replace(".", ":").ToLower()
            break
        }
         
        "An instance of LogicalUnit" {
            if ($LUN -ne "") {
                $LUNs += $LUN + $SS + $ST + $LUsed + $CMDDev + $TC + $SI + $GAD + $DPPool
                $LUN = ""; $SS = ""; $ST = ""; $LUsed = ","; $CMDDev = ","; $TC = ","; $SI = ","; $GAD = ","
                $DPPool = ""
            }
             
            # We are finally at LDEV Level 2 deep
            $l = 2
            break
        }
         
        "displayName" {
            $LUN += "," + $hash[1]
            break
        }
        "objectID" {
            if ($l -eq 2) {
            # Storage serial
                $SS = "," + $hash[1].Split('.')[2]
            # Storage type/model
                $ST = "," + $hash[1].Split('.')[1]
            }
            break
        }
		
        "commandDevice" {
            switch ($hash[1]) {
                "false" {$CMDDev = "," + "0"}
                "true" {$CMDDev = "," + "1"}
            }
            $CMDDev = "," + $hash[1]
            break
        }
        "trueCopyVolumeType" {
            $TC = "," + $hash[1]
            break
        }
        "shadowImageVolumeType" {
            $SI = "," + $hash[1]
            break
        }
        "globalActiveDeviceVolumeType" {
            $GAD = "," + $hash[1]
            break
        }
               
        "consumedCapacityInKB" {
            $LUsed = "," + $hash[1].Replace(".", "")/1024/1024 
            break 
        }
           
        "dpPoolID" {
            $DPPool = "," + $hash[1]
            break
        }
    }  
}
 
# Must process final line as it was not created by the main loop
 
if ($LUN -ne "") {
    $LUNs += $LUN + $SS + $ST + $LUsed + $CMDDev + $TC + $SI + $GAD + $DPPool
}

 
foreach ($wc in $WWNs) {
    foreach ($lc in $LUNs) {
        Add-Content $out_csv $Hst$OS$wc$lc
    }
}
 
Write-Host "Done."

UPD 2017.07.20. Multiple “&amp” issues in examples and sources were fixed (hopefully). Pastebin links to the script sources were added.

Advertisements

About mezzantrop

10 years of experience in large SAN and storage environments: mainly Hitachi, HP and Brocade. Now I am a proud SAN/storage IBMer. Empty – expect-like tool author. FreeBSD enthusiast.
This entry was posted in My projects, Storage, Storage Automation and tagged , , , , , , , , , . Bookmark the permalink.

4 Responses to Fetching data from Hitachi Command Suite with HiCommandCLI, PowerShell and Python

  1. Pingback: Online command-line reference for scripting | #define me human

  2. mezzantrop says:

    Here are hcs_hosts2ldevs.ps1 script additions to support TrueCopy, ShadowImage and GlobalActiveDevice fields. See the post for details.

    Like

  3. Chris says:

    Hi,

    When running the script I get the following errors. Have you also seen this ?

    2x Unexpected token and 1x An empty pipe is not allowed ?

    PS C:\Windows\system32> $flist = &amp; $hcs_cli $hcs_url GetHost “subtarget=LogicalUnit” -u $hcs_user -p $hcs_pass |
    At line:1 char:29
    + $flist = &amp; $hcs_cli $hcs_url GetHost “subtarget=LogicalUnit” …
    + ~~~~~~~~
    Unexpected token ‘$hcs_url’ in expression or statement.
    At line:1 char:38
    + $flist = &amp; $hcs_cli $hcs_url GetHost “subtarget=LogicalUnit” …
    + ~~~~~~~
    Unexpected token ‘GetHost’ in expression or statement.
    At line:1 char:97
    + … $hcs_url GetHost “subtarget=LogicalUnit” -u $hcs_user -p $hcs_pass |
    + ~
    An empty pipe element is not allowed.
    + CategoryInfo : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : UnexpectedToken

    Like

    • mezzantrop says:

      Hi Chris, I see the error. And I suppose it is because of broken encoding. I have spotted multiple “&amp” artefacts in my post and in your comment. I have fixed hcs_hosts2ldevs.ps1 quickly (so you can try it again), but other examples also must be rewritten. Examples are updated, also I have returned links to Pastebin.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s