What is the name of a new drive in my FreeBSD box

or a quick way to list GEOM configuration in FreeBSD

Drives related information questions, including that one in the title, may be tricky for a new (and sometimes for an experienced) FreeBSD user, and even Handbook advises to see the “dmesg” output to check the drives when adding new drives:

“Inspect `/var/run/dmesg.boot` to ensure the new disk was found. In this example, the newly added SATA drive will appear as `ada1`.”

Actually, there are many methods and utilities to check drives, and one of the most powerful tool is “camcontrol”. But what if you are dealing with GEOM stack or checking a partition? The solution is to check GEOM configuration:

sysctl -n kern.geom.conftxt

or

sysctl -n kern.geom.confxml

Yes, this is not a panacea, but it can give a hint.

I have written a simple script (geom_show) which can be used to process GEOM configuration and print out basic information about volumes: mediasize, sectorsize, stripesize and etc. It doesn’t not require root privileges and runs without any additional dependencies. And, hah, it makes me feel like an old programmer because it actually consists of pure AWK code and a wrapper in Bourne shell 😉

I hope, it can help someone to make things simpler, and, please, let me know if it doesn’t.

#!/bin/sh

# Copyright (c) 2017 Mikhail Zakharov <zmey20000@yahoo.com>
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright
#    notice, this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.

#
# Show GEOM configuration
#
# Typical usage:
# 	geom_show.sh -l -c DISK
#
# 2017.07.25	 1.0

# Defaults ---------------------------------------------------------------------
ofs=","					# Output Field Separator is "comma"
show_headers=true			# Show headers in output

# ------------------------------------------------------------------------------
usage()
{
	error_code=$1			# Error code on exit() to the caller
	error_text="$2"			# Short error message to print
	usage_help="Usage:
  geom_show.sh -k [-s separator]
	List disk names known by the kernel.
  geom_show.sh -g [-s separator]
  geom_show.sh -g -c CLASS [-s separator]
	List GEOM provider names of all or any particular class.
  geom_show.sh -l -c CLASS | -p PROVIDER [-h] [-s separator]
	Show details of GEOM provider(s). Filter entries by CLASS or exact
	PROVIDER name. Suppress headers with -h key.
  geom_show.sh -d [-s separator]
	Dump raw GEOM configuration.
Use [-s separator] to specify output field separator. Default is comma: (,)."

	[ "$error_text" ] && printf "Error: $error_text\n"
	printf "$usage_help\n"
	exit $error_code
}

# -----------------------------------------------------------------------------
awk="/usr/bin/awk"
sed="/usr/bin/sed"
sysctl="/sbin/sysctl"

# Error messages --------------------------------------------------------------
err_kgld_flags="Options -k, -g, -l or -d cannot be combined together."
err_no_flags="Specify -k, -g, -l or -d option to show disks configuration."
err_cp_flags="Specify -c, -p flag or both."

geom_show='#!/usr/bin/awk -f
# Copyright (c) 2017 Mikhail Zakharov <zmey20000@yahoo.com>
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright
#    notice, this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.
#
# Parse GEOM configuration
#
# Typical usage:
# sysctl -n kern.geom.confxml | /path/to/geom_show.awk -v "class=DISK"
#
# Other options:
#	-v provider="ada0"	Specify GEOM provider to show
#	-v noheader="yes"	Omit header
#	-v short="yes"		Short form prints only providers name(s)
# 2017.07.25	v 1.0
BEGIN {
	FS = ">|<"
	if (!ofs)
		ofs = ","
	OFS = ofs
	gprov = provider
	gclass = class
	if (!gprov && !gclass)
		short = "yes"
	if (gclass && !gprov && !short && noheader != "yes")
		print_header(gclass)
}
function expanded_length(s) { gsub("\t", "        ", s); return length(s) }
function print_header(gclass) {
	common_header = "Class" ofs "Provider" ofs "Mediasize" ofs \
		"Sectorsize" ofs "Stripesize" ofs "Stripeoffset"
	disk_header = common_header ofs "Heads" ofs "Sectors" ofs "RPM" ofs \
		"Ident" ofs "Description"
	md_header =  common_header ofs "Heads" ofs "Sectors" ofs "Compression" \
		ofs "Access" ofs "Type"
	part_header = common_header ofs "Start" ofs "End" ofs "Index" ofs "Type"
	if (gclass == "DISK")
		print disk_header
	else if (gclass == "MD")
		print md_header
	else if (gclass == "PART")
		print part_header
	else print common_header
}
/<name>/,/<\/name>/ {
	depth = expanded_length($1)
	if (depth == 4)
		if ($3 == gclass || !gclass) {
		# We have found requested GEOM Class
			in_class = 1
			current_gclass = $3
		} else
			in_class = 0
	if (in_class && depth == 10) {
		if (gprov == $3) {
			# GEOM class for the given provider is detected
			gclass = current_gclass
			if (gclass && !short && noheader != "yes")
				print_header(gclass)
		}
		# This should be the GEOM Name
		gname = $3
	}
}
/<mediasize>/ || /<sectorsize>/ || /<stripesize>/ || /<stripeoffset>/ {
	if (!short && in_class && expanded_length($1) == 10)
	# These are common, default fields for all providers
			gdata = gdata ofs $3
	next
}
/<fwheads>/ || /<fwsectors>/ {
	if (!short && in_class && expanded_length($1) == 12)
		if (gclass == "DISK" || gclass == "MD")
			gdata = gdata ofs $3
	next
}
/<rotationrate>/ || /<ident>/ || /<descr>/ {
	if (!short && in_class && expanded_length($1) == 12 && gclass == "DISK")
		gdata = gdata ofs $3
	next
}
/<compression>/ || /<access>/ {
	if (!short && in_class && expanded_length($1) == 12 && gclass == "MD")
		gdata = gdata ofs $3
	next
}
/<type>/ {
	if (!short && in_class && expanded_length($1) == 12)
		if (gclass == "MD" || gclass == "PART")
			gdata = gdata ofs $3
	next
}
/<start>/ || /<end>/ || /<index>/ {
	if (!short && in_class && expanded_length($1) == 12 && gclass == "PART")
		gdata = gdata ofs $3
	next
}
/<\/provider>/  {
	if (in_class && expanded_length($1) == 8) {
		if (!gprov || gprov == gname)
			if (current_gclass != "LABEL" &&
				current_gclass != "DEV" ||
				current_gclass == gclass)
					if (!short)
						print gclass ofs gname gdata
					else {
						if (!entry_1)
							printf gname
						else
							printf ofs gname
						entry_1 = 1
					}
		gdata = ""
	}
}
END {
	if (short)
		printf "\n"
}'

# ------------------------------------------------------------------------------
while getopts "kgldc:p:hs:" flag
do
	case "$flag" in
		k)      k_flag=true ;;
		g)	g_flag=true ;;
		l)	l_flag=true ;;
		d)	d_flag=true ;;
		c)	c_flag=true; c_arg="$OPTARG" ;;
		p)	p_flag=true; p_arg="$OPTARG";;
		h)	h_flag=true ;;
		s)	s_flag=true; s_arg="$OPTARG" ;;
		*)      usage 1 ;;
	esac
done

[ ! $k_flag ] && [ ! $g_flag ] && [ ! $l_flag ] && [ ! $d_flag ] &&
	usage 1 "$err_no_flags"
[ $s_flag ] && ofs="$s_arg"

# geom_show.sh -k [-s separator] -----------------------------------------------
if [ $k_flag ] ; then
	[ $g_flag ] || [ $l_flag ] || [ $d_flag ] && usage 1 "$err_kgld_flags"

	$sysctl -n kern.disks | $sed "s/ /$ofs/g"
	exit 0
fi

# geom_show.sh -g [-s separator] | geom_show.sh -g -c CLASS [-s separator] -----
if [ $g_flag ] ; then
	[ $k_flag ] || [ $l_flag ] || [ $d_flag ] && usage 1 "$err_kgld_flags"
	[ $c_flag ] && gclass="$c_arg"

	"$sysctl" -n kern.geom.confxml |
		"$awk" -v ofs="$ofs" -v short="yes" -v class="$gclass" \
			"$geom_show"
	exit 0
fi

# geom_show.sh -l -c CLASS | -p PROVIDER [-h] [-s separator] -------------------
if [ $l_flag ] ; then
	[ $g_flag ] || [ $k_flag ] || [ $d_flag ] && usage 1 "$err_kgld_flags"
	[ ! $c_flag ] && [ ! $p_flag ] && usage 1 "$err_cp_flags"
	[ $c_flag ] && gclass="$c_arg"
	[ $p_flag ] && gprov="$p_arg"
	[ $h_flag ] && noheader="yes" 

	"$sysctl" -n kern.geom.confxml |
		"$awk" -v ofs="$ofs" -v class="$gclass" \
			-v provider="$gprov" -v noheader="$noheader" \
			"$geom_show"
	exit 0
fi

# geom_show.sh -d [-s separator] -----------------------------------------------
if [ $d_flag ] ; then
	[ $k_flag ] || [ $g_flag ] || [ $l_flag ] && usage 1 "$err_kgld_flags"

	geom_dump=`$sysctl -n kern.geom.conftxt | sed -e "s/ /$ofs/g"`
	printf "$geom_dump\n"
	exit 0
fi
#!/usr/bin/awk -f

# Copyright (c) 2017 Mikhail Zakharov <zmey20000@yahoo.com>
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright
#    notice, this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.

#
# Parse GEOM configuration
#
# Typical usage:
# sysctl -n kern.geom.confxml | /path/to/geom_show.awk -v "class=DISK"
#
# Other options:
# 	-v provider="ada0"	Specify GEOM provider to show
#	-v noheader="yes"	Omit header
#	-v short="yes"		Short form prints only provider name(s)

# 2017.07.25	v 1.0

BEGIN {
	FS = ">|<"

	if (!ofs)
		ofs = ","
	OFS = ofs

	gprov = provider
	gclass = class

	if (!gprov && !gclass)
		short = "yes"

	if (gclass && !gprov && !short && noheader != "yes")
		print_header(gclass)
}

function expanded_length(s) { gsub("\t", "        ", s); return length(s) }

function print_header(gclass) {
	common_header = "Class" ofs "Provider" ofs "Mediasize" ofs \
		"Sectorsize" ofs "Stripesize" ofs "Stripeoffset"
	disk_header = common_header ofs "Heads" ofs "Sectors" ofs "RPM" ofs \
		"Ident" ofs "Description"
	md_header =  common_header ofs "Heads" ofs "Sectors" ofs "Compression" \
		ofs "Access" ofs "Type"
	part_header = common_header ofs "Start" ofs "End" ofs "Index" ofs "Type"

	if (gclass == "DISK")
		print disk_header
	else if (gclass == "MD")
		print md_header
	else if (gclass == "PART")
		print part_header
	else print common_header
}

/<name>/,/<\/name>/ {
	depth = expanded_length($1)

	if (depth == 4)
		if ($3 == gclass || !gclass) {
		# We have found requested GEOM Class
			in_class = 1
			current_gclass = $3
		} else
			in_class = 0

	if (in_class && depth == 10) {
		if (gprov == $3) {
			# GEOM class for the given provider is detected
			gclass = current_gclass
			if (gclass && !short && noheader != "yes")
				print_header(gclass)
		}
		# This should be the GEOM Name
		gname = $3
	}
}

/<mediasize>/ || /<sectorsize>/ || /<stripesize>/ || /<stripeoffset>/ {
	if (!short && in_class && expanded_length($1) == 10)
	# These are common, default fields for all providers
			gdata = gdata ofs $3
	next
}

/<fwheads>/ || /<fwsectors>/ {
	if (!short && in_class && expanded_length($1) == 12)
		if (gclass == "DISK" || gclass == "MD")
			gdata = gdata ofs $3
	next
}

/<rotationrate>/ || /<ident>/ || /<descr>/ {
	if (!short && in_class && expanded_length($1) == 12 && gclass == "DISK")
		gdata = gdata ofs $3
	next
}

/<compression>/ || /<access>/ {
	if (!short && in_class && expanded_length($1) == 12 && gclass == "MD")
		gdata = gdata ofs $3
	next
}

/<type>/ {
	if (!short && in_class && expanded_length($1) == 12)
		if (gclass == "MD" || gclass == "PART")
			gdata = gdata ofs $3
	next
}

/<start>/ || /<end>/ || /<index>/ {
	if (!short && in_class && expanded_length($1) == 12 && gclass == "PART")
		gdata = gdata ofs $3
	next
}

/<\/provider>/  {
	if (in_class && expanded_length($1) == 8) {
		if (!gprov || gprov == gname)
			if (current_gclass != "LABEL" &&
				current_gclass != "DEV" ||
				current_gclass == gclass)
					if (!short)
						print gclass ofs gname gdata
					else {
						if (!entry_1)
							printf gname
						else
							printf ofs gname
						entry_1 = 1
					}
		gdata = ""
	}
}

END {
	if (short)
		printf "\n"
}

See the code on the GitHub and on the project page of my blog.

UPD 2017.07.26. There is a nice update on diskinfo by Allan Jude.

Advertisements
Posted in My projects, Storage, Storage Automation | Tagged , , , , , , , | 3 Comments

Excel formulas syntax

Once, I thought embedding SQL requests into a code was the most boring thing I could imagine. Now I’m losing all my marbles while trying to push MS Excel formulas into my Python script.

Posted in Impressions, IRL | Tagged , , , | Leave a comment

Accessing IBM SVC/Storwize devices with Python and SSH

Recently I have posted a simple example of how to fetch storage related data from IBM Spectrum Control with the help of Python and REST API.

Now lets turn the power of Python to access IBM Storwize or SVC system directly with SSH and run “lssystem” command on it. In the example below I use a very nice “paramiko” module, which handles SSH protocol and simplifies the task.

As you can see, nothing difficult here:

#!/usr/bin/python3

# -----------------------------------------------------------------------------
# "THE BEER-WARE LICENSE" (Revision 42):
# zmey20000@yahoo.com wrote this file. As long as you retain this notice you
# can do whatever you want with this stuff. If we meet some day, and you think
# this stuff is worth it, you can buy me a beer in return Mikhail Zakharov
# -----------------------------------------------------------------------------

import paramiko


target = '192.168.1.1'
login = 'mylogin'
password = 'mypassword'
command = 'lssystem -delim \,'


def ssh_exec(command, target, user, password, port=22):
    client = paramiko.SSHClient()
    client.set_missing_host_key_policy(paramiko.AutoAddPolicy())

    try:
        client.connect(target, username=user, password=password, port=port)
    except:
        print('FATAL: Unable to log in')
        exit(1)

    stdin, stdout, stderr = client.exec_command(command)

    error = stderr.read()
    if error:
        error = error.decode('US-ASCII')
        print('Error running the command:{}'.format(error))
        client.close()
        return 0

    data = stdout.read()
    client.close()

    return data.decode('US-ASCII')


lssystem = ssh_exec(command, target, login, password)
print(lssystem)

Posted in IRL, Storage, Storage Automation, Tips & tricks | Tagged , , , , , , , | Leave a comment

Getting data from IBM Spectrum Control. RESTful API usage example in Python

Searching for a handy way to fetch data from IBM Spectrum Control (earlier versions are called Tivoli Storage Productivity Center (TPC)), I have found a perfect IBM storage blog: https://storagemvp.wordpress.com.

Among other interesting topics it describes several methods to export SC/TPC data, so I have immediately contacted it’s author, Dominic Pruitt, and he gave me useful advises and hints. Thank you very much, Dominic!

Because of it’s simplicity, the most attractive way for me is to use RESTful API.

It is rather new interface to Spectrum Control and that’s why it is not very well documented. Nevertheless it’s totally enough to start coding simple data exporter in Python.

Below is the example I wrote using brilliant “requests” library which makes HTTP as friendly as possible. It collects and lists information about Storage systems configured under Spectrum Control.

#!/usr/bin/python3

# -----------------------------------------------------------------------------
# "THE BEER-WARE LICENSE" (Revision 42):
# zmey20000@yahoo.com wrote this file. As long as you retain this notice you
# can do whatever you want with this stuff. If we meet some day, and you think
# this stuff is worth it, you can buy me a beer in return Mikhail Zakharov
# -----------------------------------------------------------------------------

import requests

username = 'admin'
password = 'password'

base_url = 'https://sc.server.local:9569/srm/'
login_form = base_url + 'j_security_check'

rest_root = base_url + 'REST/api/v1/'
rest_StorageSystems = rest_root + 'StorageSystems'

StorageSystems = [
	{
		'Name': 'Name', 'ID': 'ID', 'Type': 'Type', 'Model': 'Model',
		'Firmware': 'Firmware', 'IP Address': 'IP Address',
		'Serial Number': 'Serial Number', 'Vendor': 'Vendor',
        'Pool Capacity': 'Pool Capacity', 'Used Pool Space': 'Used Pool Space',
        'Available Pool Space': 'Available Pool Space'
	}
]


def get_restful(requests_session, url):
    rq = requests_session.get(url)
    if rq.status_code != 200:
        print('Unable to open: {}, status code: {}'.format(url, r.status_code))
        exit(1)

    content_type = rq.headers.get('content-type')
    if content_type != 'application/json':
        print('Unsupported Content-Type: {}. We want JSON'.format(content_type))
        exit(1)

    return rq.json()


s = requests.Session()
s.verify = False

print('Logging into Spectrum Control', flush=True)
r = s.post(login_form, data={'j_username': username, 'j_password': password})
if r.status_code != 200:
    print("Can't open login form. Status code: {}".format(r.status_code))
    exit(1)

print('Checking if we can speak RESTful API', flush=True)
get_restful(s, rest_root)

print('Requesting Storage Systems information', flush=True)
tpc_storages = get_restful(s, rest_StorageSystems)

# Parse storage systems and save essential fields  
for storage in tpc_storages:
	StorageSystems.append(
		{
			'Name': storage['Name'], 'ID': storage['id'],
			'Type': storage['Type'], 'Model': storage['Model'],
			'Firmware': storage['Firmware'],
			'IP Address': storage['IP Address'], 
			'Serial Number': storage['Serial Number'],
			'Vendor': storage['Vendor'],
			'Pool Capacity': storage['Pool Capacity'], 
			'Used Pool Space': storage['Used Pool Space'],
			'Available Pool Space': storage['Available Pool Space']
		}
	)

print(StorageSystems)
Posted in Storage, Storage Automation | Tagged , , , , , , , | 1 Comment

D81S – simple SAN visualisation tool

D81S – my SAN visualisation tool, which I develop to build a full map of a Fibre Channel Storage Area Network by tracing all paths from HBAs to storage logical devices. D81S scans SAN switches and storage systems to create a database of all volumes which are accessible by hosts and the same LUNs provided by storage systems.

Two years ago I was working on it with passion, but later I have lost my accesses to most parts of that storage environment. So I’m unable to continue my work on it, so I decided to to share the source.

If anybody is interested in it, I can help to install it on his environment, test or even continue to develop it.

d81s_screenshot

Posted in D81S, My projects, Storage, Storage Automation | Tagged , , , , , , | Leave a comment

Another challenge for those, who can exit vi

Using ed, write a hello_world.txt file with two lines “Hello World!” and “ed is the best editor”. Don’t forget to exit 🙂

Posted in Offtop | Tagged , , | 1 Comment

First sketches of the new BeaST Grid family storage system

“The BeaST Grid” is a work name of the reliable storage cluster concept. It will consist of a few Controller Nodes with optional internal drives and several Drive only Nodes. All nodes are commodity computers with internal drives.

Controller Nodes will also be able to work in driveless, standalone mode and therefore may be used as storage virtualizators for other storage systems.

First version of the BeaST Grid will be based on the BeaST Classic with RAID system.

Posted in BeaST, My projects, Storage | Tagged , , , , , , | 2 Comments