Printing lines side by side in shell

Everybody knows, there is paste command in every *NIX to print two files side by side or join which is obviously used to join text-files by their key-fields. And there are a lot of examples and tutorials over Internet, so the subject is as trivial as cat-ting files together with right command-line options.

Ah, wait… What if we have textual data stored not in files but in two variables as a result, for example, of piping or redirection?

Imagine there’s no heaven two variables:

a="aa qq
bb ww
cc ee"

b="11 44
22 55
33 66"

And we want to see this as the result:

aa qq | 11 44
bb ww | 22 55
cc ee | 33 66

It’s not an issue having Bash and arrays:

#!/usr/bin/env bash

a="aa qq
bb ww
cc ee"

b="11 44
22 55
33 66"

IFS=$'\n' read -d '' -a a_arr <<< "$a"
IFS=$'\n' read -d '' -a b_arr <<< "$b"

for  i in ${!a_arr[@]}; do
	echo ${a_arr[i]} "|" ${a_arr[i]}
done

The main trick is done with read command which stores lines in arrays, while the rest is just a loop which scans arrays to print lines with the same index.

But life is not always easy and sometimes one could still deal with pure old Bourne shell without any hints on arrays where set command was utilised to create $@ array of $1 to $n elements. Agree, this is not always handy.

So, in the worst case it’s always possible to use a counter variable and an external tool to scan over the strings. For example, awk:

#!/bin/sh

a="aa qq
bb ww
cc ee"

b="11 44
22 55
33 66"

cnt=0
echo "$a" | while read -r ln; do
	cnt=`expr $cnt + 1`
	printf "$ln | "
	printf "$b" | awk -v c="$cnt" 'NR==c {print}'
done

or even like this:

#!/bin/sh

a="aa qq
bb ww
cc ee"

b="11 44
22 55
33 66"

for l in $(seq 1 `wc -l <<< "$a"`); do
	awk -v l="$l" 'BEGIN {ORS=" | "} NR==l {print}' <<< "$a"
	awk -v l="$l" 'NR==l {print}' <<< "$b"
done

Keeping in mind external tools and languages, there are much more possibilities to do the task, but lets stop on this for a while 🙂

Advertisements
Posted in Tips & tricks | Tagged , , , , | Leave a comment

Configuring Linux server to work with the BeaST storage system over iSCSI protocol

It’s weekend, so I continue to work on the BeaST storage system.

Here is my small HOW-TO describing multipath connection with the BeaST from a RHEL/Centos Linux server via iSCSI: Configuring Linux server to work with the BeaST storage system over iSCSI protocol.pdf

Posted in BeaST, My projects, Storage | Tagged , , , , , , , , , | Leave a comment

Of course, not all of the scientists are so kind

My funny story as follows. A long time ago I discovered my expect-like tool “Empty” is mentioned in a paper buried somewhere in a book full of scientific articles. Of course, the book wasn’t free. But I was young and so pleasantly surprised to be mentioned somewhere outside my blog that I started my digging.

After a quick investigation I found the exact article and both names of its authors. Unfortunately, it was a dead end, because it wasn’t possible to discover their contacts. But I easily found the email address of that respected professor on the cover of the book. I wrote to him immediately and asked if he could send me the article or at least its fragment where my name and utility are mentioned. Unfortunately, he didn’t reply. After some time, I requested him again. And he asked me what I wanted from him. I repeated my question for the third time. Since then he completely ignored me.

Years later I found the book, scrolled the article to the desired paragraph with my name and read something stunning like that: “Mikhail Zakharov’s utility named “Empty” can’t be used to achieve the goal of the article”.

That pretty much explained me why the professor kept silence 🙂

Posted in empty, Impressions, IRL, Offtop | Tagged , , | Leave a comment

Children often ask me what I do at work (as a storage systems administrator)

And here is the answer for them right in the The Amazing World of Gumball – one of the funniest cartoon ever! See the “hacking” moment in the Safety episode (S03E32) where Anais talks about storage administration:

The transcript:
Gumball: Ugh! We need a code. How are we gonna get in?
Anais: Don't worry. I have my ways. [Types] H [Types] A [Types] C [Types] K: Hack. Press enter.
[The door opens]
Gumball: Really? "Hack"?
Anais: Just kidding. Actually, I bypassed the storage controller, created remote replication-
Gumball: [Nods] Mhm. [Nods] Mhm. [Nods] Mhm.
Anais: Mirrored data on two storage systems, connected USP V storage system with the lower level class AMS array-
Gumball: [Steps back and nods] Mhm. Mhm.
Anais: So virtualized volumes were presented to the host as original ones. Thus, storage system topology...

Known tasks for the Hitachi storage system administrator, aren’t they?! Can you imagine my shock hearing these words in the cartoon?

Unfortunately, guys, this gem is in Russian translation only.

The original text is much closer to incoherent hacker mumbling, but it’s also good enough mentioning EMC VNX, ESXi and VSANs:

The transcript:
Gumball: Ugh! We need a code. How are we gonna get in?
Anais: Don't worry. I have my ways. [Types] H [Types] A [Types] C [Types] K: Hack. Press enter.
[The door opens]
Gumball: Really? "Hack"?
Anais: Just kidding. Actually, I bypassed the storage controller, tapped directly into the VNX array head, decrypted the Nearline SAS disks–
Gumball: [Nods] Mhm. [Nods] Mhm. [Nods] Mhm.
Anais: Injected the flash drivers into the network's fiber pack, before disabling the IDS, routed incoming traffic through a bunch of offshore proxies–
Gumball: [Steps back and nods] Mhm. Mhm.
Anais: Accessed the ESXi server cluster in the primary datacenter, and disabled the Inter-VSAN routing in the layer three–

Here is the original transcription source, Russian one is my unofficial translation.

Posted in Impressions, IRL, Misc, Storage, Storage Automation | Tagged , , , , , , , , , , , , , | Leave a comment

A nice way to return string values from functions in Bash

What can be as trivial as returning a value from a function in a scripting language? For those who has ever dealt with Bash, the answer turns out to be not so obvious. First, there are multiple options to do it and each of them has their own different features. Second, most of them are… let’s say, evil and if you have developed the sixth sense you will feel daemons are wandering around there. Want to see why?

Let us start with classic example and try to return some integer value:

#!/bin/bash

ret_byte_test()
{
echo "Functions can return bytes [0..255]"
return 257
}

ret_byte_test
echo Like this overflowed one: $?

Yes, right, there is an integer overflow, so this code prints:

Functions can return bytes [0..255]
Like this overflowed one: 1

UPD 2018.07.09 BEGIN – Bourne shell implementation in FreeBSD has just surprised me pleasantly as there is no integer overflow on the byte boundary:

shell_return_FreeBSD
UPD 2018.07.09 END

Someone could think it’s possible to pass strings the same way:

#!/bin/bash

ret_str_test()
{
echo "- Say something!"
return "- ... Something"
}

ret_str_test
echo "Nothing good:" $?

Unfortunately, he is wrong because return command in Bash can only return status codes which are obviously integer values:

- Say something!
./example2.sh: line 6: return: - ... Something: numeric argument required
Nothing good: 255

Actually, there are two popular ways to return string from functions. First of them involves global variables which values are changed inside functions:

#!/bin/bash

ret_global_test()
{
global="Use a global variable, Luke!"
}

global=""
ret_global_test
echo $global

And the output is:

Use a global variable, Luke!

Yes, this method works, but it burdens you with all the disadvantages and potential dangers related to global variables usage. Haven’t you been warned about evil yet?

The second well-known way is to use command substitutions. This method works slower but potentially safer:

#!/bin/bash

ret_str_test()
{
echo "Command substitutions help a lot!"
}

mystr=$(ret_str_test)
echo "$mystr"

Output looks predictable good:

Command substitutions help a lot!

In many cases this code works just perfectly well. But what if we must forcible quit the script from a function? Something like this:

#!/bin/bash

exit_test()
{
echo "Exit from shell returns unsigned byte status [0..255] to the caller process"
exit 42
}

exit_test
echo "After exit() this part is never reached!"

And the output is:

Exit from shell returns unsigned byte status [0..255] to the caller process

Of cause, there are no references in the output to the part that is never reached. And as everything goes well so far, let’s be brave and try to return some value from the function using command substitution:

#!/bin/bash

echo "Self destruction sequence initialized: 3..2.."

exit_test()
{
echo "Are you safe because o exit()?"
exit 57
echo "Yes, sir, of cause!"
}

my_error=$(exit_test)
echo $my_error

echo "Nope, you are not: 1.. BOOM!"

And the result is:

Self destruction sequence initialized: 3..2..
Are you safe because o exit()?
Nope, you are not: 1.. BOOM!

Here, we can clearly see exit successfully terminates the function, but it doesn’t have any visible effect on the outer code! It looks like exit is cursed by evil and that’s why it’s completely ignored in the main part of the script: afterwords we can even check the $my_error variable which contains the string with the existential question. Nice isn’t it?

Actually, it happens because of well known shell feature of running command substitutions in sub-shells. So, exit actually terminates only the sub-shell process but the main shell continues to execute all the rest of the commands in the script.

Yes, it’s still possible to check $? variable for status code (which is 57 in the last example) and behave accordingly, for example, to quite the main process. So keep this feature in mind when you involve command substitutions and don’t even try using exit to terminate the main process from the sub-shell.

Tacking between both methods, charmed exit and common sense I have finally discovered more or less elegant way to put everything together:

#!/bin/bash

nice_function() {
[ $1 ] || { echo "Wrong usage"; exit 1; }

myvar="Let us return something"

eval $1='$myvar'
}

nice_function result
echo $result

And the output is:

Let us return something

So, if you can tolerate constructions with $1 and eval in the function, everything should look much better doesn’t it?

Note 1, curly braces are used here to run a group of commands in the current shell process.
Note 2, final semicolon in the grouping construction with curly braces is mandatory.

P.S. All methods and constructions shown above should also work well in Bourne shell (sh).

Posted in My projects, Tips & tricks, Misc | Tagged , , , , , , , , | Leave a comment

Sorting Tumblrs you are following by their last activity

It seems, migration to GitLab stimulated me to code more in unusual for me languages and environments. First, it pushed me to do something in PowerShell: A short script to change audit rules on a file or a directory in Microsoft Windows.

Now, I started writing even more unexpected script: tfsort.js – Sort Tumblrs you are following by their last activity. So if you are Tumblr user and you are following blogs that are dead already for a long time, you can try to filter and unfollow them.

Someday it will be a Firefox plugin, maybe. But now you have to do everything yourself:

  1. Set English locale in your Tumblr settings
  2. Open a page with Tumblrs you are following
  3. Copy the tfsort.js source code into your Firefox Web Console: Menu -> Web Developer -> Web Console
  4. Press Enter to run it and wait for results!

It should work with Firefox, other browsers are untested.

Beware of unstable alpha stage code which must be run directly in the browser’s console, but as for me it’s a nice way to learn modern web-development technologies on the fly and create something more or less useless 🙂

Posted in My projects, Tips & tricks | Tagged , , , , , , , | Leave a comment

A short script to change audit rules on a file or a directory in Microsoft Windows

Suddenly I have got a lot of work setting audit rules on multiple assorted files on Wintel servers. This short script may help me to do things much quicker. Will check it tomorrow:

# -----------------------------------------------------------------------------
# "THE BEER-WARE LICENSE" (Revision 42):
# zmey20000@yahoo.com wrote this file. As long as you retain this notice you
# can do whatever you want with this stuff. If we meet some day, and you think
# this stuff is worth it, you can buy me a beer in return Mikhail Zakharov
# -----------------------------------------------------------------------------

#
# v1.0 2018.06.13 zmey20000@yahoo.com Initial release
#

param (
[string]$Operation = "Get",
[string]$Target = "",
[string]$User = "",
[string]$Permissions = "",
[string]$Type = ""
)

function Usage() {
Write-Host
Write-Host "Get or Set Audit rules on a file or a directory"
Write-Host
Write-Host "Usage:"
Write-Host " fp_audit.ps1 [-Operation Get] -Target C:\full\path"
Write-Host " fp_audit.ps1 -Operation Set -Target C:\full\path -User Name -Permissions Permissions -Type Failure|Success"
Write-Host
Write-Host "Example:"
Write-Host " fp_audit.ps1 -Operation Set -Target C:\Temp\file.txt -User Everyone -Permissions FullControl -Type Failure"
Write-Host
}

if ($Target -and $User -and $Permissions -and $Type -and $Operation -eq "Set") {
# Set-Acl
$AuditRule = New-Object System.Security.AccessControl.FileSystemAuditRule($User, $Permissions, $Type)
$TargetACL = Get-Acl $Target -Audit
$TargetACL.AddAuditRule($AuditRule)
$TargetACL | Set-Acl
} elseif ($Target) {
# Get-Acl
Get-Acl $Target -Audit | Format-List
} else {
# Wrong params
Usage
exit 1
}

It requires administrative rights to run.

Many thanks https://technochat.in for a good example.

Posted in My projects, Tips & tricks | Tagged , , , | 1 Comment