The BeaST storage system with ZFS and CTL HA, latest news

Finally, I did it! The BeaST storage system with ZFS and CTL HA works in ALUA mode with zpools balanced over controllers. The BeaST Quorum automates Failover/Failback operations.

Yes, I have something to say now:

Read the full description on the BeaST project page: The BeaST Classic – dual-controller storage system with ZFS and CTL HA

Known limitations:

• I have to use virtual shared drives for cache (in-memory cache mirroring works badly for now, but I will win this battle someday).
• After a controller failure/recovery occurs, a special offline procedure must be used to re-balance zpools over controllers.

As usual, the BeaST is on early development stage, do not run it in production!

I need testers to check if it works for someone else then me 🙂

Fighting with ZFS to make zpools failover/failback possible

After success with single zpool switching, I’m trying now to create a reliable and balanced construction with two controllers and two pools.

When one of the controllers is dead, its pool successfully migrates to the other controller. But when the controller recovers, zpool rebalancing is needed.

And here rises a huge problem of “detaching” a pool from the running controller online. I tried to send commands with ctladm to stop the appropriate LUN and even remove/create it again. But it doesn’t work properly for me at least for now. Unbelievably, but it seems the only reliable way to detach a LUN from the frontend is to stop ctld!

It turns out, the BeaST will have offline rebalancing procedure for now. Until the offline rebalancing take place, the BeaST will have to work with one active controller and the other forwarding data only.

The second issue is related to ZIL mirroring. Adding appropriate definitions to ctl.conf doesn’t work properly for CTL HA configuration for many reasons, and the main of them is that HA mode requires right the same LUNs to be defined on both nodes. And it looks impossible to exclude ZIL LUNs from HA configuration. I tried to start two instances of ctld with different confiuration files for backend ZIL mirroring and front-end ports, but it doesn’t help me to avoid CTL HA issue on backend.

Then I have tried to use gmirror + geom gated as a mirroring transport, but something strange happens with gmirror when ggate device looses connection with the dead controller. Yes, gmirror detects that remote ggate is detached, but it doesn’t want to drop it and continues to wait for something!

Finally, I replaced it with legacy iSCSI port – istgt. It works quite stable, but sometimes drops connection. Fortunately, gmirror detects it well and it’s possible to restore ZIL-mirroring on the fly.

So, now there are two different iSCSI stacks in the BeaST! 🙂 And I’m not a half way to make everything work, as it seems the gmirror sometimes is a cause of kernel panic.

UPD 2017.05.14:

So “gmirror + something” chains are not very stable for mirroring ZIL. And yes, HAST doesn’t look sutable for the BeaST purposes it it’s based on ggate and it creates one-way replication. Also I’d prefer to keep same device names on both sides of replication.

Let’s see  if “shared drive” for ZIL make everything more stable.

Quick MS Excel IsTime() and TypeIs() functions solution

I never thought I would write hints for MS Excel VBA, nevertheless here is the first one.

It turns out, there is a scripting problem with identifying value as a time in a cell on a worksheet. I was curious to find out, there is no built-in IsTime() for the task in the wide range of IsDataType() functions: IsNumeric(), IsText(), IsLogical(), etc. And the most funny thing about it, the IsDate() function is exists!

A brief internet search showed me that people are facing the same issue while suggested solutions do not look well.

The problem is that “time” values in such cells are actually have “Double” type, which is actually an 8-byte floating point number. These numbers are just masked by one of the “time formats” to present them nicely to users. Obviously IsNumeric() function returns True for such values.

UPD. The similar situation with IsDate(), that’s why I had to put it before IsText(), otherwise it may be detected as text.

Below are two helper functions: IsTime() and TypeIs() I have wrote to handle cell data type identifying processes:

' TypeIs() and IsTime() Functions

' "THE BEER-WARE LICENSE" (Revision 42):
' <zmey20000@yahoo.com> wrote this file. As long as you retain this notice you
' can do whatever you want with this stuff. If we meet some day, and you think
' this stuff is worth it, you can buy me a beer in return.    Mikhail Zakharov

Public Function IsTime(TargetCell As Variant) As Boolean
    IsTime = False
    If InStr(TargetCell.NumberFormat, "h:m") And TypeName(TargetCell.Value) = "Double" Then IsTime = True
End Function

Public Function TypeIs(TargetCell As Variant) As String
    Select Case True
        Case IsEmpty(TargetCell.Value)
            TypeIs = "Empty"
        Case IsTime(TargetCell)
            TypeIs = "Time"
        Case IsNumeric(TargetCell.Value)
            TypeIs = "Numeric"
        Case WorksheetFunction.IsLogical(TargetCell.Value)
            TypeIs = "Logical"
        Case WorksheetFunction.IsErr(TargetCell.Value)
            TypeIs = "Error"
        Case IsDate(TargetCell.Value)
            TypeIs = "Date"
        Case WorksheetFunction.IsText(TargetCell.Value)
            TypeIs = "Text"
        Case Else
            TypeIs = "Unknown"
    End Select
End Function

Usage is simple:

Debug.Print TypeIs(Cells(507, 42))
Time
Debug.Print IsTime(Cells(507, 42))
True

Fresh BSD Magazine with my article as a headline!

Boasting now as the fresh BSD Magazine puts my paper “Nearly Online Zpool Switching Between Two FreeBSD Machines” on top of the issue’s cover. My text is about problems of ZFS storage pool sharing between computers and solutions of overcoming these obstacles.

To read the paper, download the issue from BSD Magazine site, or see the author’s version of the article on my blog: “Approaching online zpool switching between two FreeBSD machines“.

We all know that ZFS pool is designed to be used by a single machine only at any particular point of time, and when we want to access a zpool from another system we have to export it and import to the new server. This disadvantage makes simultaneous direct usage of the pool quite impossible for reliable dual-controllered storage systems.

Many thanks to Ewa and the BSD Magazine team for their perfect work!

Conway’s Game of Life for Excel

I always wanted to use MS Excel for a really, really useful project! You know, something like these amazing tools, which allow you to convert a picture into an array of cells on the Excel spreadsheet. So, don’t thinking much I have started to invent a bicycle write yet another Conway’s Game of Life implementation for MS Excel.

The primary idea was to get fun as well as to get or recall some VBA knowledge, which is useful for creating various reports in Excel. My algorithm is simple, straightforward, obviously not optimal, and, I think, it will never be enhanced 🙂

Download it here.

Short memo for SSL/TLS debugging and session decrypting

When trying to automate recurring or annoying operations on a network client, you must be lucky to have a special SDK, well-known documentation on API and plain-text protocols. Otherwise, it often happens that you have to start capturing tcpdump outputs to study application layer dataflows. But things may turn even worse if you must look inside the encrypted connection.

Here are some useful hints, which I have collected during the recent session on a Java based application.

First of all, run the application with network debugging:

# java -Djavax.net.debug=all MySSLJavaApplication

Yes, it allows to see just everything you need in hex and plain-text.

Read this to get help on the available network debugging options. Detailed example of the SSl/TLS connection debugging session is here.

Sometimes it is useful to disassemble one or more java class to see it’s code:

# javap -c -constants MySSLJavaApplication

See this perfect example with detailed descriptions.

Browsers are able to store all the keys, so we can literally intercept TLS encrypted session. Do you feel more secure now? 😉 Set SSLKEYLOGFILE environment variable, then run Firefox or Chrome:

# SSLKEYLOGFILE=~/sslkeys.log firefox

Here is SSLKEYLOGFILE format description along with instructions for Wireshark on how to use it to decrypt TLS connections.

Use ssldump if you are lucky to have a Private key in advance:

# ssldump -k /path/to/private.key -i 'interface' -dnq 'expression'

There is a super nice typical problem scenario #4…

… according to the NetApp® Manageability SDK Printable Online Help, October 2016 | 215-11740_A0, page 147 exists on DataONTAP prior to 7.3/7.2.3P1 versions 🙂