A single write locks the entire SQLite database for both: reading and writing

This fact is quite an unpleasant thing for a database with a lot of writes and relative seldom reads like, for example, a small packet capturing tool I write for my router.

The classical mechanics of SQLite locking is described here, but if you face the same problem as me, you probably would want to enable WAL – Write-ahead logging feature, which can solve the issue of SQLite DB locking:

The traditional rollback journal works by writing a copy of the original unchanged database content into a separate rollback journal file and then writing changes directly into the database file. In the event of a crash or ROLLBACK, the original content contained in the rollback journal is played back into the database file to revert the database file to its original state. The COMMIT occurs when the rollback journal is deleted.

The WAL approach inverts this. The original content is preserved in the database file and the changes are appended into a separate WAL file. A COMMIT occurs when a special record indicating a commit is appended to the WAL. Thus a COMMIT can happen without ever writing to the original database, which allows readers to continue operating from the original unaltered database while changes are simultaneously being committed into the WAL. Multiple transactions can be appended to the end of a single WAL file.

To enable WAL, execute:

sqlite> PRAGMA journal_mode=WAL;

Good news, it’s persistent and required to be run once.

Also additional tuning can be done, like setting: PRAGMA journal_size_limit=N, PRAGMA wal_checkpoint(<MODE>) or PRAGMA synchronous=FLAG, but it’s strongly advised to read documentation before changing something.

Finally, as always, remember to call sqlite3_finalize() after the prepared statement is no longer needed, or you risk locking the database which in the case of WAL leads to the checkpoint starvation and uncontrolled grows of the WAL file.

Posted in Tips & tricks | Tagged , , , , , , | Leave a comment

RAIDZ on-disk format vs. small blocks

Possible ZFS performance problems with small blocks, block alignment and much more explained by Mike Gerdts.

Posted in Copy-paste, Storage | Tagged , , , , , | Leave a comment

Modelling basic architectures of data storage systems

Nowadays majority of the modern and powerful block-level storage systems around the world are built to work in expensive Fibre Channel or more cheaper iSCSI SAN environments. Independent of their class, capacity and performance they are created on well-known principles, technologies and architectures. Furthermore some of these systems are using common servers for their controller hardware with Linux or even AIX as storage operating systems on-board.

In this paper I will describe some of the basic SAN oriented data storage architectures and also model their design using FreeBSD (https://www.freebsd.org) as the operating system and VirtualBox (https://www.virtualbox.org) as the hardware emulator. All the specific FreeBSD solutions in this document are based on my BeaST storage architecture (https://mezzantrop.wordpress.com/portfolio/the-beast) project publications.

Posted in BeaST, My projects | Tagged , , , , , , , | 1 Comment

What I have learned about science

Having a number of scientists in my family and after reading thousands of sci-fi stories in my youth, I always had some kind of romantic attitude to the topic. I used to think of scientists as the people whose lives are continuous adventures of explorers and researchers and who belong to a community with the only requirement – the thirst for knowledge, and the only prize – euphoria when you can’t hold back the exclamation of “Eureka” after discovering something new. Needless to say, “Bison”, the famous Nikolai Timofeev-Resovsky, was the impersonation of this ideal.

Much later, in the last university years, I started my career as a laboratory assistant of the Department of Mathematics and Statistics but quickly, after a couple of years, abandoned the path due to the science devaluation in Russia, mass corruption and general decline of the education system.

Since then, I have not had anything to do with science until the moment I tried to find a resource to publish my new paper about data storage architectures and their possible implementation using open-source code. Yes, the article was amazing, seriously, the text was (and still is) quite scientific, educational and potentially useful for IT engineers, it also looks pretty solid. I wanted it to be free and open for readers, but I thought of a larger audience than my personal blog can provide. So in my forties I called to arms my old beliefs and decided to try an IT scientific journal.

With a little googling, I quickly discovered that in general you must be a certified scientist to publish something in such a journal, because if you are not a certified scientist, you are not supposed to be able to conduct scientific research and, as a result, you cannot write an article of satisfactory quality. This works the similar way if you start from the other end: you must publish a research to be a scientist.

The best way to break the vicious circle is to submit your university email address to the online editorial system of the desired scientific journal. And you are not a scientist if you fail. Sad news for those, who as me, lost connections with the university years ago and can’t provide now a valid email address. However, appreciate the irony: I was the one at that time responsible for developing the whole Internet infrastructure of Alma mater including the email server as well as the pack of other servers and firewalls. But there is good news on the other hand, now we have a new definition: scientist (n.) – a person with approved email.

For those who, by all means, still want to prove they are scientists, there is an option to pay money for the article submission. I didn’t check if it works, but I don’t want to pay for something I want to share for free. Most probably, the journal will take the money. And who won’t? But from now on they will start calling you an independent researcher. Live with it.

As for me, I’m going to miss the opportunity to be a real scientist. Instead, I will post the article on some popular platform where it can be easily reached by the target audiences – IT-geeks prefer searching internet for the free information rather than paying money to journals for it.

Posted in Impressions, IRL, Offtop | Tagged , , | Leave a comment

The BeaST Grid storage early stage

I completely devoted my vacation to the development of the BeaST Grid family of my storage architecture and in the early days I was naive enough to plan a release. The Grid family was, and still is, the essential step for the whole BeaST architecture because I intended to use it as the cornerstone to develop and test the BeaST back-end arbitrator that manages drives or drive-nodes online connection/reconnection operation. The other reason to work on it is that the BeaST Grid doesn’t require direct drive attachment. This fact makes the Grid family more easy to use.

BeaST_Grid_RAID_CTLHA_AS-to-BE

The BeaST Grid storage system architecture AS-to-BE

Any back-end operations are very sensitive because on any re-attachment of the known drives, GEOM framework by default starts automatic RAID recovering procedure on the both nodes simultaneous. This obviously leads to data corruption. So, the back-end arbitrator task is to prevent the storage system collapse.

Unfortunately, I have lost most of the time trying to stabilise the VirtualBox environment on my FreeBSD server, and I still experience issues with the virtual networks on it.

The second problem I happened to face was that the BeaST started to crash on all the failover operations. I checked and desperately reinstalled the BeaST and the whole environment multiple times, I even rewrote part of my BQ arbitrator daemon, but nothing helped. I was unsure as to what the real cause of the BeaST crashes was, whether it was my fault, the issues of the VirtualBox port or FreeBSD OS itself.

Loosing all my hopes I tested the most primitive operations of the CTL HA role switching and found the possible problem with CTL HA on the FreeBSD 12.x kernels. I still hope it is an issue of my setup, though the similar configuration works quite well on the FreeBSD 11.x. So, for now, I have to confess, the BeaST works on FreeBSD 11.x but not on 12.x. As good news, I can say, Allan Jude saw my post on Facebook and probably he well point out to me what’s happened with the CTL HA in 12.x.

Nevertheless, the time is already wasted and I have written only the early draft of the Grid family guide to fix the progress, which still doesn’t include the back-end arbitrator. The other point is, that the GEOM labels created on the client are propagated to the BeaST storage system. So, they can interfere with the storage configuration. And this is another task to be solved. Unfortunately, my vacation is almost at an end and I am afraid, I really won’t have enough free time until the winter break to implement all the changes.

BeaST_Grid_RAID_CTLHA_AS_IS

The BeaST Grid Storage system architecture AS-IS

So, looking at the pictures, there are a lot of things to be done on the Christmas holidays to move the BeaST Grid from the AS IS state to the AS-to-BE one. I very much hope, I manage this 🙂

Posted in BeaST, BeaST Quorum, My projects, Storage | Tagged , , , , , , | Leave a comment

FreeBSD 12.x CTL HA kernel panic

Recently I discovered the issue with kern.cam.ctl.ha_role kernel variable.

At first, I wrongly suspected, that the problem appears when both iSCSI target and iSCSI initiator are configured on the system, but now, I have discovered that, the kernel panic happens every time I try changing the kern.cam.ctl.ha_role during the runtime.

Guys, if you please, try setting something like this:

ctl_load="YES"
kern.cam.ctl.ha_role=1
kern.cam.ctl.ha_id=1
kern.cam.ctl.ha_mode=2
kern.cam.ctl.ha_peer="listen your_IP_address:7777"

on your FreeBSD 12.x box in /boot/loader.conf, then reboot.

When you get the root prompt again, run:

sysctl kern.cam.ctl.ha_role=0 or sysctl kern.cam.ctl.ha_role=1

Report if you got kernel panic. 

I also prepared an image of the VirtualBox machine which is configured to test the issue.  Download it here, import, check for the network settings of the virtual machine in VirtualBox as well as in the FreeBSD /etc/rc.conf and /boot/loader.conf and optionally in /etc/ctl.conf.

Login is root with an empty password. Try playing with sysctl kern.cam.ctl.ha_role and see if you get kernel panics.

If you face the same issue, I’ll report the bug.

P.S. The most interesting thing is that FreeBSD 11.x doesn’t have this issue.

Posted in BeaST, My projects | Tagged , , | 1 Comment

The BeaST storage concept side-notes

  • The BeaST Quorum daemon updated to version 1.2. Actually, I added only auto-detection of the block-size of the drive to work with sectors larger than 512 bytes.
  • While trying to setup the Grid configuration of the BeaST storage system concept I faced weird issues with iSCSI links on the host-only virtual network (!) that show up as often timeouts and reconnections on the back-end! I tried different host-systems with VirualBox 6.1 and 5.2, but got similar results. So, after drinking the full cup of rage and wondering if it was my fault, the problem in VirtualBox or an issue in FreeBSD iSCSI implementation I started thinking of possibilities to find a 4 or 5 bare metal servers to accomplish the test. But, honestly, I don’t think I’ll find them in a nearest future.
  • Also I have suspicions, that if a machine is iSCSI initiator and iSCSI target at the same time, i.e. runs as a client and a server, then changes on the ctl server-side (for example, setting kern.cam.ctl.ha_role) affect client-side iscsictl even when dealing with different LUNs. But, I can’t be sure, because it can be an effect of the issues with iSCSI on the host-only virtual network.
  • Finally, I have to write it down to fix up the not very clear paragraph of ctl(4) manual: sysctl variables kern.cam.ctl.ha_role=0 on one node and kern.cam.ctl.ha_role=1 on the other must always be defined (regardless of how and if ha_role=primary or ha_role=secondary LUN options were set for each LUN).
Posted in BeaST, BeaST Quorum, Storage | Tagged , , , , , | 2 Comments

A quick memo on creating virtual machines using VirtualBox CLI

Sorry people, I have to leave it here because at my age I still have trouble organising records in a way that allows me to find notes without being surprised by the cryptic scribbles written on them*.

VirtualBox has a nice GUI, but having a headless server configuration forces me to use CLI and VBoxManage utility to maintain properties of virtual machines.

First of all, register a machine of a suitable OS type:

# VBoxManage createvm --name UnixWare-7.1.4p-clean --ostype Linux26 --register

Yes, I took UnixWare 7.1.4+ installation as a bit less trivial example, then the Linux one.

The full list of supported OS types can be listed with:

# VBoxManage list ostypes

Configure the most important parameters of the machine including memory (1G) , first-boot device (dvd) and network adapter:

# VBoxManage modifyvm UnixWare-7.1.4p-clean --memory 1024 --acpi on --boot1 dvd --nic1 bridged --bridgeadapter1 bge0

Create a new virtual hard-drive (10GB) in this particular case:

# VBoxManage createhd --filename /data/private/virtualbox/drives/UnixWare-7.1.4p-clean.vdi --size 10240

Create a storage controller, supported by the OS to be installed. A simple IDE, one for example:

# VBoxManage storagectl UnixWare-7.1.4p-clean --name "IDE Controller" --add ide --controller PIIX4

Attache the virtual drive to the controller:

# VBoxManage storageattach UnixWare-7.1.4p-clean --storagectl "IDE Controller" --port 0 --device 0 --type hdd --medium /data/private/virtualbox/drives/UnixWare-7.1.4p-clean.vdi

Attach a DVD drive with an installation media:

# VBoxManage storageattach UnixWare-7.1.4p-clean --storagectl "IDE Controller" --port 0 --device 1 --type dvddrive --medium /data/public/distrib/nix/UnixWare/UW714+_ISL_1.0.0Ds.iso

Tune all additional parameters if needed. Please, don’t thoughtlessly copy/paste the command below, as for example, in the case of UnixWare 7.1.4+ I specially had to turn off USB, limit CPU to 1 and disable 3D acceleration:

# VBoxManage modifyvm UnixWare-7.1.4p-clean --vram 256 --ioapic on --hpet on --apic on --x2apic on --hwvirtex on --nestedpaging on --vtxvpid on --vtxux on --pae on --cpus 1 --accelerate3d off --usb off --audio none

Enable remote display (either RDP or VNC, depending on VirtualBox configuration):

# VBoxManage modifyvm UnixWare-7.1.4p-clean --vrde on

If required, check virtual machines and their parameters. Note, omitting “–long” keyword will give a short list with virtual machine names:

# VBoxManage list --long vms

And finally start the virtual machine and do whatever is needed:

# VBoxHeadless --startvm UnixWare-7.1.4p-clean

* For anyone who intended to avise me a solution for online note organisers, don’t take it serious, the above statement is a joke.

Posted in IRL, Tips & tricks | Tagged , , , | 2 Comments

Compare and patch two regular- or device- files block by block

You won’t believe, my retro-scripting week continues!

After creating the major update of my Bourne shell HTTPd server – Supermatic, I have tinkered the old bfcmp script, which I wrote several years to to make my life easier when comparing (and patching) binary files and filesystems dumps of the BeaST storage system, I worked on hard that time.

But this time, I used bfcmp for checking image files that got transferred corrupted in responses from one of the earlier versions of all the same Supermatic server.

Posted in My projects | Tagged , , , , | Leave a comment

Supermatic! It works again!

Great news for all* the fans of the best** in the class web-server! A couple of minutes ago, our ingenious developers published a new version of amazing Bourne shell Supermatic web-server!

Among other beloved features, Supermatic v0.9 now supports:

  • PHP language
  • Directory listing
  • Authentication

Hurry up to download it here (The number of copies remaining is limited and is rapidly decreasing).

Read more info about Supermatic on the project page.

Congratulations guys! Hooray!

* if there are any

** According to the creators

Posted in IRL, My projects | Tagged , , , , | 1 Comment