StrongBox Reference Manual

For StrongBox 1.0-b-12

Last Updated Saturday December 27th, 2004

*Work in Progress*


Table of Contents

Basic Orientation 3

Basic design principles 3

Similarities to other Linux Distributions 3

Differences from Other Linux Distributions 3

How the free version of StrongBox is less secure 7

StrongBox System Files 7

Bundles 8

Patches 8

Boot Configuration Files 9

Module archives 9

Application Bundle Configuration Files 9

Signatures 10

Change Control and Management 11

Important administrative options: 11

StrongBox Configuration File 11

Administrative modes: 11

MySQL Logging: 12

The StrongBox Initial Boot 13

Add-Ons 13

Boot parameters 13

Partition Discovery and Hardware Detection 15

StrongBox Boot Config Files 15

StrongBox OS Bundles 17

Ramdisk Types 18

StrongBox Patches 18

Putting it all together and preboot 18

The StrongBox OS Boot Process 19

Basic OS services 19

2 Stage boot process 20

The vmount system 20

Filesystems and Data Storage 21

Best practices in Data Storage 21

The StrongBox vmount system 21

Filesystem Layout 22

Vmount configuration file structure: 23

Available vmount types and options 24

Calling vmount manually 25

Config and boot partition(s) Structure 26

Bundle Partitions Structure 26

Data Partition(s) Structure 27

Run Partitions Structure 27

StrongBox Application Subsystem 28

StrongBox Static Application Bundles 28

Application Bundle Configuration and Patches 28

Linux-vserver configuration: 30

Filesystem Layout and Data Storage 31

Administrative and non-administrative modes in bundles 31

Authentication and Security 32

Domain model vs. Distributed model 32

“Hard” security, “Soft” security, secrecy and trust 33

Users and Administrators 33

Sudo 34

IP access lists and port knocking 34

Good practices prevent bad surprises 35

The Linux-Vserver System 36

Linux Capability Set 36

Using dynamic bundles 36

Setting up dynamic bundles using debootstrap 37

Deployment Considerations 37

Backups and Recovery 38

Upgrades and Patches 38

Assimilating existing linux deployments 38




Basic Orientation

Basic design principles

The StrongBox is designed around the ideas presented in the “Partially Embedded Computing Manifesto”:

http://strongboxlinux.com/docs/PEC%20Manifesto.html

It's meant as a reference implementation of these concepts, as well as a full working operating platform for Linux-Based applicances.

Similarities to other Linux Distributions
Differences from Other Linux Distributions

StrongBox Administration Commands:
These commands are StrongBox commands that are either StrongBox specific, or wrappers for standard unix commands that make using them simpler in a StrongBox environment.

Patch and Bundle Management:

Adminstrative Functionality:

Vmount system:

User/security management aids:

Log Management:

Documentation:

SSH, remote admin access, etc

Certificate and Key Management:

Scripts used by StrongBox Automated processes and other scripts:

vserver commands (mostly from the vserver project, www.linux-vserver.org):

Wrappers for standard unix commands that deal with StrongBox differences

Minor scripts used to help building bundles, etc

How the free version of StrongBox is less secure

This is a note that I felt I had to add in, to prevent people from relying too heavily on the free version of StrongBox. Because the free version of StrongBox has an open certificate authority, with free certificates available to all, a number of the measures that are taken to secure StrongBox become ineffective. Specifically, everything that relies on the validity of certain x.509 keys can be compromised by any attacker with access to the filesystems of a StrongBox server, or local access to the machine.

In order to make StrongBox secure, you need to do one of two thing things:

  1. Alter the StrongBox initrd to use your own CA, not the StrongBox free CA.

  2. Obtain a commercial version of StrongBox, which uses a much more secure Helmer Computing Corporation root ca.

  3. Obtain a commercial version of StrongBox using your Organization's own CA (the most secure option).

StrongBox is open source software, and we cannot, nor will we, ever force you to buy StrongBox. The StrongBox OS and updates are available for free, and will continue to be. However, if you like StrongBox as a product, and would like to see it continue as a product, please consider buying a commercial version of StrongBox, or contributing in some other way (such as purchasing consulting services, contributing developers' time, etc).

At Helmer Computing Corporation, we recognize that it's not just our own efforts that have created StrongBox. It's also the efforts of countless open source developers over the last 30+ years. As such, a portion of every purchase of StrongBox will go to those developers, especially those that developed the breaking technologies that are the most crucial to StrongBox. You support of StrongBox will help to keep that open source ecosystem alive!

StrongBox System Files

StrongBox operates in a modular way, with a very different OS and security model. The following is a description of the pieces and how they're implemented.

Note: There's been a nomenclature change in the beta series, from modules to bundles (to reduce confusion with kernel modules). This affects pathnames, variable names, everything. There is some backwards compatibility logic built in to the new config reading commands to deal with old installations.

Bundles

There are many types of bundles, but all follow particular rules. OS bundles are generally in tar.gz format (unix archive). Application bundles are generally either cloop (compressed loop file) or squashfs (a compressed loop filesystem).

Bundles contain a /imageinfo directory that contains metainformation about the image. There is some redundancy in these files: normally, this was added in to allow configuration saves to happen much more quickly. They includes:

Application bundles are generally run in a vserver, and have a configuration applied at boot time. They're found on the StrongBox partition, at /ramdisks/<version number>/boot/

OS Bundles are unpacked during the boot process, and also have a configuration applied at boot time. They're located on the bundles partition at <bundlename>-<version>.<extension>

Patches

Patches are a differential from a bundle to the currently working copy. They contain changed files, and metadata about changed files. Metadata is contained in the /patchinfo directory, and includes:

OS Bundle patches are located on the patch partition at /patches/<version>/patch-strongbox-<date>...

Application Bundle patches are located on the bundle data partition at <installedname>/<version>/patch-<bundlename>-<bundleversion>-date...

Boot Configuration Files

These files contain basic information about how to use a bundle. They can contain various information about how to boot StrongBox. They contain information about:

Generally, these configuration files are automatically generated by strongbox.conf. However, it's possible to write them manually, especially to create the “preboot” and “default” files (see section on boot process).

The preboot config file is created in the same directory as you have your “current” boot file. It will be run as a script, and will run before the boot file is loaded. You must sign this file with your key whenever you change it. Possible uses include: loading extra modules, running specific commands, etc.

The default” config file is the default script that's used if all others fail to excecute. This is useful if you want a particular backup partition to boot, or some other action to happen in boot fails for a particular box.

Module archives

There are two types of kernel module archives. The first is a standard tar.gz archive. The second is a squashfs archive. They're autodetected at boot-time, and are unpacked into the StrongBox ramdisk, based on what the running kernel version is. They can be located on any partition on the system at /kernels/<kernel-version>/modules.{tar.gz|squashfs}.

Note, that a limited set of modules (specifically, all the device drivers) is included in the StrongBox initrd. This allows boot-time autodetection to be meaningful, as devices are detected and their corresponding modules are autoloaded. If a module archive for the appropriate kernel is not found, these modules are copied into the ramdisk before boot.

Application Bundle Configuration Files

These files contain startup information about a bundle. They contain:

Application bundle configuration files are located on the modules partition at: /<installedname>/version/config-<date>... The ones that are currently in use are linked to /<installedname>/config /<installedname>/previous, etc. These are covered in depth in “application bundle configurations and patches”.

Signatures

Signatures exist in the same directory as the file that they're regarding. They have the extension .sha1-pkcs7, with a possible integer behind it (.sha1-pkcs7.1, etc). It may, additionally, be compressed (note: not implemented yet).

A signature is in s/mime pkcs7 format. It includes: mime headers, a signed message, and the certificate that signed the message. The first line of the signed message is always the sha1sum of the file. Subsequent lines all begin with a hash mark (“#”), and are optional. The first such line is the date: #DATE=<system date format>, and the second line is #SIZE=<size in bytes>. Lines after the date are treated as text comments on the file that's being signed.

The primary function of these files is security. Every strongbox component that is loaded from disk is implicitly distrusted. Before it's used, the signature is verified, and the certificate in the signature is verified against its store of certificates for validity. Then, the certificate is verified for containing the appropriate OIDs that allow particular StrongBox functions, to see if it's able to sign that variety of file. Finally, if all this checks out, then the sha1sum of the file contained in the signature is checked against the appropriate file in the filesystem.

The secondary function of these files is change control. Every change that is committed to a StrongBox system causes the user to be prompted to add new comments regarding the changes, such as why the changes were done, and what files were touched by the change, etc. This change log data is currently very ad-hoc: support of a more formal change-control infrastructure will be incorporated in a future version.

Change Control and Management

Change control and management currently use the metadata information about patches and bundles, and the signatures, to authenticate and manage change control. Various operations are supported, including:

Also, the signatures on patch files provide more extensive, human-readeable version information. Signer, date and comment information are kept from patch to patch, allowing a complete stream of comments on the last patch, describing in detail how it got ot be where it is. This is essential, and should provide enough information to troubleshoot where and when a problem occurred, and how to get back and fix it.

Upgrades and roll-backs are a part of the change control and management. Currently, a basic interface to rollbacks is provided. Note, rollbacks only include bundle and configuration data, they do not include other filesystem data.

Configuration information is archived on disk, with dates and version numbers attached. Configurations can either be rolled back to the previous version easily or to a specific archived version by specifying an exact configuration file name.

Important administrative options:

StrongBox Configuration File

A number of aspects of the behaviour of StrongBox can be modified in /etc/strongbox.conf. The comments in this file should provide more than adequate information on what effect changes will have on the installation. See http://strongboxlinux.com/docs/strongbox.conf.

Administrative modes:

One aspect of StrongBox security is the use of administrative and non-administrative modes for the main StrongBox OS and for bundles. This functionality is, by default, used for StrongBox bundles, but not the main OS. To turn it on in the main OS, change “ADMIN_LOCK” to “true” in /etc/strongbox.conf. This will make sure that the /etc directory is locked by default on StrongBox.

To lock the main OS, run:

setadm off

To unlock it, run:

setadm on

Note: if you unlock it using setadm on, your shell creates a lock on the admin directories, so that they will not be locked until someone explicitly issues a setadm off again! Locking occurs so that if one administrator is doing a number of changes, other users making changes don't suddenly result in the files that they're editing not being writeable. However, if you leave a terminal open, and don't specifically run setadm off, that lock could hang around indefinitely.

Some files need to be writeable at runtime, even when the /etc directory is locked. These files are referred to as “writecached” files, and can be specified in strongbox.conf. By default /etc/admins (the admin home directories), /etc/lvm (the lvm configuration files) and /etc/fstab are writeable even when /etc is locked.

A number of the StrongBox utilities will automatically lock and unlock StrongBox as necessary. These include:

admuser

passwd

edit

bundle

strongbox

Bundles also have administrative modes. More information about this can be found in administrative and non-administrative modes in bundles.

MySQL Logging:

StrongBox is set up to easily allow logging to a mysql server. This is the preferred method, as it allows the greatest amount of flexibility for log viewing, archiving and servicing.

The logging subsystem should detect the connection to the mysql server automatically, and start using it once it's available. By default, it looks for a mysql server running in a bundle called mysql, and tries to connect through a local socket to there. To set up logging, you should edit /etc/strongbox.conf if necessary and insert the appropriate information for your mysql server. Then simply run this:

mysql -u root -p < /etc/syslog-ng/mysql-layout.sql

with other options (such as -h hostname) as appropriate to your installation. This will set up a new database called “syslog”, and will give insert and query permissions to the user “logger” for it, with no password. To find out if you're successful, try running querylog, and see if it can connect successfully.

Some files in /etc need to be writeable at runtime. In order to facilitate this, you should add these files to /etc/strongbox-writecache.

The StrongBox Initial Boot

The base StrongBox OS runs entirely out of RAM and/or Virtual memory. This design choice has a couple of drawbacks, and several advantages. The main drawback of this design is that StrongBox always requires a larger amount of Memory to run. Depending on the type of ramdisk that is used, this memory may be swapped out to disk or may not.

This ramdisk is not intended to provide applications and a full suite of packages. It's intended to provide only the tools required to administer the computer remotely, and manage hardware and administrative access control. Other functions are provided by StrongBox bundles, and by StrongBox add-ons.

In order to make this happen, the StrongBox initrd builds the ramdisk on the fly. It also exhibits some intelligence in making sure that the “best” boot image that's available to it is booted up, by searching through all of the possible options, and preferring the most recent ones.

Add-Ons

Add-ons are small packages of additional software for the main StrongBox OS. They run off of the disk, and contain packages would make the ramdisk itself too large. They encompass commonly used functions that may require closer access to the hardware.

To mount an addon, run: vmount opt:<name>.

To mount all addons, run vmount opt.

NB: in order to use services contained in add-ons, they must run after the vmount subsystem. To do this, you should ideally add the service startup information to /etc/conf.d/local.start.

Boot parameters

In addition to standard linux flags, the following parameters affect the boot process:

Partition Discovery and Hardware Detection

The first thing that the StrongBox initrd does is take stock of all the partitions that are available to it. Each partition is mounted read-only, and all the partitions are scanned for files in the appropriate locations. Raid partitions of the type “raid autodetect” (code 0xfd) that are properly initalized should be autodetected (unless some devices are missing). Lvm partitions should also be autodetected. If you use these with SCSI or SATA disks, make sure you pass a correct module=xxx parameter to StrongBox.

If the initrd is unable to the first listed config file (or /configs/current by default) on partitions that it has autodetected, hwsetup is run, which will load any device driver modules that it deems appropriate. Then, the partition discovery will be attempted again, with the intird using all the config files listed, not just the first one.

The reason for this is to make sure that, if possible, the initrd is using the most recent config file available, even if the disk requires the loading of modules that haven't been specified on the command-line. However, if you require special modules to be loaded, it's better to specify modules=[module_name] on the commandline instead -> the boot will be much faster, and you won't run the risk of a crash due to misbehaving hardware being initialized too early.

StrongBox Boot Config Files

The first thing that StrongBox does after mounting partitions is look for config files. By default, the following files will be searched for (in order):

The configuration file contains rules governing the size and type of Ramdisk created, which OS Bundle is used, and which patch file is used. These files can reside on any partition on the system, as long as that partition found during the discovery phase.

A given config file can contain multiple variations that it can attempt to boot. This can be used to try one type of ramdisk, and then another, or look on multiple disk partitions for it's files. It can also be used for other variations, such as Ramdisk size, or the setting of certain environment variables. The following variables may be present:

The functions will be tried in order. If none of them work, a config will be marked as “failed” by the boot process, and others will be tried.

Boot configuration files are located on the patch partition at: /configs/archive/config-<date>... The ones that are currently in use are liked to /configs/current, /configs/previous and configs/lastgood. The “lastgoodfile” is created the first time a configuration save is made after boot. It's created based on the /imageinfo/bootinfo file that is created by the initrd, and thus is verifiable as having worked.

A new config file has been added in the beta series: preboot. This is a place where you can enter a script that will be excecuted at boottime. Note: it may get excecuted more than once, so ideally, if it does some fixed activity (like mounting a disk), it should set a variable that it can find later to know that it's already excecuted. A typical example of such a script would be:



[[ ! -n “$PREBOOTRUN” ]] && PREBOOTRUN=true || return 0

modprobe docprobe

modprobe aacraid

mountdiscs



This would probe for disc-on-chip modules, and adaptec scsi discs, and then mount all valid partitions. It would also run only once, because of the first line. A more complicated example might be:

[[ -f /mnt/sda1/my-modules/mymodules.squashfs ]] && \

MODULESSQUASHFS=/mnt/sda1/my-modules/mymodules.squashfs

loadmodules my-hardware

This would use a particular archive for StrongBox modules, and load the “my-hardware” module from it (loadmodules will try it's own module store, and then look for a complete modules.squashfs file if it can't find it). It will also use that module archive at boot time, instead of the one that it would normally find.

The “default” script can also have special instructions put into it. The default script will run after other scripts have failed.

You could use this to boot another OS entirely:

[[ -b /mnt/hda1 ]] || return 1

unmountdiscs

mkdir -p /newroot

mount /dev/hda1 /newroot || return 1

pivotroot_exit

Note: this does mean that any administrator with a certificate that has the “config” capability can excecute arbitrary code at bootup. This is an important consideration when deploying keys from your organization's CA.

More information is available by looking at /configs/default on the cdrom.

StrongBox OS Bundles

There is presently only one type of OS bundle, which is a squashfs based bundle. In this bundle, most files (including /lib, /bin, /sbin and /usr) are located in a squashfs loop device, to save space. It currently takes up a footprint of approximately 35MB, plus a little extra space for new files and configuration changes. The default RAMDISK size is 42MB, as a tribute to the late, great Douglas Adams.

Ramdisk Types

The default Ramdisk type is tmpfs. This creates a dynamically resizeable boot volume out of virtual memory (RAM and SWAP). This is the best general purpose ramdisk, as it scales the best to low memory situations. With TMPFS, you can boot a system that has only 64MB of RAM, if you have a pre-initialized swap partition. It also makes the most efficient use of memory possible, and works quite reliably and quickly.

Having CONFIG_RDPLITSIZE option in your config, you have the ability to have administrative and non-administrative modes in your OS. This works with any type of Ramdisk, and adds additional security to it. The other types of RAMDISKS have advantages for some applications, including.

a) They will always remain in RAM. This has the added side effect of allowing you to access the StrongBox OS even in some cases where one or more disks are failing... including potentially where it's virtual memory resides.

b) It allows you to set the immutable bits on some files.

Three other types, and are mostly for legacy installations. They are: dmrd: a concatenation of ramdisks using device mapper, lvmrd, which was used for 2.4 kernels before device-mapper existed, and bigrd, which uses a large-size ramdisk with an ext2 filessystem. Other types are being developed, including an encrypted ramdisk (based on dm-crypt).

StrongBox Patches

StrongBox patches are simply a set of changes to the OS Bundle (see patches, under StrongBox System Files). This is mostly configuration data, but could also include upgrades to certain pieces of software, and actual program data in certain situations. By default, only /etc and parts of /var are saved. Additional includes or excludes can be made in strongbox.conf

Patches are generally between 100 and 500K, and contain all the information necessary to run the computer. Warning! They frequently contain sensitive data such as passwords, and care needs to be taken with where they go and who can read them! By default, all strongbox patches are readeable only by root.

Putting it all together and preboot

Currently, a simple pre-boot check is done to make sure that the ramdisk that has been built is viable, including running ldconfig, and making sure that /bin/bash links correctly. It also create the extra partition for /etc, which can be remounted read-only on the fly for better security (see adminmode). Also, if the ramdisk is not tmpfs, a number of sensitive system files are marked as immutable, as an extra security measure. This options can be turned off by passing “nochattr” to the boot loader.

The StrongBox OS Boot Process

After the initrd has finished and handed over control to the StrongBox OS, the strongbox OS boots as a standard linux distribution, albeheit with some differences. In the Strongbox system, nothing should be put into the fstab: all mounts should be done by the vmount system. Unlike a regular linux system, we want all the mounting and checking of hard disks to take place after the system is up and accessible: this allows us to administer the system even in the case of catastrophic disk failures, etc.

Basic OS services

Gentoo is used as the base for the StrongBox initrd, and this boot process is followed. A number of services are provided here:



There are a number of services that can optionally be used:

There are also a number of utilities, such as vlan and bonding utilities, filesystem utilities, lufs, fuse, submount, dhcprelay, quota utils, usermode linux, links, etc that are provided.

Then, a number of services and utilities are provided by add-ons:

2 Stage boot process

Once the basic services are started, the boot process forks. You can now log into the box, and use it fully. However, all the application bundles are starting up in the background at this point.

The intention here is to have your StrongBox system be fully administerable, even if there is a problem with one of the subsystems coming. up. This is yet another area of redundancy provided by the system.

The vmount system

The vmount system was developed to make it easier to deal with a large number of partitions and shares of partitions. It implements recursive mounting, dependancy checking, and sharing filesystem resources in a fairly intuitive way.

Once the basic OS services are started, vmount comes up and checks all partitions that are listed in /etc/conf.d/vmount, and loads up the bundle config partition. Then, the app-system loads and checks the signatures on all the StrongBox bundles in preparation for the application subsystem starting.

Filesystems and Data Storage

Best practices in Data Storage

Best practices in data storage would require the following:

These features are implemented using the StrongBox vmount system.

The StrongBox vmount system

Data storage in StrongBox is intended to be as flexible and as portable as possible. To accomplish this aim, lvm and the vmount system are used extensively. LVM allows the creation of virtual, resizeable partitions on which data can be stored. This allows some storage to be allocated at install time, with extra storage available for those parts of the operating system that need it, when they need it. Also, volumes can be moved between physical devices while in use, allowing easier removal/insertion of hot-pluggable devices (firewire, scsi, sata, etc). Combined with reiserfs, which allows the growing of partitions at runtime, it's very easy to manage data storage in StrongBox. EVMS is also included, although it is not supported for use in configuration volumes.

Another concept the vmounts implements is failover. Multiple volumes can be allocated to a given task, and, if the first one fails, the second volume will automatically be mounted. Note: there's certain cases where this behaviour is not desired: you should take a careful look at the vmount files in this case, to make sure that unwanted actions are not taken.

The vmount configuration files are located at /etc/vmounts. In there, there's a few that are created by StrongBox automatically as needed, (such as boot and dynamic), some that are made by the installation of bundles (each bundle has it's own vmount file) and some that are used generically for storage (such as data and run).

Filesystem Layout

StrongBox uses it's own filesystem layout that makes it easier to deal with backups. Because backups are only going to happen of certain files, having the files divided by the type of storage that it represents, as opposed to having data simply being associated with the application, makes sense.

As a general rule, there are many types of storage that are required by StrongBox bundles:

Temporary storage is generally allocated from tmpfs, and is allocated on the fly where it's needed.

Space for temporary files and cache is generally allocated from a shared temporary partition: /dev/data/tmp by default. It's name in vmount is run:tmpshare., and it's mounted on /data/tmp.

Space for data files is generally allocated from the “data” vmount. It will automatically detect if there is a particular partition allocated to that service, and, if it's not, allocate it from a generic /dev/data/data device. The vmount name for a data partition is data:<service>, the service being, for example, mail or mysql. The generic data volume is mounted at /data/source, and individual data partitions will be mounted at /data/<partition>.

Space for application files is by at /bundledata and /bundlesmounts. The static bundles should all be in /bundledata (if they're loop devices: they can also be lvm partitions), and they should be mounted at /bundlesmounts/<bundle name>-<bundle version>. From there, they're bound to whichever bundle needs them.

StrongBox configuration files are in various locations. The boot configuration files are not mounted, and are only remounted when they're specifically required. This is both an extra security measure, and a means of making sure that the OS boots correctly in the case of a hard reset, as it generally ensures that that filesystem is not in a read/write state at the time of the crash. When the boot volumes are mounted, they're mounted at /mnt/boot and /mnt/patch (which may or may not be the same volume). These mounts automatically generated and re-generated by the strongbox console, and their vmount names are boot:os and boot:config respectively.

The bunde configuration files are all located at /modules, which should be mounted as modules:config during the boot process. Each individual bundle configuration will be at /bundles/<vserver name>. Note: the bundless partition is remounted read-only and read-write during the boot process of the bundles. Again, this is to ensure that during a hard crash, these configuration files are not part of an in-use filesystem, and thus subject to more data corruption.

Removeable storage is implemented using the submount daemon (see http://submount.sf.net). Cdroms are mounted using /etc/vmounts/cdrom. The defaults in these files are probably useful for a lot of situations, but you may well need to alter them for your situation (for instance, uncommenting the second cdrom device if you have more than one).

Removeable USB storage is mounted and unmounted directly from hotplug, so they should require no manual intervention. They will show up at /mnt/usb, /mnt/usb1, depending on how many you have in. Also, if USB_LOGIN is selected in the strongbox.conf file, links will be created to hashes of any keys on those usb volumes at /etc/keys

Vmount configuration file structure:

The first word in a vmount file is the name of the mount. The vmount's name would be: <filename>:<linename>. <filename>:all and <filename> are equivalent: they both refer to every line in the file.

If a line in a vmount file contains <name>:<somethingelse> then it's a failover mount. It will be called when the line at <name> fails to mount only. You cannot and should not mount both the mount and the failover at the same time: if you need both at times, then you should probably be referencing another virtual mount, that will choose the correct version automatically.

The second item in the file is the type. A list of available types if provided in the next section.

The third item is a source. This can be a virtual mount (referenced like bundles:mybundle-1.0.1), a directory, a file, a device or the special word “none”. Not all mount types support all files/devices.

The fourth item is the destination. This can either be a directory, a file (if the mount type is a bind, and the source is also a file), or DEST, which specifies the user/program passed destination of the vmount.

After that, all items are treated as options.

Available vmount types and options

Here is a quick list of vmount types and options:

Vmount types:

Vmount options:

Variable substitution:

Most vmount options and line items support variable substitutions. Any variable in the environment is substituted using @variablename@ in the vmount file. Also the keyword DEST or dest refers to the destination of a given mount, when vmount is called with a particular destination.

Calling vmount manually

Use vmount <configfile>:<mountline> to mount a particular vmount. For instance:

vmount cdrom

or

vmount cdrom:all

will mount everything in /etc/vmounts/cdrom. By default, this contains a submount entry for the cdrom, allowing easier insertion/removal of the cd.

vmount opt:monitor

will mount the “monitor” bundle, which contains various network monitoring utilities (including ntop, snort etc).

Vmount is intelligent, and will not mount thing that are already mounted, and will mount using –bind things that are already mounted elsewhere. It also implements locking, to prevent multiple vmount/fsck operations from conflicting with eachother.

Vfsck is used to run fsck on vmounts. It will only check partitions that are not in use elsewhere. Vfsck all is run at boot by default.

Look in /etc/conf.d/vmount for what virtual mounts are mounted and automatically checked at boot-time.

Config and boot partition(s) Structure

The StrongBox config partition contains information about the OS that is booted at config time. Edits to this structure do not directly affect the running OS, but affect what will be booted the next time the machine is restarted. They are the virtual directories boot:config and boot:os respectively, which may represent the same volume. If you run vmount boot, these will be mounted to /mnt/boot and /mnt/patch.

The configs directory, /configs, contains all the configuration files that tell the computer how to use the other files. See here for more information on the boot process.

The patches directory, /patches, contains all the patches available, in separate directories for each OS version that they correspond to.

The /ramdisks directory contains the StrongBox OS bundles. These bundles stored in /ramdisks/<version>/boot/root-...tar.gz

The /kernels directory contains the StrongBox kernel(s). The format is:

/kernels/<kernel version>/filename

The files are: bzImage (The booting kernel), modules.squashfs (for kernel modules) and optionally modules.tar.gz (for modules as tar archives).

The /initrds directory contains initrds for StrongBox. They all have an embedded kernel version that they should be used for, and are named in the format:

/initrds/initrd-<version>-<kernel version>.gz

If the initrd ends with -comm.gz, then it's speficially for the commercial version of StrongBox, and will not boot configs/patches signed with non-commercial information.

Bundle Partitions Structure

Bundle config partitions contain information about running bundles. It is the virtual mount point bundles:config, and will mount at /bundles. Each directory here corresponds to a bundle name. /<bundlename>/config is always the current running configuration file for the bundle, and this file can be edited directly, preferably using the “edit” command, so that signatures can be more easily maintained (this file must always be signed, or else the bundle will not function).

This file is generally a link into the directory /<bundlename>/<bundleversion version>/config-<date>, which is the real configuration file. You can relink this to a previous configuration file to quickly roll back/change a bundle configuration.

You may also have a bundle data partition. It is the vmount bundles:data, and will mount at /bundlesdata. This partition generally contains a number of squashfs files that will be mounted as loop devices for static bundles, with the format:

/bundlename-<version>.squashfs

In the bundle data partition, you can create a directory called /opt. If you then copy the /opt directory of the cdrom into it, you can use these addons on the fly with StrongBox. This is the vmount bundles:opt.

Data Partition(s) Structure

The data partitions are mounted using “vmount data”. There may be only one, or there may be many partitions that are represented here. This depends on how the administrator wishes to structure information storage in the server. The general format is to have a partition assigned for a function (such as mysql), and then have each bundle share a portion of the partition in it's filesystem. This allows a great amount of flexibility, and allows certain functions not to get in eachother's way (mail spools can't get in the way of the sql databases, or vice-versa).

All the data:xxx partitions are mounted at /data, and can be viewed there. This allows for easier backups of the entire server using rsync, and quick perusal of how much space each type of data is consuming on the server.

By default, the data:xxx partitions look for a partition named /dev/data/servicename. If this partition doesn't exist, it will attempt to mount a subdirectory /dev/data/data instead. This way, for simplicity, a single LVM partition called “/dev/data/data” can be created to automatically act as a catchall for all data storage on the system.

Run Partitions Structure

Several partitions are used during runtime by many services. They are: run:tempshare, run:logshare, run:cache.

run:tempshare is mounted at /run/tmp and provides temporary files for all bundles. By default, it looks for a partition named /dev/data/tmp.

run:logshare is mounted at /run/locks, and provides a place for logfiles for all bundles. By default, it looks for /dev/data/logs

run:cache is used by the admin subsystem to make backups of all configuration files. It mounts at /run/cache

StrongBox Application Subsystem

Application bundles start after StrongBox has fully booted. There are many reasons for this, but the simplest is that it prevents disk or application problems from interfering with the administration of the StrongBox system. As the system is starting, logs are separated from the main system logs: all logs from this subsystem go into /var/log/vservers, and may be labelled with the name of the vserver that generated them.

StrongBox Static Application Bundles

Static application bundles are the most secure type of application bundle that's available on StrongBox. These bundles are implemented as a compressed loop device that contains all of the application code needed to run hundreds of applications. With static bundles, instances of the bundles are created based on the original, each with it's own configuration, information, services, files, etc. This allows quick and easy separation of StrongBox services into independent secure contexts, allowing for much higher security and portability.

Currently, these secure contexts are based on the linux-vserver project (http://www.linux-vserver.org). In the future, there will be options for creating application bundles based on usermode linux, and possibly other virtualization technologies (such as qemu, and vmware). However, linux-vserver bundles give the highest flexibility and security for the lowest performance hit, and so are used by default, and will continue to be the default.

A linux-vserver based application bundle can be based on any linux distribution, as long as it's at least somewhat binary compatible with the StrongBox kernel. Debian unstable and Gentoo are known to work with only minor alterations, and other distributions, such as Fedora, SuSE and Mandrake should work fine as well.

Primitive support for creating application bundles is provided by the mkiso script. Note, any static application bundles that you create are not supported!

Application Bundle Configuration and Patches

In order to deal with a static (read-only) application bundle, the requirement is there to have configuration and patches based on the bundle for all the information unique to that instance. The most central piece of this is the configuration file, that is generated using the bundle management section of the strongbox console.

This file contains the following variables:

Linux-vserver configuration:

If you wish to configure the capabilities and ip information of the vserver yourself, you'll need to manually edit the configuration files. They will be at /etc/vservers/<vserver_name> . See www.linux-vserver.org for more information on how to do this: this is beyond the scope of this document.

If you wish to run a standard inittab on the box, then you should remove the “inittab” file from this directory. If this file exists, it will overwrite the vserver's inittab. This is done to prevent all the vserver services from coming up, which are likely to fail and cause errors. On one hand, this allows the box to start up more like a regular server. On the other hand, it requires a more complicated configuration of the bundle to be done, and probably requires turning off a number of services. (such as services that check local filesystems).

Note that full linux-vserver style servers should work fine under strongbox, although they're not currently supported by the strongbox application bundle scripts. However, this code path is not recently tested, and may have annoyances or bugs. Please report any problems here to bugs@strongboxlinux.com.

Filesystem Layout and Data Storage

For information on how the vmount system works, see the “StrongBox Vmount System” section.

By default, bundles will generally come with a set of default mounts for everything in temporary storage. Some of the temporary storage locations are saved during a configuration save (bundle <bundlename> saveconfig), and some are not, depending on the configuration of the bundle. In general, only /etc/, some directories under /var/lib, and occasionally /dev should be saved in a configuration file, although some bundle design may require a different format for this.

In general, directories under /var/lib will require uncommenting a line from the bundles vmount file, if you intend to use that data. This is done to ensure that no permanent storage of data is done unless you specify it, saving valuable resources on your server. A number of applications (such as mysql, postgres, ldap) have an entry (or several) in the vmount file for them: it's up to you to enable the entries, or change them as desired, to suit the needs of your server.

In many cases, there exists a “firstrun” entry for a particular line in a mount file. This is generally done when there is a share/bind mount for something. The “firstrun” entry will create the directory, and generally has a “copy::dest” directive as well, to copy the information from the bundle's directory into the created directory. Subsequently, the directory exists, so no copy will be done.

Administrative and non-administrative modes in bundles

The “LOCKCONFIG” variable in the configuration file controls whether configuration file locking is done at boot time. Currently, this only controls /etc, and makes /etc read-only until a “setadm on <bundlename>” is issued, or “configuration mode” is entered in the strongbox console. Conversely, “setadm default <bundlename>” locks the bundle if “LOCKCONFIG” is set, and “setadm off <bundlename> locks any bundle, regardless of whether LOCKCONFIG is on in the configuration file or not.

Also, administrative mode in the strongbox console will run the “/bundles/<name>/scripts/configmode” script, if one exists and is excecutable. There is no default script here, but it can be used as a convenience.

Authentication and Security

Authentication and Security in strongbox are complex and interesting subjects, as they are in any OS. However, because of the multi-layer security model in Strongbox, with virtual machines running inside machines, the authentication needs are even more complicated. Indeed, every virtual machine could have it's own authentication system, which can make it very confusing.

Domain model vs. Distributed model

The domain authentication model is based around the principle that a number of computers have the same authentication needs, and that need should be met by a subset of the systems involved. There's three major systems for this: NIS, LDAP and Kerberos. In some cases, these systems may even be combined for certain functions. The advantage of a system like this is that it's easier to manage once you have more than two computers involved. The disadvantage is that it introduces external points of failure that can prevent other systems from working.

A distributed authentication model uses autentication information on each system involved. It may vary from system to system, and service to service. Generally, this is useful for small installations of one or two computers, but doesn't scale very well.

A type of distributed authentication that also has aspects of domain authentication is the use of digital certificates. These allow a common authentication token, present on a single machine, to be used for authenticating to other machines. Through the use of certificate revocation lists, and careful management, it's possible to make it work like a domain authentication model, yet without having the same single point of failure.

StrongBox has, to varying degrees, support for all of these. Support for LDAP and X.509 certificate authentication, specifically, is very good. How to implement a system like this, however, is left up to the individual administrator, and is outside the scope of this document.

“Hard” security, “Soft” security, secrecy and trust

Soft security is a term I like to use to describe security that is implemented by a piece of software: it's soft, in the sense, that if the software malfunctions, or one gains control over that software, then a compromise is trivial. An example of this type of security is UNIX passwords, or a Windows domain.

Hard security is a term used to describe security that must be broken directly, and cannot be backdoored through software flaws. An example of this type of security is encryption (assuming that the scheme used is effective).

Secrecy is a relative term that describes how much privileged information a security service has to have in the system that it's running on to perform the authentication. The StrongBox system boot sequence, when it's checking digital signatures, this is a high secrecy system: the system doesn't need access to the private keys to authenticate the signatures, only the public certificate of the root CA. A plain text password file is an example of very low secrecy system: it requires direct access to the passwords involved. A unix password file is middle of the road: it requires access only to hashes of the passwords, which are somewhat less sensitive than the passwords themselves.

A domain authentication is generally medium secrecy on the domain server, and high secrecy on the client (except for some domain models that cache too much privileged information on the client). The security is good as far as a local system is concerned, as a local system could be overriden to not use the security model, but, because of the high amount of secrecy on the client, one system can be compromized without compromizing others.

The net result is that a domain authentication model must trust some systems much more than others. Which systems are trusted the most depends on how the system is implemented. Any systems that are not trusted should contain as little information as possible about the authentication system.

Users and Administrators

The main StrongBox OS is really only intended to allow administrators to log in. As such, authentication for these users should have a) a high level of security and b) if possible, no single point of failure. Certificate or public key based authentication using ssh is ideal. How to set this up is described in the document “Keys to the kingdom”. Generally, it's recommended to have more than one authentication method for users, such as USB key plus ssh key, or passwords and ssh.

Each StrongBox bundle can have it's own authentication system for users of those systems. Probably the ideal way of doing this for a number of bundles is using ldap: assign one bundle to be an ldap server, and have all other bundles authenticate to it. This allows the bundles to have a no trust of eachother, but a high trust of a single bundle (which should not be remotely accessible), and complete secrecy from one bundle to another, making the compromise of one piece of the system less damaging.

Sudo

One greatly underused piece of software in unix is sudo. Sudo is similar to su, in that it allows users to run commands as root. The major difference with sudo is that it has a very specific list of commands that those users are allowed to run. Sudo then denies attempts to access commands that are not specifically authorised, and emails the administrator about the breach.

Another thing that sudo provides is terminal timeouts. When administrative commands have not been used for a certain amount of time, a password is required for commands to be excecuted. This means that unattended terminals reqire a password in order to access root commands, if the administrator is consistently using sudo for all administrative commands.

Finally, sudo also provides logging. Each command that is excecuted using sudo is logged to the system log, including the username, the date, and the time.

StrongBox has provided a simple script called “sudowrapper”. This is a very quick script, and it allows you to wrap commands so that user accounts will automatically use those command through sudo. To use this, run:

sudowrapper wrap mkswap

After this, any time a user calls “mkswap” then sudo will attempt to run the command for them as root. Note: this doesn't set up access rules: you need to do that yourself, using “edit /etc/suduoers” or “visudo”.

IP access lists and port knocking

Access control is another facet of authentication. IP access lists have been used for a long time, especially for sensitive systems like routers and domain authentication servers, or sensitive servers like file servers or accounting servers. The general rule of thumb is: if it doesn't need to be accessed from a particular location, it should be allowed to be accessed.

By default, the firewall on StrongBox allows all traffic out, but only ssh and dhcp client traffic in. If you want anything else to function, you'll have to activate it yourself. There is a nice administration interface in the webmin bundle for doing this. Or, you can edit manually, in /etc/shorewall/rules. Note: there are many, many options for firewalling in shorewall, try “man shorewall” for some ideas, or consult “http://www.shorewall.net” for more details.

There exists a standard problem with IP access lists, which is services that need to be available remotely. There's a number of solutions for this, including ipsec. See http://www.openswan.org for more information on on the ipsec implementation used in StrongBox.

However, sometimes ipsec is too heavy, and what is needed is a simple replacement for ip access lists; something to prevent the service from appearing available to prying eyes (such as a compromised server scanning for an open port), yet allow it to be accessible from any IP. An example of such a solution is port knocking.

StrongBox linux is the first linux distribution that we know of to incorporate port knocking by default. All you need to do to turn it on for ssh access is edit /etc/knocker.conf to add in your port knocking password, and restart the syslog daemon (/etc/init.d/syslog-ng restart). This implementation is called simply “knocker” and is specific to strongbox. In the author's view, this system is very secure, and is a good addition to security for more sensitive systems. However, port knocking is not a perfect system. For more information about port knocking in StrongBox, and how it works, see:

http://strongboxlinux.com/projects/portknockingindex.html

For a dissenting opinion about port knocking (why you shouldn't use it) see:

http://www.newsforge.com/article.pl?sid=04/08/02/1954253

Note that many of the complaints in this article do not apply to the StrongBox implementation of it.

Good practices prevent bad surprises

There are many ways, despite the best of intentions, that you can make a StrongBox system inaccessible, or accessible to the wrong people. Some good rules of thumb:

The Linux-Vserver System

This section needs to be fleshed out. For now, go to http://www.linux-vserver.org

Linux Capability Set

This is very critical to the functioning of linux-vserver. The kernel documentation on the subject available at http://strongboxlinux.com/docs/capability.h. The default capability set is quite limited: the intention is for it to be “safe”, in the sense that it is very unlikely to result in a compromize of the box by compromizing a single bundle.

To change the capability set for a bundle, edit /etc/vservers/<bundlename>/bcapabilities, and restart the bundle. A list of all capabilities is included at /etc/vservers/<bundlename>/bcapabilities.all

Using dynamic bundles

StrongBox Dynamic bundles are possible, which are very much like regular vservers, if you set the “DYNAMIC_BUNDLE” flag in the bundle config. If you do this, they will operate almost identically to regular vservers, although some of the StrongBox functionality, like ip management and inittab management, still work (although they can be turned off).

Setting up dynamic bundles using debootstrap

The easiest way to create a dynamic bundle would be with debootstrap. Set up a static bundle normally in the strongbox console. Don't start it however. When it's created, run:

lvcreate -n <bundlename> -L 2G data

mkreiserfs /dev/data/<bundlename>

Then, find /etc/vmounts/bundlename file for it with the following information:

module bundle /dev/data/<bundlename> DEST

pts devpts none dev/pts

proc proc none module/proc

tmp share run:tempshare module/var/tmp options::mode=777

tmp2 share run:tempshare module/tmp options::mode=777

Then mount it into an existing bundle like so:

vmount <bundlename>:module /vservers/oldbundle/root/

Then run:

bundle oldbundle enter

debootstrap sarge /root

When debootstrap is done, you should unmount it, using: vumount <bundlename>. Edit the config file, using edit /bundles/<bundlename>/config, or in the strongbox console, and set “DYNAMIC_BUNDLE” to “true”. Then, start the bundle normally.

Deployment Considerations

There are many ways to deploy strongbox. Because of it's low dependance on particular media, it should be relatively easy to deploy in various different ways. A cdrom for boot, 64MB flash media for storing configurations would be adequate for a firewall. Indeed, the entire installation can be cut down to approximately 50MB, allowing for very small embedded deployments.

On the other end of the spectrum, having many different disks is possible with LVM and software RAID. Autodetection of RAID and LVM at boot-time allows you to have a very small real partition for the kernel and initrd (approximately 10MB), simply so that the BIOS is able to see a portion of the OS.

Backups and Recovery

Some kind of backup strategy should be in place with StrongBox installations. One possible strategy is net backup, using rsync. This can easily support incremental or full backups, and can perform checksum operations to ensure the integrity of a backup. Another strategy is a tape backup, however, without a tape library, this strategy is better suited towards easily accessible systems. There are a number of commercial backup packages that can be used with StrongBox as well: these should ideally run within a bundle, rather than on the core StrongBox OS, but that decision is left to an individual administrator.

Upgrades and Patches

Upgrades to StrongBox components should be relatively painless. When a new OS bundle is copied to the the boot volume, one can migrate it in the StrongBox console. Choose “Manage Patches”, then your patch version, the patch itself, and then “migrate patch”. You'll then be asked which specific ramdisk to migrate from/to. You will then be asked to resolve any conflicts between files in the patch and files in the new ramdisk. The net result of this process is that you will have a new patch based on the old one, built for the new ramdisk. At this point, simply add this patch to your boot configuration, and reboot.

If the OS doesn't work as expected, you can roll back, by adding the “config=previous” boot argument to the bootloader.

Upgrading a bundle is a similar process. Run the command “bundle <bundlename> upgrade <version number>. This will present the same “migratepatch” command, and allow you to start using the new patch and bundle version after you stop and start the bundle. If the new bundle doesn't work as expected, simply stop the bundle, and run “bundle <vserver name> start previous” to load it up with the previous bundle and patch.

Methods for updating files on the OS ramdisk and the bundles without rebooting/restarting the bundle are in the works.

Assimilating existing linux deployments

With StrongBox, it's possible to run an existing linux depoyment in a dynamic application bundle. There is no howto for this, and it's very installation dependant. However, the linux-vserver tools including in StrongBox do come a long way on this path. Here's some considerations:

  1. Having free disk space available is ideal. If you don't have any, StrongBox comes with the commands resize2fs, resize_reiserfs and ntfsresize for dealing with this. These are command-line tools, and come with reams of warnings, but, if you don't cancel them or try and reset a system while they're running, they're mostly quite safe.

  2. Follow the instructions for creating a dynamic bundle, only without setting the vmount information or creating an lvm volume for it. Instead, you need to convert the mount information from the existing installation. To do this, look in /etc/fstab, and make a vmount file as follows.

This gawk script should work for most cases:

gawk 'BEGIN { a = 1 } ; $1 !~ /^#/ && $1 !~ /^$/ {print "mount"a"\t"$3"\t"$1"\t/vservers/server_name"$2"\toptions::"$4 ; a++ } ' /etc/fstab > /etc/vmounts/server_name

However, you'll want to remove cdroms, floppies, that kind of thing, and only keep the most essential pieces.

  1. chroot to /vservers/server_name and disable the startup of all disk related scripts (this is distribution dependant, but rc-update, or chkconfig is probably your friend for this)

  2. try bundle bundle_name start

It may well fail the first time and require some tweaking to get it to run. But, the principle is there. Redhat 7.3, Plesk, Debian testing, Debian Unstable, Gentoo, etc. are known to work fine in this configuration, and I'm sure there's many other distributions that will work correctly as well.