How-To's site users contribute that others may find helpful...
Backups, ways to backup Debian
Each backup How-To should be on a specific topic.
"How to Backup Debian servers" -any takers?
"How to Backup Debian with Backupninja - home user"
"How to Backup Debian with Backupninja - business"
The home user how to make backups as easy as:
#backupninja -n
*Overview
Backups the never ending problem. Follow this How-To if you have:
a) a cd/dvd burner, (to use backupninja's makecd) OR
b) two linux pc's on a LAN, (to use backupninja's rdiff) OR
c) want an example of backupninja you can adapt to use:
-- a spare hard drive be it internal or usb (you can rdiff to a local directory/partition)
-- push backups over the net to another pc/server (that you have permissions to ssh into) (rdiff if trusted or duplicity to encrypt the data)
We will install, setup and use backupninja to create backups over a LAN and/or create iso images you can burn to cd/dvd using your favourite burning program.
Note: If mixing i386/amd64 and etch/sarge then see this post
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
*Information
There is very good documentation at backupninja and a guide to usage at debianhelp.co.uk (thanks to david23 for the link) and of course:
$man backupninja
$man ninjahelper
This excellent documentation and the way backupninja integrates backup into the sytem while providing multiple ways to backup at the same time are all good reasons to use it.
This How-To will not attempt to cover all that is available in the above, rather it aims to set up a easy backup system using cd/dvd and/or another pc on a home LAN.
Next Installation
How to Backup Debian with Backupninja - home user
If using Sarge:
I would suggest the backports version as it has ninjahelper which is an ncurses helper that sets up a few things like ssh keys the easy way.
You can install the stable sarge versions of rdiff-backup, duplicity, hwinfo, mkisofs, cdrecord, dvd+rw-tools which are suggests and any depends you don't already have.
Backports instructions will show how to use backports safely.
Details:
#apt-get update
#apt-get install backupninja rdiff-backup duplicity hwinfo mkisofs cdrecord dvd+rw-tools
Next Set up
How to Backup Debian with Backupninja - home user
What good would making backups be if you never tested the restore function or didn't even know where it was? Kind of like making a boot/rescue floppy and never booting with it till the system won't start on it's own any more! Do you remember how that story ended?
Unfortunately for us there is no nice graphical or semi-graphical helper that I could find at this time in Sarge to do this. There are a few being developed but they are in alpha stage at this time, not what you want to trust data to.
So it is left to the independant tools used and mostly command line stuff ie. read the man page. This is the reason I used 'rdiff-backup' and 'makecd' methods since they can partly use standard copy commands and your favourite filemanager.
makecd
Test the cd/dvd after burning, use the check data function of the burner software during burning. Place it in another drive and mount it. Copy random files to a temp folder and see if everything is ok. Optical disks have a limited lifespan, have you checked any disks you made 6+ years ago lately?
rdiff-backup
The beauty of rdiff-backup is that it creates a mirror and diffs. For the mirror normal tools can be used to copy the data to a test directory. Try using fish in konqueror over the network to copy a file out of /backup on the host machine to a local temp folder. Enter this at the location bar : "fish://root@hostIP" or "fish://user@rhostname" you need the remote password for that user and ssh access through any firewalls. If you know of any other programs that use fish insert here.
rdiff-backup can also be called upon to do time based rollback, see the man page and be carefull, test in a temp folder and don't overwrite anything it will happily whipe out whole directory tree's if you use --force, remember linux is a unix it thinks you know what you want to do.
How to Backup Debian with Backupninja - home user
The rdiff-backup set up bellow is based around having two linux pc's on a LAN.
I have my main box and my wifes laptop connnected via a LAN HS switch to each other and to an ADSL modem/router. She keeps my backups and I keep hers which include the family /pics database that she maintains.
Or if you have a trusted friend you could keep each others backups by using rdiff-backup (secure since it uses ssh) over the net or better still duplicity which encrypts the data then you don't have to trust them. Restoring data with duplicity may prove harder since it is not a plain mirror.
However rdiff-backup may just as easily be used locally to an internal hard drive or a usb drive mounted at /backup. Use a different unit from the data source otherwise you won't protect against disk failure or a seperate partition mounted at /backup and then use makecd to create iso images to burn to cd/dvd size allowing. Take these to work and stash them in your desk in case of fire/theft.
See the makecd section if you just want to prepare iso images for burning to cd/dvd. I have not tested if backupninja splits the images to the correct size for the type of disk.
//////////////////////////////////////////////////////////////////////////
Details:
/etc/backupninja.conf
comment out (place # in front) or change these lines as needed using your favourite text editor:
#reportemail = root
usecolors = yes
now it turns out the when line "when = everyday at 01:00" is the script default and will not stop backup attempts if simply commented out. So if you want to stop these automatic backup attempts then the cron job that is run every hour needs to be commented out:
/etc/cron.d/backupninja
#0 * * * * root if [ -x /usr/sbin/backupninja ]; then /usr/sbin/backupninja; fi
__Logic__
Since this is a home machine and not likely to be on and unused at 01:00 every night I don't want a string of failed backup attempts. With this setup we will run the backup when we are good and ready ie when the source and destination machines are on and unused (only light loaded is fine too) and the correct cd/dvd is in the drive.
Have you ever read root's mail as a home user? If you do then leave it as is. Otherwise /var/log/backupninja.log is the place to look, to ensure the backup completed every time you run:
#backupninja -n
color is up to you, I have a yellow root terminal backround so I set it to 'no'
//////////////////////////////////////////////////////////////////////////
/etc/backup.d/
We have installed a version of backupninja that includes ninjahelper in the Installation section of this How-To so that is what we will use to populate this directory (roll down to next section). Or you can copy files from /usr/share/doc/backupninja/examples or get my examples here.
-Note do not leave old files lying around it this directory as backupninja will not like it and tries to execute everything that does not have a .disabled extension.
-If your text editor likes to leave .old .bak or text.txt~ laying around then clean up behind it or turn that feature off!
-Files are executed in order so :
10.sys.disabled
20.makecd.disabled
30.sh
90.rdiff
91.rdiff
will try to run: 10 (but disabled) then 20 (but disabled) then 30.sh which does things that will be used in 90.rdiff then 91.rdiff.
_______________________________________________________________________
Just run ninjahelper, as root, in your favourite term (xterm, konsole etc):
#ninjahelper
An ncurses program should appear.
Select 'new' -> 'sys' use spacebar to toggle selection:
[x] packages
[x] partitions
[x] hardware
hit enter, a new selection should appear in the main menu
: 1 '/etc/backup.d/10.sys'.
_______________________________________________________________________
Just run ninjahelper, as root, in your favourite term (xterm, konsole etc):
#ninjahelper
Select 'new' -> 'makecd' answer questions:
>cd or dvd
>y
>/home/backupninja (if you have enough space here or /tmp/backupninja)
>backup.iso
>/dev/hdx (not required as we only making iso)
>/home/your_user_name (or /backup)
>enter any excludes like /home/user/Desktop/Trash
hit enter, a new selection should appear: 2 '/etc/backup.d/20.makecd'.
Select this and press enter again then 'run'
Use your favourite burning app to burn /home/backupninja/backup.iso to a disk.
_______________________________________________________________________
Just run ninjahelper, as root, in your favourite term (xterm, konsole etc):
#ninjahelper
Select 'new' -> 'rdiff' -> 'src' add include lines:
include = /root
include = /boot
include = /etc
include = /var/backups
include = /var/lib/dpkg/status*
include = /home
#or
#include = /home/'user1'
#include = /home/'user2'
>hit enter add exclude lines:
exclude = /home/*/.gnupg
exclude = /home/*/.local/share/Trash
exclude = /home/*/.Trash
exclude = /home/*/.thumbnails
exclude = /home/*/Desktop/Trash
exclude = /home/*/.mozilla/*/*/Cache
exclude = /home/*/.mozilla/*/*/Cache.Trash
exclude = /home/*/.beagle
exclude = /home/*/.aMule
exclude = /home/*/gtk-gnutella-downloads
exclude = /home/*/.giFT
exclude = /root/*/.gnupg
exclude = /root/*/.local/share/Trash
exclude = /root/*/.Trash
exclude = /root/*/.thumbnails
exclude = /root/*/Desktop/Trash
exclude = /root/*/.mozilla/*/*/Cache
exclude = /root/*/.mozilla/*/*/Cache.Trash
>hit enter 'des' add destination lines:
60D
/backup/'hostname' (you local hostname will appear here)
IP_of_dest_host (10.0.0.2 etc or hostname like 'otherbox' as found in /etc/hosts)
root
>hit enter 'conn' this will then test the destination and transfer the ssh keys. Follow the messages all must go well here or you won't be able to transfer data to the destination. Create the directory if it asks.
3 '/etc/backup.d/90.rdiff should appear in the main menu.
_______________________________________________________________________
Finished
You are now ready to run each menu item by selecting it, press enter, and selecting 'run'.
Or to run all the enabled actions when it suites you (say weekly).
Open your favourite term (xterm, konsole etc) and do:
$su
Password:
#backupninja -n
When backupninja is finished (the prompt returns) always check that the backup has worked by reading /var/log/backupninja.log and checking the destination and testing Restore.
Make it a habit to do the backup at the intervals that you have decided, the more frequently the smaller and quicker the rdiff will run.
Backupninja will attempt to continue if the network drops out and returns. But if you 'Ctrl+C' it's ass then it obviously won't finish and rdiff-backup's need to be run initially at least twice successfully, once to create the mirror then once to create the diff so do it twice for this first time.
A first run of rdiff backup took me 1h34 (3.62GB) of pics across 100M LAN through a switch. The second run takes less than 2min.
Next Restore
#/etc/backup.d/10.sys.disabled
##### Gav - Disabled as I don't like running sfdisk every back up
##### too easy to partition the disk instead of dumping partition table.
##### See the shell script for dpkg selections and the much safer df.
##### Run only once on a new system/disk after running the other backups first!
# this config file will save various reports of vital system information.
# by default, all the reports are enabled and are saved in /var/backups.
# requires dpkg, sfdisk, and hwinfo
# (1) a list of all the packages installed and removed.
# this file can be used to restore the state of installed packages
# by running "dpkg --set-selections < dpkg-selections.txt
# (2) the partition table of all disks.
# this partition table can be used to format another disk of
# the same size. this can be handy if using software raid and
# you have a disk go bad. just replace the disk and partition it
# by running "sfdisk /dev/sdb < partitions.sdb.txt"
# (MAKE SURE YOU PARTITION THE CORRECT DISK!!!)
# (3) hardware information.
# detailed information on most important aspects of the hardware.
# here the defaults were commented out:
packages = yes
# packagesfile = /var/backups/dpkg-selections.txt
partitions = yes
# partitionsfile = /var/backups/partitions.*.txt
hardware = yes
# hardwarefile = /var/backups/hardware.txt
#/etc/backup.d/20.makecd.disabled
# TYP is cd or dvd AS WELL AS the disk inside!!
burnertype = dvd
# not yet supported
system = no
# iso or burn to cd/dvd?
isoonly = yes
# location for image file
# where you have sufficient space
backupdir = /tmp/backup
# image filename
imagefile = bak_hostname_24Jul06.iso
# cd/dvd burner device
device = /dev/hdx
# dirs/files to include in the backup
# or /home/user or what you want to backup
target = /backup
# directories/files to be excluded
exclude = /proc
exclude = /sys
exclude = /dev
#/etc/backup.d/30.sh
# This script can run most any command normally run on the command line.
# Add anything usefull that you would like to do during a backup. But all it does as it stands
#is place files in /var/backup. So these will still need to be backed up by rdiff and friends
# (1) a list of all the packages installed and removed.
# this file can be used to restore the state of installed packages
# by running "dpkg --set-selections < dpkg-selections.txt"
dpkg --get-selections > /var/backups/dpkg-selections.txt
# (2) used with along with /etc/fstab and /var/backups/partitions.hdx.txt
# to decide on how to partition a new disk during a recovery.
# possibly by running #"sfdisk /dev/sdx < partitions.hdx.txt"
# (MAKE SURE YOU PARTITION THE CORRECT DISK!!!)
df > /var/backups/df.txt
#/etc/backup.d/90.rdiff
# options = --force
# when = everyday at 02
[source]
type = local
keep = 60D
include = /root
include = /boot
include = /etc
include = /var/backups
include = /var/lib/dpkg/status*
include = /home
#or
#include = /home/'user1'
#include = /home/'user2'
#/home excludes
exclude = /home/*/.gnupg
exclude = /home/*/.local/share/Trash
exclude = /home/*/.Trash
exclude = /home/*/.thumbnails
exclude = /home/*/Desktop/Trash
exclude = /home/*/.mozilla/*/*/Cache
exclude = /home/*/.mozilla/*/*/Cache.Trash
exclude = /home/*/.beagle
exclude = /home/*/.aMule
exclude = /home/*/gtk-gnutella-downloads
exclude = /home/*/.giFT
#/root excludes
exclude = /root/*/.gnupg
exclude = /root/*/.local/share/Trash
exclude = /root/*/.Trash
exclude = /root/*/.thumbnails
exclude = /root/*/Desktop/Trash
exclude = /root/*/.mozilla/*/*/Cache
exclude = /root/*/.mozilla/*/*/Cache.Trash
######################################################
## destination section
## (where the files are copied to)
[dest]
type = remote
directory = /backup/'local_hostname'
host = 'dest IP or hostname'
user = root
Just basic shh information.
Secure SHell allows a user to securely login to a remote machine from a terminal and execute commands as if sitting at the rhost's keyboard.
To login to a remote machine:
$ssh user@rhost
//////////////////////////////////////////////////////////////////////
*Sources of info:
$man ssh
"Setting Up SSH Keys"
//////////////////////////////////////////////////////////////////////
*More secure
The connection data is encrypted so ssh can be used over an insecure network (read the internet).
The sshd runs in the backround and listens for connections on port 22 by default. So if you don't want anyone to be able to login to your machine remotely then you have to change the default behaviour or block that port/protocol on your favourite firewall. Having said that if you have chosen all your account passwords wisely then you are quite secure, see bellow.
You can of course be more secure by not running the sshd at all if you never plan to use it or not allowing password only login's (by using keys). Or setting up your firewall to only accept connections on that port/protocol from specific hosts, MAC addresses or IP address that you would like to use ssh from.
For sshd exposed to the internet:
Beware that dictionary attacks are fairly common. I have counted >11 different IP addresses each trying this in a week and I don't run a well publicised site! So for improved security I would suggest one or more of the following:
1. Use a non standard port since port 22 is obviously targeted.
2. Don't allow direct root login since then user_name=root and now only the password has to be guessed. Login as user then su, now 3 things have to be guessed.
3. Don't use real 'dictionary' words for user_names or passwords. eg Jared+exponential may seem hard to guess but both exist. Perhaps jaredss5+Supe537surfe or the such would be a better way to construct user+pass.
4. Commonly tried user_names are: user, test, future, admin, root, www, web. So beware that you have not left one of these lying around and that they don't have ssh rights.
To see if ssh is running:
$ps -A|grep ssh
4036 ? 00:00:00 ssh-agent
4263 ? 00:00:00 sshd
sshd is the daemon
ssh-agent starts with X not much use without sshd.
//////////////////////////////////////////////////////////////////////
*
SSH keys allow you to login to a remote host without a password. To set up ssh keys, follow these steps:
Use ssh-keygen to set up keys for logging in without a password.
user@host:~$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/user/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/user/.ssh/id_dsa.
Your public key has been saved in /home/user/.ssh/id_dsa.pub.
The key fingerprint is:
6d:e9:f1:da:a5:78:70:af:92:f9:dc:ae:12:9e:51:f4 user@host
Append the contents of id_dsa.pub to the authorized_keys file in your home directory on a remote host. Note that the user names do not need to be the same on the two hosts.
user@host:~$ scp ~/.ssh/id_dsa.pub remote_user@remote_host:
user@host:~$ ssh remote_user@remote_host
remote_user@remote_host:~$ cat id_dsa.pub >> .ssh/authorized_keys
remote_user@remote_host:~$ rm id_dsa.pub
The next time you ssh to remote_user@remote_host, you will be logged in without needing to enter a password.
To login from remote_host to host, just run ssh-keygen on remote_host and append the contents of the id_dsa.pub to the authorized_keys file on host.
Alternatively, to login without a password between multiple (trusted!) hosts, run ssh-keygen on each host. Then append the contents of id_dsa.pub on each host to a single authorized_keys file. Finally, copy the authorized_keys file to the .ssh directory on each host.
Various methods of disk encryption/decryption users may find helpful.
This How-To assumes the user is familiar with Debian install procedures, basic command line tools and a console based editor of the user's choice. Familiarity with dm_crypt, cryptsetup, initial ramdisk functions and LVM2 is strongly advised. Some reading on chroot jails is encouraged. If you trash your machine using the techniques outlined below, you get to keep the pieces, and people will laugh and point. At you.
Think about what you are doing, and be careful.
Comments Welcome
Why do this?
I travel a lot.
In my travels, I frequently pass through security checkpoints carrying my trusty laptop.
In some of those checkpoints, I'm instructed by the (sometimes not too nice) underpaid, poorly educated, and clueless security person to boot my computer to show the world that it is indeed a computer and not some nefarious evildoing device.
"OK", I say, hit the start button, and spend the next 30 minutes explaining to this person and a growing crowd of other mistrustful security types what GRUB is, what Linux is... that it's really a computer, and I mean no harm. Meanwhile, three other people have booted their machines, some form of windows immediately came up, and they were immediately waved through. Hmm. I should be having a beer on the other side, not doing this.
Then one day I had a laptop stolen. That was not fun. I had to notify a lot of people, change a lot of passwords, I had to treat my keyring as blown, etc, etc.. I was not even carrying big time corporate secrets, but there was enough info in the various caches on that machine where some knowledgeable person could have gotten to my bank account. You get the idea.
Now I boot a completely encrypted Debian system (including swap), using a USB stick as the /boot partition to supply GRUB, the kernel, and a ramdisk. When the initial ram disk is loaded, I'm challenged for a passphrase that unlocks the encrypted portion of the hard drive. If the USB stick is not present at boot, the system boots to directly to Windows(tm), without passing through GRUB.
This will help in situations where a 'maid' or 'maintenance person' for a hotel boots your machine... heh.. I came back to my room once to find my slab lit up, when I left it powered down and charging... I'm serious about this!
Start with a laptop that can boot from the USB – sometimes this is a hidden feature that you don't find until you boot the machine with a usb stick in the plug and access the BIOS with the keyboard.
Get a flash drive – I use a 512 Mg Sandisk Cruzer Micro, but just about anything will do – it has to be a stick you can reformat, though (almost all of 'em are), tough, and big enough to put a /boot partition on. Quality counts here.
At the parts store pick up a nice, new UNFORMATTED hard drive (I got a nice Hitachi 100 Gig Drive for about $100.00 US, but prices will vary) for that machine, smile sweetly “no” when they offer to install for you and scurry home, package seal intact.
If you paid the Microsoft Tax when you bought the machine, then you should have a copy of the install disk around somewhere. If you threw it away, go buy a copy of Microsoft(tm) Windows(tm) XP(tm), from a legit dealer, NOT a pirate. I do not advise using pirate software. I DO advise using good software, so this may seem like an insane thing to do: you'll see why in a little bit.
Back up anything important you need. This is going to be a fresh install.
Replace your drive.
Install Microsoft(tm) Windows(tm) XP(tm), on a small partition at the beginning of the drive, remembering that this pig will take an enormous amount of space:
You will need about 10 gig for it to work with Service Pack 2, your laptop drivers and all the Microsoft(tm) updates.
Go online, get all the latest updates for that machine, etc...
Change the desktop around from the default, like all Windows(tm) "power users" do...
Stick AVG Free on it, and maybe a couple small games, something to put a lot of icons on the desktop to make it look like a hard-working Microsoft(tm) install. Don't throw the Microsoft(tm) disk in the dustbin just yet, you'll need it a bit later.
Fun Stuff:
1. Use Windows(tm) to partition and format the USB Stick. My SanDisk Cruzer came with all kinds of weird stuff on it that SanDisk claimed made it better, with their proprietary software. All it did was take up space. Get rid of that crap, and make sure you have a partition table on it that Windows(tm) can read, with one primary partition. You'll reformat the primary partition with ext2 later, anyway, but you have to make sure the partition table can be read by your BIOS at boot time, and if your BIOS was built around wintel (like mine) it's best practice to go with Microsoft's idiot routine for partitioning it. Remove the stick when you're done and set it aside for now.
2. Determine how you're going to partition the rest of your hard drive. I've elected to go with one encrypted swap partition and the balance of the disk as an encrypted base with LVM2 over the top of it, so that I can mount each LVM2 partition with limited permissions, and LVM2 allows me to resize as necessary without reinstalling. You may decide to place filesystems directly on encrypted partitions, and there's certainly nothing wrong with that. It's your call. Modify this procedure accordingly.
The swap partition size is determined by the amount of RAM I have available:
2.5 X (amount of RAM) = (swap partition size) => 2.5 Gig on this box for swap partition.
3. Do a BASE INSTALL of Debian Etch (kernel 2.6) (I used the netinstall ISO) to what will eventually be your swap partition, DO NOT define a swap partition in the install process, DO NOT flag any linux partition as bootable in the partitioning phase, but DO partition the entire disk. This disk will wind up looking something like this, with the ENTIRE Debian system on /dev/hda2, GRUB in the MBR of /dev/hda and no swap:
The free space thingy is a Microsoft(tm) idiocy, irritating, but not to really worry about.
4. Boot to your shiny new Debian, get your connection going, set up APT, and do:
apt-get update
apt-get dist-upgrade
If you get a new kernel during this process, reboot to it. 'nuff said.
5. You will need some tools:
apt-get cryptsetup hashalot lvm2 yaird reiserfsprogs
Then go off line.
6. You will need to get rid of another tool:
apt-get remove --purge mkinitramfs-tools. (initramfs-tools depends on udev to build it's ramdisk, yaird does not. I find that with the crazy boot configuration we're going to be building, udev gets in the way of a clean boot with initramfs-tools. Not to worry, udev just kicks in later in the boot process ;)
7. Prepare your USB stick:
Insert the stick. The stick's primary partition should be identified as /dev/sda1. Format that partition:
mkfs.ext2 /dev/sda1
Do not use ext3, or any other journalling filesystem. Journals do a lot of writing, and flash memory will eventually wear out with that kind of activity. Remove the stick and set it aside for now.
8. Prepare the hard drive partition for encryption:
We are going to encrypt /dev/hda5. First, we need to randomize it, so that the encryption hides in it like a mist in a cloud of random data. Therefore, we have to completely fill /dev/hda5 with random noise:
dd if=/dev/urandom of=/dev/hda5
This will take a long time to complete: my drive took about fourteen hours, at roughly 2Mg/sec. It's cpu intensive, so it generates heat. Prop up your laptop, put a fan on it, and go to the pub or something. Be aware that this method of generating noise on the drive may not give the best randomness, but there is a time stricture, and this method will definitely give you a strong base.
if=/dev/random would be the best with standard software tools, but it may take weeks, months, or even years to fill a large drive from the command line.
Don't use shred for this. Shred samples the random pattern only once, then writes the same pattern over and over again. The end result is not random noise. Effective erasure, but not random noise.
9. Create an encrypted mapping for /dev/hda5:
Depending on your kernel, you may have to modprobe the encryption algorithm you wish to use and the hash function, etc (the algorithm and hash used here are examples - the algorithm and hash you use are your secrets - do a little research on the subject):
modprobe twofish
modprobe sha512
cryptsetup -y create cryptdisk /dev/hda5 -c twofish-cbc-essiv:sha512 -h sha512
- the passphrase you give here will be the passphrase you enter at every boot from here on out. Pick a good, strong passphrase that you can remember. Don't write it down.
/dev/mapper should now have two items in it: control and cryptdisk
10. Set up LVM2 on the encrypted device:
First, we have to define what devices we want LVM to use - edit /etc/lvm/lvm.conf:
Under '#Advanced settings' you need to have the line:
types = [ "device-mapper", 16 ]
This lets LVM map on top of an already mapped device.
In the filter section, you will want to exclude the rest of the hard drive for LVM and the cdrom:
filter = [ "r|/dev/cdrom|", "r|/dev/hda*|" ]
When we're done with the edit, restart lvm: /etc/init.d/lvm restart
11. Define our LVM2 partitions:
pvcreate /dev/mapper/cryptdisk
vgcreate vgwhatever /dev/mapper/cryptdisk
lvcreate -v -L 300Mg -n lvroot vgwhatever
lvcreate -v -L 5Gt -n lvusr vgwhatever
lvcreate -v -L 3Gt -n lvvar vgwhatever
lvcreate -v -L 400Mg -n lvtmp vgwhatever
-Now use this command to find out how much room we have left:
vgdisplay vgwhatever | grep "Free"
We'll have a line like:
Free PE / Size (umptyscratch1) / (umptyscratch2)
Use the PE number (umtyscratch1) to utilize the the remainder of the disk for your /home partition, thusly:
lvcreate -v -l (umptyscratch1) -n lvhome vgwhatever
12. Check your work and make it permanent:
"ls /dev/mapper" should now yield seven items:
control vgwhatever-lvhome vgwhatever-lvtmp vgwhatever-lvusr
cryptdisk vgwhatever-lvroot vgwhatever-lvvar
Now edit /etc/crypttab to contain the following line:
cryptdisk /dev/hda5 none verify,cipher=blowfish,hash=sha512
13. And we want to test the system to this point (I highly recommend you do this now.):
make a new initrd:
yaird -o /boot/crypttest.img
edit your /boot/grub/menu.lst and make a new stanza for the kernel you're running OUTSIDE of the automagic kernels list:
title Debian GNU/Linux, kernel 2.6.17dc03 (or whatever your kernel is)
root (hd0,1)
kernel /boot/vmlinuz-2.6.17dc03 root=/dev/hda2 ro
initrd /boot/crypttest.img
savedefault
boot
any errors to this point will make themselves known, and we'll be able to take corrective action before you experience any data loss ;-)
Reboot, selecting the new entry in GRUB. You'll be asked for the passphrase. At this point, the kernel and ramdisk are loaded. Remember that in the future... Enter your passphrase, verify it, and the machine will continue to boot, and all 7 items in /dev/mapper should be defined in the process. If you goof with the passphrase, you'll be asked for it again. If you enter it wrong exactly the same way twice, the kernel will not map your encrypted partition correctly. THIS IS IMPORTANT TO KNOW. You'll use it to your advantage later.
If all worked well, replace the original initrd for your kernel with the one you built with yaird:
mv /boot/initrd.img-2.6.17dc03 (or whatever the original image was) /boot/initrd.img-2.6.17dc03.bak
cp /boot/crypttest.img /boot/initrd.img-2.6.17dc03 (or whatever the original image was)
...and remove the extra stanza in /boot/grub/menu.lst. Now you should be able to boot off normal options in GRUB, the exception being that you will now be asked for a passphrase every boot. Try it.
When it works to your satisfaction, do:
rm /boot/initrd.img-2.6.17dc03.bak
rm /boot/crypttest.img
Getting closer. We have defined a place we want to transfer our system to, that is fully encrypted. All data (including filesystem definition and information) that is written to /dev/hda5 will be encrypted, and only you have the key to unlock it. We've proven to our satisfaction that given the right passphrase on boot, the system will reliably map /dev/hda5 every time to our encryption specification.
Next: Transfer the system to our encrypted partition, and place the /boot partition on the usb stick.
1. Format the LVM2 partitions:
mkfs.ext3 vgwhatever-lvroot
mkfs.ext3 vgwhatever-lvusr
mkfs.ext3 vgwhatever-lvvar
mkfs.ext3 vgwhatever-lvtmp
mkreiserfs vgwhatever-lvhome
2. Mount the future root directory, and make empty static directories on it:
mkdir /mnt/tmp
(lost+found will be created on the ext3 filesystem when it's mounted for the first time, as it will for all remaining ext3 filesystems as you mount them first time)
mount /dev/mapper/vgwhatever-lvroot /mnt/tmp
cd /mnt/tmp
mkdir boot home mnt proc sys tmp usr var
cd /
3. Copy necessary elements fron the root directory into the new root directory:
cp -a bin cdrom dev etc initrd* lib media opt root sbin srv vmlinuz* /mnt/tmp
4. Mount the rest of the new filesystems at appropriate places on the new root and copy those directories over:
mount vgwhatever-lvhome /mnt/tmp/home
cp -a /home/* /mnt/tmp/home
mount vgwhatever-lvusr /mnt/tmp/usr
cp -a /usr/* /mnt/tmp/usr
mount vgwhatever-lvvar /mnt/tmp/var
cp -a /var/* /mnt/tmp/var
mount vgwhatever-lvtmp /mnt/tmp/tmp
cp -a /tmp/* /mnt/tmp/tmp
(this should return file not found, because there should not be anything in /tmp at this point)
(check yourself here - you should have an exact mirror of your system on the new root with the exception of /sys /proc and /boot)
5. Mount the USB stick and copy /boot to it:
Insert the stick.
mount /dev/sda1 /mnt/tmp/boot
cp -a /boot/* /mnt/tmp/boot
umount /dev/sda1
6. Chroot Magic!
chroot /mnt/tmp /bin/bash (ooh, cool! we've just jumped into our new, encrypted system! Let's breathe some life into it...)
mount proc -t proc /proc
mount sys -t sysfs /sys
edit the new system's /etc/fstab to reflect the new system's configuration:
# /etc/fstab: static file system information.
#
#
proc /proc proc defaults 0 0
/dev/mapper/vgwhatever-lvroot / ext3 defaults,errors=remount-ro 0 1
/dev/mapper/vgwhatever-lvusr /usr ext3 defaults 0 2
/dev/mapper/vgwhatever-lvvar /var ext3 defaults 0 2
/dev/mapper/vgwhatever-lvhome /home reiserfs defaults 0 2
/dev/mapper/vgwhatever-lvtmp /tmp ext3 defaults 0 2
/dev/hdc /media/cdrom0 udf,iso9660 user,noauto 0 0
/dev/sda1 /boot ext2 defaults 0 2
Now, mount /dev/sda1 using the new fstab:
mount /dev/sda1 (heh,heh... the system is truly alive, now, and fully operational - let's make it bootable!)
grub-install --recheck /dev/sda (this will write the grub call into the stick's MBR)
Edit /boot/grub/menu.lst to point GRUB to the stick:
# kopt=root=/dev/mapper/vgwhatever-lvroot ro
# groot=(hd0,0)
...and run update-grub (this will magically tell GRUB where all the kernels are ON THE STICK)
Now, we have to make another initrd:
yaird -o /boot/crypttest2.img (this will build an image that takes into account the new configuration)
... and add a stanza to /boot/grub/menu.lst OUTSIDE THE AUTOMAGIC KERNELS LIST:
title Debian GNU/Linux, kernel 2.6.17dc03
root (hd0,0)
kernel /vmlinuz-2.6.17dc03 root=/dev/mapper/vgwhatever-lvroot ro
initrd /crypttest2.img
savedefault
boot
Lastly, comment out the /dev/sda1 line in /etc/fstab (this will prevent fsck beating up the stick unnecessarily)
7. Back out of the chroot jail:
umount /dev/sda1
umount sys
umount proc
exit
8. Take a deep breath, and reboot with the stick in the USB plug, selecting USB primary boot in your BIOS along the way, and selecting the new stanza in GRUB. (note: GRUB takes longer to load from the stick - no problem. It still loads.)
When the passphrase question comes up, remove the stick, put it safely in your pocket (make this action a habit), do the passphrase call and response, and watch the machine boot into the new system.
Heh.. there are now three complete operating systems on the machine, two bootable from the machine's hard disk, and one completely encrypted system bootable from the USB stick.
A note about udev: If we catch a complaint during boot from the stick something like:
.udev/ already exists on static /dev!
Then boot into the old configuration and do:
mount /dev/mapper/vgwhatever-lvroot /mnt/tmp
cd /mnt/tmp
rm -r /dev/.udev
cd /
umount /dev/mapper/vgwhatever-lvroot
and try again... This will actually depend on what version of udev we're running and kernel version. Be patient. Even Linus has said that udev is a mess right now. People upstream are working their collective tails off on it...
Next: cleanup, adding swap, and operating tips...
Once you have proven a reliable boot process from the usb stick to the encrypted partition, with all your mapped devices mounted in their proper places, it's time to make all this permanent:
1. Boot from the stick, remembering to remove the stick at passphrase time (this is not absolutely necessary, but I've gotten into the habit of putting the stick away at the earliest opportunity. If I leave the stick in the machine while it's unattended, a not-so-casual observer gets another clue...). Get used to this. It's called discipline. Cryptography is only as good as the discipline that handles it!
2. Insert the stick.
Uncomment the /dev/sda1 line in /etc/fstab and mount the stick with:
mount /dev/sda1
Clean up the /boot partition as outlined in the first system check, under Fun Stuff, step 13. Now you're cleaning up the USB stick. Remove reference to Windows on /boot/grub/menu.lst, as you can't boot Windows on the hard drive using this method anyway. Bill's loss.
umount /dev/sda1
remove the stick.
comment out /dev/sda1 in /etc/fstab again.
3. We need swap to make the system sing.
Destroy the old Debian install:
shred -n 3 -v /dev/hda2 (if you want to do the whole 38 passes, be my guest - it'll take some time though...)
fill /dev/hda2 with random noise:
dd if=/dev/urandom of=/dev/hda2
(Note: As of this writing (17 July 2006), on Debian Sid the old cryptsetup method of making swap with a random key (normally /dev/random) does not work. I will outline how to set up the swap using a static key and later point to a work-around method for this bug. There is a debate percolating about the best way to define encrypted swap, and a lot of it has to do with software suspend. The user can redefine her/his swap space later, if so desired, quite easily. This How-To is not going to get into the arcana of suspending to encrypted swap. This is an initial setup.)
make a key in /etc/keys with hashalot:
hashalot -s whateversaltphraseyouwanttouse -x hashyouwant > /etc/keys/swapkey
Give it an immediately forgettable, very secure passphrase when asked - preferably an ascii mix, and long. You will never need that passphrase again, just the key generated. There are other methods for key generation, most notably with gpg, but the results are the same.
map the partition with cryptsetup:
cryptsetup create swap /dev/hda2 -d /etc/keys/swapkey -c twofish-cbc-essiv:sha256 -h sha512
(This key is sitting on an encrypted partition, remember. Only one passphrase will be needed to unlock the entire drive ;->)
make a swap filesystem on it:
mkswap /dev/mapper/swap
make it permanent:
edit /etc/crypttap and add the line:
swap /dev/hda2 /etc/keys/swapkey cipher=blowfish,hash-sha512
edit /etc/fstab and just before the /dev/cdrom line (no reason - makes it pretty) add the line:
/dev/mapper/swap none swap sw 0 0
and do:
swapon -a
Heh, heh. Completely encrypted Debian system. Booted from USB stick... 90 Gig of real estate to play in... Uh, oh.. need to fix the MBR in the hard drive!
GRUB bits in the MBR of the hard drive now point to nothing. Machine won't boot without the stick.
Put the Microsoft(tm) install disc in the cd drive and reboot to it (we'll have to rearrange the BIOS to tell it to call the CDROM first).
Select Repair Installation Using Console when it asks, and do:
fixmbr
Say y (for yes) after the gruesome warning.
Exit the console thingy, and the machine will magically boot the Windows(tm) "os".
Now then: Put the USB stick in the plug, and reboot - we need to look at the BIOS.
Call BIOS config at boot time, select USB first, hard drive second, and cdrom last in boot order.
Lock it down with a password, if it allows us to (False security on my machine - BIOS settings can be easily reset by replacing the cmos battery. Be aware of that. It's a very low chicken wire fence, but still...).
Now boot to completely encrypted Debian using the USB stick, and boot to Windows when the stick's not present.
Try it out a couple times, back and forth. Amaze our friends. Bask in our glory a little bit...
Go back to work..
permissions on our new /tmp are wrong (root wrote it, remember). A desktop install like KDE will not be able to use it and it will crash out:
chmod 1777 /tmp
Now KDE can write to and read /tmp (and other folks as well)
Everything will work now, but we need to make /tmp a little more secure (not perfect at all, like the BIOS password, it keeps the clueless from making us stupid):
edit /etc/fstab and change the mount option for /tmp from 'defaults' to 'noexec,nosuid' This keeps script kiddies from punching us up from our /tmp partition ;->, then do:
mount -o remount /tmp
Then add the next two lines to /etc/apt/apt.conf
DPkg::Pre-Install-Pkgs {"mount -o remount,exec /tmp";};
DPkg::Post-Invoke {"mount -o remount /tmp";};
This will allow APT to remount /tmp as necessary during it's operations.
WE'RE READY TO ROCK!
Go online. Continue installing Debian, putting what you like on the machine. Go offline. Configure your stuff. Setup your firewall. Surf. Have fun!
Operational notes:
1. Normally, leave the /dev/sda1 line in /etc/fstab commented out. When building, installing or removing kernel, or updating the GRUB Package, uncomment that line, mount /dev/sda1, and perform your operations normally. Yaird will from this point forward operate normally, and transparently (yaird will be called to build the initrd during kernel install by dpkg), and the stick will take care of itself through update-grub. No more fiddling with it.
Done with such operations, comment the /dev/sda1 line back out of /etc/fstab, and umount /dev/sda1.
2. The stick can be cloned, so protect it accordingly - no keys or passphrases are kept on the stick, however. Cloning the stick actually may be a good idea, if the clone is kept in a safe place as a form of backup. Imagine you lose your usb key...
References:
A good discussion on limiting partition permissions can be found here:
This was the basis for this How-To. I've modified it to boot from USB, and procedurally removed the necessity for utilizing a live cd:
Work-around for random key on encrypted swap:
About LVM2:
Docs:
/usr/share/doc/cryptsetup
/usr/share/doc/lvm2
man hashalot
man random
Help more people speak like us!
...various topics
Just basic conventions and terminolgy often used in Debian and Linux.
----------------------------------------------------------------------------------
Note some terms and symbols have different meanings in different contexts. A living language changes over time so legacy (recent history) terms may have different meanings to different people, please add yours. These are only based on what I have come across and understood so I may have been completely wrong, as always corrections and additions most welcome this is a book (community) page so just hit the edit tab and leave a comment on what you fixed.
////////////////////////////////////////////////////////////////////////////////////////////////////////////
*Language
foo
meaning some program. Like 'x' in math a place holder used to make a statement generic. So "gksu foo" could mean "gksu Konqueror" or "gksu Nautilus".
bar
friend of foo (see above) sometimes used together "foo bar" or "foobar" again as in math but a different variable place holder like 'y'.
root
1)the administrative user that has all power to do things on a linux/unix computer.
2)"/" is the root symbol, pronounced root, and often meaning the begining (or base point) of a *nix file system.
*nix
meaning all things ending in "nix" mostly referring to "Linux" "unix" (or "bsd" variants?)
app
abbreviation of "application" or program sometimes referred to as clients.
read the man page
means open your favourite terminal application (xterm, konsole, linux console etc) and then type:
"man foo"
see foo above, in this case it is the name of the app that you are trying to use so try:
$ man ls
Most *nix programs have this very handy manual page installed along with the program. They are your friend and should always be consulted BEFORE you run any unfamiliar command, that includes help given here and elsewhere. A small typo by the helper could cause you to make a big mistake so check what all those options do and you will learn more.
KDE programs often do not have a man page, they have their own help system, a rare KDE failing IMO. How about gnome and other desktops?
////////////////////////////////////////////////////////////////////////////////////////////////////////////
*Conventions
run mount:
$ mount
means open your favourite terminal application (xterm, konsole, linux console etc) and then type "mount" and press enter (return key). The "$" normally appears as the prompt when logged in as a normal user.
run ifconfig, as root:
# ifconfig
means open your favourite terminal application (xterm, konsole, linux console etc) and login or gain root. Then type "ifconfig" and press enter (return key). The "#" normally appears as the prompt when logged in as the root user.
gain root
login as the root user or run the program with root permissions. Most often done by opening your favourite terminal application (xterm, konsole, linux console etc) and typing "su" pressing enter (return key) and entering the requested root password. May also be accomplished by dropping to the Linux console and logging in as the root user (press "Ctrl+Alt+F1" if in X, "Alt+F7" gets you back to X). Or by using kdesu, gksu or xsu to run the program as the root user.
////////////////////////////////////////////////////////////////////////////////////////////////////////////
A place for tricky hardware usage in Debian.
Well if you were lucky enough to snag on of these exceptional mice in you're Christmas stocking (like me, thanks to my wonderful wife) or you're looking to buy one now that they are a more reasonable price (for top of the line anyways). Then here's the breakdown for usage in Sarge:
(Please note there are 2 'models' of these mice the original was green with hard plastic all round and the current revised model is all black with a carbon fibre look and rubber grip. I speak here of the latter, very sexy, not sure if any firmware/operational differences exit. A google of 'gentoo+G7+wiki' or 'G7+ubuntu+forum' will provide heaps of confusing detail but not for Sarge.)
1.Out the box all basic mice functions will work, great! Point and click I didn't even change XF86Config-4 from the PS/2 setup mouse I was using.
--Picked up by the usual XF86Config-4 'mouse' driver and usb setup.
--The hardware 3 step res works fine.
--The thumb button was picked up as button 8 and so could be mapped using your preferred manner (I was using xmodmap and imwheel) or see ~/.xbinkeysrc bellow.
2.Try:
$>xev
and click the thumb and tilt wheels while over the window to see what is detected and what button number.
3.Ah the tilt buttons... are not picked/assigned using the 2.6.8 debian kernel and xfree86 4.3.0 no matter what combination of drivers I tried with what config and I spent a day and a half trying ;-( so long story short if you absolutely have to have the use of these your system can no longer remain pure Sarge/Stable.
######################################################################
The following was a huge mission and X doesn't seem to be all that stable any more so unless you absolutely have to have the tilt buttons working (and have spare hours to waste) perhaps wait for the next Debian stable release, not too far off now I hope, or even possibly upgrade everything to unstable...you have been warned.
Strongly Suggest to change /etc/inittab so that X does not start on boot look for this line and change 2 to 1 (remember to change it back later):
id:1:initdefault:
**I finally got the tilt working with the following in Sarge using www.backports.org
linux-image-2.6.18-3-686
linux-headers for above needed for nvidia build.
linux-kbuild-2.6.18
x11-common 6.9.0.dfsg
xserver-xorg 6.9.0.dfsg
As you can imagine there are many, many dependancy issues to resolve and in the proccess I ended up removing most of the X apps and reinstalling them along with KDE 4:3.5.0-3bpo1 (why not it is pretty). Of course if your using accelerated video hardware then the older Nvidia drivers in Sarge will not work well with the newer Xorg so I got 'NVIDIA-Linux-x86-1.0-9746-pkg1.run' from the nvidia driver site. Many hours in a console later and $> xev now registers the tilt buttons with the following setup.
######################################################################
$> cat /proc/bus/input/devices|less
I: Bus=0003 Vendor=046d Product=c51a Version=4101
N: Name="Logitech USB Receiver"
P: Phys=usb-0000:00:03.0-1/input0
S: Sysfs=/class/input/input1
H: Handlers=mouse0 event1 ts0
B: EV=7
B: KEY=ffff0000 0 0 0 0 0 0 0 0
B: REL=143
I: Bus=0003 Vendor=046d Product=c51a Version=4101
N: Name="Logitech USB Receiver"
P: Phys=usb-0000:00:03.0-1/input1
S: Sysfs=/class/input/input2
H: Handlers=kbd event2
B: EV=f
B: KEY=c0002 400 0 0 1 f80 78000 6639fa d84157ad 8e0000 0 0 0
B: REL=40
B: ABS=1 0
To repeatedly get the event number and optionally update xorg.conf try this little
script(needs xorg setup below): www.debianhelp.org/node/4390
######################################################################
/etc/X11/xorg.conf
Section "InputDevice"
Identifier "LogitechG7"
Driver "evdev"
Option "Device" "/dev/input/event1"
Option "Name" "Logitech USB Receiver"
Option "Phys" "usb-*/input0"
Option "SendCoreEvents"
EndSection
Apparently the 'Device' line should not be needed in newer x.org but mine always complained about no device specifed, the trouble is that the usb system registers two "Logitech USB Receiver" with no way to tell the difference except that the mouse event is always at "usb-*/input0". Of course if you boot with different usb peripherals plugged in the event number may change and X will freeze (once it has successfully locked up the keyboard) so I ssh in from another machine and issue a reboot.
######################################################################
/home/user/.xbindkeysrc
"/usr/bin/X11/xvkbd -xsendevent -text "\[Alt_L]\[Left]""
m:0x0 + b:6
"/usr/bin/X11/xvkbd -xsendevent -text "\[Alt_L]\[Right]""
m:0x0 + b:7
"/usr/bin/X11/xvkbd -xsendevent -text "\[Alt_L]\[Up]""
m:0x0 + b:8
The spacing quotation marks and new lines (6 of) must all be retained exactly as above.
Of course if you just want to use the thumb button (8) mapped to your choice of command most likely "\[Alt_L]\[Left]"" for -back- in your browser use:
"/usr/bin/X11/xvkbd -xsendevent -text "\[Alt_L]\[Up]""
m:0x0 + b:8
Also of course for this to work:
#> aptitude install xvkbd xbindkeys
######################################################################
/home/user/.kde/Autostart/xmodscript
#!/bin/sh
#Logitech G7 mouse bindings see ~/.xbindkeysrc
xbindkeys
Create an executable script to start xbindkeys when kde starts, or place in your window managers startup scripts or $>xbinkeys
######################################################################
Well after all that a very slick mouse that performs well once you get used to using short quick taps in the tilt buttons to only get 1 click not 3 ;-) Have fun.
#!/bin/sh
#A little script that reports the event# to use for a Logitech G7 mouse and
#optionally replaces the xorg event# with the detected event#.
#I had fun writing this and use it to update xorg.conf since my mouse turns up as
#a different event# every boot!
#May work for other Logitech mice with the some adjustments.
#Usage:
#Please make a backup of your xorg.conf!
#cp /etc/X11/xorg.conf /etc/X11/xorg.conf~
#To find the the correct event number just save as executable script (eg /home/user/sg7).
#Then call the script: $ /home/user/sg7
EVENTLINE=$(cat /proc/bus/input/devices|grep -A 3 "Logitech USB Receiver"|grep "Handlers=mouse0")
echo "proc line: $EVENTLINE"
EVENTSTR=$(echo $EVENTLINE|grep -o event.)
echo "proc string: $EVENTSTR"
echo ""
XORGEVENTLINE=$(cat /etc/X11/xorg.conf|grep event)
echo "xorg.conf line: $XORGEVENTLINE"
XORGEVENTSTR=$(echo $XORGEVENTLINE|grep -o event.)
echo "xorg.conf string: $XORGEVENTSTR"
echo ""
#To overwrite xorg.conf call the script with "edit" as the only argument.
#Save as executable script (eg /root/scripts/sg7).
#Then call the script: $ /root/scripts/sg7 edit
#Needs an already setup xorg.conf where "event#" only occurs ONCE.
#Needs permission to overwrite xorg.conf, I use 'super' for this.
#Since you plan to run this as root understand the commands, since if there is
#a typo or paste error anything could happen...
if [ "$1" = "edit" ]; then
echo "running sed: "
sed -i s/event./$EVENTSTR/ /etc/X11/xorg.conf
XORGEVENTLINE=$(cat /etc/X11/xorg.conf|grep event)
echo "after sed xorg.conf: $XORGEVENTLINE"
fi