// you’re reading...

The Intra-Web-O-Sphere!

How To: Backup your Mac to a Dreamhost Server (part 1 of 2)

  • The Intra-Web-O-Sphere!
  • geek
  • technology
By michael lee ⋅ March 9, 2007 ⋅ Post a comment

Time: Two 30 minute session, with a break between.
Geek Level: Played Dungeons and Dragons, but didn’t own your own dice.
Written for: Mac OSX 10.4.8 (should work on OS 10.2 and up), Dreamhost server account

Ever since my complete meltdown at the beginning of the year, I’ve been like a backup zealot. All of my computers are now on a regular back-up rotation with SuperDuper, and I’ve built an automated off-site backup plan too. This post will walk you through how to build an off-site backup of your data.

Some background

With so many companies offering website hosting, providers have started to compete on both price and storage space. Dreamhost has a plan that offers 170 GB of server space for $8 a month. For just a second, don’t think about that as a website server, think of it as a hard drive that you can rent for $8 a month, only this hard drive is maintained by somebody else, and they keep a backup copy of everything on it, so that if it crashes, your data gets completely restored. As a bonus, this extra hard drive you’re renting from Dreamhost keeps growing in size by 1 gig every week. Even if you’re not planning on building a website, that kind of cheap storage space is well worth the money.

There are two huge advantages to off-site backup on a server. First, if something catastrophic happens, like a fire or theft, I need to have my critical files stored in some other location. Secondly, this enables me to access those critical files from any computer, anywhere. I don’t need to worry about bringing my lecture notes to class with me - I can access them on the server when I get there.

So, I’ve decided to use the extra storage space on my Dreamhost account to keep regular backups of all my critical data. I keep a copy of my Documents folder on my Mac synced to the server, as well as a full copy of my audio hard drive, and all of my samples. Today, I’ll show you how to setup the same backup system I use, and how to automate it so that it backs up your data even if you forget.

(Note: Dreamhost doesn’t allow you to keep pirated media on their servers. If you have a ton of BitTorrent movies and TV shows on your computer, and you sync them up to the server, you’ll get a friendly little email from Dreamhost asking you to take them down)

This tutorial has a pretty high geek rating - it requires you to use some programs you might not be familiar with, like Terminal, and the Applescript Editor. It’s written to be understood by casual computer users, so here’s the deal - I’ll gladly offer personalized tech support to anyone who opens a new Dreamhost account using this link: Dreamhost Signup. I get a little kick-back from them if you do, which in turn makes me a happy little tech-support guy. If you sign up for a new account, I’ll work with you to get this working right.

The Plan

  1. Signup for a Dreamhost Account
  2. Create a new user with SSH login
  3. Using shared keys, enable SSH access to the server without having to supply a password
  4. Run rsync from the terminal to create a back-up of your home folder on the server
  5. Create an applescript that runs rsync automatically
  6. Use iCal to run the applescript backup daily, without doing a thing!

Before we begin, please do a local backup of your computer to an external drive, using SuperDuper or some other applications. We’re not doing anything here that will break your computer if you follow the directions, but it’s always a good idea to backup before you do anything new to your system.

1. Signup for a Dreamhost Account

Ready? Click here. Pick whichever plan works for you - the basic plan is probably enough for most people. Sweet! You’re already 1/6th of the way done!

2. Create a new user with SSH login

When you created your new Dreamhost account, you created a default admin user. The first thing we’re going to do is add another new user that will only be used for backups. Think of users as separate drawers in a filing cabinet. Your account can have a bunch of users, each with their own drawer. Each drawer has its own separate login and password. This way, your backup data stays separate from any data you’re using for a website.

Dreamhost will send you an email with details about your new account. There are a few things you need to write down and keep handy. First, it will tell you which server your account is on. It will be something like admiralty.dreamhost.com. Whenever this tutorial uses the word SERVER, replace it with the info from that email.

add userNext, the email will give you details on how to access the control panel for your hosting plan. Login, and look for the “Users” tab on the left sidebar menu. Then, click “add a new user”.

When the new page loads, under “Type of User Account” select shell. This will allow you to login directly to your server and pass files back and forth.

ssh account

Write down your user name and password. Again, whenever this tutorial uses the words USER or PASSWORD, replace them with the user name and password you’ve just created.

It’ll take 10 minutes or so for Dreamhost to get the new user up and running, so now would be a good time to brew some coffee and put on The Dailies CD.

3. Using shared keys, enable SSH access to the server without having to supply a password

Ok, ready? It’s about to get very geeky. Don’t be scared. Most of you think of computer applications as graphic windows that you click in and drag things around in (think iTunes). Those aren’t really the applications - they just control the applications. The applications are running under the hood. In the next few steps, we’re going to use some applications that don’t have graphic interfaces - you operate them by typing commands, which the application then carries out. That’s the only difference. Don’t let it scare you. If it makes you feel more comfortable, just copy and paste the text that’s here, and only change things like the SERVER, USER or PASSWORD.

Launch the “Terminal” application. It’s in your Applications folder, inside a folder called “Utilities”. Whoah! Check that out! Your geek rating just went up 4 points.

You’re going to enter some text commands into the terminal. You should see a string of text in the window, followed by a $ dollar sign (called a command prompt). Type the commands after that $. You can simply copy and paste what’s here, remembering to replace SERVER, USER and PASSWORD with your own info.

Let’s make sure the SSH login to your account is working. Type the following:

ssh USER@SERVER

You will get a warning saying that the authenticity of the host can’t be verified. Type “yes”. Next, you will be prompted for your password. Type it, and hit enter. Congratulations, you are now connected to your new server! Now type “exit” to close the connection.

In order for our automatic backup to work, we need to be able to connect to the server without having to enter in the password each time. Rather than retype the whole article, let me point you to the Dreamhost support wiki article. It’s pretty clear, just type in the commands as they appear. Replace example.com in their tutorial with your SERVER info, and you’ll be fine. To see if it worked correctly, exit out of your server, then type

ssh USER@SERVER

This time, it should connect you without asking for a password. Sweet!

4. Run rsync from the terminal to create a back-up of your home folder on the server

Ok, now we’re ready to do the first major backup of your data. We’re going to use an application called “rsync” to do that backup. Rsync doesn’t have a graphic interface, it runs from the terminal. It’s job is to compare two different files or folders, called a SOURCE and a DESTINATION, and to make changes to the destination. The advantage of rsync is that it only sends that data that needs to be changed, which makes our backup much quicker. Enter the following into the terminal, all on one line (don’t worry if it automatically makes line breaks for you, just don’t hit return until you’ve entered the whole thing)

rsync -avz ~/Documents/ USER@SERVER:~/laptop.bak/Documents/

Here’s what you told it to do:

  • “rsync” tells the terminal which program to use
  • “-avz” tells rsync which settings to use. This set of letters tells it to preserve some important information about the file permissions, and also tells it to compress the data so that it requires less bandwidth over your network connection.
  • “~/Documents/” gives rsync the SOURCE it should look at. In this case, it’s gathering data from the Documents folder in your user account. If you want to backup your entire account, replace this with just “~/”, and do the same to the following step.
  • Finally, “USER@SERVER:~/laptop.bak/Documents/” tells rsync where to put the files. USER@SERVER should look familiar - it’s the login to your dreamhost account. The last part of the command, “:~/laptop.bak/Documents/” creates a new folder on your server, called “laptop.bak”. Since I’m backing up many different computers, this helps me keep everything straight. It puts the data from your Documents folder into a similar folder on the server. If you changed the source folder to “~/” then change this to “USER@SERVER:/~/laptop.bak/”

This first time running the sync may take quite a while, depending on how much data you have, and how fast your network connection is. You may want to leave it running overnight. The good news is, once you’ve done this sync one time, it will only have to transfer the data that’s changed. It generally takes less than a minute for me to run the backup, unless I’ve altered a ton of data.

It’s perfectly fine to leave this project right where it is, and pick up the second half later. In fact, I’m going to save it for a second post. Rest content in the knowledge that your critical data has been backed up to a remote server. Next post, I’ll show to automate the backup, so that you can forget all about it. While you’re waiting, why not take a minute to digg this post.

(continue reading part 2)

Print This PostTags: , ,

Discussion

60 comments for “How To: Backup your Mac to a Dreamhost Server (part 1 of 2)”

  1. quick question:
    SuperDuper looks great. What are the bonuses to upgrading to the full version for 28 bucks? Is it necessary?

    D

  2. The registered version will let you do “smart backups”, which means it only writes the data that has changed. It makes for a much quicker back than having to rewrite the whole drive every time.

    It also lets you do scheduled backups, so that you can set to automatically work, instead of having to tell it every time.

    The free versions works great, but the full version is more likely to be actually used, since it’s so much quicker. And any backup scheme that you don’t actually use is useless, obviously.

  3. Howdy.

    I have also been using rsync to dreamhost as a backup solution for about 4 months now.

    It would be a good thing to note that backing up via rsync does have some limitations, especially for files with Apple’s extended attributes. More specifically with my experience a good example would be a textClipping file. It will appear to back up the clippings file, but the text clipping will be empty.

    The solution is to add the -E option to the rsync command. HOWEVER, this only works when going from Mac to Mac as the -E option is supported only on Mac computers (or computers that have the Apple modified rsync running on them.)

    Dreamhost rsync does NOT support the -E option. What that means is that some of your files are really not adequately backed up. Most of them will be, though as most files don’t rely on the resource fork anymore.

    You can google “rsync macosx” for lots of discussion on this and take a peek at: http://lists.apple.com/archives/macos-x-server/2006/Apr/msg00183.html

  4. Oh and additionally, I have a few –exclude statements so I don’t copy things like cache and trash…

    –exclude Podcasts/ –exclude .ssh/ –exclude Network?Trash?Folder/ –exclude *.tmp –exclude .gaim/ –exclude .java/ –exclude .jpi_cache/ –exclude *.cache –exclude .Trash/ –exclude Caches/ –exclude Cache/ –exclude cache/

  5. [quote comment="68299"]Oh and additionally, I have a few –exclude statements so I don’t copy things like cache and trash…

    –exclude Podcasts/ –exclude .ssh/ –exclude Network?Trash?Folder/ –exclude *.tmp –exclude .gaim/ –exclude .java/ –exclude .jpi_cache/ –exclude *.cache –exclude .Trash/ –exclude Caches/ –exclude Cache/ –exclude cache/[/quote]

    The excludes are good to note. I will have to add several of those myself.

  6. [...] />Michael Lee at the Addison Road blog has put together a nice tutorial for doing an online backup to a DreamHost account. DreamHost is a well known shared hosting [...]

  7. I get an error when it comes to step 4 and the rsync command:

    rsync: mkdir “/home/USER/mbp.bak/Documents” failed: No such file or directory (2)

    (I’ve replaced the actual username with USER here.)

    Is this because there’s no “home” folder in the new users directory? I’ll try creating some directories by hand, see what that does…

    I’m on Mac OS X 10.4.9, btw…

  8. [quote post="1266"]Dreamhost rsync does NOT support the -E option.[/quote]

    KiltBear, I left a similar comment on Part 2, but I thought you’d be interested in this solution. A patched version of rsync that preserves resource forks by using the AppleDouble (._filename) format. Here’s the link:

    http://www.onthenet.com.au/~q/rsync/

  9. Mac Backup To Dreamhost…

    When I posted a brief comparison to Amazon S3 and Dreamhost one thing I didn’t do was suggest how one could actually use Dreamhost in a similar way as S3, like backups. For S3 Jungle Disk is one option for backups. Now there is a how to on back…

  10. Oops. Of course there’s no “home” folder in the “USER” folder. It’s the other way around. Still, for some strange reason rsync doesn’t want to make that directory…

    I created the mpb.bak/Documents directories on my own, and now rsync is working just fine…

  11. [...] Lee at the Addison Road blog has put together a nice tutorial for doing an online backup to a DreamHost account. DreamHost is a well known shared hosting [...]

  12. [...] all of you who found this blog via the flurry of links today from various tech sites picking up on Mike’s excellent tutorial.   My name is Chad, and I’m one of the contributing authors around here, and I’m going [...]

  13. [...] How To: Backup your Mac to a Dreamhost Server (part 1 of 2) at Addison Road How to backup to Dreamhost using the terminal. (tags: dreamhost backup howto osx) [...]

  14. [...] How To: Backup your Mac to a Dreamhost Server (part 1 of 2) at Addison Road Quickly back up your mac to a dreamhost server. (tags: mac) [...]

  15. [...] Link Original: How To: Backup your Mac to a Dreamhost Server (part 1 of 2) at Addison Road [...]

  16. [...] Lee at the Addison Road blog has put together a nice tutorial for doing an online backup to a DreamHost account. DreamHost is a well known shared hosting [...]

  17. How to backup your Mac to a Dreamhost server…

    There are two huge advantages to off-site backup on a server. First, if something catastrophic happens, like a fire or theft, I need to have my critical files stored in some other location. Secondly, it enables you to access those……

  18. [...] Lee at the Addison Road blog has put together a nice tutorial for doing an online backup to a DreamHost account. DreamHost is a well known shared hosting [...]

  19. [...] was looking recently for a reliable way to backup my computer, and came across this article, which seemed like the perfect fit. It’s geeky, it’s solid and simple, and it makes use [...]

  20. [...] The Unix synchronization command, rsync, is a nice solution for off-site backup since the transfer happens encrypted over SSH, and it’ll work with most web site hosts, or with the home-built SSH server you set up at your sister’s in Florida. Check our previously-posted rsync primer for how to write an rsync script that will send your Mac’s files to a remote server for backup. See also how one chap schedules rsync jobs to Dreamhost with iCal. [...]

  21. Thanks for this! As a long time Dreamhost member, I’ve thought about doing this for a while, but wasn’t quite sure how best to go about it. Then I came across this article, and much to my surprise, noticed Addision Road and thought, hey, that’s Mike Lee! What a small world! Anyway, I hope all is going well with you guys. Take care! :)

  22. [...] Part 1 of the tutorial shows how to create a new user with shell access, how to enable SSH access to the server without having to supply a password and how to use rsync from the terminal to create a backup of your home folder on the server. [...]

  23. duplicity is also worth considering unless you trust the staff at dreamhost with all of your personal data. The reason for trying duplicity over rsync is that it encrypts the data. All the other advantages remain.

  24. Hey Jake! Good to hear from you! Where are you these days?

  25. [...] Part 1 of the tutorial shows how to create a new user with shell access, how to enable SSH access to the server without having to supply a password and how to use rsync from the terminal to create a backup of your home folder on the server. [...]

  26. [...] How To: Backup your Mac to a Dreamhost ServerGood basic tutorial for using rsync utility to backup to a remote server. If I was geekier I’d try duplicity for compression and encryption… but this is a good start I think. As noted in the comments it’s probably better to use a cronjob rather than using an iCal alarm to trigger the backup.(Tags: backup, mac, dreamhost, osx, rsync, howto, tutorial) [...]

  27. [...] primeiro post amosa paso a paso desde como crear un novo usuario con acceso SSH e logo, utilizar rsync desde o [...]

  28. 2 things..

    First you might want to look at rsnapshot. It uses rsync to make incremental backups… would be great for this type of thing…

    The next thing you might want to try for security is to make a secure disk image stored on the dreamhost server. Use webdav to mount the server then mount the diskimage… dreamhost shouldn’t be able to see that … I did something like this once on a clients server that wasn’t paying regularly…

  29. [quote comment="68373"][quote post="1266"]Dreamhost rsync does NOT support the -E option.[/quote]

    [quote]KiltBear, I left a similar comment on Part 2, but I thought you’d be interested in this solution. A patched version of rsync that preserves resource forks by using the AppleDouble (._filename) format. Here’s the link:

    http://www.onthenet.com.au/~q/rsync//quote

    Thanks Andy, that is great. I am trying it out now. Looks promising!

  30. [...] to my hosting provider, DreamHost.  I was inspired by Michael Lee’s aptly titled article on How to Backup Your Mac to a DreamHost Server and while I didn’t do all the technical coding work, I managed back up my music files [...]

  31. This was very interesting. I should have thought about this before!!

    Well, people has already mentioned the problem with the forks, and that is quite a trouble.

    In my case I just found out that I have too many strange crap in my home folders that I don’t want to backup (movies, songs , and so on). So I am trying to use zsh completion stuff to do the backup only for my small files (less than 2Mb):

    rsync -Ravz ~/Desktop/**/*(.LM-2) User@Host:~/iBook.bak/

    But this is a very unoptimal solution…

    In fact, I should ratther organise up my home directory stuff and I should place all my personal things in the Document folder instead of having it on Desktop and many other places.

  32. [...] met AppleScript en Automator, en wel op niveau dat mijn oude buurman het zou kunnen (voorbeeld: backup’en als een Unix-koning, maar dan voor [...]

  33. [...] besteiras. Através do blog não oficial da Dreamhost e do delicious do Leo Faoro eu descobri um bom tutorial para fazer backup de pastas do meu [...]

  34. [...] http://addisonrd.com/WordPress/2007/03/09/how-to-backup-your-mac-to-a-dreamhost-server-part-1-of-2/ [...]

  35. [...] available to me now. So I backed up all my photos and documents. I used this handy little tutorial. Granted this backup tutorial only works on macs, I am confident there is a similar windows tool [...]

  36. I have a dreamhost account and like it very much (we use it for SVN).

    But are you guys sure you want to be putting your credit card information and every personal document and email you have on someone else’s remote server?

    You are asking for a lot of trouble.

    You would have to take all highly private documents and photos out of the folder you are backing up and put them somewhere else.

    Otherwise, a very clever solution.

    Enough people do it, though, and Dreamhost will be out of business or have to ban such backups.

  37. [...] that I’ll also start using S3 and JungleDisk as a backup server as well. Even though there are cheaper solutions, I don’t need to backup off-site but once a month so it works well for [...]

  38. I keep trying to make this work, but I can’t get past entering “ssh USER@SERVER” in the Terminal.

    I did successfully create a new User account at DHost. I chose the “Shell account - allows FTP plus ssh/telnet access.” I’m not sure what “Shell Type” to select, so I just left it as the default, which was “/bin/bash”.

    However, I’m unclear what the “SERVER” name is. Is that what DHost calls the “Machine” column in the list of Users?

    In any case, when I open a new Terminal window and I type “ssh mybackup@otto” (where “otto” is the Machine name), I get the following in the Terminal window:
    otto: No address associated with nodename Smith-Mac-G5:~ rso$

    And that’s where I’ve been stuck for weeks. Any ideas?

  39. [quote comment="88014"]
    … I’m not sure what “Shell Type” to select, so I just left it as the default, which was “/bin/bash”.[/quote]

    That’s fine - that’s what it should be.

    [quote]However, I’m unclear what the “SERVER” name is. Is that what DHost calls the “Machine” column in the list of Users?

    In any case, when I open a new Terminal window and I type “ssh mybackup@otto” (where “otto” is the Machine name), I get the following in the Terminal window:
    otto: No address associated with nodename Smith-Mac-G5:~ rso$ [/quote]

    Make sure that you use the entire machine name that dreamhost supplied you. In other words, don’t just use “otto”, use “otto.dreamhost.com”. For the example you used, the full command would be “ssh mybackup@otto.dreamhost.com”

    You might also be running into some problems on the Dreamhost end. They’ve had pretty much a nightmare run this week, with hardware crashing and several key components of their network going haywire. So, even if you are doing everything correctly, you might still be running into some login problems.

    You can keep track of how the actual server is doing by checking the dreamhost status blog:

    http://www.dreamhoststatus.com

  40. Thanks Michael. I was able to get to the point where I can now login without a password via the Terminal. (By the way, this is the very first time I’ve opened the Terminal app, and that’s after Mac’ing for 16 years).

    However, when I enter this line:
    [otto]$ rsync -avz ~/Documents/ scotts@otto.dreamhost.com:~/laptop.bak/Documents/

    … it does two strange things. One, it asked me for my password:
    scotts@otto.dreamhost.com’s password:

    I thought since I was able to ssh without a password, it wouldn’t ask for that again.

    The second strange thing is that it “fails” :

    building file list … rsync: link_stat “/home/scotts/Documents/.” failed: No such file or directory (2)
    done
    sent 29 bytes received 20 bytes 8.91 bytes/sec
    total size is 0 speedup is 0.00
    rsync error: some files could not be transferred (code 23) at main.c(789)
    [otto]$

    (I changed the server name above for security).

    Any ideas? I had a friend who’s hot on the Terminal look at it, and he’s baffled as well.

    Scott

  41. That’s very, very odd. rsync uses the SSH login protocol, so I can’t for the life of me think of any reason why ssh would work with the shared keys (no login), and rsync wouldn’t.

  42. AHA! I think I see what happened.

    From what you copied / pasted into comment above, it looks like you’re actually running the rsync command while logged in to your dreamhost server! Is that so? Once you’ve done an SSH into your server, you’re actually running commands on the server, not on your local computer.

    You need to exit (type ‘exit’, or just close the terminal and reopen it again), make sure that the command prompt shows the name of your local computer, and THEN run the rsync command.

    The rsync command needs to be from FROM the local computer, TO the server.

  43. That was it, Michael. Wow, good troubleshooting. And, it eliminated that odd asking for my password again, too. I just opened a new Terminal window, entered the rsync line, and it took about 3 minutes to do this “Building List…” and then said “Done.” After that, it started listing each file individually as it, presumably, backed it up to Dreamhost.

    I’m already seeing how this would be a good time to clean up and reorganize my Documents folder. As the rsync lists every single file, I find myself thinking, “I still need *that*?”

    thanks again for helping me crack the Terminal for the very first time.

    Scott

  44. You’re welcome - glad to help out.

  45. Hi! Great write-up! Thank you.

    I have a problem though, when I run the “rsync” command, it copies all of my “Home” folders to the dreamhost server… BUT… it only copies the actual folders…. no files…. all the folders on the dreamhost server are empty. Can someone help me???

    I tried to back up my “Home” folder using the “~/” which is about 160GB… All the Folders (empty) show up on the dreamhost server only 0.02MB… ???

    Please help me.

    Thanks in advance

  46. I have the same problem :

    rsync: mkdir [dir name] failed: No such file or directory (2)
    Does anyone know how to fix it without create by hand all the directories ? it’s strange that rsync cannot creat recursively the directories …

  47. Dreamhost admits encriptation?
    for the backups . . .

  48. This is a rather novice question, but how does one get stuff off Dreamhost once it’s there? Won’t rsync wipe the Dreamhost copy so it matches the local copy?

  49. Sorry, i don’t speak english, a little… little
    I speak spanish, but…
    if my local copy is encrypted, remote copy is encrypted (with rsync),it’s ok.

    The question is, if server in EEUU (dreamhost) don’t want problem with the info no readable (is encrypted) stored in your servers.
    Some rules about this?

  50. Just read up on DreamHost’s terms of service:

    …
    6 The customer agrees to make use of DreamHost Web Hosting servers primarily for the purpose of hosting a website, and associated email functions. Data uploaded must be primarily for this purpose.. DreamHost Web Hosting servers are not intended as a data backup or archiving service. DreamHost Web Hosting reserves the right to negotiate additional charges with the Customer and/or the discontinuation of the backups/archives at their discretion. …

  51. ok, thx

  52. [...] doing a bit of reading on rsync, I came across this post detailing setting up a Mac to use rsync with DreamHost.  Exactly what I’m looking to do, [...]

  53. Looks like Dreamhost is stopping this: http://www.dreamhoststatus.com/2007/10/17/policy-clarification-personal-storage-back-ups/

    lame.

  54. Bummer. I added a disclaimer.

  55. [...] spring, I posted this tutorial on how to use rsync to upload data to a Dreamhost server. It’s still the most popular article [...]

  56. It doesn’t look prohibited so much as Officially Unsupported.

    Thanks for this tutorial. I plan on exploiting this myself soon, supported or not.

  57. [...] This post taught me everything I needed to know to get rsync working with Bluehost (the instructions are for Dreamhost, but you get the idea). Be sure to check out the part for running a backup without logging in. Up until recently, I’ve used the terminal to periodically run a backup–yet there’s always room for improvement. [...]

  58. [...] into a number of different solutions including Mozy, Box.net, and even a manual upload using my Dreamhost account - but none of those solutions met all my needs. I wanted something simple to use and a solution [...]

  59. [...] How To: Backup your Mac to a Dreamhost Server (part 1 of 2) [...]

  60. thanks for this one!

Post a comment