[rescue] Oracle making just a little harder to keep old machines in use

[Rich Kulawiec]

On Mon, May 10, 2010 at 08:57:08PM +0100, Mike Meredith wrote:
> On Mon, 10 May 2010 08:00:53 -0400, Rich Kulawiec wrote:
> > Tar doesn't have code to cope with filesystems that are changing
> > as they're being read.
> 
> Actually neither does (ufs)dump ... the Solaris manual page
> specifically warns against dumping filesystems mounted read/write, and
> the FreeBSD 8.0 manual page hints that the same situation may cause
> problems.

Actually, yes, it does (have code to cope with filesystems that are
changing) -- the manual page has been wrong since the days of SunOS.
Not that the code is bulletproof, but it was developed back in the
days of BSD 4.0 by folks at Stanford and Purdue who didn't like the
idea of going through the umount-dump-mount sequence.  If memory
serves (and sheesh it's been a long time since I've hacked on that)
most of the relevant stuff is in dumptraverse.c and dumponline.c.
I've been relying on it for decades, and while it *is* possible to
create pathological cases which will break it, these are extremely
unlikely to arise on production systems.  And given the vagaries
of timing involved, successive dumps/ufsdump (say, on consecutive
days) will almost certainly sweep up anything abandoned by their
predcessors.

> Filesystem snapshots are of course the answer to the problem here. 

They are definitely an answer to some problems, but I don't see how
they solve (some of) not mine.  Maybe it's just because I don't know
how to use ZFS snapshots properly yet.

For example: I have many ZFS filesystems of size 1-5T, occupied
30% to 70%, where 90% of that will never change.   (It's static
data, loaded once and then read a kazillion times.) So, using dump's
levels as an example, just for illustration, a sequence something like:

	week 1
	0 1 2 3 4 5 6
	week 2
	1 2 3 4 5 6 7
	week 3
	1 2 3 4 5 6 7
	week 4
	1 2 3 4 5 6 7

will snarf everything once a month -- still arguably overkill, but
let's run with it -- and will then pick up all the stuff that changes
on a day-to-day basis.  I don't NEED to snapshot all that unchanging
data over and over and over again: I just want the deltas, which are
1-2 orders of magnitude smaller, and are the critical work-product that
I'll need to restore in the event of system failure or user error.

> > It runs in userspace instead of reading the raw device.
> 
> Both run in userspace; reading the raw device might give some
> advantages, but personally I prefer file-by-file backup ... whilst you
> are not guaranteed consistency, at least it copes gracefully with the
> inconsistencies unlike *dump which generates errors (although it
> usually continues), and possibly generates erroneous backups[0].

Dump actually tries very hard to create output that's logically
self-consistent.  I've run it on live filesystems for 25 years and
have yet to come across an image that I can't restore.  Of course,
I can hardly claim to have attempted every possible case, so perhaps
there were some and I just didn't happen to try to use them.  I *do* know
how to pause it and create pathological conditions that will break it,
but those are pretty unlikely to occur in production, and very likely
to occur repeatedly.

> > While it has some incremental backup capabilities, it's not readily
> > usable in the same way that dump's incrementals are.  Yeah, you can
> > wrap some shell around it and get close, but it's awkward.
> 
> Indeed. However you will almost certainly be wrapping any backup tool
> in a shell script anyway ... unless you're using some 'advanced' backup
> package such as Legato (which gives you a pretty Windows GUI and a
> whole other set of problems).

I concur.  Legato holds down the #2 spot in my all-time list of "worst
pieces of software I have ever had to deal with on 'nix systems".  Far
better are perl or shell scripts, or Amanda, or anything else.  (25-ish
years ago, I published a C program on comp.sources.unix that is just such
as a wrapper, and included tape labeling as a defensive measure.)

> For instance you often want to quiesce certain applications ... put
> Oracle into "hot backup mode", stop a MySQL slave from applying
> transactions. Or manually exclude log directories from backups.
> 
> You also (once you have more than about 2-3 servers to backup) want to
> improve on the reporting. A simple cron mail becomes tricky to handle
> once you get enough filesystems to monitor.

All true.  I think it's best, no matter what the backup method, to use the
native capabilities of databases et.al. to produce self-consistent
backups of their contents, and then back *those* up.

> > What I'll argue that we (FSVO "we") need is a portable version of
> > dump that runs on the appropriate set of currently-extant open-source
> 
> Perhaps. The problem of producing a portable dump format isn't trivial.

Strongly agreed.  Having fought that particular problem in the past,
and having spent a little time recently assessing its scope, it would
like require significant engineering effort.  I'm on the fence as to
whether or not it's worth it, but leaning toward "yes".


---Rsk