Announcement: buttersink - like rsync for btrfs snapshots

[Ames Cornish]

I've written a utility to help me with using btrfs send and receive for
backups or other synchronization, and I'd love to get feedback on it.

As of this release, buttersink will synchronize a set of read-only snapshots
in a btrfs filesystem to an Amazon S3 bucket, and vice-versa. It
intelligently picks parent snapshots to "diff" from, so that a minimal amount
of data needs to be sent over the wire and stored in the backend.

The utility is on PyPi as "buttersink", and the GitHub page is here:
https://github.com/AmesCornish/buttersink.

Thanks in advance for any feedback!

- Ames


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Jim Salter]

How has it been for reliability?

I wrote a btrsync app a while back, and the app /itself/ worked fine,
but the btrfs send / btrfs receive itself proved problematic. Since
btrfs would keep a partial receive - with no easy way to tell whether a
receive WAS partial or full - I would inevitably end up with interrupted
sends causing a problem that couldn't be resolved without manually
deleting snapshots on the target end haphazardly until I nailed the
incomplete one.

Post by Ames Cornish
I've written a utility to help me with using btrfs send and receive for
backups or other synchronization, and I'd love to get feedback on it.
As of this release, buttersink will synchronize a set of read-only snapshots
in a btrfs filesystem to an Amazon S3 bucket, and vice-versa. It
intelligently picks parent snapshots to "diff" from, so that a minimal amount
of data needs to be sent over the wire and stored in the backend.
https://github.com/AmesCornish/buttersink.
Thanks in advance for any feedback!
- Ames
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Jim Salter]

To any core btrfs devs who are listening and care - the unreliability of
btrfs send/receive is IMO the single biggest roadblock to adoption of
btrfs as a serious next-gen FS.

I can live with occasional corner-case performance issues, I can even
live with (very) occasional filesystem corruption... IF I can rely on
replication to keep my data safe on another box. Without the
replication, there's just no reasonable case to be made to replace ZFS.

Jim,
btrfs send reliability has been an issue, though I've been able to
successfully use it for my backups. buttersink usually detects the
errors and will either move the destination snapshot to mark it as
partial/failed (for btrfs), or cancel and delete the partial upload
(for S3). I've also found that it helps to wait a while (e.g. 30
seconds) after any volume deletes before trying the send/sync. I hope
btrfs-progs will get more reliable, too.
- Ames

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Clemens Eisserer]

Hi Jim,

Post by Jim Salter
To any core btrfs devs who are listening and care - the unreliability of
btrfs send/receive is IMO the single biggest roadblock to adoption of btrfs
as a serious next-gen FS.

For you it are send/receive deficiencies, however there are many other
feature/enhancement requests having top priority for other users.

Fact is, just crying for improvements won't make things better - if
you want a feature/enhancement done, you have several options:
- Implement/fix it yourself
- Go to your distribution provider with whom you have a support
contract and make him implement what you would like to see
- Sponsor somebody yourself to get it sone.

Post by Jim Salter
with (very) occasional filesystem corruption... IF I can rely on replication
to keep my data safe on another box. Without the replication, there's just
no reasonable case to be made to replace ZFS.

... and now the usual ZFS comparison to make devs jealous ;)

Best regards, Clemens
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Ames Cornish]

Jim,

btrfs send reliability has been an issue, though I've been able to
successfully use it for my backups. buttersink usually detects the
errors and will either move the destination snapshot to mark it as
partial/failed (for btrfs), or cancel and delete the partial upload
(for S3). I've also found that it helps to wait a while (e.g. 30
seconds) after any volume deletes before trying the send/sync. I hope
btrfs-progs will get more reliable, too.

- Ames
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html