Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support for gocryptfs, ecryptfs, borg, duplicity, and rsync #374

Open
wants to merge 82 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 71 commits
Commits
Show all changes
82 commits
Select commit Hold shift + click to select a range
5024923
support for ecryptfs; this requires sequential file writing using mul…
bgemmill Aug 7, 2016
1b36299
Merge remote-tracking branch 'upstream/master'
bgemmill Aug 8, 2016
6a1ee48
prevent binary blob spam in the logs when using set/getxattr
bgemmill Aug 8, 2016
15ac1e7
fix requirements
bgemmill Aug 10, 2016
db9f9b0
implement proper mtime handling so rsync can work over acd_cli.
bgemmill Aug 10, 2016
076dec0
bugfix: turns out reinterpret casting floats to bytes and back via st…
Aug 11, 2016
9184a80
workaround: due to ecryptfs' bug, reporting incorrect file sizes when…
bgemmill Aug 13, 2016
45ed3fa
bugfix: who's sick of index errors when writing? this guy. turns out …
bgemmill Aug 15, 2016
6e30a93
implement truncate and ftruncate since rsync wants these too sometimes.
bgemmill Aug 15, 2016
e806362
only truncates that shorten are safe without reading the file from ac…
bgemmill Aug 15, 2016
8d87e06
The corner cases around implementing truncate when we only have a (pa…
bgemmill Aug 16, 2016
06efeca
Use memory as a backing for buffers only if the writing size is 1G (c…
bgemmill Aug 17, 2016
f1428f8
expand the path -> node caching to solve large delays with readdir + …
bgemmill Aug 18, 2016
4f9bc8b
remove tz-aware timestamp references since we're handing mtime in xat…
bgemmill Aug 21, 2016
d75fc54
helper methods for debugging, cleanup of log and comments
bgemmill Aug 21, 2016
c25a2ea
duplicity support: opportunistic reading and file size query if we're…
bgemmill Aug 22, 2016
1f2b4c6
borgbackup support by allowing manual flushes, deduplicated with rele…
bgemmill Sep 11, 2016
0520af1
Merge branch 'master' of https://github.com/yadayada/acd_cli
bgemmill Sep 11, 2016
d135b51
retry fuse operations if there's a failure on amazon's side and the r…
bgemmill Sep 12, 2016
d867f53
clean up codes when we retry
bgemmill Sep 13, 2016
d7bc917
Amazon has any number of ways to make file operations fail.
bgemmill Sep 13, 2016
f4121e8
turns out the 1000 response is thrown rather then returned
bgemmill Sep 14, 2016
ed4b5a5
Merge branch 'master' of https://github.com/yadayada/acd_cli
bgemmill Sep 24, 2016
94b9d9f
Merge branch 'master' of https://github.com/yadayada/acd_cli
bgemmill Oct 13, 2016
1c537c0
Merge branch 'master' of https://github.com/yadayada/acd_cli
bgemmill Nov 6, 2016
18671ea
we can get 1000 error codes on writing too.
bgemmill Nov 7, 2016
58c6bd4
handle more error codes when reading, and generally retry on 400s.
bgemmill Nov 9, 2016
788b862
Merge branch 'master' of https://github.com/yadayada/acd_cli
bgemmill Nov 9, 2016
282aff2
Merge branch 'master' of https://github.com/yadayada/acd_cli
bgemmill Nov 22, 2016
c1f58a4
handle 503 codes when amazon doesn't get a request body
bgemmill Dec 2, 2016
cf8ab28
Merge branch 'master' of https://github.com/yadayada/acd_cli
bgemmill Dec 2, 2016
1de45d9
Merge branch 'master' of https://github.com/yadayada/acd_cli
bgemmill Dec 25, 2016
5124699
verify that file renames and moves happen since amazon can drop some …
bgemmill Dec 26, 2016
6fb1a06
move tempfile uploading to multipart streams to see if that helps wit…
bgemmill Jan 8, 2017
0928976
extra logging around renames to catch when amazon drops rename requests
bgemmill Jan 8, 2017
6843dc1
mode, uid, and gid
bgemmill Jan 12, 2017
b36aeda
Merge branch 'master' of https://github.com/yadayada/acd_cli
bgemmill Jan 12, 2017
cf3cbb7
blksize, blocks
bgemmill Jan 12, 2017
b14b36e
Merge branch 'master' of https://github.com/yadayada/acd_cli
bgemmill Jan 16, 2017
be9a0a2
symlink, readlink
bgemmill Jan 19, 2017
b2b6d4b
caching of symlink targts
bgemmill Jan 19, 2017
879d652
invalidate the path resolving cache less to help with rsync --partial…
bgemmill Jan 21, 2017
bb1bec8
clean log spam
bgemmill Jan 21, 2017
1ef5657
add elements to the resolve cache on creation
bgemmill Jan 22, 2017
067a887
The entirety of resolve caching can be removed if we apply gerph's PR…
bgemmill Jan 24, 2017
cc47811
turns out it's faster with both
bgemmill Jan 25, 2017
2b75256
cache node ids instead of nodes to fix 0 file size issues
bgemmill Jan 29, 2017
4c19184
cache write buffer length so long uploads won't hold up getattr calls
bgemmill Feb 4, 2017
3b5044b
hit sqlite less
bgemmill Feb 6, 2017
8c0e506
tidy and make flush a noop again
bgemmill Feb 10, 2017
c5401ac
credit
bgemmill Feb 10, 2017
a167351
cache nodes by id for faster getattr/getxattr calls in large directories
bgemmill Feb 14, 2017
572f32f
Merge branch 'master' of https://github.com/yadayada/acd_cli
bgemmill Feb 14, 2017
885b4d0
store symlinks targets in file contents to ultimately allow for longe…
bgemmill Feb 15, 2017
dc47fe2
cache files' content, starting with small files and symlinks
bgemmill Feb 19, 2017
d428544
retry on 429 rate throttling errors
bgemmill Feb 21, 2017
3a88b05
release file handle lock during file writes to amazon
bgemmill Feb 23, 2017
f760f46
rely on rsync's later chmod to save an amazon call per file/folder cr…
bgemmill Feb 26, 2017
6098ebc
the folders table no longer exists
bgemmill Feb 26, 2017
a3e291e
turns out rsync needs this for proper change detection in some cases.
bgemmill Feb 27, 2017
ac80bec
releasing the fh lock can lead to 409 concurrent modification errors …
bgemmill Mar 1, 2017
c9c0719
sparse file support
bgemmill Mar 3, 2017
3d2c26a
sparse file support at ends of files too
bgemmill Mar 4, 2017
96735d5
put flush back to try to solve plex issues
bgemmill Mar 6, 2017
26325db
fix hanging write buffers full of 0s after a truncate if no fh's are …
bgemmill Mar 7, 2017
0bfca6d
Merge branch 'master' of https://github.com/yadayada/acd_cli
bgemmill Mar 7, 2017
89c33e3
lazy xattr writing and general cleanup
bgemmill Mar 7, 2017
b084226
tidy
bgemmill Mar 15, 2017
1a4b91e
speed up smaller syncs by not using the disk
bgemmill Mar 16, 2017
1d68ecf
make resolve recursive to cache intermediate results
bgemmill Mar 16, 2017
a5fa452
fix schema for properties table
bgemmill Mar 16, 2017
f60ba35
prevent xattr cache from falling out of sync on failed acd calls
bgemmill Mar 21, 2017
2583cb0
the fuse subdir module sometimes leaves trailing slashes on directory…
bgemmill Mar 29, 2017
9c08d52
Merge branch 'master' of https://github.com/yadayada/acd_cli
bgemmill Apr 16, 2017
c381f5d
Merge branch 'master' of https://github.com/yadayada/acd_cli
bgemmill May 13, 2017
e5a7592
Merge branch 'master' of https://github.com/yadayada/acd_cli
bgemmill May 26, 2017
e966440
Merge branch 'master' of https://github.com/yadayada/acd_cli
bgemmill May 26, 2017
a598ef9
Merge branch 'master' of https://github.com/yadayada/acd_cli
bgemmill May 27, 2017
3f3bed8
remove legacy symlink handling since amazon purged properties during …
bgemmill May 27, 2017
e2dcdf7
Merge branch 'master' of https://github.com/yadayada/acd_cli
bgemmill May 29, 2017
e2554a0
docs
bgemmill May 30, 2017
6b014d6
Merge branch 'master' of https://github.com/yadayada/acd_cli
bgemmill May 30, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 22 additions & 17 deletions acd_cli.py
Original file line number Diff line number Diff line change
@@ -1,35 +1,33 @@
#!/usr/bin/env python3
import sys
import os
import json
import argparse
import json
import logging
import logging.handlers
import os
import re
import signal
import sys
import time
import re
import appdirs

from collections import namedtuple
from configparser import ConfigParser
from functools import partial
from multiprocessing import Event

from pkgutil import walk_packages

import appdirs
from pkg_resources import iter_entry_points

import acdcli
from acdcli import plugins
from acdcli.api import client
from acdcli.api.common import RequestError, is_valid_id
from acdcli.cache import format, db
from acdcli.cache.db import CacheConsts
from acdcli.utils import hashing, progress
from acdcli.utils.conf import get_conf
from acdcli.utils.threading import QueuedLoader
from acdcli.utils.time import *

# load local plugin modules (default ones, for developers)
from acdcli import plugins

for importer, modname, ispkg in walk_packages(path=plugins.__path__, prefix=plugins.__name__ + '.',
onerror=lambda x: None):
if not ispkg:
Expand Down Expand Up @@ -121,11 +119,13 @@ def pprint(d: dict):
# Glue functions (API, cache)
#


class CacheConsts(object):
CHECKPOINT_KEY = 'checkpoint'
LAST_SYNC_KEY = 'last_sync'
MAX_AGE = 30
def sync_owner_id():
global cache
owner_id = cache.KeyValueStorage.get(CacheConsts.OWNER_ID)
if not owner_id:
owner_id = acd_client.get_owner_id()
cache.KeyValueStorage[CacheConsts.OWNER_ID] = owner_id
return owner_id


def sync_node_list(full=False, to_file=None, from_file=None) -> 'Union[int, None]':
Expand Down Expand Up @@ -194,12 +194,14 @@ def sync_node_list(full=False, to_file=None, from_file=None) -> 'Union[int, None
print()
if to_file:
out.close()
sync_owner_id()


def old_sync() -> 'Union[int, None]':
global cache
cache.drop_all()
cache = db.NodeCache(CACHE_PATH)
sync_owner_id()
try:
folders = acd_client.get_folder_list()
folders.extend(acd_client.get_trashed_folders())
Expand Down Expand Up @@ -1146,11 +1148,14 @@ def mount_action(args: argparse.Namespace):
import acdcli.acd_fuse
acdcli.acd_fuse.mount(args.path, dict(acd_client=acd_client, cache=cache,
nlinks=args.nlinks, autosync=asp,
settings_path=SETTINGS_PATH),
settings_path=SETTINGS_PATH,
umask = args.umask,
uid = args.uid,
gid = args.gid
),
ro=args.read_only, foreground=args.foreground,
nothreads=args.single_threaded,
nonempty=args.nonempty, modules=args.modules,
umask=args.umask,gid=args.gid,uid=args.uid,
allow_root=args.allow_root, allow_other=args.allow_other,
volname=args.volname)

Expand Down
Loading