drive: Rework token aquisition into config framework and store token in config file

README.md

@@ -1,11 +1,13 @@
 Rclone
 ======
 
+[![Logo](http://rclone.org/rclone-120x120.png)](http://rclone.org/)
+
 Sync files and directories to and from
 
-  * Openstack Swift / Rackspace cloud files / Memset Memstore
-  * Amazon S3
   * Google Drive
+  * Amazon S3
+  * Openstack Swift / Rackspace cloud files / Memset Memstore
   * The local filesystem
 
 Features
@@ -16,6 +18,11 @@ Features
   * Copy mode to just copy new/changed files
   * Sync mode to make a directory identical
   * Check mode to check all MD5SUMs
+  * Can sync to and from network, eg two different Drive accounts
+
+Home page
+
+  * http://rclone.org/
 
 Install
 -------
@@ -30,8 +37,9 @@ Or alternatively if you have Go installed use
 
     go get github.com/ncw/rclone
 
-and this will build the binary in `$GOPATH/bin`.  You can then modify
+and this will build the binary in `$GOPATH/bin`.
-the source and submit patches.
+
+You can then modify the source and submit patches.
 
 Configure
 ---------
@@ -75,9 +83,6 @@ Choose a number from below, or type in your own value
  * US Region, Northern Virginia only.
  * Leave location constraint empty.
  2) https://s3-external-1.amazonaws.com
- * US West (Oregon) Region
- * Needs location constraint us-west-2.
- 3) https://s3-us-west-2.amazonaws.com
 [snip]
  * South America (Sao Paulo) Region
  * Needs location constraint sa-east-1.
@@ -89,8 +94,6 @@ Choose a number from below, or type in your own value
  1) 
  * US West (Oregon) Region.
  2) us-west-2
- * US West (Northern California) Region.
- 3) us-west-1
 [snip]
  * South America (Sao Paulo) Region.
  9) sa-east-1
@@ -240,11 +243,57 @@ Google drive
 
 Paths are specified as drive://path  Drive paths may be as deep as required.
 
-FIXME describe how to set up initially
+The initial setup for drive involves getting a token from Google drive
+which you need to do in your browser.  The `rclone config` walks you
+through it.
 
-So to copy a local directory to a drive directory called backup
+Here is an example of how to make a remote called `drv`
 
-rclone sync /home/source s3://backup
+```
+$ ./rclone config
+n) New remote
+d) Delete remote
+q) Quit config
+e/n/d/q> n
+name> drv
+What type of source is it?
+Choose a number from below
+ 1) swift
+ 2) s3
+ 3) local
+ 4) drive
+type> 4
+Google Application Client Id - leave blank to use rclone's.
+client_id> 
+Google Application Client Secret - leave blank to use rclone's.
+client_secret> 
+Remote config
+Go to the following link in your browser
+https://accounts.google.com/o/oauth2/auth?access_type=&approval_prompt=&client_id=XXXXXXXXXXXX.apps.googleusercontent.com&redirect_uri=urn%3XXXXX%3Awg%3Aoauth%3XX.0%3Aoob&response_type=code&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive&state=state
+Log in, then type paste the token that is returned in the browser here
+Enter verification code> X/XXXXXXXXXXXXXXXXXX-XXXXXXXXX.XXXXXXXXX-XXXXX_XXXXXXX_XXXXXXX
+--------------------
+[drv]
+client_id = 
+client_secret = 
+token = {"AccessToken":"xxxx.xxxxxxx_xxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"1/xxxxxxxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxx","Expiry":"2014-03-16T13:57:58.955387075Z","Extra":null}
+--------------------
+y) Yes this is OK
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+```
+
+You can then use it like this
+
+    rclone lsd drv://
+    rclone ls drv://
+
+To copy a local directory to a drive directory called backup
+
+    rclone copy /home/source drv://backup
+
+Google drive stores modification times accurate to 1 ms.
 
 License
 -------
@@ -255,25 +304,27 @@ COPYING file included in this package).
 Bugs
 ----
 
-Save the google drive auth in this config file too!
+  * Doesn't sync individual files yet, only directories.
-
+  * Drive: Sometimes get: Failed to copy: Upload failed: googleapi: Error 403: Rate Limit Exceeded
-Describe how to do the google auth.
+    * quota is 100.0 requests/second/user
+  * Empty directories left behind with Local and Drive
+    * eg purging a local directory with subdirectories doesn't work
 
 Contact and support
 -------------------
 
 The project website is at:
 
-- https://github.com/ncw/rclone
+  * https://github.com/ncw/rclone
 
 There you can file bug reports, ask for help or contribute patches.
 
 Authors
 -------
 
-- Nick Craig-Wood <nick@craig-wood.com>
+  * Nick Craig-Wood <nick@craig-wood.com>
 
 Contributors
 ------------
 
-- Your name goes here!
+  * Your name goes here!

drive/drive.go

@@ -9,12 +9,6 @@ package drive
 // FIXME list directory should list to channel for concurrency not
 // append to array
 
-// FIXME perhaps have a drive setup mode where we ask for all the
-// params interactively and store them all in one file
-// - don't need to store client* apparently
-
-// NB permissions of token file is too open
-
 // FIXME need to deal with some corner cases
 // * multiple files with the same name
 // * files can be in multiple directories
@@ -22,14 +16,13 @@ package drive
 // * files with / in name
 
 import (
-	"errors"
+	"encoding/json"
 	"flag"
 	"fmt"
 	"io"
 	"log"
 	"mime"
 	"net/http"
-	"os"
 	"path"
 	"strings"
 	"sync"
@@ -40,36 +33,106 @@ import (
 	"github.com/ncw/rclone/fs"
 )
 
+// Constants
+const (
+	rcloneClientId     = "202264815644.apps.googleusercontent.com"
+	rcloneClientSecret = "X4Z3ca8xfWDb1Voo-F9a7ZxJ"
+	driveFolderType    = "application/vnd.google-apps.folder"
+)
+
+// Globals
+var (
+	// Flags
+	driveFullList = flag.Bool("drive-full-list", true, "Use a full listing for directory list. More data but usually quicker.")
+)
+
 // Register with Fs
 func init() {
 	fs.Register(&fs.FsInfo{
 		Name:   "drive",
 		NewFs:  NewFs,
+		Config: Config,
 		Options: []fs.Option{{
 			Name: "client_id",
-			Help: "Google Application Client Id.",
+			Help: "Google Application Client Id - leave blank to use rclone's.",
-			Examples: []fs.OptionExample{{
-				Value: "202264815644.apps.googleusercontent.com",
-				Help:  "rclone's client id - use this or your own if you want",
-			}},
 		}, {
 			Name: "client_secret",
-			Help: "Google Application Client Secret.",
+			Help: "Google Application Client Secret - leave blank to use rclone's.",
-			Examples: []fs.OptionExample{{
-				Value: "X4Z3ca8xfWDb1Voo-F9a7ZxJ",
-				Help:  "rclone's client secret - use this or your own if you want",
-			}},
-		}, {
-			Name: "token_file",
-			Help: "Path to store token file.",
-			Examples: []fs.OptionExample{{
-				Value: path.Join(fs.HomeDir, ".gdrive-token-file"),
-				Help:  "Suggested path for token file",
-			}},
 		}},
 	})
 }
 
+// Configuration helper - called after the user has put in the defaults
+func Config(name string) {
+	// See if already have a token
+	tokenString := fs.ConfigFile.MustValue(name, "token")
+	if tokenString != "" {
+		fmt.Printf("Already have a drive token - refresh?\n")
+		if !fs.Confirm() {
+			return
+		}
+	}
+
+	// Get a drive transport
+	t, err := newDriveTransport(name)
+	if err != nil {
+		log.Fatalf("Couldn't make drive transport: %v", err)
+	}
+
+	// Generate a URL for the user to visit for authorization.
+	authUrl := t.Config.AuthCodeURL("state")
+	fmt.Printf("Go to the following link in your browser\n")
+	fmt.Printf("%s\n", authUrl)
+	fmt.Printf("Log in, then type paste the token that is returned in the browser here\n")
+
+	// Read the code, and exchange it for a token.
+	fmt.Printf("Enter verification code> ")
+	authCode := fs.ReadLine()
+	_, err = t.Exchange(authCode)
+	if err != nil {
+		log.Fatalf("Failed to get token: %v", err)
+	}
+
+}
+
+// A token cache to save the token in the config file section named
+type tokenCache string
+
+// Get the token from the config file - returns an error if it isn't present
+func (name tokenCache) Token() (*oauth.Token, error) {
+	tokenString, err := fs.ConfigFile.GetValue(string(name), "token")
+	if err != nil {
+		return nil, err
+	}
+	if tokenString == "" {
+		return nil, fmt.Errorf("Empty token found - please reconfigure")
+	}
+	token := new(oauth.Token)
+	err = json.Unmarshal([]byte(tokenString), token)
+	if err != nil {
+		return nil, err
+	}
+	return token, nil
+
+}
+
+// Save the token to the config file
+//
+// This saves the config file if it changes
+func (name tokenCache) PutToken(token *oauth.Token) error {
+	tokenBytes, err := json.Marshal(token)
+	if err != nil {
+		return err
+	}
+	tokenString := string(tokenBytes)
+	old := fs.ConfigFile.MustValue(string(name), "token")
+	if tokenString != old {
+		fs.ConfigFile.SetValue(string(name), "token", tokenString)
+		fs.SaveConfig()
+	}
+	return nil
+}
+
 // FsDrive represents a remote drive server
 type FsDrive struct {
 	svc         *drive.Service // the connection to the drive server
@@ -141,19 +204,6 @@ func (m *dirCache) Flush() {
 
 // ------------------------------------------------------------
 
-// Constants
-const (
-	//	defaultDriveTokenFile = ".google-drive-token" // FIXME root in home directory somehow
-	driveFolderType = "application/vnd.google-apps.folder"
-)
-
-// Globals
-var (
-	// Flags
-	driveAuthCode = flag.String("drive-auth-code", "", "Pass in when requested to make the drive token file.")
-	driveFullList = flag.Bool("drive-full-list", true, "Use a full listing for directory list. More data but usually quicker.")
-)
-
 // String converts this FsDrive to a string
 func (f *FsDrive) String() string {
 	return fmt.Sprintf("Google drive root '%s'", f.root)
@@ -214,39 +264,15 @@ OUTER:
 	return
 }
 
-// Ask the user for a new auth
+// Makes a new drive transport from the config
-func MakeNewToken(t *oauth.Transport) error {
+func newDriveTransport(name string) (*oauth.Transport, error) {
-	if *driveAuthCode == "" {
-		// Generate a URL to visit for authorization.
-		authUrl := t.Config.AuthCodeURL("state")
-		fmt.Fprintf(os.Stderr, "Go to the following link in your browser\n")
-		fmt.Fprintf(os.Stderr, "%s\n", authUrl)
-		fmt.Fprintf(os.Stderr, "Log in, then re-run this program with the -drive-auth-code parameter\n")
-		fmt.Fprintf(os.Stderr, "You only need this parameter once until the drive token file has been created\n")
-		return errors.New("Re-run with --drive-auth-code")
-	}
-
-	// Read the code, and exchange it for a token.
-	//fmt.Printf("Enter verification code: ")
-	//var code string
-	//fmt.Scanln(&code)
-	_, err := t.Exchange(*driveAuthCode)
-	return err
-}
-
-// NewFs contstructs an FsDrive from the path, container:path
-func NewFs(name, path string) (fs.Fs, error) {
 	clientId := fs.ConfigFile.MustValue(name, "client_id")
 	if clientId == "" {
-		return nil, errors.New("client_id not found")
+		clientId = rcloneClientId
 	}
 	clientSecret := fs.ConfigFile.MustValue(name, "client_secret")
 	if clientSecret == "" {
-		return nil, errors.New("client_secret not found")
+		clientSecret = rcloneClientSecret
-	}
-	tokenFile := fs.ConfigFile.MustValue(name, "token_file")
-	if tokenFile == "" {
-		return nil, errors.New("token-file not found")
 	}
 
 	// Settings for authorization.
@@ -257,7 +283,22 @@ func NewFs(name, path string) (fs.Fs, error) {
 		RedirectURL:  "urn:ietf:wg:oauth:2.0:oob",
 		AuthURL:      "https://accounts.google.com/o/oauth2/auth",
 		TokenURL:     "https://accounts.google.com/o/oauth2/token",
-		TokenCache:   oauth.CacheFile(tokenFile),
+		TokenCache:   tokenCache(name),
+	}
+
+	t := &oauth.Transport{
+		Config:    driveConfig,
+		Transport: http.DefaultTransport,
+	}
+
+	return t, nil
+}
+
+// NewFs contstructs an FsDrive from the path, container:path
+func NewFs(name, path string) (fs.Fs, error) {
+	t, err := newDriveTransport(name)
+	if err != nil {
+		return nil, err
 	}
 
 	root, err := parseDrivePath(path)
@@ -266,22 +307,10 @@ func NewFs(name, path string) (fs.Fs, error) {
 	}
 	f := &FsDrive{root: root, dirCache: newDirCache()}
 
-	t := &oauth.Transport{
-		Config:    driveConfig,
-		Transport: http.DefaultTransport,
-	}
-
 	// Try to pull the token from the cache; if this fails, we need to get one.
-	token, err := driveConfig.TokenCache.Token()
+	token, err := t.Config.TokenCache.Token()
 	if err != nil {
-		err := MakeNewToken(t)
+		return nil, fmt.Errorf("Failed to get token: %s", err)
-		if err != nil {
-			return nil, fmt.Errorf("Failed to authorise: %s", err)
-		}
-	} else {
-		if *driveAuthCode != "" {
-			return nil, fmt.Errorf("Only supply -drive-auth-code once")
-		}
 	}
 	t.Token = token
 

notes.txt

@@ -4,7 +4,7 @@ Todo
   * FIXME: ls without an argument for buckets/containers?
   * FIXME: More -dry-run checks for object transfer
   * Might be quicker to check md5sums first? for swift <-> swift certainly, and maybe for small files
-  * Ignoring the pseudo directories
+  * swift: Ignoring the pseudo directories
     * if object.PseudoDirectory {
       * fmt.Printf("%9s %19s %s\n", "Directory", "-", fs.Remote())
   * Make Account wrapper
@@ -20,20 +20,6 @@ Todo
   * Add max object size to fs metadata - 5GB for swift, infinite for local, ? for s3
     * tie into -max-size flag
 
-Drive
-  * Do we need the secrets or just the code?  If just the code then
-    can make a web service which does the request on the clients
-    behalf so don't need to expose the client secrets
-  * Apparently we don't need -drive-client-id or -drive-client-secret once we have a token
-  * Make a cgi which we send the user to
-    * It has the client secrets
-    * It gets google to authenticate
-    * It receives the token back
-    * It displays the token to the user to paste in to the code
-    * Should be https really
-  * Sometimes get: Failed to copy: Upload failed: googleapi: Error 403: Rate Limit Exceeded
-    * quota is 100.0 requests/second/user
-
 Ideas
   * could do encryption - put IV into metadata?
   * optimise remote copy container to another container using remote
@@ -45,7 +31,6 @@ Ideas
       * Google cloud storage: https://developers.google.com/storage/
       * rsync over ssh
       * dropbox: https://github.com/nickoneill/go-dropbox (no MD5s)
-  * grive seems to have its secrets in the source code which would make things easier!
 
 Need to make directory objects otherwise can't upload an empty directory
   * Or could upload empty directories only?
@@ -57,17 +42,6 @@ s3
     * Otherwise can set metadata
   * Returns etag and last modified in bucket list
 
-
 Bugs
 
-local & drive need to delete directories
-
-2013/01/18 16:31:32 Waiting for deletions to finish
-2013/01/18 16:31:32 z3: FIXME Skipping directory
-2013/01/18 16:31:32 z3/x: Deleted
-2013/01/18 16:31:32 Deleting path
-2013/01/18 16:31:32 Rmdir failed: remove z3: directory not empty
-
-------------------------------------------------------------
-
 Non verbose - not sure number transferred got counted up? CHECK