drive: Rework token aquisition into config framework and store token in config file

2024-11-25 09:54:44 +01:00 · 2014-03-16 14:01:17 +00:00 · 2014-03-16 14:01:17 +00:00 · b4dd693d23
commit b4dd693d23
parent 1b3a49929b
3 changed files with 182 additions and 128 deletions
--- a/README.md
+++ b/README.md
@ -1,11 +1,13 @@
 Rclone
 ======

+[![Logo](http://rclone.org/rclone-120x120.png)](http://rclone.org/)
+
 Sync files and directories to and from

-  * Openstack Swift / Rackspace cloud files / Memset Memstore
-  * Amazon S3
  * Google Drive
+  * Amazon S3
+  * Openstack Swift / Rackspace cloud files / Memset Memstore
  * The local filesystem

 Features
@ -16,6 +18,11 @@ Features
  * Copy mode to just copy new/changed files
  * Sync mode to make a directory identical
  * Check mode to check all MD5SUMs
+  * Can sync to and from network, eg two different Drive accounts
+
+Home page
+
+  * http://rclone.org/

 Install
 -------
@ -30,8 +37,9 @@ Or alternatively if you have Go installed use

    go get github.com/ncw/rclone

-and this will build the binary in `$GOPATH/bin`.  You can then modify
-the source and submit patches.
+and this will build the binary in `$GOPATH/bin`.
+
+You can then modify the source and submit patches.

 Configure
 ---------
@ -75,9 +83,6 @@ Choose a number from below, or type in your own value
 * US Region, Northern Virginia only.
 * Leave location constraint empty.
 2) https://s3-external-1.amazonaws.com
- * US West (Oregon) Region
- * Needs location constraint us-west-2.
- 3) https://s3-us-west-2.amazonaws.com
 [snip]
 * South America (Sao Paulo) Region
 * Needs location constraint sa-east-1.
@ -89,8 +94,6 @@ Choose a number from below, or type in your own value
 1) 
 * US West (Oregon) Region.
 2) us-west-2
- * US West (Northern California) Region.
- 3) us-west-1
 [snip]
 * South America (Sao Paulo) Region.
 9) sa-east-1
@ -240,11 +243,57 @@ Google drive

 Paths are specified as drive://path  Drive paths may be as deep as required.

-FIXME describe how to set up initially
+The initial setup for drive involves getting a token from Google drive
+which you need to do in your browser.  The `rclone config` walks you
+through it.

-So to copy a local directory to a drive directory called backup
+Here is an example of how to make a remote called `drv`

-rclone sync /home/source s3://backup
+```
+$ ./rclone config
+n) New remote
+d) Delete remote
+q) Quit config
+e/n/d/q> n
+name> drv
+What type of source is it?
+Choose a number from below
+ 1) swift
+ 2) s3
+ 3) local
+ 4) drive
+type> 4
+Google Application Client Id - leave blank to use rclone's.
+client_id> 
+Google Application Client Secret - leave blank to use rclone's.
+client_secret> 
+Remote config
+Go to the following link in your browser
+https://accounts.google.com/o/oauth2/auth?access_type=&approval_prompt=&client_id=XXXXXXXXXXXX.apps.googleusercontent.com&redirect_uri=urn%3XXXXX%3Awg%3Aoauth%3XX.0%3Aoob&response_type=code&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive&state=state
+Log in, then type paste the token that is returned in the browser here
+Enter verification code> X/XXXXXXXXXXXXXXXXXX-XXXXXXXXX.XXXXXXXXX-XXXXX_XXXXXXX_XXXXXXX
+--------------------
+[drv]
+client_id = 
+client_secret = 
+token = {"AccessToken":"xxxx.xxxxxxx_xxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"1/xxxxxxxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxx","Expiry":"2014-03-16T13:57:58.955387075Z","Extra":null}
+--------------------
+y) Yes this is OK
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+```
+
+You can then use it like this
+
+    rclone lsd drv://
+    rclone ls drv://
+
+To copy a local directory to a drive directory called backup
+
+    rclone copy /home/source drv://backup
+
+Google drive stores modification times accurate to 1 ms.

 License
 -------
@ -255,25 +304,27 @@ COPYING file included in this package).
 Bugs
 ----

-Save the google drive auth in this config file too!
-
-Describe how to do the google auth.
+  * Doesn't sync individual files yet, only directories.
+  * Drive: Sometimes get: Failed to copy: Upload failed: googleapi: Error 403: Rate Limit Exceeded
+    * quota is 100.0 requests/second/user
+  * Empty directories left behind with Local and Drive
+    * eg purging a local directory with subdirectories doesn't work

 Contact and support
 -------------------

 The project website is at:

- https://github.com/ncw/rclone
+  * https://github.com/ncw/rclone

 There you can file bug reports, ask for help or contribute patches.

 Authors
 -------

- Nick Craig-Wood <nick@craig-wood.com>
+  * Nick Craig-Wood <nick@craig-wood.com>

 Contributors
 ------------

- Your name goes here!
+  * Your name goes here!
--- a/drive/drive.go
+++ b/drive/drive.go
@ -9,12 +9,6 @@ package drive
 // FIXME list directory should list to channel for concurrency not
 // append to array

-// FIXME perhaps have a drive setup mode where we ask for all the
-// params interactively and store them all in one file
-// - don't need to store client* apparently
-
-// NB permissions of token file is too open
-
 // FIXME need to deal with some corner cases
 // * multiple files with the same name
 // * files can be in multiple directories
@ -22,14 +16,13 @@ package drive
 // * files with / in name

 import (
-	"errors"
+	"encoding/json"
 	"flag"
 	"fmt"
 	"io"
 	"log"
 	"mime"
 	"net/http"
-	"os"
 	"path"
 	"strings"
 	"sync"
@ -40,36 +33,106 @@ import (
 	"github.com/ncw/rclone/fs"
 )

+// Constants
+const (
+	rcloneClientId     = "202264815644.apps.googleusercontent.com"
+	rcloneClientSecret = "X4Z3ca8xfWDb1Voo-F9a7ZxJ"
+	driveFolderType    = "application/vnd.google-apps.folder"
+)
+
+// Globals
+var (
+	// Flags
+	driveFullList = flag.Bool("drive-full-list", true, "Use a full listing for directory list. More data but usually quicker.")
+)
+
 // Register with Fs
 func init() {
 	fs.Register(&fs.FsInfo{
 		Name:   "drive",
 		NewFs:  NewFs,
+		Config: Config,
 		Options: []fs.Option{{
 			Name: "client_id",
-			Help: "Google Application Client Id.",
-			Examples: []fs.OptionExample{{
-				Value: "202264815644.apps.googleusercontent.com",
-				Help:  "rclone's client id - use this or your own if you want",
-			}},
+			Help: "Google Application Client Id - leave blank to use rclone's.",
 		}, {
 			Name: "client_secret",
-			Help: "Google Application Client Secret.",
-			Examples: []fs.OptionExample{{
-				Value: "X4Z3ca8xfWDb1Voo-F9a7ZxJ",
-				Help:  "rclone's client secret - use this or your own if you want",
-			}},
-		}, {
-			Name: "token_file",
-			Help: "Path to store token file.",
-			Examples: []fs.OptionExample{{
-				Value: path.Join(fs.HomeDir, ".gdrive-token-file"),
-				Help:  "Suggested path for token file",
-			}},
+			Help: "Google Application Client Secret - leave blank to use rclone's.",
 		}},
 	})
 }

+// Configuration helper - called after the user has put in the defaults
+func Config(name string) {
+	// See if already have a token
+	tokenString := fs.ConfigFile.MustValue(name, "token")
+	if tokenString != "" {
+		fmt.Printf("Already have a drive token - refresh?\n")
+		if !fs.Confirm() {
+			return
+		}
+	}
+
+	// Get a drive transport
+	t, err := newDriveTransport(name)
+	if err != nil {
+		log.Fatalf("Couldn't make drive transport: %v", err)
+	}
+
+	// Generate a URL for the user to visit for authorization.
+	authUrl := t.Config.AuthCodeURL("state")
+	fmt.Printf("Go to the following link in your browser\n")
+	fmt.Printf("%s\n", authUrl)
+	fmt.Printf("Log in, then type paste the token that is returned in the browser here\n")
+
+	// Read the code, and exchange it for a token.
+	fmt.Printf("Enter verification code> ")
+	authCode := fs.ReadLine()
+	_, err = t.Exchange(authCode)
+	if err != nil {
+		log.Fatalf("Failed to get token: %v", err)
+	}
+
+}
+
+// A token cache to save the token in the config file section named
+type tokenCache string
+
+// Get the token from the config file - returns an error if it isn't present
+func (name tokenCache) Token() (*oauth.Token, error) {
+	tokenString, err := fs.ConfigFile.GetValue(string(name), "token")
+	if err != nil {
+		return nil, err
+	}
+	if tokenString == "" {
+		return nil, fmt.Errorf("Empty token found - please reconfigure")
+	}
+	token := new(oauth.Token)
+	err = json.Unmarshal([]byte(tokenString), token)
+	if err != nil {
+		return nil, err
+	}
+	return token, nil
+
+}
+
+// Save the token to the config file
+//
+// This saves the config file if it changes
+func (name tokenCache) PutToken(token *oauth.Token) error {
+	tokenBytes, err := json.Marshal(token)
+	if err != nil {
+		return err
+	}
+	tokenString := string(tokenBytes)
+	old := fs.ConfigFile.MustValue(string(name), "token")
+	if tokenString != old {
+		fs.ConfigFile.SetValue(string(name), "token", tokenString)
+		fs.SaveConfig()
+	}
+	return nil
+}
+
 // FsDrive represents a remote drive server
 type FsDrive struct {
 	svc         *drive.Service // the connection to the drive server
@ -141,19 +204,6 @@ func (m *dirCache) Flush() {

 // ------------------------------------------------------------

-// Constants
-const (
-	//	defaultDriveTokenFile = ".google-drive-token" // FIXME root in home directory somehow
-	driveFolderType = "application/vnd.google-apps.folder"
-)
-
-// Globals
-var (
-	// Flags
-	driveAuthCode = flag.String("drive-auth-code", "", "Pass in when requested to make the drive token file.")
-	driveFullList = flag.Bool("drive-full-list", true, "Use a full listing for directory list. More data but usually quicker.")
-)
-
 // String converts this FsDrive to a string
 func (f *FsDrive) String() string {
 	return fmt.Sprintf("Google drive root '%s'", f.root)
@ -214,39 +264,15 @@ OUTER:
 	return
 }

-// Ask the user for a new auth
-func MakeNewToken(t *oauth.Transport) error {
-	if *driveAuthCode == "" {
-		// Generate a URL to visit for authorization.
-		authUrl := t.Config.AuthCodeURL("state")
-		fmt.Fprintf(os.Stderr, "Go to the following link in your browser\n")
-		fmt.Fprintf(os.Stderr, "%s\n", authUrl)
-		fmt.Fprintf(os.Stderr, "Log in, then re-run this program with the -drive-auth-code parameter\n")
-		fmt.Fprintf(os.Stderr, "You only need this parameter once until the drive token file has been created\n")
-		return errors.New("Re-run with --drive-auth-code")
-	}
-
-	// Read the code, and exchange it for a token.
-	//fmt.Printf("Enter verification code: ")
-	//var code string
-	//fmt.Scanln(&code)
-	_, err := t.Exchange(*driveAuthCode)
-	return err
-}
-
-// NewFs contstructs an FsDrive from the path, container:path
-func NewFs(name, path string) (fs.Fs, error) {
+// Makes a new drive transport from the config
+func newDriveTransport(name string) (*oauth.Transport, error) {
 	clientId := fs.ConfigFile.MustValue(name, "client_id")
 	if clientId == "" {
-		return nil, errors.New("client_id not found")
+		clientId = rcloneClientId
 	}
 	clientSecret := fs.ConfigFile.MustValue(name, "client_secret")
 	if clientSecret == "" {
-		return nil, errors.New("client_secret not found")
-	}
-	tokenFile := fs.ConfigFile.MustValue(name, "token_file")
-	if tokenFile == "" {
-		return nil, errors.New("token-file not found")
+		clientSecret = rcloneClientSecret
 	}

 	// Settings for authorization.
@ -257,7 +283,22 @@ func NewFs(name, path string) (fs.Fs, error) {
 		RedirectURL:  "urn:ietf:wg:oauth:2.0:oob",
 		AuthURL:      "https://accounts.google.com/o/oauth2/auth",
 		TokenURL:     "https://accounts.google.com/o/oauth2/token",
-		TokenCache:   oauth.CacheFile(tokenFile),
+		TokenCache:   tokenCache(name),
+	}
+
+	t := &oauth.Transport{
+		Config:    driveConfig,
+		Transport: http.DefaultTransport,
+	}
+
+	return t, nil
+}
+
+// NewFs contstructs an FsDrive from the path, container:path
+func NewFs(name, path string) (fs.Fs, error) {
+	t, err := newDriveTransport(name)
+	if err != nil {
+		return nil, err
 	}

 	root, err := parseDrivePath(path)
@ -266,22 +307,10 @@ func NewFs(name, path string) (fs.Fs, error) {
 	}
 	f := &FsDrive{root: root, dirCache: newDirCache()}

-	t := &oauth.Transport{
-		Config:    driveConfig,
-		Transport: http.DefaultTransport,
-	}
-
 	// Try to pull the token from the cache; if this fails, we need to get one.
-	token, err := driveConfig.TokenCache.Token()
+	token, err := t.Config.TokenCache.Token()
 	if err != nil {
-		err := MakeNewToken(t)
-		if err != nil {
-			return nil, fmt.Errorf("Failed to authorise: %s", err)
-		}
-	} else {
-		if *driveAuthCode != "" {
-			return nil, fmt.Errorf("Only supply -drive-auth-code once")
-		}
+		return nil, fmt.Errorf("Failed to get token: %s", err)
 	}
 	t.Token = token

--- a/notes.txt
+++ b/notes.txt
@ -4,7 +4,7 @@ Todo
  * FIXME: ls without an argument for buckets/containers?
  * FIXME: More -dry-run checks for object transfer
  * Might be quicker to check md5sums first? for swift <-> swift certainly, and maybe for small files
-  * Ignoring the pseudo directories
+  * swift: Ignoring the pseudo directories
    * if object.PseudoDirectory {
      * fmt.Printf("%9s %19s %s\n", "Directory", "-", fs.Remote())
  * Make Account wrapper
@ -20,20 +20,6 @@ Todo
  * Add max object size to fs metadata - 5GB for swift, infinite for local, ? for s3
    * tie into -max-size flag

-Drive
-  * Do we need the secrets or just the code?  If just the code then
-    can make a web service which does the request on the clients
-    behalf so don't need to expose the client secrets
-  * Apparently we don't need -drive-client-id or -drive-client-secret once we have a token
-  * Make a cgi which we send the user to
-    * It has the client secrets
-    * It gets google to authenticate
-    * It receives the token back
-    * It displays the token to the user to paste in to the code
-    * Should be https really
-  * Sometimes get: Failed to copy: Upload failed: googleapi: Error 403: Rate Limit Exceeded
-    * quota is 100.0 requests/second/user
-
 Ideas
  * could do encryption - put IV into metadata?
  * optimise remote copy container to another container using remote
@ -45,7 +31,6 @@ Ideas
      * Google cloud storage: https://developers.google.com/storage/
      * rsync over ssh
      * dropbox: https://github.com/nickoneill/go-dropbox (no MD5s)
-  * grive seems to have its secrets in the source code which would make things easier!

 Need to make directory objects otherwise can't upload an empty directory
  * Or could upload empty directories only?
@ -57,17 +42,6 @@ s3
    * Otherwise can set metadata
  * Returns etag and last modified in bucket list

-
 Bugs

-local & drive need to delete directories
-
-2013/01/18 16:31:32 Waiting for deletions to finish
-2013/01/18 16:31:32 z3: FIXME Skipping directory
-2013/01/18 16:31:32 z3/x: Deleted
-2013/01/18 16:31:32 Deleting path
-2013/01/18 16:31:32 Rmdir failed: remove z3: directory not empty
-
------------------------------------------------------------
-
 Non verbose - not sure number transferred got counted up? CHECK