extern/rclone
1
0
Fork 0
mirror of https://github.com/rclone/rclone.git synced 2024-11-22 00:13:49 +01:00
Code Issues Packages Projects Releases Wiki Activity

onedrive: add metadata support

Browse Source 
This change adds support for metadata on OneDrive. Metadata (including
permissions) is supported for both files and directories.

OneDrive supports System Metadata (not User Metadata, as of this writing.) Much
of the metadata is read-only, and there are some differences between OneDrive
Personal and Business (see table in OneDrive backend docs for details).

Permissions are also supported, if --onedrive-metadata-permissions is set. The
accepted values for --onedrive-metadata-permissions are read, write, read,write, and
off (the default). write supports adding new permissions, updating the "role" of
existing permissions, and removing permissions. Updating and removing require
the Permission ID to be known, so it is recommended to use read,write instead of
write if you wish to update/remove permissions.

Permissions are read/written in JSON format using the same schema as the
OneDrive API, which differs slightly between OneDrive Personal and Business.
(See OneDrive backend docs for examples.)

To write permissions, pass in a "permissions" metadata key using this same
format. The --metadata-mapper tool can be very helpful for this.

When adding permissions, an email address can be provided in the User.ID or
DisplayName properties of grantedTo or grantedToIdentities. Alternatively, an
ObjectID can be provided in User.ID. At least one valid recipient must be
provided in order to add a permission for a user. Creating a Public Link is also
supported, if Link.Scope is set to "anonymous".

Note that adding a permission can fail if a conflicting permission already
exists for the file/folder.

To update an existing permission, include both the Permission ID and the new
roles to be assigned. roles is the only property that can be changed.

To remove permissions, pass in a blob containing only the permissions you wish
to keep (which can be empty, to remove all.)

Note that both reading and writing permissions requires extra API calls, so if
you don't need to read or write permissions it is recommended to omit --onedrive-
metadata-permissions.

Metadata and permissions are supported for Folders (directories) as well as
Files. Note that setting the mtime or btime on a Folder requires one extra API
call on OneDrive Business only.

OneDrive does not currently support User Metadata. When writing metadata, only
writeable system properties will be written -- any read-only or unrecognized keys
passed in will be ignored.

TIP: to see the metadata and permissions for any file or folder, run:

rclone lsjson remote:path --stat -M --onedrive-metadata-permissions read

See the OneDrive backend docs for a table of all the supported metadata
properties.
This commit is contained in:
nielash 2024-02-22 09:17:14 -05:00 committed by Nick Craig-Wood
parent 4e07a72dc7
commit 1473de3f04
8 changed files with 2093 additions and 69 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

144
backend/onedrive/api/types.go
View File

@ -7,7 +7,7 @@ import (
)
const (
timeFormat = `"` + time.RFC3339 + `"`
timeFormat = `"` + "2006-01-02T15:04:05.999Z" + `"`
// PackageTypeOneNote is the package type value for OneNote files
PackageTypeOneNote = "oneNote"
@ -40,17 +40,17 @@ var _ error = (*Error)(nil)
// Identity represents an identity of an actor. For example, and actor
// can be a user, device, or application.
type Identity struct {
DisplayName string `json:"displayName"`
ID string `json:"id"`
DisplayName string `json:"displayName,omitempty"`
ID string `json:"id,omitempty"`
}
// IdentitySet is a keyed collection of Identity objects. It is used
// to represent a set of identities associated with various events for
// an item, such as created by or last modified by.
type IdentitySet struct {
User Identity `json:"user"`
Application Identity `json:"application"`
Device Identity `json:"device"`
User Identity `json:"user,omitempty"`
Application Identity `json:"application,omitempty"`
Device Identity `json:"device,omitempty"`
}
// Quota groups storage space quota-related information on OneDrive into a single structure.
@ -150,16 +150,15 @@ type FileFacet struct {
// facet can be used to specify the last modified date or created date
// of the item as it was on the local device.
type FileSystemInfoFacet struct {
CreatedDateTime Timestamp `json:"createdDateTime"` // The UTC date and time the file was created on a client.
LastModifiedDateTime Timestamp `json:"lastModifiedDateTime"` // The UTC date and time the file was last modified on a client.
CreatedDateTime Timestamp `json:"createdDateTime,omitempty"` // The UTC date and time the file was created on a client.
LastModifiedDateTime Timestamp `json:"lastModifiedDateTime,omitempty"` // The UTC date and time the file was last modified on a client.
}
// DeletedFacet indicates that the item on OneDrive has been
// deleted. In this version of the API, the presence (non-null) of the
// facet value indicates that the file was deleted. A null (or
// missing) value indicates that the file is not deleted.
type DeletedFacet struct {
}
type DeletedFacet struct{}
// PackageFacet indicates that a DriveItem is the top level item
// in a "package" or a collection of items that should be treated as a collection instead of individual items.
@ -168,6 +167,101 @@ type PackageFacet struct {
Type string `json:"type"`
}
// SharedType indicates a DriveItem has been shared with others. The resource includes information about how the item is shared.
// If a Driveitem has a non-null shared facet, the item has been shared.
type SharedType struct {
Owner IdentitySet `json:"owner,omitempty"` // The identity of the owner of the shared item. Read-only.
Scope string `json:"scope,omitempty"` // Indicates the scope of how the item is shared: anonymous, organization, or users. Read-only.
SharedBy IdentitySet `json:"sharedBy,omitempty"` // The identity of the user who shared the item. Read-only.
SharedDateTime Timestamp `json:"sharedDateTime,omitempty"` // The UTC date and time when the item was shared. Read-only.
}
// SharingInvitationType groups invitation-related data items into a single structure.
type SharingInvitationType struct {
Email string `json:"email,omitempty"` // The email address provided for the recipient of the sharing invitation. Read-only.
InvitedBy *IdentitySet `json:"invitedBy,omitempty"` // Provides information about who sent the invitation that created this permission, if that information is available. Read-only.
SignInRequired bool `json:"signInRequired,omitempty"` // If true the recipient of the invitation needs to sign in in order to access the shared item. Read-only.
}
// SharingLinkType groups link-related data items into a single structure.
// If a Permission resource has a non-null sharingLink facet, the permission represents a sharing link (as opposed to permissions granted to a person or group).
type SharingLinkType struct {
Application *Identity `json:"application,omitempty"` // The app the link is associated with.
Type LinkType `json:"type,omitempty"` // The type of the link created.
Scope LinkScope `json:"scope,omitempty"` // The scope of the link represented by this permission. Value anonymous indicates the link is usable by anyone, organization indicates the link is only usable for users signed into the same tenant.
WebHTML string `json:"webHtml,omitempty"` // For embed links, this property contains the HTML code for an <iframe> element that will embed the item in a webpage.
WebURL string `json:"webUrl,omitempty"` // A URL that opens the item in the browser on the OneDrive website.
}
// LinkType represents the type of SharingLinkType created.
type LinkType string
const (
ViewLinkType LinkType = "view" // ViewLinkType (role: read) A view-only sharing link, allowing read-only access.
EditLinkType LinkType = "edit" // EditLinkType (role: write) An edit sharing link, allowing read-write access.
EmbedLinkType LinkType = "embed" // EmbedLinkType (role: read) A view-only sharing link that can be used to embed content into a host webpage. Embed links are not available for OneDrive for Business or SharePoint.
)
// LinkScope represents the scope of the link represented by this permission.
// Value anonymous indicates the link is usable by anyone, organization indicates the link is only usable for users signed into the same tenant.
type LinkScope string
const (
AnonymousScope LinkScope = "anonymous" // AnonymousScope = Anyone with the link has access, without needing to sign in. This may include people outside of your organization.
OrganizationScope LinkScope = "organization" // OrganizationScope = Anyone signed into your organization (tenant) can use the link to get access. Only available in OneDrive for Business and SharePoint.
)
// PermissionsType provides information about a sharing permission granted for a DriveItem resource.
// Sharing permissions have a number of different forms. The Permission resource represents these different forms through facets on the resource.
type PermissionsType struct {
ID string `json:"id"` // The unique identifier of the permission among all permissions on the item. Read-only.
GrantedTo *IdentitySet `json:"grantedTo,omitempty"` // For user type permissions, the details of the users & applications for this permission. Read-only.
GrantedToIdentities []*IdentitySet `json:"grantedToIdentities,omitempty"` // For link type permissions, the details of the users to whom permission was granted. Read-only.
Invitation *SharingInvitationType `json:"invitation,omitempty"` // Details of any associated sharing invitation for this permission. Read-only.
InheritedFrom *ItemReference `json:"inheritedFrom,omitempty"` // Provides a reference to the ancestor of the current permission, if it is inherited from an ancestor. Read-only.
Link *SharingLinkType `json:"link,omitempty"` // Provides the link details of the current permission, if it is a link type permissions. Read-only.
Roles []Role `json:"roles,omitempty"` // The type of permission (read, write, owner, member). Read-only.
ShareID string `json:"shareId,omitempty"` // A unique token that can be used to access this shared item via the shares API. Read-only.
}
// Role represents the type of permission (read, write, owner, member)
type Role string
const (
ReadRole Role = "read" // ReadRole provides the ability to read the metadata and contents of the item.
WriteRole Role = "write" // WriteRole provides the ability to read and modify the metadata and contents of the item.
OwnerRole Role = "owner" // OwnerRole represents the owner role for SharePoint and OneDrive for Business.
MemberRole Role = "member" // MemberRole represents the member role for SharePoint and OneDrive for Business.
)
// PermissionsResponse is the response to the list permissions method
type PermissionsResponse struct {
Value []*PermissionsType `json:"value"` // An array of Item objects
}
// AddPermissionsRequest is the request for the add permissions method
type AddPermissionsRequest struct {
Recipients []DriveRecipient `json:"recipients,omitempty"` // A collection of recipients who will receive access and the sharing invitation.
Message string `json:"message,omitempty"` // A plain text formatted message that is included in the sharing invitation. Maximum length 2000 characters.
RequireSignIn bool `json:"requireSignIn,omitempty"` // Specifies whether the recipient of the invitation is required to sign-in to view the shared item.
SendInvitation bool `json:"sendInvitation,omitempty"` // If true, a sharing link is sent to the recipient. Otherwise, a permission is granted directly without sending a notification.
Roles []Role `json:"roles,omitempty"` // Specify the roles that are to be granted to the recipients of the sharing invitation.
RetainInheritedPermissions bool `json:"retainInheritedPermissions,omitempty"` // Optional. If true (default), any existing inherited permissions are retained on the shared item when sharing this item for the first time. If false, all existing permissions are removed when sharing for the first time. OneDrive Business Only.
}
// UpdatePermissionsRequest is the request for the update permissions method
type UpdatePermissionsRequest struct {
Roles []Role `json:"roles,omitempty"` // Specify the roles that are to be granted to the recipients of the sharing invitation.
}
// DriveRecipient represents a person, group, or other recipient to share with using the invite action.
type DriveRecipient struct {
Email string `json:"email,omitempty"` // The email address for the recipient, if the recipient has an associated email address.
Alias string `json:"alias,omitempty"` // The alias of the domain object, for cases where an email address is unavailable (e.g. security groups).
ObjectID string `json:"objectId,omitempty"` // The unique identifier for the recipient in the directory.
}
// Item represents metadata for an item in OneDrive
type Item struct {
ID string `json:"id"` // The unique identifier of the item within the Drive. Read-only.
@ -181,7 +275,7 @@ type Item struct {
Size int64 `json:"size"` // Size of the item in bytes. Read-only.
ParentReference *ItemReference `json:"parentReference"` // Parent information, if the item has a parent. Read-write.
WebURL string `json:"webUrl"` // URL that displays the resource in the browser. Read-only.
Description string `json:"description"` // Provide a user-visible description of the item. Read-write.
Description string `json:"description,omitempty"` // Provides a user-visible description of the item. Read-write. Only on OneDrive Personal. Undocumented limit of 1024 characters.
Folder *FolderFacet `json:"folder"` // Folder metadata, if the item is a folder. Read-only.
File *FileFacet `json:"file"` // File metadata, if the item is a file. Read-only.
RemoteItem *RemoteItemFacet `json:"remoteItem"` // Remote Item metadata, if the item is a remote shared item. Read-only.
@ -193,6 +287,21 @@ type Item struct {
// Location *LocationFacet `json:"location"` // Location metadata, if the item has location data. Read-only.
Package *PackageFacet `json:"package"` // If present, indicates that this item is a package instead of a folder or file. Packages are treated like files in some contexts and folders in others. Read-only.
Deleted *DeletedFacet `json:"deleted"` // Information about the deleted state of the item. Read-only.
Malware *struct{} `json:"malware,omitempty"` // Malware metadata, if the item was detected to contain malware. Read-only. (Currently has no properties.)
Shared *SharedType `json:"shared,omitempty"` // Indicates that the item has been shared with others and provides information about the shared state of the item. Read-only.
}
// Metadata represents a request to update Metadata.
// It includes only the writeable properties.
// omitempty is intentionally included for all, per https://learn.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_update?view=odsp-graph-online#request-body
type Metadata struct {
Description string `json:"description,omitempty"` // Provides a user-visible description of the item. Read-write. Only on OneDrive Personal. Undocumented limit of 1024 characters.
FileSystemInfo *FileSystemInfoFacet `json:"fileSystemInfo,omitempty"` // File system information on client. Read-write.
}
// IsEmpty returns true if the metadata is empty (there is nothing to set)
func (m Metadata) IsEmpty() bool {
return m.Description == "" && m.FileSystemInfo == &FileSystemInfoFacet{}
}
// DeltaResponse is the response to the view delta method
@ -216,6 +325,12 @@ type CreateItemRequest struct {
ConflictBehavior string `json:"@name.conflictBehavior"` // Determines what to do if an item with a matching name already exists in this folder. Accepted values are: rename, replace, and fail (the default).
}
// CreateItemWithMetadataRequest is like CreateItemRequest but also allows setting Metadata
type CreateItemWithMetadataRequest struct {
CreateItemRequest
Metadata
}
// SetFileSystemInfo is used to Update an object's FileSystemInfo.
type SetFileSystemInfo struct {
FileSystemInfo FileSystemInfoFacet `json:"fileSystemInfo"` // File system information on client. Read-write.
@ -223,7 +338,7 @@ type SetFileSystemInfo struct {
// CreateUploadRequest is used by CreateUploadSession to set the dates correctly
type CreateUploadRequest struct {
Item SetFileSystemInfo `json:"item"`
Item Metadata `json:"item"`
}
// CreateUploadResponse is the response from creating an upload session
@ -419,6 +534,11 @@ func (i *Item) GetParentReference() *ItemReference {
return i.ParentReference
}
// MalwareDetected returns true if OneDrive has detected that this item contains malware.
func (i *Item) MalwareDetected() bool {
return i.Malware != nil
}
// IsRemote checks if item is a remote item
func (i *Item) IsRemote() bool {
return i.RemoteItem != nil

951
backend/onedrive/metadata.go Normal file
View File

@ -0,0 +1,951 @@
package onedrive
import (
"context"
"encoding/json"
"errors"
"fmt"
"net/http"
"strings"
"time"
"github.com/rclone/rclone/backend/onedrive/api"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/lib/dircache"
"golang.org/x/exp/slices" // replace with slices after go1.21 is the minimum version
)
const (
dirMimeType = "inode/directory"
timeFormatIn = time.RFC3339
timeFormatOut = "2006-01-02T15:04:05.999Z" // mS for OneDrive Personal, otherwise only S
)
// system metadata keys which this backend owns
var systemMetadataInfo = map[string]fs.MetadataHelp{
"content-type": {
Help: "The MIME type of the file.",
Type: "string",
Example: "text/plain",
ReadOnly: true,
},
"mtime": {
Help: "Time of last modification with S accuracy (mS for OneDrive Personal).",
Type: "RFC 3339",
Example: "2006-01-02T15:04:05Z",
},
"btime": {
Help: "Time of file birth (creation) with S accuracy (mS for OneDrive Personal).",
Type: "RFC 3339",
Example: "2006-01-02T15:04:05Z",
},
"utime": {
Help: "Time of upload with S accuracy (mS for OneDrive Personal).",
Type: "RFC 3339",
Example: "2006-01-02T15:04:05Z",
ReadOnly: true,
},
"created-by-display-name": {
Help: "Display name of the user that created the item.",
Type: "string",
Example: "John Doe",
ReadOnly: true,
},
"created-by-id": {
Help: "ID of the user that created the item.",
Type: "string",
Example: "48d31887-5fad-4d73-a9f5-3c356e68a038",
ReadOnly: true,
},
"description": {
Help: "A short description of the file. Max 1024 characters. Only supported for OneDrive Personal.",
Type: "string",
Example: "Contract for signing",
},
"id": {
Help: "The unique identifier of the item within OneDrive.",
Type: "string",
Example: "01BYE5RZ6QN3ZWBTUFOFD3GSPGOHDJD36K",
ReadOnly: true,
},
"last-modified-by-display-name": {
Help: "Display name of the user that last modified the item.",
Type: "string",
Example: "John Doe",
ReadOnly: true,
},
"last-modified-by-id": {
Help: "ID of the user that last modified the item.",
Type: "string",
Example: "48d31887-5fad-4d73-a9f5-3c356e68a038",
ReadOnly: true,
},
"malware-detected": {
Help: "Whether OneDrive has detected that the item contains malware.",
Type: "boolean",
Example: "true",
ReadOnly: true,
},
"package-type": {
Help: "If present, indicates that this item is a package instead of a folder or file. Packages are treated like files in some contexts and folders in others.",
Type: "string",
Example: "oneNote",
ReadOnly: true,
},
"shared-owner-id": {
Help: "ID of the owner of the shared item (if shared).",
Type: "string",
Example: "48d31887-5fad-4d73-a9f5-3c356e68a038",
ReadOnly: true,
},
"shared-by-id": {
Help: "ID of the user that shared the item (if shared).",
Type: "string",
Example: "48d31887-5fad-4d73-a9f5-3c356e68a038",
ReadOnly: true,
},
"shared-scope": {
Help: "If shared, indicates the scope of how the item is shared: anonymous, organization, or users.",
Type: "string",
Example: "users",
ReadOnly: true,
},
"shared-time": {
Help: "Time when the item was shared, with S accuracy (mS for OneDrive Personal).",
Type: "RFC 3339",
Example: "2006-01-02T15:04:05Z",
ReadOnly: true,
},
"permissions": {
Help: "Permissions in a JSON dump of OneDrive format. Enable with --onedrive-metadata-permissions. Properties: id, grantedTo, grantedToIdentities, invitation, inheritedFrom, link, roles, shareId",
Type: "JSON",
Example: "{}",
},
}
// rwChoices type for fs.Bits
type rwChoices struct{}
func (rwChoices) Choices() []fs.BitsChoicesInfo {
return []fs.BitsChoicesInfo{
{Bit: uint64(rwOff), Name: "off"},
{Bit: uint64(rwRead), Name: "read"},
{Bit: uint64(rwWrite), Name: "write"},
}
}
// rwChoice type alias
type rwChoice = fs.Bits[rwChoices]
const (
rwRead rwChoice = 1 << iota
rwWrite
rwOff rwChoice = 0
)
// Examples for the options
var rwExamples = fs.OptionExamples{{
Value: rwOff.String(),
Help: "Do not read or write the value",
}, {
Value: rwRead.String(),
Help: "Read the value only",
}, {
Value: rwWrite.String(),
Help: "Write the value only",
}, {
Value: (rwRead | rwWrite).String(),
Help: "Read and Write the value.",
}}
// Metadata describes metadata properties shared by both Objects and Directories
type Metadata struct {
fs *Fs // what this object/dir is part of
remote string // remote, for convenience when obj/dir not in scope
mimeType string // Content-Type of object from server (may not be as uploaded)
description string // Provides a user-visible description of the item. Read-write. Only on OneDrive Personal
mtime time.Time // Time of last modification with S accuracy.
btime time.Time // Time of file birth (creation) with S accuracy.
utime time.Time // Time of upload with S accuracy.
createdBy api.IdentitySet // user that created the item
lastModifiedBy api.IdentitySet // user that last modified the item
malwareDetected bool // Whether OneDrive has detected that the item contains malware.
packageType string // If present, indicates that this item is a package instead of a folder or file.
shared *api.SharedType // information about the shared state of the item, if shared
normalizedID string // the normalized ID of the object or dir
permissions []*api.PermissionsType // The current set of permissions for the item. Note that to save API calls, this is not guaranteed to be cached on the object. Use m.Get() to refresh.
queuedPermissions []*api.PermissionsType // The set of permissions queued to be updated.
permsAddOnly bool // Whether to disable "update" and "remove" (for example, during server-side copy when the dst will have new IDs)
}
// Get retrieves the cached metadata and converts it to fs.Metadata.
// This is most typically used when OneDrive is the source (as opposed to the dest).
// If m.fs.opt.MetadataPermissions includes "read" then this will also include permissions, which requires an API call.
// Get does not use an API call otherwise.
func (m *Metadata) Get(ctx context.Context) (metadata fs.Metadata, err error) {
metadata = make(fs.Metadata, 17)
metadata["content-type"] = m.mimeType
metadata["mtime"] = m.mtime.Format(timeFormatOut)
metadata["btime"] = m.btime.Format(timeFormatOut)
metadata["utime"] = m.utime.Format(timeFormatOut)
metadata["created-by-display-name"] = m.createdBy.User.DisplayName
metadata["created-by-id"] = m.createdBy.User.ID
if m.description != "" {
metadata["description"] = m.description
}
metadata["id"] = m.normalizedID
metadata["last-modified-by-display-name"] = m.lastModifiedBy.User.DisplayName
metadata["last-modified-by-id"] = m.lastModifiedBy.User.ID
metadata["malware-detected"] = fmt.Sprint(m.malwareDetected)
if m.packageType != "" {
metadata["package-type"] = m.packageType
}
if m.shared != nil {
metadata["shared-owner-id"] = m.shared.Owner.User.ID
metadata["shared-by-id"] = m.shared.SharedBy.User.ID
metadata["shared-scope"] = m.shared.Scope
metadata["shared-time"] = time.Time(m.shared.SharedDateTime).Format(timeFormatOut)
}
if m.fs.opt.MetadataPermissions.IsSet(rwRead) {
p, _, err := m.fs.getPermissions(ctx, m.normalizedID)
if err != nil {
return nil, fmt.Errorf("failed to get permissions: %w", err)
}
m.permissions = p
if len(p) > 0 {
fs.PrettyPrint(m.permissions, "perms", fs.LogLevelDebug)
buf, err := json.Marshal(m.permissions)
if err != nil {
return nil, fmt.Errorf("failed to marshal permissions: %w", err)
}
metadata["permissions"] = string(buf)
}
}
return metadata, nil
}
// Set takes fs.Metadata and parses/converts it to cached Metadata.
// This is most typically used when OneDrive is the destination (as opposed to the source).
// It does not actually update the remote (use Write for that.)
// It sets only the writeable metadata properties (i.e. read-only properties are skipped.)
// Permissions are included if m.fs.opt.MetadataPermissions includes "write".
// It returns errors if writeable properties can't be parsed.
// It does not return errors for unsupported properties that may be passed in.
// It returns the number of writeable properties set (if it is 0, we can skip the Write API call.)
func (m *Metadata) Set(ctx context.Context, metadata fs.Metadata) (numSet int, err error) {
numSet = 0
for k, v := range metadata {
k, v := k, v
switch k {
case "mtime":
t, err := time.Parse(timeFormatIn, v)
if err != nil {
return numSet, fmt.Errorf("failed to parse metadata %q = %q: %w", k, v, err)
}
m.mtime = t
numSet++
case "btime":
t, err := time.Parse(timeFormatIn, v)
if err != nil {
return numSet, fmt.Errorf("failed to parse metadata %q = %q: %w", k, v, err)
}
m.btime = t
numSet++
case "description":
if m.fs.driveType != driveTypePersonal {
fs.Debugf(m.remote, "metadata description is only supported for OneDrive Personal -- skipping: %s", v)
continue
}
m.description = v
numSet++
case "permissions":
if !m.fs.opt.MetadataPermissions.IsSet(rwWrite) {
continue
}
var perms []*api.PermissionsType
err := json.Unmarshal([]byte(v), &perms)
if err != nil {
return numSet, fmt.Errorf("failed to unmarshal permissions: %w", err)
}
m.queuedPermissions = perms
numSet++
default:
fs.Debugf(m.remote, "skipping unsupported metadata item: %s: %s", k, v)
}
}
if numSet == 0 {
fs.Infof(m.remote, "no writeable metadata found: %v", metadata)
}
return numSet, nil
}
// toAPIMetadata converts object/dir Metadata to api.Metadata for API calls.
// If btime is missing but mtime is present, mtime is also used as the btime, as otherwise it would get overwritten.
func (m *Metadata) toAPIMetadata() api.Metadata {
update := api.Metadata{
FileSystemInfo: &api.FileSystemInfoFacet{},
}
if m.description != "" && m.fs.driveType == driveTypePersonal {
update.Description = m.description
}
if !m.mtime.IsZero() {
update.FileSystemInfo.LastModifiedDateTime = api.Timestamp(m.mtime)
}
if !m.btime.IsZero() {
update.FileSystemInfo.CreatedDateTime = api.Timestamp(m.btime)
}
if m.btime.IsZero() && !m.mtime.IsZero() { // use mtime as btime if missing
m.btime = m.mtime
update.FileSystemInfo.CreatedDateTime = api.Timestamp(m.btime)
}
return update
}
// Write takes the cached Metadata and sets it on the remote, using API calls.
// If m.fs.opt.MetadataPermissions includes "write" and updatePermissions == true, permissions are also set.
// Calling Write without any writeable metadata will result in an error.
func (m *Metadata) Write(ctx context.Context, updatePermissions bool) (*api.Item, error) {
update := m.toAPIMetadata()
if update.IsEmpty() {
return nil, fmt.Errorf("%v: no writeable metadata found: %v", m.remote, m)
}
opts := m.fs.newOptsCallWithPath(ctx, m.remote, "PATCH", "")
var info *api.Item
err := m.fs.pacer.Call(func() (bool, error) {
resp, err := m.fs.srv.CallJSON(ctx, &opts, &update, &info)
return shouldRetry(ctx, resp, err)
})
if err != nil {
fs.Debugf(m.remote, "errored metadata: %v", m)
return nil, fmt.Errorf("%v: error updating metadata: %v", m.remote, err)
}
if m.fs.opt.MetadataPermissions.IsSet(rwWrite) && updatePermissions {
m.normalizedID = info.GetID()
err = m.WritePermissions(ctx)
if err != nil {
fs.Errorf(m.remote, "error writing permissions: %v", err)
return info, err
}
}
// update the struct since we have fresh info
m.fs.setSystemMetadata(info, m, m.remote, m.mimeType)
return info, err
}
// RefreshPermissions fetches the current permissions from the remote and caches them as Metadata
func (m *Metadata) RefreshPermissions(ctx context.Context) (err error) {
if m.normalizedID == "" {
return errors.New("internal error: normalizedID is missing")
}
p, _, err := m.fs.getPermissions(ctx, m.normalizedID)
if err != nil {
return fmt.Errorf("failed to refresh permissions: %w", err)
}
m.permissions = p
return nil
}
// WritePermissions sets the permissions (and no other metadata) on the remote.
// m.permissions (the existing perms) and m.queuedPermissions (the new perms to be set) must be set correctly before calling this.
// m.permissions == nil will not error, as it is valid to add permissions when there were previously none.
// If successful, m.permissions will be set with the new current permissions and m.queuedPermissions will be nil.
func (m *Metadata) WritePermissions(ctx context.Context) (err error) {
if !m.fs.opt.MetadataPermissions.IsSet(rwWrite) {
return errors.New("can't write permissions without --onedrive-metadata-permissions write")
}
if m.normalizedID == "" {
return errors.New("internal error: normalizedID is missing")
}
// compare current to queued and sort into add/update/remove queues
add, update, remove := m.sortPermissions()
fs.Debugf(m.remote, "metadata permissions: to add: %d to update: %d to remove: %d", len(add), len(update), len(remove))
_, err = m.processPermissions(ctx, add, update, remove)
if err != nil {
return fmt.Errorf("failed to process permissions: %w", err)
}
err = m.RefreshPermissions(ctx)
fs.Debugf(m.remote, "updated permissions (now has %d permissions)", len(m.permissions))
if err != nil {
return fmt.Errorf("failed to get permissions: %w", err)
}
m.queuedPermissions = nil
return nil
}
// sortPermissions sorts the permissions (to be written) into add, update, and remove queues
func (m *Metadata) sortPermissions() (add, update, remove []*api.PermissionsType) {
new, old := m.queuedPermissions, m.permissions
if len(old) == 0 || m.permsAddOnly {
return new, nil, nil // they must all be "add"
}
for _, n := range new {
if n == nil {
continue
}
if n.ID != "" {
// sanity check: ensure there's a matching "old" id with a non-matching role
if !slices.ContainsFunc(old, func(o *api.PermissionsType) bool {
return o.ID == n.ID && slices.Compare(o.Roles, n.Roles) != 0 && len(o.Roles) > 0 && len(n.Roles) > 0
}) {
fs.Debugf(m.remote, "skipping update for invalid roles: %v (perm ID: %v)", n.Roles, n.ID)
continue
}
if m.fs.driveType != driveTypePersonal && n.Link != nil && n.Link.WebURL != "" {
// special case to work around API limitation -- can't update a sharing link perm so need to remove + add instead
// https://learn.microsoft.com/en-us/answers/questions/986279/why-is-update-permission-graph-api-for-files-not-w
// https://github.com/microsoftgraph/msgraph-sdk-dotnet/issues/1135
fs.Debugf(m.remote, "sortPermissions: can't update due to API limitation, will remove + add instead: %v", n.Roles)
remove = append(remove, n)
add = append(add, n)
continue
}
fs.Debugf(m.remote, "sortPermissions: will update role to %v", n.Roles)
update = append(update, n)
} else {
fs.Debugf(m.remote, "sortPermissions: will add permission: %v %v", n, n.Roles)
add = append(add, n)
}
}
for _, o := range old {
newHasOld := slices.ContainsFunc(new, func(n *api.PermissionsType) bool {
if n == nil || n.ID == "" {
return false // can't remove perms without an ID
}
return n.ID == o.ID
})
if !newHasOld && o.ID != "" && !slices.Contains(add, o) && !slices.Contains(update, o) {
fs.Debugf(m.remote, "sortPermissions: will remove permission: %v %v (perm ID: %v)", o, o.Roles, o.ID)
remove = append(remove, o)
}
}
return add, update, remove
}
// processPermissions executes the add, update, and remove queues for writing permissions
func (m *Metadata) processPermissions(ctx context.Context, add, update, remove []*api.PermissionsType) (newPermissions []*api.PermissionsType, err error) {
for _, p := range remove { // remove (need to do these first because of remove + add workaround)
_, err := m.removePermission(ctx, p)
if err != nil {
return newPermissions, err
}
}
for _, p := range add { // add
newPs, _, err := m.addPermission(ctx, p)
if err != nil {
return newPermissions, err
}
newPermissions = append(newPermissions, newPs...)
}
for _, p := range update { // update
newP, _, err := m.updatePermission(ctx, p)
if err != nil {
return newPermissions, err
}
newPermissions = append(newPermissions, newP)
}
return newPermissions, err
}
// fillRecipients looks for recipients to add from the permission passed in.
// It looks for an email address in identity.User.ID and DisplayName, otherwise it uses the identity.User.ID as r.ObjectID.
// It considers both "GrantedTo" and "GrantedToIdentities".
func fillRecipients(p *api.PermissionsType) (recipients []api.DriveRecipient) {
if p == nil {
return recipients
}
ids := make(map[string]struct{}, len(p.GrantedToIdentities)+1)
isUnique := func(s string) bool {
_, ok := ids[s]
return !ok && s != ""
}
addRecipient := func(identity *api.IdentitySet) {
r := api.DriveRecipient{}
id := ""
if strings.ContainsRune(identity.User.ID, '@') {
id = identity.User.ID
r.Email = id
} else if strings.ContainsRune(identity.User.DisplayName, '@') {
id = identity.User.DisplayName
r.Email = id
} else {
id = identity.User.ID
r.ObjectID = id
}
if !isUnique(id) {
return
}
ids[id] = struct{}{}
recipients = append(recipients, r)
}
for _, identity := range p.GrantedToIdentities {
addRecipient(identity)
}
if p.GrantedTo != nil && p.GrantedTo.User != (api.Identity{}) {
addRecipient(p.GrantedTo)
}
return recipients
}
// addPermission adds new permissions to an object or dir.
// if p.Link.Scope == "anonymous" then it will also create a Public Link.
func (m *Metadata) addPermission(ctx context.Context, p *api.PermissionsType) (newPs []*api.PermissionsType, resp *http.Response, err error) {
opts := m.fs.newOptsCall(m.normalizedID, "POST", "/invite")
req := &api.AddPermissionsRequest{
Recipients: fillRecipients(p),
RequireSignIn: m.fs.driveType != driveTypePersonal, // personal and business have conflicting requirements
Roles: p.Roles,
}
if m.fs.driveType != driveTypePersonal {
req.RetainInheritedPermissions = false // not supported for personal
}
if p.Link != nil && p.Link.Scope == api.AnonymousScope {
link, err := m.fs.PublicLink(ctx, m.remote, fs.DurationOff, false)
if err != nil {
return nil, nil, err
}
p.Link.WebURL = link
newPs = append(newPs, p)
if len(req.Recipients) == 0 {
return newPs, nil, nil
}
}
if len(req.Recipients) == 0 {
fs.Debugf(m.remote, "skipping add permission -- at least one valid recipient is required")
return nil, nil, nil
}
if len(req.Roles) == 0 {
return nil, nil, errors.New("at least one role is required to add a permission (choices: read, write, owner, member)")
}
if slices.Contains(req.Roles, api.OwnerRole) {
fs.Debugf(m.remote, "skipping add permission -- can't invite a user with 'owner' role")
return nil, nil, nil
}
newP := &api.PermissionsResponse{}
err = m.fs.pacer.Call(func() (bool, error) {
resp, err = m.fs.srv.CallJSON(ctx, &opts, &req, &newP)
return shouldRetry(ctx, resp, err)
})
return newP.Value, resp, err
}
// updatePermission updates an existing permission on an object or dir.
// This requires the permission ID and a role to update (which will error if it is the same as the existing role.)
// Role is the only property that can be updated.
func (m *Metadata) updatePermission(ctx context.Context, p *api.PermissionsType) (newP *api.PermissionsType, resp *http.Response, err error) {
opts := m.fs.newOptsCall(m.normalizedID, "PATCH", "/permissions/"+p.ID)
req := api.UpdatePermissionsRequest{Roles: p.Roles} // roles is the only property that can be updated
if len(req.Roles) == 0 {
return nil, nil, errors.New("at least one role is required to update a permission (choices: read, write, owner, member)")
}
newP = &api.PermissionsType{}
err = m.fs.pacer.Call(func() (bool, error) {
resp, err = m.fs.srv.CallJSON(ctx, &opts, &req, &newP)
return shouldRetry(ctx, resp, err)
})
return newP, resp, err
}
// removePermission removes an existing permission on an object or dir.
// This requires the permission ID.
func (m *Metadata) removePermission(ctx context.Context, p *api.PermissionsType) (resp *http.Response, err error) {
opts := m.fs.newOptsCall(m.normalizedID, "DELETE", "/permissions/"+p.ID)
opts.NoResponse = true
err = m.fs.pacer.Call(func() (bool, error) {
resp, err = m.fs.srv.CallJSON(ctx, &opts, nil, nil)
return shouldRetry(ctx, resp, err)
})
return resp, err
}
// getPermissions gets the current permissions for an object or dir, from the API.
func (f *Fs) getPermissions(ctx context.Context, normalizedID string) (p []*api.PermissionsType, resp *http.Response, err error) {
opts := f.newOptsCall(normalizedID, "GET", "/permissions")
permResp := &api.PermissionsResponse{}
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(ctx, &opts, nil, &permResp)
return shouldRetry(ctx, resp, err)
})
return permResp.Value, resp, err
}
func (f *Fs) newMetadata(remote string) *Metadata {
return &Metadata{fs: f, remote: remote}
}
// returns true if metadata includes a "permissions" key and f.opt.MetadataPermissions includes "write".
func (f *Fs) needsUpdatePermissions(metadata fs.Metadata) bool {
_, ok := metadata["permissions"]
return ok && f.opt.MetadataPermissions.IsSet(rwWrite)
}
// returns a non-zero btime if we have one
// otherwise falls back to mtime
func (o *Object) tryGetBtime(modTime time.Time) time.Time {
if o.meta != nil && !o.meta.btime.IsZero() {
return o.meta.btime
}
return modTime
}
// adds metadata (except permissions) if --metadata is in use
func (o *Object) fetchMetadataForCreate(ctx context.Context, src fs.ObjectInfo, options []fs.OpenOption, modTime time.Time) (createRequest api.CreateUploadRequest, err error) {
createRequest = api.CreateUploadRequest{ // we set mtime no matter what
Item: api.Metadata{
FileSystemInfo: &api.FileSystemInfoFacet{
CreatedDateTime: api.Timestamp(o.tryGetBtime(modTime)),
LastModifiedDateTime: api.Timestamp(modTime),
},
},
}
meta, err := fs.GetMetadataOptions(ctx, o.fs, src, options)
if err != nil {
return createRequest, fmt.Errorf("failed to read metadata from source object: %w", err)
}
if meta == nil {
return createRequest, nil // no metadata or --metadata not in use, so just return mtime
}
if o.meta == nil {
o.meta = o.fs.newMetadata(o.Remote())
}
o.meta.mtime = modTime
numSet, err := o.meta.Set(ctx, meta)
if err != nil {
return createRequest, err
}
if numSet == 0 {
return createRequest, nil
}
createRequest.Item = o.meta.toAPIMetadata()
return createRequest, nil
}
// Fetch metadata and update updateInfo if --metadata is in use
// modtime will still be set when there is no metadata to set
func (f *Fs) fetchAndUpdateMetadata(ctx context.Context, src fs.ObjectInfo, options []fs.OpenOption, updateInfo *Object) (info *api.Item, err error) {
meta, err := fs.GetMetadataOptions(ctx, f, src, options)
if err != nil {
return nil, fmt.Errorf("failed to read metadata from source object: %w", err)
}
if meta == nil {
return updateInfo.setModTime(ctx, src.ModTime(ctx)) // no metadata or --metadata not in use, so just set modtime
}
if updateInfo.meta == nil {
updateInfo.meta = f.newMetadata(updateInfo.Remote())
}
newInfo, err := updateInfo.updateMetadata(ctx, meta)
if newInfo == nil {
return info, err
}
return newInfo, err
}
// Fetch and update permissions if --metadata is in use
// This is similar to fetchAndUpdateMetadata, except it does NOT set modtime or other metadata if there are no permissions to set.
// This is intended for cases where metadata may already have been set during upload and an extra step is needed only for permissions.
func (f *Fs) fetchAndUpdatePermissions(ctx context.Context, src fs.ObjectInfo, options []fs.OpenOption, updateInfo *Object) (info *api.Item, err error) {
meta, err := fs.GetMetadataOptions(ctx, f, src, options)
if err != nil {
return nil, fmt.Errorf("failed to read metadata from source object: %w", err)
}
if meta == nil || !f.needsUpdatePermissions(meta) {
return nil, nil // no metadata, --metadata not in use, or wrong flags
}
if updateInfo.meta == nil {
updateInfo.meta = f.newMetadata(updateInfo.Remote())
}
newInfo, err := updateInfo.updateMetadata(ctx, meta)
if newInfo == nil {
return info, err
}
return newInfo, err
}
// updateMetadata calls Get, Set, and Write
func (o *Object) updateMetadata(ctx context.Context, meta fs.Metadata) (info *api.Item, err error) {
_, err = o.meta.Get(ctx) // refresh permissions
if err != nil {
return nil, err
}
numSet, err := o.meta.Set(ctx, meta)
if err != nil {
return nil, err
}
if numSet == 0 {
return nil, nil
}
info, err = o.meta.Write(ctx, o.fs.needsUpdatePermissions(meta))
if err != nil {
return info, err
}
err = o.setMetaData(info)
if err != nil {
return info, err
}
// Remove versions if required
if o.fs.opt.NoVersions {
err := o.deleteVersions(ctx)
if err != nil {
return info, fmt.Errorf("%v: Failed to remove versions: %v", o, err)
}
}
return info, nil
}
// MkdirMetadata makes the directory passed in as dir.
//
// It shouldn't return an error if it already exists.
//
// If the metadata is not nil it is set.
//
// It returns the directory that was created.
func (f *Fs) MkdirMetadata(ctx context.Context, dir string, metadata fs.Metadata) (fs.Directory, error) {
var info *api.Item
var meta *Metadata
dirID, err := f.dirCache.FindDir(ctx, dir, false)
if err == fs.ErrorDirNotFound {
// Directory does not exist so create it
var leaf, parentID string
leaf, parentID, err = f.dirCache.FindPath(ctx, dir, true)
if err != nil {
return nil, err
}
info, meta, err = f.createDir(ctx, parentID, dir, leaf, metadata)
if err != nil {
return nil, err
}
if f.driveType != driveTypePersonal {
// for some reason, OneDrive Business needs this extra step to set modtime, while Personal does not. Seems like a bug...
fs.Debugf(dir, "setting time %v", meta.mtime)
info, err = meta.Write(ctx, false)
}
} else if err == nil {
// Directory exists and needs updating
info, meta, err = f.updateDir(ctx, dirID, dir, metadata)
}
if err != nil {
return nil, err
}
// Convert the info into a directory entry
parent, _ := dircache.SplitPath(dir)
entry, err := f.itemToDirEntry(ctx, parent, info)
if err != nil {
return nil, err
}
directory, ok := entry.(*Directory)
if !ok {
return nil, fmt.Errorf("internal error: expecting %T to be a *Directory", entry)
}
directory.meta = meta
f.setSystemMetadata(info, directory.meta, entry.Remote(), dirMimeType)
dirEntry, ok := entry.(fs.Directory)
if !ok {
return nil, fmt.Errorf("internal error: expecting %T to be an fs.Directory", entry)
}
return dirEntry, nil
}
// createDir makes a directory with pathID as parent and name leaf with optional metadata
func (f *Fs) createDir(ctx context.Context, pathID, dirWithLeaf, leaf string, metadata fs.Metadata) (info *api.Item, meta *Metadata, err error) {
// fs.Debugf(f, "CreateDir(%q, %q)\n", dirID, leaf)
var resp *http.Response
opts := f.newOptsCall(pathID, "POST", "/children")
mkdir := api.CreateItemWithMetadataRequest{
CreateItemRequest: api.CreateItemRequest{
Name: f.opt.Enc.FromStandardName(leaf),
ConflictBehavior: "fail",
},
}
m := f.newMetadata(dirWithLeaf)
m.mimeType = dirMimeType
numSet := 0
if len(metadata) > 0 {
numSet, err = m.Set(ctx, metadata)
if err != nil {
return nil, m, err
}
if numSet > 0 {
mkdir.Metadata = m.toAPIMetadata()
}
}
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(ctx, &opts, &mkdir, &info)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return nil, m, err
}
if f.needsUpdatePermissions(metadata) && numSet > 0 { // permissions must be done as a separate step
m.normalizedID = info.GetID()
err = m.RefreshPermissions(ctx)
if err != nil {
return info, m, err
}
err = m.WritePermissions(ctx)
if err != nil {
fs.Errorf(m.remote, "error writing permissions: %v", err)
return info, m, err
}
}
return info, m, nil
}
// updateDir updates an existing a directory with the metadata passed in
func (f *Fs) updateDir(ctx context.Context, dirID, remote string, metadata fs.Metadata) (info *api.Item, meta *Metadata, err error) {
d := f.newDir(dirID, remote)
_, err = d.meta.Set(ctx, metadata)
if err != nil {
return nil, nil, err
}
info, err = d.meta.Write(ctx, f.needsUpdatePermissions(metadata))
return info, d.meta, err
}
func (f *Fs) newDir(dirID, remote string) (d *Directory) {
d = &Directory{
fs: f,
remote: remote,
size: -1,
items: -1,
id: dirID,
meta: f.newMetadata(remote),
}
d.meta.normalizedID = dirID
return d
}
// Metadata returns metadata for a DirEntry
//
// It should return nil if there is no Metadata
func (o *Object) Metadata(ctx context.Context) (metadata fs.Metadata, err error) {
err = o.readMetaData(ctx)
if err != nil {
fs.Logf(o, "Failed to read metadata: %v", err)
return nil, err
}
return o.meta.Get(ctx)
}
// DirSetModTime sets the directory modtime for dir
func (f *Fs) DirSetModTime(ctx context.Context, dir string, modTime time.Time) error {
dirID, err := f.dirCache.FindDir(ctx, dir, false)
if err != nil {
return err
}
d := f.newDir(dirID, dir)
return d.SetModTime(ctx, modTime)
}
// SetModTime sets the metadata on the DirEntry to set the modification date
//
// If there is any other metadata it does not overwrite it.
func (d *Directory) SetModTime(ctx context.Context, t time.Time) error {
btime := t
if d.meta != nil && !d.meta.btime.IsZero() {
btime = d.meta.btime // if we already have a non-zero btime, preserve it
}
d.meta = d.fs.newMetadata(d.remote) // set only the mtime and btime
d.meta.mtime = t
d.meta.btime = btime
_, err := d.meta.Write(ctx, false)
return err
}
// Metadata returns metadata for a DirEntry
//
// It should return nil if there is no Metadata
func (d *Directory) Metadata(ctx context.Context) (metadata fs.Metadata, err error) {
return d.meta.Get(ctx)
}
// SetMetadata sets metadata for a Directory
//
// It should return fs.ErrorNotImplemented if it can't set metadata
func (d *Directory) SetMetadata(ctx context.Context, metadata fs.Metadata) error {
_, meta, err := d.fs.updateDir(ctx, d.id, d.remote, metadata)
d.meta = meta
return err
}
// Fs returns read only access to the Fs that this object is part of
func (d *Directory) Fs() fs.Info {
return d.fs
}
// String returns the name
func (d *Directory) String() string {
return d.remote
}
// Remote returns the remote path
func (d *Directory) Remote() string {
return d.remote
}
// ModTime returns the modification date of the file
//
// If one isn't available it returns the configured --default-dir-time
func (d *Directory) ModTime(ctx context.Context) time.Time {
if !d.meta.mtime.IsZero() {
return d.meta.mtime
}
ci := fs.GetConfig(ctx)
return time.Time(ci.DefaultTime)
}
// Size returns the size of the file
func (d *Directory) Size() int64 {
return d.size
}
// Items returns the count of items in this directory or this
// directory and subdirectories if known, -1 for unknown
func (d *Directory) Items() int64 {
return d.items
}
// ID gets the optional ID
func (d *Directory) ID() string {
return d.id
}
// MimeType returns the content type of the Object if
// known, or "" if not
func (d *Directory) MimeType(ctx context.Context) string {
return dirMimeType
}

147
backend/onedrive/metadata.md Normal file
View File

@ -0,0 +1,147 @@
OneDrive supports System Metadata (not User Metadata, as of this writing) for
both files and directories. Much of the metadata is read-only, and there are some
differences between OneDrive Personal and Business (see table below for
details).
Permissions are also supported, if `--onedrive-metadata-permissions` is set. The
accepted values for `--onedrive-metadata-permissions` are `read`, `write`,
`read,write`, and `off` (the default). `write` supports adding new permissions,
updating the "role" of existing permissions, and removing permissions. Updating
and removing require the Permission ID to be known, so it is recommended to use
`read,write` instead of `write` if you wish to update/remove permissions.
Permissions are read/written in JSON format using the same schema as the
[OneDrive API](https://learn.microsoft.com/en-us/onedrive/developer/rest-api/resources/permission?view=odsp-graph-online),
which differs slightly between OneDrive Personal and Business.
Example for OneDrive Personal:
```json
[
{
"id": "1234567890ABC!123",
"grantedTo": {
"user": {
"id": "ryan@contoso.com"
},
"application": {},
"device": {}
},
"invitation": {
"email": "ryan@contoso.com"
},
"link": {
"webUrl": "https://1drv.ms/t/s!1234567890ABC"
},
"roles": [
"read"
],
"shareId": "s!1234567890ABC"
}
]
```
Example for OneDrive Business:
```json
[
{
"id": "48d31887-5fad-4d73-a9f5-3c356e68a038",
"grantedToIdentities": [
{
"user": {
"displayName": "ryan@contoso.com"
},
"application": {},
"device": {}
}
],
"link": {
"type": "view",
"scope": "users",
"webUrl": "https://contoso.sharepoint.com/:w:/t/design/a577ghg9hgh737613bmbjf839026561fmzhsr85ng9f3hjck2t5s"
},
"roles": [
"read"
],
"shareId": "u!LKj1lkdlals90j1nlkascl"
},
{
"id": "5D33DD65C6932946",
"grantedTo": {
"user": {
"displayName": "John Doe",
"id": "efee1b77-fb3b-4f65-99d6-274c11914d12"
},
"application": {},
"device": {}
},
"roles": [
"owner"
],
"shareId": "FWxc1lasfdbEAGM5fI7B67aB5ZMPDMmQ11U"
}
]
```
To write permissions, pass in a "permissions" metadata key using this same
format. The [`--metadata-mapper`](https://rclone.org/docs/#metadata-mapper) tool can
be very helpful for this.
When adding permissions, an email address can be provided in the `User.ID` or
`DisplayName` properties of `grantedTo` or `grantedToIdentities`. Alternatively,
an ObjectID can be provided in `User.ID`. At least one valid recipient must be
provided in order to add a permission for a user. Creating a Public Link is also
supported, if `Link.Scope` is set to `"anonymous"`.
Example request to add a "read" permission:
```json
[
{
"id": "",
"grantedTo": {
"user": {},
"application": {},
"device": {}
},
"grantedToIdentities": [
{
"user": {
"id": "ryan@contoso.com"
},
"application": {},
"device": {}
}
],
"roles": [
"read"
]
}
]
```
Note that adding a permission can fail if a conflicting permission already
exists for the file/folder.
To update an existing permission, include both the Permission ID and the new
`roles` to be assigned. `roles` is the only property that can be changed.
To remove permissions, pass in a blob containing only the permissions you wish
to keep (which can be empty, to remove all.)
Note that both reading and writing permissions requires extra API calls, so if
you don't need to read or write permissions it is recommended to omit
`--onedrive-metadata-permissions`.
Metadata and permissions are supported for Folders (directories) as well as
Files. Note that setting the `mtime` or `btime` on a Folder requires one extra
API call on OneDrive Business only.
OneDrive does not currently support User Metadata. When writing metadata, only
writeable system properties will be written -- any read-only or unrecognized keys
passed in will be ignored.
TIP: to see the metadata and permissions for any file or folder, run:
```
rclone lsjson remote:path --stat -M --onedrive-metadata-permissions read
```

188
backend/onedrive/onedrive.go
View File

@ -4,6 +4,7 @@ package onedrive
import (
"context"
_ "embed"
"encoding/base64"
"encoding/hex"
"encoding/json"
@ -29,6 +30,7 @@ import (
"github.com/rclone/rclone/fs/fserrors"
"github.com/rclone/rclone/fs/fshttp"
"github.com/rclone/rclone/fs/hash"
"github.com/rclone/rclone/fs/log"
"github.com/rclone/rclone/fs/operations"
"github.com/rclone/rclone/fs/walk"
"github.com/rclone/rclone/lib/atexit"
@ -93,6 +95,9 @@ var (
// QuickXorHashType is the hash.Type for OneDrive
QuickXorHashType hash.Type
//go:embed metadata.md
metadataHelp string
)
// Register with Fs
@ -103,6 +108,10 @@ func init() {
Description: "Microsoft OneDrive",
NewFs: NewFs,
Config: Config,
MetadataInfo: &fs.MetadataInfo{
System: systemMetadataInfo,
Help: metadataHelp,
},
Options: append(oauthutil.SharedOptions, []fs.Option{{
Name: "region",
Help: "Choose national cloud region for OneDrive.",
@ -173,7 +182,8 @@ Choose or manually enter a custom space separated list with all scopes, that rcl
Value: "Files.Read Files.ReadWrite Files.Read.All Files.ReadWrite.All offline_access",
Help: "Read and write access to all resources, without the ability to browse SharePoint sites. \nSame as if disable_site_permission was set to true",
},
}}, {
},
}, {
Name: "disable_site_permission",
Help: `Disable the request for Sites.Read.All permission.
@ -356,6 +366,16 @@ It is recommended if you are mounting your onedrive at the root
(or near the root when using crypt) and using rclone |rc vfs/refresh|.
`, "|", "`"),
Advanced: true,
}, {
Name: "metadata_permissions",
Help: `Control whether permissions should be read or written in metadata.
Reading permissions metadata from files can be done quickly, but it
isn't always desirable to set the permissions from the metadata.
`,
Advanced: true,
Default: rwOff,
Examples: rwExamples,
}, {
Name: config.ConfigEncoding,
Help: config.ConfigEncodingHelp,
@ -639,7 +659,8 @@ Examples:
opts := rest.Opts{
Method: "GET",
RootURL: graphURL,
Path: "/drives/" + finalDriveID + "/root"}
Path: "/drives/" + finalDriveID + "/root",
}
var rootItem api.Item
_, err = srv.CallJSON(ctx, &opts, nil, &rootItem)
if err != nil {
@ -679,6 +700,7 @@ type Options struct {
AVOverride bool `config:"av_override"`
Delta bool `config:"delta"`
Enc encoder.MultiEncoder `config:"encoding"`
MetadataPermissions rwChoice `config:"metadata_permissions"`
}
// Fs represents a remote OneDrive
@ -711,6 +733,17 @@ type Object struct {
id string // ID of the object
hash string // Hash of the content, usually QuickXorHash but set as hash_type
mimeType string // Content-Type of object from server (may not be as uploaded)
meta *Metadata // metadata properties
}
// Directory describes a OneDrive directory
type Directory struct {
fs *Fs // what this object is part of
remote string // The remote path
size int64 // size of directory and contents or -1 if unknown
items int64 // number of objects or -1 for unknown
id string // dir ID
meta *Metadata // metadata properties
}
// ------------------------------------------------------------
@ -751,8 +784,10 @@ var retryErrorCodes = []int{
509, // Bandwidth Limit Exceeded
}
var gatewayTimeoutError sync.Once
var errAsyncJobAccessDenied = errors.New("async job failed - access denied")
var (
gatewayTimeoutError sync.Once
errAsyncJobAccessDenied = errors.New("async job failed - access denied")
)
// shouldRetry returns a boolean as to whether this resp and err
// deserve to be retried. It returns the err as a convenience
@ -971,8 +1006,17 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
f.features = (&fs.Features{
CaseInsensitive: true,
ReadMimeType: true,
WriteMimeType: false,
CanHaveEmptyDirectories: true,
ServerSideAcrossConfigs: opt.ServerSideAcrossConfigs,
ReadMetadata: true,
WriteMetadata: true,
UserMetadata: false,
ReadDirMetadata: true,
WriteDirMetadata: true,
WriteDirSetModTime: true,
UserDirMetadata: false,
DirModTimeUpdatesOnWrite: false,
}).Fill(ctx, f)
f.srv.SetErrorHandler(errorHandler)
@ -998,7 +1042,7 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
})
// Get rootID
var rootID = opt.RootFolderID
rootID := opt.RootFolderID
if rootID == "" {
rootInfo, _, err := f.readMetaDataForPath(ctx, "")
if err != nil {
@ -1065,6 +1109,7 @@ func (f *Fs) newObjectWithInfo(ctx context.Context, remote string, info *api.Ite
o := &Object{
fs: f,
remote: remote,
meta: f.newMetadata(remote),
}
var err error
if info != nil {
@ -1123,11 +1168,11 @@ func (f *Fs) CreateDir(ctx context.Context, dirID, leaf string) (newID string, e
return shouldRetry(ctx, resp, err)
})
if err != nil {
//fmt.Printf("...Error %v\n", err)
// fmt.Printf("...Error %v\n", err)
return "", err
}
//fmt.Printf("...Id %q\n", *info.Id)
// fmt.Printf("...Id %q\n", *info.Id)
return info.GetID(), nil
}
@ -1216,8 +1261,9 @@ func (f *Fs) itemToDirEntry(ctx context.Context, dir string, info *api.Item) (en
// cache the directory ID for later lookups
id := info.GetID()
f.dirCache.Put(remote, id)
d := fs.NewDir(remote, time.Time(info.GetLastModifiedDateTime())).SetID(id)
d.SetItems(folder.ChildCount)
d := f.newDir(id, remote)
d.items = folder.ChildCount
f.setSystemMetadata(info, d.meta, remote, dirMimeType)
entry = d
} else {
o, err := f.newObjectWithInfo(ctx, remote, info)
@ -1378,7 +1424,6 @@ func (f *Fs) ListR(ctx context.Context, dir string, callback fs.ListRCallback) (
}
return list.Flush()
}
// Shutdown shutdown the fs
@ -1479,6 +1524,9 @@ func (f *Fs) Rmdir(ctx context.Context, dir string) error {
// Precision return the precision of this Fs
func (f *Fs) Precision() time.Duration {
if f.driveType == driveTypePersonal {
return time.Millisecond
}
return time.Second
}
@ -1618,12 +1666,19 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
// Copy does NOT copy the modTime from the source and there seems to
// be no way to set date before
// This will create TWO versions on OneDrive
err = dstObj.SetModTime(ctx, srcObj.ModTime(ctx))
// Set modtime and adjust metadata if required
_, err = dstObj.Metadata(ctx) // make sure we get the correct new normalizedID
if err != nil {
return nil, err
}
return dstObj, nil
dstObj.meta.permsAddOnly = true // dst will have different IDs from src, so can't update/remove
info, err := f.fetchAndUpdateMetadata(ctx, src, fs.MetadataAsOpenOptions(ctx), dstObj)
if err != nil {
return nil, err
}
err = dstObj.setMetaData(info)
return dstObj, err
}
// Purge deletes all the files in the directory
@ -1678,12 +1733,12 @@ func (f *Fs) Move(ctx context.Context, src fs.Object, remote string) (fs.Object,
},
// We set the mod time too as it gets reset otherwise
FileSystemInfo: &api.FileSystemInfoFacet{
CreatedDateTime: api.Timestamp(srcObj.modTime),
CreatedDateTime: api.Timestamp(srcObj.tryGetBtime(srcObj.modTime)),
LastModifiedDateTime: api.Timestamp(srcObj.modTime),
},
}
var resp *http.Response
var info api.Item
var info *api.Item
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(ctx, &opts, &move, &info)
return shouldRetry(ctx, resp, err)
@ -1692,11 +1747,18 @@ func (f *Fs) Move(ctx context.Context, src fs.Object, remote string) (fs.Object,
return nil, err
}
err = dstObj.setMetaData(&info)
err = dstObj.setMetaData(info)
if err != nil {
return nil, err
}
return dstObj, nil
// Set modtime and adjust metadata if required
info, err = f.fetchAndUpdateMetadata(ctx, src, fs.MetadataAsOpenOptions(ctx), dstObj)
if err != nil {
return nil, err
}
err = dstObj.setMetaData(info)
return dstObj, err
}
// DirMove moves src, srcRemote to this remote at dstRemote
@ -2032,6 +2094,7 @@ func (o *Object) Size() int64 {
// setMetaData sets the metadata from info
func (o *Object) setMetaData(info *api.Item) (err error) {
if info.GetFolder() != nil {
log.Stack(o, "setMetaData called on dir instead of obj")
return fs.ErrorIsDir
}
o.hasMetaData = true
@ -2071,9 +2134,40 @@ func (o *Object) setMetaData(info *api.Item) (err error) {
o.modTime = time.Time(info.GetLastModifiedDateTime())
}
o.id = info.GetID()
if o.meta == nil {
o.meta = o.fs.newMetadata(o.Remote())
}
o.fs.setSystemMetadata(info, o.meta, o.remote, o.mimeType)
return nil
}
// sets system metadata shared by both objects and directories
func (f *Fs) setSystemMetadata(info *api.Item, meta *Metadata, remote string, mimeType string) {
meta.fs = f
meta.remote = remote
meta.mimeType = mimeType
if info == nil {
fs.Errorf("setSystemMetadata", "internal error: info is nil")
}
fileSystemInfo := info.GetFileSystemInfo()
if fileSystemInfo != nil {
meta.mtime = time.Time(fileSystemInfo.LastModifiedDateTime)
meta.btime = time.Time(fileSystemInfo.CreatedDateTime)
} else {
meta.mtime = time.Time(info.GetLastModifiedDateTime())
meta.btime = time.Time(info.GetCreatedDateTime())
}
meta.utime = time.Time(info.GetCreatedDateTime())
meta.description = info.Description
meta.packageType = info.GetPackageType()
meta.createdBy = info.GetCreatedBy()
meta.lastModifiedBy = info.GetLastModifiedBy()
meta.malwareDetected = info.MalwareDetected()
meta.shared = info.Shared
meta.normalizedID = info.GetID()
}
// readMetaData gets the metadata if it hasn't already been fetched
//
// it also sets the info
@ -2111,7 +2205,7 @@ func (o *Object) setModTime(ctx context.Context, modTime time.Time) (*api.Item,
opts := o.fs.newOptsCallWithPath(ctx, o.remote, "PATCH", "")
update := api.SetFileSystemInfo{
FileSystemInfo: api.FileSystemInfoFacet{
CreatedDateTime: api.Timestamp(modTime),
CreatedDateTime: api.Timestamp(o.tryGetBtime(modTime)),
LastModifiedDateTime: api.Timestamp(modTime),
},
}
@ -2175,18 +2269,19 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
}
if resp.StatusCode == http.StatusOK && resp.ContentLength > 0 && resp.Header.Get("Content-Range") == "" {
//Overwrite size with actual size since size readings from Onedrive is unreliable.
// Overwrite size with actual size since size readings from Onedrive is unreliable.
o.size = resp.ContentLength
}
return resp.Body, err
}
// createUploadSession creates an upload session for the object
func (o *Object) createUploadSession(ctx context.Context, modTime time.Time) (response *api.CreateUploadResponse, err error) {
func (o *Object) createUploadSession(ctx context.Context, src fs.ObjectInfo, modTime time.Time) (response *api.CreateUploadResponse, err error) {
opts := o.fs.newOptsCallWithPath(ctx, o.remote, "POST", "/createUploadSession")
createRequest := api.CreateUploadRequest{}
createRequest.Item.FileSystemInfo.CreatedDateTime = api.Timestamp(modTime)
createRequest.Item.FileSystemInfo.LastModifiedDateTime = api.Timestamp(modTime)
createRequest, err := o.fetchMetadataForCreate(ctx, src, opts.Options, modTime)
if err != nil {
return nil, err
}
var resp *http.Response
err = o.fs.pacer.Call(func() (bool, error) {
resp, err = o.fs.srv.CallJSON(ctx, &opts, &createRequest, &response)
@ -2237,7 +2332,7 @@ func (o *Object) uploadFragment(ctx context.Context, url string, start int64, to
// var response api.UploadFragmentResponse
var resp *http.Response
var body []byte
var skip = int64(0)
skip := int64(0)
err = o.fs.pacer.Call(func() (bool, error) {
toSend := chunkSize - skip
opts := rest.Opts{
@ -2304,14 +2399,17 @@ func (o *Object) cancelUploadSession(ctx context.Context, url string) (err error
}
// uploadMultipart uploads a file using multipart upload
func (o *Object) uploadMultipart(ctx context.Context, in io.Reader, size int64, modTime time.Time, options ...fs.OpenOption) (info *api.Item, err error) {
// if there is metadata, it will be set at the same time, except for permissions, which must be set after (if present and enabled).
func (o *Object) uploadMultipart(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (info *api.Item, err error) {
size := src.Size()
modTime := src.ModTime(ctx)
if size <= 0 {
return nil, errors.New("unknown-sized upload not supported")
}
// Create upload session
fs.Debugf(o, "Starting multipart upload")
session, err := o.createUploadSession(ctx, modTime)
session, err := o.createUploadSession(ctx, src, modTime)
if err != nil {
return nil, err
}
@ -2344,12 +2442,25 @@ func (o *Object) uploadMultipart(ctx context.Context, in io.Reader, size int64,
position += n
}
return info, nil
err = o.setMetaData(info)
if err != nil {
return info, err
}
if !o.fs.opt.MetadataPermissions.IsSet(rwWrite) {
return info, err
}
info, err = o.fs.fetchAndUpdatePermissions(ctx, src, options, o) // for permissions, which can't be set during original upload
if info == nil {
return nil, err
}
return info, o.setMetaData(info)
}
// Update the content of a remote file within 4 MiB size in one single request
// This function will set modtime after uploading, which will create a new version for the remote file
func (o *Object) uploadSinglepart(ctx context.Context, in io.Reader, size int64, modTime time.Time, options ...fs.OpenOption) (info *api.Item, err error) {
// (currently only used when size is exactly 0)
// This function will set modtime and metadata after uploading, which will create a new version for the remote file
func (o *Object) uploadSinglepart(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (info *api.Item, err error) {
size := src.Size()
if size < 0 || size > int64(fs.SizeSuffix(4*1024*1024)) {
return nil, errors.New("size passed into uploadSinglepart must be >= 0 and <= 4 MiB")
}
@ -2380,7 +2491,8 @@ func (o *Object) uploadSinglepart(ctx context.Context, in io.Reader, size int64,
return nil, err
}
// Set the mod time now and read metadata
return o.setModTime(ctx, modTime)
info, err = o.fs.fetchAndUpdateMetadata(ctx, src, options, o)
return info, o.setMetaData(info)
}
// Update the object with the contents of the io.Reader, modTime and size
@ -2395,17 +2507,17 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
defer o.fs.tokenRenewer.Stop()
size := src.Size()
modTime := src.ModTime(ctx)
var info *api.Item
if size > 0 {
info, err = o.uploadMultipart(ctx, in, size, modTime, options...)
info, err = o.uploadMultipart(ctx, in, src, options...)
} else if size == 0 {
info, err = o.uploadSinglepart(ctx, in, size, modTime, options...)
info, err = o.uploadSinglepart(ctx, in, src, options...)
} else {
return errors.New("unknown-sized upload not supported")
}
if err != nil {
fs.PrettyPrint(info, "info from Update error", fs.LogLevelDebug)
return err
}
@ -2416,8 +2528,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
fs.Errorf(o, "Failed to remove versions: %v", err)
}
}
return o.setMetaData(info)
return nil
}
// Remove an object
@ -2769,4 +2880,11 @@ var (
_ fs.Object = (*Object)(nil)
_ fs.MimeTyper = &Object{}
_ fs.IDer = &Object{}
_ fs.Metadataer = (*Object)(nil)
_ fs.Metadataer = (*Directory)(nil)
_ fs.SetModTimer = (*Directory)(nil)
_ fs.SetMetadataer = (*Directory)(nil)
_ fs.MimeTyper = &Directory{}
_ fs.DirSetModTimer = (*Fs)(nil)
_ fs.MkdirMetadataer = (*Fs)(nil)
)

464
backend/onedrive/onedrive_internal_test.go Normal file
View File

@ -0,0 +1,464 @@
package onedrive
import (
"context"
"encoding/json"
"fmt"
"testing"
"time"
_ "github.com/rclone/rclone/backend/local"
"github.com/rclone/rclone/backend/onedrive/api"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/operations"
"github.com/rclone/rclone/fstest"
"github.com/rclone/rclone/fstest/fstests"
"github.com/rclone/rclone/lib/random"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"golang.org/x/exp/slices" // replace with slices after go1.21 is the minimum version
)
// go test -timeout 30m -run ^TestIntegration/FsMkdir/FsPutFiles/Internal$ github.com/rclone/rclone/backend/onedrive -remote TestOneDrive:meta -v
// go test -timeout 30m -run ^TestIntegration/FsMkdir/FsPutFiles/Internal$ github.com/rclone/rclone/backend/onedrive -remote TestOneDriveBusiness:meta -v
// go run ./fstest/test_all -remotes TestOneDriveBusiness:meta,TestOneDrive:meta -verbose -maxtries 1
var (
t1 = fstest.Time("2023-08-26T23:13:06.499999999Z")
t2 = fstest.Time("2020-02-29T12:34:56.789Z")
t3 = time.Date(1994, time.December, 24, 9+12, 0, 0, 525600, time.FixedZone("Eastern Standard Time", -5))
ctx = context.Background()
content = "hello"
)
const (
testUserID = "ryan@contoso.com" // demo user from doc examples (can't share files with yourself)
// https://learn.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_invite?view=odsp-graph-online#http-request-1
)
// TestMain drives the tests
func TestMain(m *testing.M) {
fstest.TestMain(m)
}
// TestWritePermissions tests reading and writing permissions
func (f *Fs) TestWritePermissions(t *testing.T, r *fstest.Run) {
// setup
ctx, ci := fs.AddConfig(ctx)
ci.Metadata = true
_ = f.opt.MetadataPermissions.Set("read,write")
file1 := r.WriteFile(randomFilename(), content, t2)
// add a permission with "read" role
permissions := defaultPermissions()
permissions[0].Roles[0] = api.ReadRole
expectedMeta, actualMeta := f.putWithMeta(ctx, t, &file1, permissions)
f.compareMeta(t, expectedMeta, actualMeta, false)
expectedP, actualP := unmarshalPerms(t, expectedMeta["permissions"]), unmarshalPerms(t, actualMeta["permissions"])
found, num := false, 0
foundCount := 0
for i, p := range actualP {
for _, identity := range p.GrantedToIdentities {
if identity.User.DisplayName == testUserID {
// note: expected will always be element 0 here, but actual may be variable based on org settings
assert.Equal(t, expectedP[0].Roles, p.Roles)
found, num = true, i
foundCount++
}
}
if f.driveType == driveTypePersonal {
if p.GrantedTo != nil && p.GrantedTo.User != (api.Identity{}) && p.GrantedTo.User.ID == testUserID { // shows up in a different place on biz vs. personal
assert.Equal(t, expectedP[0].Roles, p.Roles)
found, num = true, i
foundCount++
}
}
}
assert.True(t, found, fmt.Sprintf("no permission found with expected role (want: \n\n%v \n\ngot: \n\n%v\n\n)", indent(t, expectedMeta["permissions"]), indent(t, actualMeta["permissions"])))
assert.Equal(t, 1, foundCount, "expected to find exactly 1 match")
// update it to "write"
permissions = actualP
permissions[num].Roles[0] = api.WriteRole
expectedMeta, actualMeta = f.putWithMeta(ctx, t, &file1, permissions)
f.compareMeta(t, expectedMeta, actualMeta, false)
if f.driveType != driveTypePersonal {
// zero out some things we expect to be different
expectedP, actualP = unmarshalPerms(t, expectedMeta["permissions"]), unmarshalPerms(t, actualMeta["permissions"])
normalize(expectedP)
normalize(actualP)
expectedMeta.Set("permissions", marshalPerms(t, expectedP))
actualMeta.Set("permissions", marshalPerms(t, actualP))
}
assert.JSONEq(t, expectedMeta["permissions"], actualMeta["permissions"])
// remove it
permissions[num] = nil
_, actualMeta = f.putWithMeta(ctx, t, &file1, permissions)
if f.driveType == driveTypePersonal {
perms, ok := actualMeta["permissions"]
assert.False(t, ok, fmt.Sprintf("permissions metadata key was unexpectedly found: %v", perms))
return
}
_, actualP = unmarshalPerms(t, expectedMeta["permissions"]), unmarshalPerms(t, actualMeta["permissions"])
found = false
var foundP *api.PermissionsType
for _, p := range actualP {
if p.GrantedTo == nil || p.GrantedTo.User == (api.Identity{}) || p.GrantedTo.User.ID != testUserID {
continue
}
found = true
foundP = p
}
assert.False(t, found, fmt.Sprintf("permission was found but expected to be removed: %v", foundP))
}
// TestUploadSinglePart tests reading/writing permissions using uploadSinglepart()
// This is only used when file size is exactly 0.
func (f *Fs) TestUploadSinglePart(t *testing.T, r *fstest.Run) {
content = ""
f.TestWritePermissions(t, r)
content = "hello"
}
// TestReadPermissions tests that no permissions are written when --onedrive-metadata-permissions has "read" but not "write"
func (f *Fs) TestReadPermissions(t *testing.T, r *fstest.Run) {
// setup
ctx, ci := fs.AddConfig(ctx)
ci.Metadata = true
file1 := r.WriteFile(randomFilename(), "hello", t2)
// try adding a permission without --onedrive-metadata-permissions -- should fail
// test that what we got before vs. after is the same
_ = f.opt.MetadataPermissions.Set("read")
_, expectedMeta := f.putWithMeta(ctx, t, &file1, []*api.PermissionsType{}) // return var intentionally switched here
permissions := defaultPermissions()
_, actualMeta := f.putWithMeta(ctx, t, &file1, permissions)
if f.driveType == driveTypePersonal {
perms, ok := actualMeta["permissions"]
assert.False(t, ok, fmt.Sprintf("permissions metadata key was unexpectedly found: %v", perms))
return
}
assert.JSONEq(t, expectedMeta["permissions"], actualMeta["permissions"])
}
// TestReadMetadata tests that all the read-only system properties are present and non-blank
func (f *Fs) TestReadMetadata(t *testing.T, r *fstest.Run) {
// setup
ctx, ci := fs.AddConfig(ctx)
ci.Metadata = true
file1 := r.WriteFile(randomFilename(), "hello", t2)
permissions := defaultPermissions()
_ = f.opt.MetadataPermissions.Set("read,write")
_, actualMeta := f.putWithMeta(ctx, t, &file1, permissions)
optionals := []string{"package-type", "shared-by-id", "shared-scope", "shared-time", "shared-owner-id"} // not always present
for k := range systemMetadataInfo {
if slices.Contains(optionals, k) {
continue
}
if k == "description" && f.driveType != driveTypePersonal {
continue // not supported
}
gotV, ok := actualMeta[k]
assert.True(t, ok, fmt.Sprintf("property is missing: %v", k))
assert.NotEmpty(t, gotV, fmt.Sprintf("property is blank: %v", k))
}
}
// TestDirectoryMetadata tests reading and writing modtime and other metadata and permissions for directories
func (f *Fs) TestDirectoryMetadata(t *testing.T, r *fstest.Run) {
// setup
ctx, ci := fs.AddConfig(ctx)
ci.Metadata = true
_ = f.opt.MetadataPermissions.Set("read,write")
permissions := defaultPermissions()
permissions[0].Roles[0] = api.ReadRole
expectedMeta := fs.Metadata{
"mtime": t1.Format(timeFormatOut),
"btime": t2.Format(timeFormatOut),
"content-type": dirMimeType,
"description": "that is so meta!",
}
b, err := json.MarshalIndent(permissions, "", "\t")
assert.NoError(t, err)
expectedMeta.Set("permissions", string(b))
compareDirMeta := func(expectedMeta, actualMeta fs.Metadata, ignoreID bool) {
f.compareMeta(t, expectedMeta, actualMeta, ignoreID)
// check that all required system properties are present
optionals := []string{"package-type", "shared-by-id", "shared-scope", "shared-time", "shared-owner-id"} // not always present
for k := range systemMetadataInfo {
if slices.Contains(optionals, k) {
continue
}
if k == "description" && f.driveType != driveTypePersonal {
continue // not supported
}
gotV, ok := actualMeta[k]
assert.True(t, ok, fmt.Sprintf("property is missing: %v", k))
assert.NotEmpty(t, gotV, fmt.Sprintf("property is blank: %v", k))
}
}
newDst, err := operations.MkdirMetadata(ctx, f, "subdir", expectedMeta)
assert.NoError(t, err)
require.NotNil(t, newDst)
assert.Equal(t, "subdir", newDst.Remote())
actualMeta, err := fs.GetMetadata(ctx, newDst)
assert.NoError(t, err)
assert.NotNil(t, actualMeta)
compareDirMeta(expectedMeta, actualMeta, false)
// modtime
assert.Equal(t, t1.Truncate(f.Precision()), newDst.ModTime(ctx))
// try changing it and re-check it
newDst, err = operations.SetDirModTime(ctx, f, newDst, "", t2)
assert.NoError(t, err)
assert.Equal(t, t2.Truncate(f.Precision()), newDst.ModTime(ctx))
// ensure that f.DirSetModTime also works
err = f.DirSetModTime(ctx, "subdir", t3)
assert.NoError(t, err)
entries, err := f.List(ctx, "")
assert.NoError(t, err)
entries.ForDir(func(dir fs.Directory) {
if dir.Remote() == "subdir" {
assert.True(t, t3.Truncate(f.Precision()).Equal(dir.ModTime(ctx)), fmt.Sprintf("got %v", dir.ModTime(ctx)))
}
})
// test updating metadata on existing dir
actualMeta, err = fs.GetMetadata(ctx, newDst) // get fresh info as we've been changing modtimes
assert.NoError(t, err)
expectedMeta = actualMeta
expectedMeta.Set("description", "metadata is fun!")
expectedMeta.Set("btime", t3.Format(timeFormatOut))
expectedMeta.Set("mtime", t1.Format(timeFormatOut))
expectedMeta.Set("content-type", dirMimeType)
perms := unmarshalPerms(t, expectedMeta["permissions"])
perms[0].Roles[0] = api.WriteRole
b, err = json.MarshalIndent(perms, "", "\t")
assert.NoError(t, err)
expectedMeta.Set("permissions", string(b))
newDst, err = operations.MkdirMetadata(ctx, f, "subdir", expectedMeta)
assert.NoError(t, err)
require.NotNil(t, newDst)
assert.Equal(t, "subdir", newDst.Remote())
actualMeta, err = fs.GetMetadata(ctx, newDst)
assert.NoError(t, err)
assert.NotNil(t, actualMeta)
compareDirMeta(expectedMeta, actualMeta, false)
// test copying metadata from one dir to another
copiedDir, err := operations.CopyDirMetadata(ctx, f, nil, "subdir2", newDst)
assert.NoError(t, err)
require.NotNil(t, copiedDir)
assert.Equal(t, "subdir2", copiedDir.Remote())
actualMeta, err = fs.GetMetadata(ctx, copiedDir)
assert.NoError(t, err)
assert.NotNil(t, actualMeta)
compareDirMeta(expectedMeta, actualMeta, true)
// test DirModTimeUpdatesOnWrite
expectedTime := copiedDir.ModTime(ctx)
assert.True(t, !expectedTime.IsZero())
r.WriteObject(ctx, copiedDir.Remote()+"/"+randomFilename(), "hi there", t3)
entries, err = f.List(ctx, "")
assert.NoError(t, err)
entries.ForDir(func(dir fs.Directory) {
if dir.Remote() == copiedDir.Remote() {
assert.True(t, expectedTime.Equal(dir.ModTime(ctx)), fmt.Sprintf("want %v got %v", expectedTime, dir.ModTime(ctx)))
}
})
}
// TestServerSideCopyMove tests server-side Copy and Move
func (f *Fs) TestServerSideCopyMove(t *testing.T, r *fstest.Run) {
// setup
ctx, ci := fs.AddConfig(ctx)
ci.Metadata = true
_ = f.opt.MetadataPermissions.Set("read,write")
file1 := r.WriteFile(randomFilename(), content, t2)
// add a permission with "read" role
permissions := defaultPermissions()
permissions[0].Roles[0] = api.ReadRole
expectedMeta, actualMeta := f.putWithMeta(ctx, t, &file1, permissions)
f.compareMeta(t, expectedMeta, actualMeta, false)
comparePerms := func(expectedMeta, actualMeta fs.Metadata) (newExpectedMeta, newActualMeta fs.Metadata) {
expectedP, actualP := unmarshalPerms(t, expectedMeta["permissions"]), unmarshalPerms(t, actualMeta["permissions"])
normalize(expectedP)
normalize(actualP)
expectedMeta.Set("permissions", marshalPerms(t, expectedP))
actualMeta.Set("permissions", marshalPerms(t, actualP))
assert.JSONEq(t, expectedMeta["permissions"], actualMeta["permissions"])
return expectedMeta, actualMeta
}
// Copy
obj1, err := f.NewObject(ctx, file1.Path)
assert.NoError(t, err)
originalMeta := actualMeta
obj2, err := f.Copy(ctx, obj1, randomFilename())
assert.NoError(t, err)
actualMeta, err = fs.GetMetadata(ctx, obj2)
assert.NoError(t, err)
expectedMeta, actualMeta = comparePerms(originalMeta, actualMeta)
f.compareMeta(t, expectedMeta, actualMeta, true)
// Move
obj3, err := f.Move(ctx, obj1, randomFilename())
assert.NoError(t, err)
actualMeta, err = fs.GetMetadata(ctx, obj3)
assert.NoError(t, err)
expectedMeta, actualMeta = comparePerms(originalMeta, actualMeta)
f.compareMeta(t, expectedMeta, actualMeta, true)
}
// helper function to put an object with metadata and permissions
func (f *Fs) putWithMeta(ctx context.Context, t *testing.T, file *fstest.Item, perms []*api.PermissionsType) (expectedMeta, actualMeta fs.Metadata) {
t.Helper()
expectedMeta = fs.Metadata{
"mtime": t1.Format(timeFormatOut),
"btime": t2.Format(timeFormatOut),
"description": "that is so meta!",
}
expectedMeta.Set("permissions", marshalPerms(t, perms))
obj := fstests.PutTestContentsMetadata(ctx, t, f, file, content, true, "plain/text", expectedMeta)
do, ok := obj.(fs.Metadataer)
require.True(t, ok)
actualMeta, err := do.Metadata(ctx)
require.NoError(t, err)
return expectedMeta, actualMeta
}
func randomFilename() string {
return "some file-" + random.String(8) + ".txt"
}
func (f *Fs) compareMeta(t *testing.T, expectedMeta, actualMeta fs.Metadata, ignoreID bool) {
t.Helper()
for k, v := range expectedMeta {
gotV, ok := actualMeta[k]
switch k {
case "shared-owner-id", "shared-time", "shared-by-id", "shared-scope":
continue
case "permissions":
continue
case "utime":
assert.True(t, ok, fmt.Sprintf("expected metadata key is missing: %v", k))
if f.driveType == driveTypePersonal {
compareTimeStrings(t, k, v, gotV, time.Minute) // read-only upload time, so slight difference expected -- use larger precision
continue
}
compareTimeStrings(t, k, expectedMeta["btime"], gotV, time.Minute) // another bizarre difference between personal and business...
continue
case "id":
if ignoreID {
continue // different id is expected when copying meta from one item to another
}
case "mtime", "btime":
assert.True(t, ok, fmt.Sprintf("expected metadata key is missing: %v", k))
compareTimeStrings(t, k, v, gotV, time.Second)
continue
case "description":
if f.driveType != driveTypePersonal {
continue // not supported
}
}
assert.True(t, ok, fmt.Sprintf("expected metadata key is missing: %v", k))
assert.Equal(t, v, gotV, actualMeta)
}
}
func compareTimeStrings(t *testing.T, remote, want, got string, precision time.Duration) {
wantT, err := time.Parse(timeFormatIn, want)
assert.NoError(t, err)
gotT, err := time.Parse(timeFormatIn, got)
assert.NoError(t, err)
fstest.AssertTimeEqualWithPrecision(t, remote, wantT, gotT, precision)
}
func marshalPerms(t *testing.T, p []*api.PermissionsType) string {
b, err := json.MarshalIndent(p, "", "\t")
assert.NoError(t, err)
return string(b)
}
func unmarshalPerms(t *testing.T, perms string) (p []*api.PermissionsType) {
t.Helper()
err := json.Unmarshal([]byte(perms), &p)
assert.NoError(t, err)
return p
}
func indent(t *testing.T, s string) string {
p := unmarshalPerms(t, s)
return marshalPerms(t, p)
}
func defaultPermissions() []*api.PermissionsType {
return []*api.PermissionsType{{
GrantedTo: &api.IdentitySet{User: api.Identity{}},
GrantedToIdentities: []*api.IdentitySet{{User: api.Identity{ID: testUserID}}},
Roles: []api.Role{api.WriteRole},
}}
}
// zeroes out some things we expect to be different when copying/moving between objects
func normalize(Ps []*api.PermissionsType) {
for _, ep := range Ps {
ep.ID = ""
ep.Link = nil
ep.ShareID = ""
}
}
func (f *Fs) resetTestDefaults(r *fstest.Run) {
ci := fs.GetConfig(ctx)
ci.Metadata = false
_ = f.opt.MetadataPermissions.Set("off")
r.Finalise()
}
// InternalTest dispatches all internal tests
func (f *Fs) InternalTest(t *testing.T) {
newTestF := func() (*Fs, *fstest.Run) {
r := fstest.NewRunIndividual(t)
testF, ok := r.Fremote.(*Fs)
if !ok {
t.FailNow()
}
return testF, r
}
testF, r := newTestF()
t.Run("TestWritePermissions", func(t *testing.T) { testF.TestWritePermissions(t, r) })
testF.resetTestDefaults(r)
testF, r = newTestF()
t.Run("TestUploadSinglePart", func(t *testing.T) { testF.TestUploadSinglePart(t, r) })
testF.resetTestDefaults(r)
testF, r = newTestF()
t.Run("TestReadPermissions", func(t *testing.T) { testF.TestReadPermissions(t, r) })
testF.resetTestDefaults(r)
testF, r = newTestF()
t.Run("TestReadMetadata", func(t *testing.T) { testF.TestReadMetadata(t, r) })
testF.resetTestDefaults(r)
testF, r = newTestF()
t.Run("TestDirectoryMetadata", func(t *testing.T) { testF.TestDirectoryMetadata(t, r) })
testF.resetTestDefaults(r)
testF, r = newTestF()
t.Run("TestServerSideCopyMove", func(t *testing.T) { testF.TestServerSideCopyMove(t, r) })
testF.resetTestDefaults(r)
}
var _ fstests.InternalTester = (*Fs)(nil)

209
docs/content/onedrive.md
View File

@ -657,6 +657,30 @@ Properties:
- Type: bool
- Default: false
#### --onedrive-metadata-permissions
Control whether permissions should be read or written in metadata.
Reading permissions metadata from files can be done quickly, but it
isn't always desirable to set the permissions from the metadata.
Properties:
- Config: metadata_permissions
- Env Var: RCLONE_ONEDRIVE_METADATA_PERMISSIONS
- Type: Bits
- Default: off
- Examples:
- "off"
- Do not read or write the value
- "read"
- Read the value only
- "write"
- Write the value only
- "read,write"
- Read and Write the value.
#### --onedrive-encoding
The encoding for the backend.
@ -670,6 +694,191 @@ Properties:
- Type: Encoding
- Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,LeftSpace,LeftTilde,RightSpace,RightPeriod,InvalidUtf8,Dot
#### --onedrive-description
Description of the remote
Properties:
- Config: description
- Env Var: RCLONE_ONEDRIVE_DESCRIPTION
- Type: string
- Required: false
### Metadata
OneDrive supports System Metadata (not User Metadata, as of this writing) for
both files and directories. Much of the metadata is read-only, and there are some
differences between OneDrive Personal and Business (see table below for
details).
Permissions are also supported, if `--onedrive-metadata-permissions` is set. The
accepted values for `--onedrive-metadata-permissions` are `read`, `write`,
`read,write`, and `off` (the default). `write` supports adding new permissions,
updating the "role" of existing permissions, and removing permissions. Updating
and removing require the Permission ID to be known, so it is recommended to use
`read,write` instead of `write` if you wish to update/remove permissions.
Permissions are read/written in JSON format using the same schema as the
[OneDrive API](https://learn.microsoft.com/en-us/onedrive/developer/rest-api/resources/permission?view=odsp-graph-online),
which differs slightly between OneDrive Personal and Business.
Example for OneDrive Personal:
```json
[
{
"id": "1234567890ABC!123",
"grantedTo": {
"user": {
"id": "ryan@contoso.com"
},
"application": {},
"device": {}
},
"invitation": {
"email": "ryan@contoso.com"
},
"link": {
"webUrl": "https://1drv.ms/t/s!1234567890ABC"
},
"roles": [
"read"
],
"shareId": "s!1234567890ABC"
}
]
```
Example for OneDrive Business:
```json
[
{
"id": "48d31887-5fad-4d73-a9f5-3c356e68a038",
"grantedToIdentities": [
{
"user": {
"displayName": "ryan@contoso.com"
},
"application": {},
"device": {}
}
],
"link": {
"type": "view",
"scope": "users",
"webUrl": "https://contoso.sharepoint.com/:w:/t/design/a577ghg9hgh737613bmbjf839026561fmzhsr85ng9f3hjck2t5s"
},
"roles": [
"read"
],
"shareId": "u!LKj1lkdlals90j1nlkascl"
},
{
"id": "5D33DD65C6932946",
"grantedTo": {
"user": {
"displayName": "John Doe",
"id": "efee1b77-fb3b-4f65-99d6-274c11914d12"
},
"application": {},
"device": {}
},
"roles": [
"owner"
],
"shareId": "FWxc1lasfdbEAGM5fI7B67aB5ZMPDMmQ11U"
}
]
```
To write permissions, pass in a "permissions" metadata key using this same
format. The [`--metadata-mapper`](https://rclone.org/docs/#metadata-mapper) tool can
be very helpful for this.
When adding permissions, an email address can be provided in the `User.ID` or
`DisplayName` properties of `grantedTo` or `grantedToIdentities`. Alternatively,
an ObjectID can be provided in `User.ID`. At least one valid recipient must be
provided in order to add a permission for a user. Creating a Public Link is also
supported, if `Link.Scope` is set to `"anonymous"`.
Example request to add a "read" permission:
```json
[
{
"id": "",
"grantedTo": {
"user": {},
"application": {},
"device": {}
},
"grantedToIdentities": [
{
"user": {
"id": "ryan@contoso.com"
},
"application": {},
"device": {}
}
],
"roles": [
"read"
]
}
]
```
Note that adding a permission can fail if a conflicting permission already
exists for the file/folder.
To update an existing permission, include both the Permission ID and the new
`roles` to be assigned. `roles` is the only property that can be changed.
To remove permissions, pass in a blob containing only the permissions you wish
to keep (which can be empty, to remove all.)
Note that both reading and writing permissions requires extra API calls, so if
you don't need to read or write permissions it is recommended to omit
`--onedrive-metadata-permissions`.
Metadata and permissions are supported for Folders (directories) as well as
Files. Note that setting the `mtime` or `btime` on a Folder requires one extra
API call on OneDrive Business only.
OneDrive does not currently support User Metadata. When writing metadata, only
writeable system properties will be written -- any read-only or unrecognized keys
passed in will be ignored.
TIP: to see the metadata and permissions for any file or folder, run:
```
rclone lsjson remote:path --stat -M --onedrive-metadata-permissions read
```
Here are the possible system metadata items for the onedrive backend.
| Name | Help | Type | Example | Read Only |
|------|------|------|---------|-----------|
| btime | Time of file birth (creation) with S accuracy (mS for OneDrive Personal). | RFC 3339 | 2006-01-02T15:04:05Z | N |
| content-type | The MIME type of the file. | string | text/plain | **Y** |
| created-by-display-name | Display name of the user that created the item. | string | John Doe | **Y** |
| created-by-id | ID of the user that created the item. | string | 48d31887-5fad-4d73-a9f5-3c356e68a038 | **Y** |
| description | A short description of the file. Max 1024 characters. Only supported for OneDrive Personal. | string | Contract for signing | N |
| id | The unique identifier of the item within OneDrive. | string | 01BYE5RZ6QN3ZWBTUFOFD3GSPGOHDJD36K | **Y** |
| last-modified-by-display-name | Display name of the user that last modified the item. | string | John Doe | **Y** |
| last-modified-by-id | ID of the user that last modified the item. | string | 48d31887-5fad-4d73-a9f5-3c356e68a038 | **Y** |
| malware-detected | Whether OneDrive has detected that the item contains malware. | boolean | true | **Y** |
| mtime | Time of last modification with S accuracy (mS for OneDrive Personal). | RFC 3339 | 2006-01-02T15:04:05Z | N |
| package-type | If present, indicates that this item is a package instead of a folder or file. Packages are treated like files in some contexts and folders in others. | string | oneNote | **Y** |
| permissions | Permissions in a JSON dump of OneDrive format. Enable with --onedrive-metadata-permissions. Properties: id, grantedTo, grantedToIdentities, invitation, inheritedFrom, link, roles, shareId | JSON | {} | N |
| shared-by-id | ID of the user that shared the item (if shared). | string | 48d31887-5fad-4d73-a9f5-3c356e68a038 | **Y** |
| shared-owner-id | ID of the owner of the shared item (if shared). | string | 48d31887-5fad-4d73-a9f5-3c356e68a038 | **Y** |
| shared-scope | If shared, indicates the scope of how the item is shared: anonymous, organization, or users. | string | users | **Y** |
| shared-time | Time when the item was shared, with S accuracy (mS for OneDrive Personal). | RFC 3339 | 2006-01-02T15:04:05Z | **Y** |
| utime | Time of upload with S accuracy (mS for OneDrive Personal). | RFC 3339 | 2006-01-02T15:04:05Z | **Y** |
See the [metadata](/docs/#metadata) docs for more info.
{{< rem autogenerated options stop >}}
## Limitations

2
docs/content/overview.md
View File

@ -40,7 +40,7 @@ Here is an overview of the major features of each cloud storage system.
| Memory | MD5 | R/W | No | No | - | - |
| Microsoft Azure Blob Storage | MD5 | R/W | No | No | R/W | - |
| Microsoft Azure Files Storage | MD5 | R/W | Yes | No | R/W | - |
| Microsoft OneDrive | QuickXorHash ⁵ | R/W | Yes | No | R | - |
| Microsoft OneDrive | QuickXorHash ⁵ | DR/W | Yes | No | R | DRW |
| OpenDrive | MD5 | R/W | Yes | Partial ⁸ | - | - |
| OpenStack Swift | MD5 | R/W | No | No | R/W | - |
| Oracle Object Storage | MD5 | R/W | No | No | R/W | - |

15
fs/log.go
View File

@ -2,6 +2,7 @@ package fs
import (
"context"
"encoding/json"
"fmt"
"log"
"os"
@ -194,3 +195,17 @@ func LogDirName(f Fs, dir string) interface{} {
}
return f
}
// PrettyPrint formats JSON for improved readability in debug logs.
// If it can't Marshal JSON, it falls back to fmt.
func PrettyPrint(in any, label string, level LogLevel) {
if GetConfig(context.TODO()).LogLevel < level {
return
}
inBytes, err := json.MarshalIndent(in, "", "\t")
if err != nil {
LogPrintf(level, label, "\n%+v\n", in)
return
}
LogPrintf(level, label, "\n%s\n", string(inBytes))
}