add support for serving Google Cloud Storage over SFTP/SCP

Each user can be mapped with a Google Cloud Storage bucket or a bucket
virtual folder
This commit is contained in:
Nicola Murino 2020-01-31 19:04:00 +01:00
parent 45a13f5f4e
commit 3491717c26
33 changed files with 1632 additions and 165 deletions

View file

@ -22,7 +22,7 @@ Full featured and highly configurable SFTP server
- Atomic uploads are configurable.
- Support for Git repositories over SSH.
- SCP and rsync are supported.
- Support for serving S3 Compatible Object Storage over SFTP.
- Support for serving local filesystem, S3 Compatible Object Storage and Google Cloud Storage over SFTP/SCP.
- Prometheus metrics are exposed.
- REST API for users management, backup, restore and real time reports of the active connections with possibility of forcibly closing a connection.
- Web based interface to easily manage users and connections.
@ -178,6 +178,7 @@ The `sftpgo` configuration file contains the following sections:
- `http_notification_url`, a valid URL. Leave empty to disable.
- `external_auth_program`, string. Absolute path to an external program to use for users authentication. See the "External Authentication" paragraph for more details.
- `external_auth_scope`, integer. 0 means all supported authetication scopes (passwords, public keys and keyboard interactive). 1 means passwords only. 2 means public keys only. 4 means key keyboard interactive only. The flags can be combined, for example 6 means public keys and keyboard interactive
- `credentials_path`, string. It defines the directory for storing user provided credential files such as Google Cloud Storage credentials. This can be an absolute path or a path relative to the config dir
- **"httpd"**, the configuration for the HTTP server used to serve REST API
- `bind_port`, integer. The port used for serving HTTP requests. Set to 0 to disable HTTP server. Default: 8080
- `bind_address`, string. Leave blank to listen on all available network interfaces. Default: "127.0.0.1"
@ -232,7 +233,8 @@ Here is a full example showing the default config in JSON format:
"http_notification_url": ""
},
"external_auth_program": "",
"external_auth_scope": 0
"external_auth_scope": 0,
"credentials_path": "credentials"
},
"httpd": {
"bind_port": 8080,
@ -270,10 +272,12 @@ Please note that to override configuration options with environment variables a
### Data provider initialization
Before starting `sftpgo serve` a data provider must be configured.
Before starting `sftpgo serve` please ensure that the configured dataprovider is properly initialized.
SQL scripts to create the required database structure can be found inside the source tree [sql](./sql "sql") directory. The SQL scripts filename is, by convention, the date as `YYYYMMDD` and the suffix `.sql`. You need to apply all the SQL scripts for your database ordered by name, for example `20190828.sql` must be applied before `20191112.sql` and so on.
Example for `sqlite`: `find sql/sqlite/ -type f -iname '*.sql' -print |sort -n|xargs cat |sqlite3 sftpgo.db`
SQL based data providers (SQLite, MySQL, PostgreSQL) requires the creation of a database containing the required tables. Memory and bolt data providers does not require an initialization.
SQL scripts to create the required database structure can be found inside the source tree [sql](./sql "sql") directory. The SQL scripts filename is, by convention, the date as `YYYYMMDD` and the suffix `.sql`. You need to apply all the SQL scripts for your database ordered by name, for example `20190828.sql` must be applied before `20191112.sql` and so on.
Example for `SQLite`: `find sql/sqlite/ -type f -iname '*.sql' -print | sort -n |xargs cat | sqlite3 sftpgo.db`
### Starting SFTGo in server mode
@ -482,13 +486,13 @@ The HTTP request has a 15 seconds timeout.
## S3 Compabible Object Storage backends
Each user can be mapped with an S3-Compatible bucket or a bucket virtual folder, this way the mapped bucket/virtual folder is exposed over SFTP/SCP.
Each user can be mapped to whole bucket or to a bucket virtual folder, this way the mapped bucket/virtual folder is exposed over SFTP/SCP.
Specifying a different `key_prefix` you can assign different virtual folders of the same bucket to different users. This is similar to a chroot directory for local filesystem. The virtual folder identified by `key_prefix` does not need to be pre-created.
Specifying a different `key_prefix` you can assign different virtual folders of the same bucket to different users. This is similar to a chroot directory for local filesystem. Each SFTP/SCP user can only access to the assigned virtual folder and to its contents The virtual folder identified by `key_prefix` does not need to be pre-created.
SFTPGo uses multipart uploads and parallel downloads for storing and retrieving files from S3.
SFTPGo tries to automatically create the mapped bucket if it does not exists but it's a better idea to pre-create the bucket and to assign to it the wanted options such as automatic encryption and authorizations.
The configured bucket must exist.
Some SFTP commands doesn't work over S3:
@ -504,6 +508,21 @@ Other notes:
- For server side encryption you have to configure the mapped bucket to automatically encrypt objects.
- A local home directory is still required to store temporary files.
## Google Cloud Storage backend
Each user can be mapped with a Google Cloud Storage bucket or a bucket virtual folder, this way the mapped bucket/virtual folder is exposed over SFTP/SCP. This backend is very similar to the S3 backend and it has the same limitations.
## Other Storage backends
Adding new storage backends it's quite easy:
- implement the [Fs interface](./vfs/vfs.go#L18 "interface for filesystem backends").
- update the user method `GetFilesystem` to return the new backend
- update the web interface and the REST API CLI
- add the flags for the new storage backed to the `portable` mode
Anyway some backends require a pay per use account (or they offer free account for a limited time period only), to be able to add support for such backends or to review pull requests please provide a test account. The test account must be available over the time to be able to maintain the backend and do basic tests before each new release.
## Portable mode
SFTPGo allows to share a single directory on demand using the `portable` subcommand:
@ -520,25 +539,29 @@ Usage:
sftpgo portable [flags]
Flags:
-C, --advertise-credentials If the SFTP service is advertised via multicast DNS this flag allows to put username/password inside the advertised TXT record
-S, --advertise-service Advertise SFTP service using multicast DNS (default true)
-d, --directory string Path to the directory to serve. This can be an absolute path or a path relative to the current directory (default ".")
-f, --fs-provider int 0 means local filesystem, 1 S3 compatible
-h, --help help for portable
-l, --log-file-path string Leave empty to disable logging
-p, --password string Leave empty to use an auto generated value
-g, --permissions strings User's permissions. "*" means any permission (default [list,download])
-C, --advertise-credentials If the SFTP service is advertised via multicast DNS this flag allows to put username/password inside the advertised TXT record
-S, --advertise-service Advertise SFTP service using multicast DNS (default true)
-d, --directory string Path to the directory to serve. This can be an absolute path or a path relative to the current directory (default ".")
-f, --fs-provider int 0 means local filesystem, 1 Amazon S3 compatible, 2 Google Cloud Storage
--gcs-bucket string
--gcs-credentials-file string Google Cloud Storage JSON credentials file
--gcs-key-prefix string Allows to restrict access to the virtual folder identified by this prefix and its contents
--gcs-storage-class string
-h, --help help for portable
-l, --log-file-path string Leave empty to disable logging
-p, --password string Leave empty to use an auto generated value
-g, --permissions strings User's permissions. "*" means any permission (default [list,download])
-k, --public-key strings
--s3-access-key string
--s3-access-secret string
--s3-bucket string
--s3-endpoint string
--s3-key-prefix string Allows to restrict access to the virtual folder identified by this prefix and its contents
--s3-key-prefix string Allows to restrict access to the virtual folder identified by this prefix and its contents
--s3-region string
--s3-storage-class string
-s, --sftpd-port int 0 means a random non privileged port
-c, --ssh-commands strings SSH commands to enable. "*" means any supported SSH command including scp (default [md5sum,sha1sum,cd,pwd])
-u, --username string Leave empty to use an auto generated value
-s, --sftpd-port int 0 means a random non privileged port
-c, --ssh-commands strings SSH commands to enable. "*" means any supported SSH command including scp (default [md5sum,sha1sum,cd,pwd])
-u, --username string Leave empty to use an auto generated value
```
In portable mode SFTPGo can advertise the SFTP service and, optionally, the credentials via multicast DNS, so there is a standard way to discover the service and to automatically connect to it.
@ -592,6 +615,10 @@ For each account the following properties can be configured:
- `s3_endpoint`, specifies s3 endpoint (server) different from AWS
- `s3_storage_class`
- `s3_key_prefix`, allows to restrict access to the virtual folder identified by this prefix and its contents
- `gcs_bucket`, required for GCS filesystem
- `gcs_credentials`, Google Cloud Storage JSON credentials base64 encoded
- `gcs_storage_class`
- `gcs_key_prefix`, allows to restrict access to the virtual folder identified by this prefix and its contents
These properties are stored inside the data provider.

View file

@ -1,6 +1,10 @@
package cmd
import (
"encoding/base64"
"fmt"
"io/ioutil"
"os"
"path/filepath"
"github.com/drakkan/sftpgo/dataprovider"
@ -29,6 +33,10 @@ var (
portableS3Endpoint string
portableS3StorageClass string
portableS3KeyPrefix string
portableGCSBucket string
portableGCSCredentialsFile string
portableGCSStorageClass string
portableGCSKeyPrefix string
portableCmd = &cobra.Command{
Use: "portable",
Short: "Serve a single directory",
@ -44,6 +52,24 @@ Please take a look at the usage below to customize the serving parameters`,
}
permissions := make(map[string][]string)
permissions["/"] = portablePermissions
portableGCSCredentials := ""
if portableFsProvider == 2 {
fi, err := os.Stat(portableGCSCredentialsFile)
if err != nil {
fmt.Printf("Invalid GCS credentials file: %v\n", err)
return
}
if fi.Size() > 1048576 {
fmt.Printf("Invalid GCS credentials file: %#v is too big %v/1048576 bytes\n", portableGCSCredentialsFile,
fi.Size())
return
}
creds, err := ioutil.ReadFile(portableGCSCredentialsFile)
if err != nil {
fmt.Printf("Unable to read credentials file: %v\n", err)
}
portableGCSCredentials = base64.StdEncoding.EncodeToString(creds)
}
service := service.Service{
ConfigDir: defaultConfigDir,
ConfigFile: defaultConfigName,
@ -73,6 +99,12 @@ Please take a look at the usage below to customize the serving parameters`,
StorageClass: portableS3StorageClass,
KeyPrefix: portableS3KeyPrefix,
},
GCSConfig: vfs.GCSFsConfig{
Bucket: portableGCSBucket,
Credentials: portableGCSCredentials,
StorageClass: portableGCSStorageClass,
KeyPrefix: portableGCSKeyPrefix,
},
},
},
}
@ -100,7 +132,8 @@ func init() {
"Advertise SFTP service using multicast DNS")
portableCmd.Flags().BoolVarP(&portableAdvertiseCredentials, "advertise-credentials", "C", false,
"If the SFTP service is advertised via multicast DNS this flag allows to put username/password inside the advertised TXT record")
portableCmd.Flags().IntVarP(&portableFsProvider, "fs-provider", "f", 0, "0 means local filesystem, 1 S3 compatible")
portableCmd.Flags().IntVarP(&portableFsProvider, "fs-provider", "f", 0, "0 means local filesystem, 1 Amazon S3 compatible, "+
"2 Google Cloud Storage")
portableCmd.Flags().StringVar(&portableS3Bucket, "s3-bucket", "", "")
portableCmd.Flags().StringVar(&portableS3Region, "s3-region", "", "")
portableCmd.Flags().StringVar(&portableS3AccessKey, "s3-access-key", "", "")
@ -109,5 +142,10 @@ func init() {
portableCmd.Flags().StringVar(&portableS3StorageClass, "s3-storage-class", "", "")
portableCmd.Flags().StringVar(&portableS3KeyPrefix, "s3-key-prefix", "", "Allows to restrict access to the virtual folder "+
"identified by this prefix and its contents")
portableCmd.Flags().StringVar(&portableGCSBucket, "gcs-bucket", "", "")
portableCmd.Flags().StringVar(&portableGCSStorageClass, "gcs-storage-class", "", "")
portableCmd.Flags().StringVar(&portableGCSKeyPrefix, "gcs-key-prefix", "", "Allows to restrict access to the virtual folder "+
"identified by this prefix and its contents")
portableCmd.Flags().StringVar(&portableGCSCredentialsFile, "gcs-credentials-file", "", "Google Cloud Storage JSON credentials file")
rootCmd.AddCommand(portableCmd)
}

View file

@ -84,6 +84,7 @@ func init() {
},
ExternalAuthProgram: "",
ExternalAuthScope: 0,
CredentialsPath: "credentials",
},
HTTPDConfig: httpd.Conf{
BindPort: 8080,
@ -178,6 +179,12 @@ func LoadConfig(configDir, configName string) error {
logger.Warn(logSender, "", "Configuration error: %v", err)
logger.WarnToConsole("Configuration error: %v", err)
}
if len(globalConf.ProviderConf.CredentialsPath) == 0 {
err = fmt.Errorf("invalid credentials path, reset to \"credentials\"")
globalConf.ProviderConf.CredentialsPath = "credentials"
logger.Warn(logSender, "", "Configuration error: %v", err)
logger.WarnToConsole("Configuration error: %v", err)
}
logger.Debug(logSender, "", "config file used: '%v', config loaded: %+v", viper.ConfigFileUsed(), getRedactedGlobalConf())
return err
}

View file

@ -119,6 +119,27 @@ func TestInvalidExternalAuthScope(t *testing.T) {
os.Remove(configFilePath)
}
func TestInvalidCredentialsPath(t *testing.T) {
configDir := ".."
confName := tempConfigName + ".json"
configFilePath := filepath.Join(configDir, confName)
config.LoadConfig(configDir, "")
providerConf := config.GetProviderConf()
providerConf.CredentialsPath = ""
c := make(map[string]dataprovider.Config)
c["data_provider"] = providerConf
jsonConf, _ := json.Marshal(c)
err := ioutil.WriteFile(configFilePath, jsonConf, 0666)
if err != nil {
t.Errorf("error saving temporary configuration")
}
err = config.LoadConfig(configDir, tempConfigName)
if err == nil {
t.Errorf("Loading configuration with credentials path must fail")
}
os.Remove(configFilePath)
}
func TestSetGetConfig(t *testing.T) {
sftpdConf := config.GetSFTPDConfig()
sftpdConf.IdleTimeout = 3

View file

@ -319,6 +319,10 @@ func (p BoltProvider) dumpUsers() ([]User, error) {
if err != nil {
return err
}
err = addCredentialsToUser(&user)
if err != nil {
return err
}
users = append(users, user)
}
return err

View file

@ -16,6 +16,7 @@ import (
"errors"
"fmt"
"hash"
"io/ioutil"
"net"
"net/http"
"net/url"
@ -85,6 +86,7 @@ var (
availabilityTicker *time.Ticker
availabilityTickerDone chan bool
errWrongPassword = errors.New("password does not match")
credentialsDirPath string
)
// Actions to execute on user create, update, delete.
@ -179,6 +181,10 @@ type Config struct {
// you can combine the scopes, for example 3 means password and public key, 5 password and keyboard
// interactive and so on
ExternalAuthScope int `json:"external_auth_scope" mapstructure:"external_auth_scope"`
// CredentialsPath defines the directory for storing user provided credential files such as
// Google Cloud Storage credentials. It can be a path relative to the config dir or an
// absolute path
CredentialsPath string `json:"credentials_path" mapstructure:"credentials_path"`
}
type keyboardAuthProgramResponse struct {
@ -268,6 +274,9 @@ func Initialize(cnf Config, basePath string) error {
return err
}
}
if err := validateCredentialsDir(basePath); err != nil {
return err
}
if config.Driver == SQLiteDataProviderName {
err = initializeSQLiteProvider(basePath)
@ -509,6 +518,25 @@ func validateFilters(user *User) error {
return nil
}
func saveGCSCredentials(user *User) error {
if user.FsConfig.Provider != 2 {
return nil
}
if len(user.FsConfig.GCSConfig.Credentials) == 0 {
return nil
}
decoded, err := base64.StdEncoding.DecodeString(user.FsConfig.GCSConfig.Credentials)
if err != nil {
return &ValidationError{err: fmt.Sprintf("could not validate GCS credentials: %v", err)}
}
err = ioutil.WriteFile(user.getGCSCredentialsFilePath(), decoded, 0600)
if err != nil {
return &ValidationError{err: fmt.Sprintf("could not save GCS credentials: %v", err)}
}
user.FsConfig.GCSConfig.Credentials = ""
return nil
}
func validateFilesystemConfig(user *User) error {
if user.FsConfig.Provider == 1 {
err := vfs.ValidateS3FsConfig(&user.FsConfig.S3Config)
@ -524,9 +552,16 @@ func validateFilesystemConfig(user *User) error {
user.FsConfig.S3Config.AccessSecret = accessSecret
}
return nil
} else if user.FsConfig.Provider == 2 {
err := vfs.ValidateGCSFsConfig(&user.FsConfig.GCSConfig, user.getGCSCredentialsFilePath())
if err != nil {
return &ValidationError{err: fmt.Sprintf("could not validate GCS config: %v", err)}
}
return nil
}
user.FsConfig.Provider = 0
user.FsConfig.S3Config = vfs.S3FsConfig{}
user.FsConfig.GCSConfig = vfs.GCSFsConfig{}
return nil
}
@ -563,6 +598,9 @@ func validateUser(user *User) error {
if err := validateFilters(user); err != nil {
return err
}
if err := saveGCSCredentials(user); err != nil {
return err
}
return nil
}
@ -704,10 +742,24 @@ func HideUserSensitiveData(user *User) User {
user.Password = ""
if user.FsConfig.Provider == 1 {
user.FsConfig.S3Config.AccessSecret = utils.RemoveDecryptionKey(user.FsConfig.S3Config.AccessSecret)
} else if user.FsConfig.Provider == 2 {
user.FsConfig.GCSConfig.Credentials = ""
}
return *user
}
func addCredentialsToUser(user *User) error {
if user.FsConfig.Provider != 2 {
return nil
}
cred, err := ioutil.ReadFile(user.getGCSCredentialsFilePath())
if err != nil {
return err
}
user.FsConfig.GCSConfig.Credentials = base64.StdEncoding.EncodeToString(cred)
return nil
}
func getSSLMode() string {
if config.Driver == PGSQLDataProviderName {
if config.SSLMode == 0 {
@ -748,6 +800,25 @@ func startAvailabilityTimer() {
}()
}
func validateCredentialsDir(basePath string) error {
if filepath.IsAbs(config.CredentialsPath) {
credentialsDirPath = config.CredentialsPath
} else {
credentialsDirPath = filepath.Join(basePath, config.CredentialsPath)
}
fi, err := os.Stat(credentialsDirPath)
if err == nil {
if !fi.IsDir() {
return errors.New("Credential path is not a valid directory")
}
return nil
}
if !os.IsNotExist(err) {
return err
}
return os.MkdirAll(credentialsDirPath, 0700)
}
func checkDataprovider() {
err := provider.checkAvailability()
if err != nil {

View file

@ -224,6 +224,10 @@ func (p MemoryProvider) dumpUsers() ([]User, error) {
}
for _, username := range p.dbHandle.usernames {
user := p.dbHandle.users[username]
err = addCredentialsToUser(&user)
if err != nil {
return users, err
}
users = append(users, user)
}
return users, err

View file

@ -232,11 +232,14 @@ func sqlCommonDumpUsers(dbHandle *sql.DB) ([]User, error) {
defer rows.Close()
for rows.Next() {
u, err := getUserFromDbRow(nil, rows)
if err == nil {
users = append(users, u)
} else {
break
if err != nil {
return users, err
}
err = addCredentialsToUser(&u)
if err != nil {
return users, err
}
users = append(users, u)
}
}

View file

@ -55,9 +55,10 @@ type UserFilters struct {
// Filesystem defines cloud storage filesystem details
type Filesystem struct {
// 0 local filesystem, 1 Amazon S3 compatible
Provider int `json:"provider"`
S3Config vfs.S3FsConfig `json:"s3config,omitempty"`
// 0 local filesystem, 1 Amazon S3 compatible, 2 Google Cloud Storage
Provider int `json:"provider"`
S3Config vfs.S3FsConfig `json:"s3config,omitempty"`
GCSConfig vfs.GCSFsConfig `json:"gcsconfig,omitempty"`
}
// User defines an SFTP user
@ -73,7 +74,7 @@ type User struct {
ExpirationDate int64 `json:"expiration_date"`
// Password used for password authentication.
// For users created using SFTPGo REST API the password is be stored using argon2id hashing algo.
// Checking passwords stored with bcrypt, pbkdf2 and sha512crypt is supported too.
// Checking passwords stored with bcrypt, pbkdf2, md5crypt and sha512crypt is supported too.
Password string `json:"password,omitempty"`
// PublicKeys used for public key authentication. At least one between password and a public key is mandatory
PublicKeys []string `json:"public_keys,omitempty"`
@ -113,6 +114,10 @@ type User struct {
func (u *User) GetFilesystem(connectionID string) (vfs.Fs, error) {
if u.FsConfig.Provider == 1 {
return vfs.NewS3Fs(connectionID, u.GetHomeDir(), u.FsConfig.S3Config)
} else if u.FsConfig.Provider == 2 {
config := u.FsConfig.GCSConfig
config.CredentialFile = u.getGCSCredentialsFilePath()
return vfs.NewGCSFs(connectionID, u.GetHomeDir(), config)
}
return vfs.NewOsFs(connectionID, u.GetHomeDir()), nil
}
@ -321,7 +326,8 @@ func (u *User) GetBandwidthAsString() string {
}
// GetInfoString returns user's info as string.
// Number of public keys, max sessions, uid and gid are returned
// Storage provider, number of public keys, max sessions, uid,
// gid, denied and allowed IP/Mask are returned
func (u *User) GetInfoString() string {
var result string
if u.LastLogin > 0 {
@ -329,7 +335,9 @@ func (u *User) GetInfoString() string {
result += fmt.Sprintf("Last login: %v ", t.Format("2006-01-02 15:04:05")) // YYYY-MM-DD HH:MM:SS
}
if u.FsConfig.Provider == 1 {
result += fmt.Sprintf("Storage: S3")
result += fmt.Sprintf("Storage: S3 ")
} else if u.FsConfig.Provider == 2 {
result += fmt.Sprintf("Storage: GCS ")
}
if len(u.PublicKeys) > 0 {
result += fmt.Sprintf("Public keys: %v ", len(u.PublicKeys))
@ -410,6 +418,12 @@ func (u *User) getACopy() User {
StorageClass: u.FsConfig.S3Config.StorageClass,
KeyPrefix: u.FsConfig.S3Config.KeyPrefix,
},
GCSConfig: vfs.GCSFsConfig{
Bucket: u.FsConfig.GCSConfig.Bucket,
CredentialFile: u.FsConfig.GCSConfig.CredentialFile,
StorageClass: u.FsConfig.GCSConfig.StorageClass,
KeyPrefix: u.FsConfig.GCSConfig.KeyPrefix,
},
}
return User{
@ -458,3 +472,7 @@ func (u *User) getNotificationFieldsAsEnvVars(action string) []string {
fmt.Sprintf("SFTPGO_USER_UID=%v", u.UID),
fmt.Sprintf("SFTPGO_USER_GID=%v", u.GID)}
}
func (u *User) getGCSCredentialsFilePath() string {
return filepath.Join(credentialsDirPath, fmt.Sprintf("%v_gcs_credentials.json", u.Username))
}

View file

@ -12,6 +12,7 @@ sudo groupadd -g 1003 sftpgrp && \
curl https://raw.githubusercontent.com/drakkan/sftpgo/master/sql/sqlite/20190828.sql | sqlite3 /home/sftpuser/conf/sftpgo.db && \
curl https://raw.githubusercontent.com/drakkan/sftpgo/master/sql/sqlite/20191112.sql | sqlite3 /home/sftpuser/conf/sftpgo.db && \
curl https://raw.githubusercontent.com/drakkan/sftpgo/master/sql/sqlite/20191230.sql | sqlite3 /home/sftpuser/conf/sftpgo.db && \
curl https://raw.githubusercontent.com/drakkan/sftpgo/master/sql/sqlite/20200116.sql | sqlite3 /home/sftpuser/conf/sftpgo.db && \
curl https://raw.githubusercontent.com/drakkan/sftpgo/master/sftpgo.json -o /home/sftpuser/conf/sftpgo.json
# Get and build SFTPGo image

9
go.mod
View file

@ -3,6 +3,8 @@ module github.com/drakkan/sftpgo
go 1.13
require (
cloud.google.com/go v0.52.0 // indirect
cloud.google.com/go/storage v1.5.0
github.com/alexedwards/argon2id v0.0.0-20190612080829-01a59b2b8802
github.com/aws/aws-sdk-go v1.28.3
github.com/cenkalti/backoff v2.2.1+incompatible // indirect
@ -10,6 +12,7 @@ require (
github.com/go-chi/chi v4.0.2+incompatible
github.com/go-chi/render v1.0.1
github.com/go-sql-driver/mysql v1.5.0
github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e // indirect
github.com/grandcat/zeroconf v0.0.0-20190424104450-85eadb44205c
github.com/lib/pq v1.3.0
github.com/mattn/go-sqlite3 v2.0.2+incompatible
@ -23,7 +26,11 @@ require (
github.com/spf13/viper v1.6.1
go.etcd.io/bbolt v1.3.3
golang.org/x/crypto v0.0.0-20200109152110-61a87790db17
golang.org/x/sys v0.0.0-20191220142924-d4481acd189f
golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a // indirect
golang.org/x/sys v0.0.0-20200122134326-e047566fdf82
golang.org/x/tools v0.0.0-20200124200720-1b668f209185 // indirect
google.golang.org/api v0.15.0
google.golang.org/genproto v0.0.0-20200122232147-0452cf42e150 // indirect
gopkg.in/natefinch/lumberjack.v2 v2.0.0
)

183
go.sum
View file

@ -1,5 +1,26 @@
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.38.0/go.mod h1:990N+gfupTy94rShfmMCWGDn0LpTmnzTp2qbd1dvSRU=
cloud.google.com/go v0.44.1/go.mod h1:iSa0KzasP4Uvy3f1mN/7PiObzGgflwredwwASm/v6AU=
cloud.google.com/go v0.44.2/go.mod h1:60680Gw3Yr4ikxnPRS/oxxkBccT6SA1yMk63TGekxKY=
cloud.google.com/go v0.45.1/go.mod h1:RpBamKRgapWJb87xiFSdk4g1CME7QZg3uwTez+TSTjc=
cloud.google.com/go v0.46.3/go.mod h1:a6bKKbmY7er1mI7TEI4lsAkts/mkhTSZK8w33B4RAg0=
cloud.google.com/go v0.50.0 h1:0E3eE8MX426vUOs7aHfI7aN1BrIzzzf4ccKCSfSjGmc=
cloud.google.com/go v0.50.0/go.mod h1:r9sluTvynVuxRIOHXQEHMFffphuXHOMZMycpNR5e6To=
cloud.google.com/go v0.52.0 h1:GGslhk/BU052LPlnI1vpp3fcbUs+hQ3E+Doti/3/vF8=
cloud.google.com/go v0.52.0/go.mod h1:pXajvRH/6o3+F9jDHZWQ5PbGhn+o8w9qiu/CffaVdO4=
cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o=
cloud.google.com/go/bigquery v1.3.0/go.mod h1:PjpwJnslEMmckchkHFfq+HTD2DmtT67aNFKH1/VBDHE=
cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE=
cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I=
cloud.google.com/go/pubsub v1.1.0/go.mod h1:EwwdRX2sKPjnvnqCa270oGRyludottCI76h+R3AArQw=
cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw=
cloud.google.com/go/storage v1.5.0 h1:RPUcBvDeYgQFMfQu1eBMq6piD1SXmLH+vK3qjewZPus=
cloud.google.com/go/storage v1.5.0/go.mod h1:tpKbwo567HUNpVclU5sGELwQWBDZ8gh0ZeosJ0Rtdos=
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
@ -16,10 +37,14 @@ github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/cenkalti/backoff v2.2.1+incompatible h1:tNowT99t7UNflLxfYYSlKYsBpXdEet03Pg2g16Swow4=
github.com/cenkalti/backoff v2.2.1+incompatible/go.mod h1:90ReRw6GdpyfrHakVjL/QHaoyV4aDUVVkXQJJJ3NXXM=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/cespare/xxhash v1.1.0 h1:a6HrQnmkObjyL+Gs60czilIUGqrzKutQD6XZog3p+ko=
github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
github.com/cespare/xxhash/v2 v2.1.1 h1:6MnRN8NT7+YBpUIWxHtefFZOKTAPgGjpQSxqLNn0+qY=
github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/coreos/bbolt v1.3.2/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk=
github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
@ -34,6 +59,8 @@ github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZm
github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954/go.mod h1:vAd38F8PWV+bWy6jNmig1y/TA+kYO4g3RSRF0IAv0no=
github.com/drakkan/pipeat v0.0.0-20200123131427-11c048cfc0ec h1:DXfzg1NXoesnFzdCyyi2uU3o1o0XiWTN2ZcpWDE7MCk=
github.com/drakkan/pipeat v0.0.0-20200123131427-11c048cfc0ec/go.mod h1:wNYvIpR5rIhoezOYcpxcXz4HbIEOu7A45EqlQCA+h+w=
github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
@ -41,6 +68,8 @@ github.com/go-chi/chi v4.0.2+incompatible h1:maB6vn6FqCxrpz4FqWdh4+lwpyZIQS7YEAU
github.com/go-chi/chi v4.0.2+incompatible/go.mod h1:eB3wogJHnLi3x/kFX2A+IbTBlXxmMeXJVKy9tTv1XzQ=
github.com/go-chi/render v1.0.1 h1:4/5tis2cKaNdnv9zFLfXzcquC9HbeZgCnxGnKrltBS8=
github.com/go-chi/render v1.0.1/go.mod h1:pq4Rr7HbnsdaeHagklXub+p6Wd16Af5l9koip1OvJns=
github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE=
@ -52,15 +81,35 @@ github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7a
github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7 h1:5ZkaAPbicIKTF2I64qf5Fh8Aa83Q/dnOafMYV0OMwjA=
github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e h1:1r7pUrabqp18hOBcwBwiTsbnFeTZHV9eER/QT5JVZxY=
github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.2 h1:6nsPYzhq5kReh6QImI3k5qWzO4PEbvbIW2cwSfR/6xs=
github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.4.0 h1:xsAVV57WRhGj6kEIi8ReJzQlHHqcBYCElAvkovg3B/4=
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/martian v2.1.0+incompatible h1:/CP5g8u/VJHijgedC/Legn3BAbAaWPgecwXBIDzw5no=
github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
github.com/google/pprof v0.0.0-20190515194954-54271f7e092f/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
github.com/google/pprof v0.0.0-20191218002539-d4f498aebedc/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg=
github.com/googleapis/gax-go/v2 v2.0.5 h1:sjZBwGj9Jlw33ImPtvFviGYvseOtDM7hkSKB7+Tv3SM=
github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ=
github.com/grandcat/zeroconf v0.0.0-20190424104450-85eadb44205c h1:svzQzfVE9t7Y1CGULS5PsMWs4/H4Au/ZTJzU/0CKgqc=
@ -68,14 +117,20 @@ github.com/grandcat/zeroconf v0.0.0-20190424104450-85eadb44205c/go.mod h1:YjKB0W
github.com/grpc-ecosystem/go-grpc-middleware v1.0.0/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs=
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk=
github.com/grpc-ecosystem/grpc-gateway v1.9.0/go.mod h1:vNeuVxBJEsws4ogUvrchl83t/GYV9WGTSLVdBhOQFDY=
github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4=
github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af h1:pmfjZENx5imkbgOkpRUYLnmbU7UEFbjtDA2hxJ1ichM=
github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k=
github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
github.com/json-iterator/go v1.1.8/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU=
github.com/jstemmer/go-junit-report v0.9.1 h1:6QPYqodiu3GuPL+7mfx+NwDdp2eTkp9IfEUpgAwUN0o=
github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk=
github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w=
github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q=
@ -124,6 +179,7 @@ github.com/prometheus/client_golang v1.3.0 h1:miYCvYqFXtl/J9FIy8eNpBfYthAEFg+Ys0
github.com/prometheus/client_golang v1.3.0/go.mod h1:hJaj2vgQTGQmVCsAACORcieXFeDPbaTKGT+JTgUa3og=
github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/prometheus/client_model v0.1.0 h1:ElTg5tNp4DqfV7UQjDqv2+RJlNzsDtvNAWccbItceIE=
github.com/prometheus/client_model v0.1.0/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/prometheus/common v0.0.0-20181113130724-41aa239b4cce/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro=
@ -138,6 +194,7 @@ github.com/prometheus/procfs v0.0.8 h1:+fpWZdT24pJBiqJdAwYBjPSk+5YmQzYNPYzQsdzLk
github.com/prometheus/procfs v0.0.8/go.mod h1:7Qr8sr6344vo1JqZ6HhLceV9o3AJ1Ff+GxbHq6oeK9A=
github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40TwIPHuXU=
github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg=
github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/rs/xid v1.2.1 h1:mhH9Nq+C1fY2l1XIpgxIiUOfNpRBYH1kKcr+qfKgjRc=
github.com/rs/xid v1.2.1/go.mod h1:+uKXf+4Djp6Md1KODXJxgGQPKngRmWyn10oCKFzNHOQ=
github.com/rs/zerolog v1.17.2 h1:RMRHFw2+wF7LO0QqtELQwo8hqSmqISyCJeFeAAuWcRo=
@ -178,34 +235,80 @@ github.com/zenazn/goji v0.9.0/go.mod h1:7S9M489iMyHBNxwZnk9/EHS098H4/F6TATF2mIxt
go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
go.etcd.io/bbolt v1.3.3 h1:MUGmc65QhB3pIlaQ5bB4LwqSj6GIonVJXpZiaKNyaKk=
go.etcd.io/bbolt v1.3.3/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
go.opencensus.io v0.22.2 h1:75k/FF0Q2YM8QYo07VPddOLBslDt1MZOdEslOHvmzAs=
go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20190820162420-60c769a6c586/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200109152110-61a87790db17 h1:nVJ3guKA9qdkEQ3TUdXI9QSINo2CUPM/cySEvw2w8I0=
golang.org/x/crypto v0.0.0-20200109152110-61a87790db17/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
golang.org/x/exp v0.0.0-20190829153037-c13cbed26979/go.mod h1:86+5VVa7VpoJ4kLfm080zCjGlMRFzhUhsZKEZO7MGek=
golang.org/x/exp v0.0.0-20191030013958-a1ab85dbe136/go.mod h1:JXzH8nQsPlswgeRAPE3MuO9GYsAcnJvJ4vnMwN/5qkY=
golang.org/x/exp v0.0.0-20191129062945-2f5052295587/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20191227195350-da58074b4299 h1:zQpM52jfKHG6II1ISZY1ZcpygvuSFZpLwfluuF89XOg=
golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a h1:7Wlg8L54In96HTWOaI4sreLJ6qfyGuvSau5el3fK41Y=
golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20190409202823-959b441ac422/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20190909230951-414d861bb4ac/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f h1:J5lckAjkw6qYlOZNj90mLYNTEKDvWeuc1yieZ8qUzUE=
golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f/go.mod h1:5qLYkcX4OjUUV8bRuDixDT3tpyyb+LUpUlRWLxfhWrs=
golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE=
golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCcRqshq8CkpyQDoeVncDDYHnLhea+o=
golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc=
golang.org/x/mod v0.1.0/go.mod h1:0QHyrYULN0/3qlju5TqG8bIK38QM8yzMo5ekMj3DlcY=
golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181220203305-927f97764cc3/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190501004415-9ce7a6920f09/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190522155817-f3200d17e092/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
golang.org/x/net v0.0.0-20190613194153-d28f0bde5980/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190724013045-ca1201d0de80/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190923162816-aa69164e4478 h1:l5EDrHhldLYb3ZRHDUhXF7Om7MvYXnkV9/iQNo1lX6g=
golang.org/x/net v0.0.0-20190923162816-aa69164e4478/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553 h1:efeOvDhwQ29Dj3SdAV/MJf8oukgn+8D8WgaCaRMchF8=
golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa h1:F+8P+gmewFQYRk6JoLQLwjBCTu3mcIURZfNkVweuRKA=
golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6 h1:pE8b58s1HRDMi8RDc79m0HISf9D4TzseP40cEA6IGfs=
golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d h1:TzXSXBo42m9gQenoE3b9BGiEpg5IG2JkU5FkPIawgtw=
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@ -214,29 +317,103 @@ golang.org/x/sys v0.0.0-20181107165924-66b7b1311ac8/go.mod h1:STP8DvDyc/dI5b8T5h
golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181205085412-a5c9d58dba9a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190624142023-c5567b49c5d0/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190726091711-fc99dfbffb4e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190924154521-2837fb4f24fe/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191220142924-d4481acd189f h1:68K/z8GLUxV76xGSqwTWw2gyk/jwn79LUL43rES2g8o=
golang.org/x/sys v0.0.0-20191220142924-d4481acd189f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191228213918-04cbcbbfeed8 h1:JA8d3MPx/IToSyXZG/RhwYEtfrKO1Fxrqe8KrkiLXKM=
golang.org/x/sys v0.0.0-20191228213918-04cbcbbfeed8/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200113162924-86b910548bc1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200122134326-e047566fdf82 h1:ywK/j/KkyTHcdyYSZNXGjMwgmDSfjglYZ3vStQ/gSCU=
golang.org/x/sys v0.0.0-20200122134326-e047566fdf82/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.2 h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs=
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190312170243-e65039ee4138/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190425150028-36563e24a262/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20190506145303-2d16b83fe98c/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20190606124116-d0a3d012864b/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190628153133-6cdbf07be9d0/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190816200558-6889da9d5479/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20190828213141-aed303cbaa74/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20190911174233-4f2ddba30aff/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191012152004-8de300cfc20a/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191113191852-77e3bb0ad9e7/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191115202509-3a792d9c32b2/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191125144606-a911d9008d1f/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191216052735-49a3e744a425/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20191216173652-a0e659d51361/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20191227053925-7b8e75db28f4 h1:Toz2IK7k8rbltAXwNAxKcn9OzqyNfMUhUNjz3sL0NMk=
golang.org/x/tools v0.0.0-20191227053925-7b8e75db28f4/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200117161641-43d50277825c/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200124200720-1b668f209185 h1:UhyNb/h6VU6sPOVb6x118tYR91HRLBUtWS01bOSFAz4=
golang.org/x/tools v0.0.0-20200124200720-1b668f209185/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE=
google.golang.org/api v0.7.0/go.mod h1:WtwebWUNSVBH/HAw79HIFXZNqEvBhG+Ra+ax0hx3E3M=
google.golang.org/api v0.8.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg=
google.golang.org/api v0.9.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg=
google.golang.org/api v0.13.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=
google.golang.org/api v0.14.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=
google.golang.org/api v0.15.0 h1:yzlyyDW/J0w8yNFJIhiAJy4kq74S+1DOLdawELNxFMA=
google.golang.org/api v0.15.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.6.1/go.mod h1:i06prIuMbXzDqacNJfV5OdTW448YApPu5ww/cMBSeb0=
google.golang.org/appengine v1.6.5 h1:tycE03LOZYQNhDpS27tcQdAzLCVMaj7QT2SXxebnpCM=
google.golang.org/appengine v1.6.5/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190502173448-54afdca5d873/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190801165951-fa694d86fc64/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
google.golang.org/genproto v0.0.0-20190911173649-1774047e7e51/go.mod h1:IbNlFCBrqXvoKpeg0TB2l7cyZUmoaFKYIwrEpbDKLA8=
google.golang.org/genproto v0.0.0-20191108220845-16a3f7862a1a/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20191115194625-c23dd37a84c9/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20191216164720-4f79533eabd1/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20191230161307-f3c370f40bfb h1:ADPHZzpzM4tk4V4S5cnCrr5SwzvlrPRmqqCuJDB8UTs=
google.golang.org/genproto v0.0.0-20191230161307-f3c370f40bfb/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20200115191322-ca5a22157cba/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20200122232147-0452cf42e150 h1:VPpdpQkGvFicX9yo4G5oxZPi9ALBnEOZblPSa/Wa2m4=
google.golang.org/genproto v0.0.0-20200122232147-0452cf42e150/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38=
google.golang.org/grpc v1.21.0/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM=
google.golang.org/grpc v1.21.1/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM=
google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
google.golang.org/grpc v1.26.0 h1:2dTRdpdFEEhJYQD8EMLB61nnrzSCTbG38PhqdhvOltg=
google.golang.org/grpc v1.26.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
gopkg.in/ini.v1 v1.51.0 h1:AQvPpx3LzTDM0AjnIRlVFwFFGC+npRopjZxLJj6gdno=
gopkg.in/ini.v1 v1.51.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
gopkg.in/natefinch/lumberjack.v2 v2.0.0 h1:1Lc07Kr7qY4U2YPouBjpCLxpiyxIVoxqXgkXLknAOE8=
@ -248,3 +425,9 @@ gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.4 h1:/eiJrUcujPVeJ3xlSWaiNi3uSVmDGBK1pDHUHAnao1I=
gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190418001031-e561f6794a2a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.1-2019.2.3 h1:3JgtbtFHMiCmsznwGVTUWbgGov+pVqnlf1dEJTNAXeM=
honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg=
rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8=

View file

@ -14,6 +14,7 @@ func getQuotaScans(w http.ResponseWriter, r *http.Request) {
}
func startQuotaScan(w http.ResponseWriter, r *http.Request) {
r.Body = http.MaxBytesReader(w, r.Body, maxRequestSize)
var u dataprovider.User
err := render.DecodeJSON(r.Body, &u)
if err != nil {

View file

@ -73,6 +73,7 @@ func getUserByID(w http.ResponseWriter, r *http.Request) {
}
func addUser(w http.ResponseWriter, r *http.Request) {
r.Body = http.MaxBytesReader(w, r.Body, maxRequestSize)
var user dataprovider.User
err := render.DecodeJSON(r.Body, &user)
if err != nil {
@ -93,6 +94,7 @@ func addUser(w http.ResponseWriter, r *http.Request) {
}
func updateUser(w http.ResponseWriter, r *http.Request) {
r.Body = http.MaxBytesReader(w, r.Body, maxRequestSize)
userID, err := strconv.ParseInt(chi.URLParam(r, "userID"), 10, 64)
if err != nil {
err = errors.New("Invalid userID")
@ -100,10 +102,10 @@ func updateUser(w http.ResponseWriter, r *http.Request) {
return
}
user, err := dataprovider.GetUserByID(dataProvider, userID)
oldPermissions := user.Permissions
oldS3AccessSecret := ""
currentPermissions := user.Permissions
currentS3AccessSecret := ""
if user.FsConfig.Provider == 1 {
oldS3AccessSecret = user.FsConfig.S3Config.AccessSecret
currentS3AccessSecret = user.FsConfig.S3Config.AccessSecret
}
user.Permissions = make(map[string][]string)
if _, ok := err.(*dataprovider.RecordNotFoundError); ok {
@ -120,13 +122,13 @@ func updateUser(w http.ResponseWriter, r *http.Request) {
}
// we use new Permissions if passed otherwise the old ones
if len(user.Permissions) == 0 {
user.Permissions = oldPermissions
user.Permissions = currentPermissions
}
// we use the new access secret if different from the old one and not empty
if user.FsConfig.Provider == 1 {
if utils.RemoveDecryptionKey(oldS3AccessSecret) == user.FsConfig.S3Config.AccessSecret ||
if utils.RemoveDecryptionKey(currentS3AccessSecret) == user.FsConfig.S3Config.AccessSecret ||
len(user.FsConfig.S3Config.AccessSecret) == 0 {
user.FsConfig.S3Config.AccessSecret = oldS3AccessSecret
user.FsConfig.S3Config.AccessSecret = currentS3AccessSecret
}
}
if user.ID != userID {

View file

@ -439,6 +439,16 @@ func compareUserFsConfig(expected *dataprovider.User, actual *dataprovider.User)
expected.FsConfig.S3Config.KeyPrefix+"/" != actual.FsConfig.S3Config.KeyPrefix {
return errors.New("S3 key prefix mismatch")
}
if expected.FsConfig.GCSConfig.Bucket != actual.FsConfig.GCSConfig.Bucket {
return errors.New("GCS bucket mismatch")
}
if expected.FsConfig.GCSConfig.StorageClass != actual.FsConfig.GCSConfig.StorageClass {
return errors.New("GCS storage class mismatch")
}
if expected.FsConfig.GCSConfig.KeyPrefix != actual.FsConfig.GCSConfig.KeyPrefix &&
expected.FsConfig.GCSConfig.KeyPrefix+"/" != actual.FsConfig.GCSConfig.KeyPrefix {
return errors.New("GCS key prefix mismatch")
}
return nil
}

View file

@ -33,6 +33,7 @@ const (
webConnectionsPath = "/web/connections"
webStaticFilesPath = "/static"
maxRestoreSize = 10485760 // 10 MB
maxRequestSize = 1048576 // 1MB
)
var (

View file

@ -3,9 +3,12 @@ package httpd_test
import (
"bytes"
"crypto/rand"
"encoding/base64"
"encoding/json"
"fmt"
"io"
"io/ioutil"
"mime/multipart"
"net"
"net/http"
"net/http/httptest"
@ -56,6 +59,7 @@ var (
defaultPerms = []string{dataprovider.PermAny}
homeBasePath string
backupsPath string
credentialsPath string
testServer *httptest.Server
providerDriverName string
)
@ -66,7 +70,10 @@ func TestMain(m *testing.M) {
logger.InitLogger(logfilePath, 5, 1, 28, false, zerolog.DebugLevel)
config.LoadConfig(configDir, "")
providerConf := config.GetProviderConf()
credentialsPath = filepath.Join(os.TempDir(), "test_credentials")
providerConf.CredentialsPath = credentialsPath
providerDriverName = providerConf.Driver
os.RemoveAll(credentialsPath)
err := dataprovider.Initialize(providerConf, configDir)
if err != nil {
@ -102,6 +109,7 @@ func TestMain(m *testing.M) {
exitCode := m.Run()
os.Remove(logfilePath)
os.RemoveAll(backupsPath)
os.RemoveAll(credentialsPath)
os.Exit(exitCode)
}
@ -250,6 +258,8 @@ func TestAddUserInvalidFsConfig(t *testing.T) {
if err != nil {
t.Errorf("unexpected error adding user with invalid fs config: %v", err)
}
os.RemoveAll(credentialsPath)
os.MkdirAll(credentialsPath, 0700)
u.FsConfig.S3Config.Bucket = "test"
u.FsConfig.S3Config.Region = "eu-west-1"
u.FsConfig.S3Config.AccessKey = "access-key"
@ -261,6 +271,32 @@ func TestAddUserInvalidFsConfig(t *testing.T) {
if err != nil {
t.Errorf("unexpected error adding user with invalid fs config: %v", err)
}
u = getTestUser()
u.FsConfig.Provider = 2
u.FsConfig.GCSConfig.Bucket = ""
_, _, err = httpd.AddUser(u, http.StatusBadRequest)
if err != nil {
t.Errorf("unexpected error adding user with invalid fs config: %v", err)
}
u.FsConfig.GCSConfig.Bucket = "test"
u.FsConfig.GCSConfig.StorageClass = "Standard"
u.FsConfig.GCSConfig.KeyPrefix = "/somedir/subdir/"
u.FsConfig.GCSConfig.Credentials = base64.StdEncoding.EncodeToString([]byte("test"))
_, _, err = httpd.AddUser(u, http.StatusBadRequest)
if err != nil {
t.Errorf("unexpected error adding user with invalid fs config: %v", err)
}
u.FsConfig.GCSConfig.KeyPrefix = "somedir/subdir/"
u.FsConfig.GCSConfig.Credentials = ""
_, _, err = httpd.AddUser(u, http.StatusBadRequest)
if err != nil {
t.Errorf("unexpected error adding user with invalid fs config: %v", err)
}
u.FsConfig.GCSConfig.Credentials = "no base64 encoded"
_, _, err = httpd.AddUser(u, http.StatusBadRequest)
if err != nil {
t.Errorf("unexpected error adding user with invalid fs config: %v", err)
}
}
func TestUserPublicKey(t *testing.T) {
@ -363,6 +399,56 @@ func TestUserS3Config(t *testing.T) {
}
}
func TestUserGCSConfig(t *testing.T) {
user, _, err := httpd.AddUser(getTestUser(), http.StatusOK)
if err != nil {
t.Errorf("unable to add user: %v", err)
}
os.RemoveAll(credentialsPath)
os.MkdirAll(credentialsPath, 0700)
user.FsConfig.Provider = 2
user.FsConfig.GCSConfig.Bucket = "test"
user.FsConfig.GCSConfig.Credentials = base64.StdEncoding.EncodeToString([]byte("fake credentials"))
user, _, err = httpd.UpdateUser(user, http.StatusOK)
if err != nil {
t.Errorf("unable to update user: %v", err)
}
_, err = httpd.RemoveUser(user, http.StatusOK)
if err != nil {
t.Errorf("unable to remove: %v", err)
}
user.Password = defaultPassword
user.ID = 0
// the user will be added since the credentials file is found
user, _, err = httpd.AddUser(user, http.StatusOK)
if err != nil {
t.Errorf("unable to add user: %v", err)
}
user.FsConfig.Provider = 1
user.FsConfig.S3Config.Bucket = "test1"
user.FsConfig.S3Config.Region = "us-east-1"
user.FsConfig.S3Config.AccessKey = "Server-Access-Key1"
user.FsConfig.S3Config.AccessSecret = "secret"
user.FsConfig.S3Config.Endpoint = "http://localhost:9000"
user.FsConfig.S3Config.KeyPrefix = "somedir/subdir"
user, _, err = httpd.UpdateUser(user, http.StatusOK)
if err != nil {
t.Errorf("unable to update user: %v", err)
}
user.FsConfig.Provider = 2
user.FsConfig.GCSConfig.Bucket = "test1"
user.FsConfig.GCSConfig.Credentials = base64.StdEncoding.EncodeToString([]byte("fake credentials"))
user, _, err = httpd.UpdateUser(user, http.StatusOK)
if err != nil {
t.Errorf("unable to update user: %v", err)
}
_, err = httpd.RemoveUser(user, http.StatusOK)
if err != nil {
t.Errorf("unable to remove: %v", err)
}
}
func TestUpdateUserNoCredentials(t *testing.T) {
user, _, err := httpd.AddUser(getTestUser(), http.StatusOK)
if err != nil {
@ -594,6 +680,8 @@ func TestUserBaseDir(t *testing.T) {
dataprovider.Close(dataProvider)
config.LoadConfig(configDir, "")
providerConf = config.GetProviderConf()
providerConf.CredentialsPath = credentialsPath
os.RemoveAll(credentialsPath)
err = dataprovider.Initialize(providerConf, configDir)
if err != nil {
t.Errorf("error initializing data provider")
@ -646,6 +734,8 @@ func TestProviderErrors(t *testing.T) {
os.Remove(backupFilePath)
config.LoadConfig(configDir, "")
providerConf := config.GetProviderConf()
providerConf.CredentialsPath = credentialsPath
os.RemoveAll(credentialsPath)
err = dataprovider.Initialize(providerConf, configDir)
if err != nil {
t.Errorf("error initializing data provider")
@ -655,7 +745,17 @@ func TestProviderErrors(t *testing.T) {
}
func TestDumpdata(t *testing.T) {
_, _, err := httpd.Dumpdata("", http.StatusBadRequest)
dataProvider := dataprovider.GetProvider()
dataprovider.Close(dataProvider)
config.LoadConfig(configDir, "")
providerConf := config.GetProviderConf()
err := dataprovider.Initialize(providerConf, configDir)
if err != nil {
t.Errorf("error initializing data provider")
}
httpd.SetDataProvider(dataprovider.GetProvider())
sftpd.SetDataProvider(dataprovider.GetProvider())
_, _, err = httpd.Dumpdata("", http.StatusBadRequest)
if err != nil {
t.Errorf("unexpected error: %v", err)
}
@ -680,6 +780,15 @@ func TestDumpdata(t *testing.T) {
}
os.Chmod(backupsPath, 0755)
}
providerConf = config.GetProviderConf()
providerConf.CredentialsPath = credentialsPath
os.RemoveAll(credentialsPath)
err = dataprovider.Initialize(providerConf, configDir)
if err != nil {
t.Errorf("error initializing data provider")
}
httpd.SetDataProvider(dataprovider.GetProvider())
sftpd.SetDataProvider(dataprovider.GetProvider())
}
func TestLoaddata(t *testing.T) {
@ -1228,16 +1337,23 @@ func TestBasicWebUsersMock(t *testing.T) {
checkResponseCode(t, http.StatusBadRequest, rr.Code)
form := make(url.Values)
form.Set("username", user.Username)
req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
b, contentType, _ := getMultipartFormData(form, "", "")
req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
req, _ = http.NewRequest(http.MethodPost, webUserPath+"/"+strconv.FormatInt(user.ID, 10), strings.NewReader(form.Encode()))
b, contentType, _ = getMultipartFormData(form, "", "")
req, _ = http.NewRequest(http.MethodPost, webUserPath+"/"+strconv.FormatInt(user.ID, 10), &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
req, _ = http.NewRequest(http.MethodPost, webUserPath+"/0", strings.NewReader(form.Encode()))
b, contentType, _ = getMultipartFormData(form, "", "")
req, _ = http.NewRequest(http.MethodPost, webUserPath+"/0", &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusNotFound, rr.Code)
req, _ = http.NewRequest(http.MethodPost, webUserPath+"/a", strings.NewReader(form.Encode()))
req, _ = http.NewRequest(http.MethodPost, webUserPath+"/a", &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusBadRequest, rr.Code)
req, _ = http.NewRequest(http.MethodDelete, userPath+"/"+strconv.FormatInt(user.ID, 10), nil)
@ -1261,90 +1377,103 @@ func TestWebUserAddMock(t *testing.T) {
form.Set("expiration_date", "")
form.Set("permissions", "*")
form.Set("sub_dirs_permissions", " /subdir:list ,download ")
b, contentType, _ := getMultipartFormData(form, "", "")
// test invalid url escape
req, _ := http.NewRequest(http.MethodPost, webUserPath+"?a=%2", strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req, _ := http.NewRequest(http.MethodPost, webUserPath+"?a=%2", &b)
req.Header.Set("Content-Type", contentType)
rr := executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
form.Set("public_keys", testPubKey)
form.Set("uid", strconv.FormatInt(int64(user.UID), 10))
form.Set("gid", "a")
b, contentType, _ = getMultipartFormData(form, "", "")
// test invalid gid
req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
form.Set("gid", "0")
form.Set("max_sessions", "a")
b, contentType, _ = getMultipartFormData(form, "", "")
// test invalid max sessions
req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
form.Set("max_sessions", "0")
form.Set("quota_size", "a")
b, contentType, _ = getMultipartFormData(form, "", "")
// test invalid quota size
req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
form.Set("quota_size", "0")
form.Set("quota_files", "a")
b, contentType, _ = getMultipartFormData(form, "", "")
// test invalid quota files
req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
form.Set("quota_files", "0")
form.Set("upload_bandwidth", "a")
b, contentType, _ = getMultipartFormData(form, "", "")
// test invalid upload bandwidth
req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
form.Set("upload_bandwidth", strconv.FormatInt(user.UploadBandwidth, 10))
form.Set("download_bandwidth", "a")
b, contentType, _ = getMultipartFormData(form, "", "")
// test invalid download bandwidth
req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
form.Set("download_bandwidth", strconv.FormatInt(user.DownloadBandwidth, 10))
form.Set("status", "a")
b, contentType, _ = getMultipartFormData(form, "", "")
// test invalid status
req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
form.Set("status", strconv.Itoa(user.Status))
form.Set("expiration_date", "123")
b, contentType, _ = getMultipartFormData(form, "", "")
// test invalid expiration date
req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
form.Set("expiration_date", "")
form.Set("allowed_ip", "invalid,ip")
b, contentType, _ = getMultipartFormData(form, "", "")
// test invalid allowed_ip
req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
form.Set("allowed_ip", "")
form.Set("denied_ip", "192.168.1.2") // it should be 192.168.1.2/32
b, contentType, _ = getMultipartFormData(form, "", "")
// test invalid denied_ip
req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
form.Set("denied_ip", "")
req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
b, contentType, _ = getMultipartFormData(form, "", "")
req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusSeeOther, rr.Code)
// the user already exists, was created with the above request
req, _ = http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
b, contentType, _ = getMultipartFormData(form, "", "")
req, _ = http.NewRequest(http.MethodPost, webUserPath, &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
req, _ = http.NewRequest(http.MethodGet, userPath+"?limit=1&offset=0&order=ASC&username="+user.Username, nil)
@ -1356,7 +1485,7 @@ func TestWebUserAddMock(t *testing.T) {
t.Errorf("Error decoding users: %v", err)
}
if len(users) != 1 {
t.Errorf("1 user is expected")
t.Errorf("1 user is expected, actual: %v", len(users))
}
newUser := users[0]
if newUser.UID != user.UID {
@ -1413,8 +1542,9 @@ func TestWebUserUpdateMock(t *testing.T) {
form.Set("expiration_date", "2020-01-01 00:00:00")
form.Set("allowed_ip", " 192.168.1.3/32, 192.168.2.0/24 ")
form.Set("denied_ip", " 10.0.0.2/32 ")
req, _ = http.NewRequest(http.MethodPost, webUserPath+"/"+strconv.FormatInt(user.ID, 10), strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
b, contentType, _ := getMultipartFormData(form, "", "")
req, _ = http.NewRequest(http.MethodPost, webUserPath+"/"+strconv.FormatInt(user.ID, 10), &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusSeeOther, rr.Code)
req, _ = http.NewRequest(http.MethodGet, userPath+"?limit=1&offset=0&order=ASC&username="+user.Username, nil)
@ -1504,8 +1634,9 @@ func TestWebUserS3Mock(t *testing.T) {
form.Set("s3_storage_class", user.FsConfig.S3Config.StorageClass)
form.Set("s3_endpoint", user.FsConfig.S3Config.Endpoint)
form.Set("s3_key_prefix", user.FsConfig.S3Config.KeyPrefix)
req, _ = http.NewRequest(http.MethodPost, webUserPath+"/"+strconv.FormatInt(user.ID, 10), strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
b, contentType, _ := getMultipartFormData(form, "", "")
req, _ = http.NewRequest(http.MethodPost, webUserPath+"/"+strconv.FormatInt(user.ID, 10), &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusSeeOther, rr.Code)
req, _ = http.NewRequest(http.MethodGet, userPath+"?limit=1&offset=0&order=ASC&username="+user.Username, nil)
@ -1552,6 +1683,97 @@ func TestWebUserS3Mock(t *testing.T) {
checkResponseCode(t, http.StatusOK, rr.Code)
}
func TestWebUserGCSMock(t *testing.T) {
user := getTestUser()
userAsJSON := getUserAsJSON(t, user)
req, _ := http.NewRequest(http.MethodPost, userPath, bytes.NewBuffer(userAsJSON))
rr := executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
err := render.DecodeJSON(rr.Body, &user)
if err != nil {
t.Errorf("Error get user: %v", err)
}
credentialsFilePath := filepath.Join(os.TempDir(), "gcs.json")
err = createTestFile(credentialsFilePath, 0)
if err != nil {
t.Errorf("unable to create credential test file: %v", err)
}
user.FsConfig.Provider = 2
user.FsConfig.GCSConfig.Bucket = "test"
user.FsConfig.GCSConfig.KeyPrefix = "somedir/subdir/"
user.FsConfig.GCSConfig.StorageClass = "standard"
form := make(url.Values)
form.Set("username", user.Username)
form.Set("home_dir", user.HomeDir)
form.Set("uid", "0")
form.Set("gid", strconv.FormatInt(int64(user.GID), 10))
form.Set("max_sessions", strconv.FormatInt(int64(user.MaxSessions), 10))
form.Set("quota_size", strconv.FormatInt(user.QuotaSize, 10))
form.Set("quota_files", strconv.FormatInt(int64(user.QuotaFiles), 10))
form.Set("upload_bandwidth", "0")
form.Set("download_bandwidth", "0")
form.Set("permissions", "*")
form.Set("sub_dirs_permissions", "")
form.Set("status", strconv.Itoa(user.Status))
form.Set("expiration_date", "2020-01-01 00:00:00")
form.Set("allowed_ip", "")
form.Set("denied_ip", "")
form.Set("fs_provider", "2")
form.Set("gcs_bucket", user.FsConfig.GCSConfig.Bucket)
form.Set("gcs_storage_class", user.FsConfig.GCSConfig.StorageClass)
form.Set("gcs_key_prefix", user.FsConfig.GCSConfig.KeyPrefix)
b, contentType, _ := getMultipartFormData(form, "", "")
req, _ = http.NewRequest(http.MethodPost, webUserPath+"/"+strconv.FormatInt(user.ID, 10), &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
b, contentType, _ = getMultipartFormData(form, "gcs_credential_file", credentialsFilePath)
req, _ = http.NewRequest(http.MethodPost, webUserPath+"/"+strconv.FormatInt(user.ID, 10), &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
err = createTestFile(credentialsFilePath, 4096)
if err != nil {
t.Errorf("unable to create credential test file: %v", err)
}
b, contentType, _ = getMultipartFormData(form, "gcs_credential_file", credentialsFilePath)
req, _ = http.NewRequest(http.MethodPost, webUserPath+"/"+strconv.FormatInt(user.ID, 10), &b)
req.Header.Set("Content-Type", contentType)
rr = executeRequest(req)
checkResponseCode(t, http.StatusSeeOther, rr.Code)
req, _ = http.NewRequest(http.MethodGet, userPath+"?limit=1&offset=0&order=ASC&username="+user.Username, nil)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
var users []dataprovider.User
err = render.DecodeJSON(rr.Body, &users)
if err != nil {
t.Errorf("Error decoding users: %v", err)
}
if len(users) != 1 {
t.Errorf("1 user is expected")
}
updateUser := users[0]
if updateUser.ExpirationDate != 1577836800000 {
t.Errorf("invalid expiration date: %v", updateUser.ExpirationDate)
}
if updateUser.FsConfig.Provider != user.FsConfig.Provider {
t.Error("fs provider mismatch")
}
if updateUser.FsConfig.GCSConfig.Bucket != user.FsConfig.GCSConfig.Bucket {
t.Error("GCS bucket mismatch")
}
if updateUser.FsConfig.GCSConfig.StorageClass != user.FsConfig.GCSConfig.StorageClass {
t.Error("GCS storage class mismatch")
}
if updateUser.FsConfig.GCSConfig.KeyPrefix != user.FsConfig.GCSConfig.KeyPrefix {
t.Error("GCS key prefix mismatch")
}
req, _ = http.NewRequest(http.MethodDelete, userPath+"/"+strconv.FormatInt(user.ID, 10), nil)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
os.Remove(credentialsFilePath)
}
func TestProviderClosedMock(t *testing.T) {
if providerDriverName == dataprovider.BoltDataProviderName {
t.Skip("skipping test provider errors for bolt provider")
@ -1571,6 +1793,8 @@ func TestProviderClosedMock(t *testing.T) {
checkResponseCode(t, http.StatusInternalServerError, rr.Code)
config.LoadConfig(configDir, "")
providerConf := config.GetProviderConf()
providerConf.CredentialsPath = credentialsPath
os.RemoveAll(credentialsPath)
err := dataprovider.Initialize(providerConf, configDir)
if err != nil {
t.Errorf("error initializing data provider")
@ -1644,9 +1868,38 @@ func createTestFile(path string, size int64) error {
os.MkdirAll(baseDir, 0777)
}
content := make([]byte, size)
_, err := rand.Read(content)
if err != nil {
return err
if size > 0 {
_, err := rand.Read(content)
if err != nil {
return err
}
}
return ioutil.WriteFile(path, content, 0666)
}
func getMultipartFormData(values url.Values, fileFieldName, filePath string) (bytes.Buffer, string, error) {
var b bytes.Buffer
w := multipart.NewWriter(&b)
for k, v := range values {
for _, s := range v {
if err := w.WriteField(k, s); err != nil {
return b, "", err
}
}
}
if len(fileFieldName) > 0 && len(filePath) > 0 {
fw, err := w.CreateFormFile(fileFieldName, filepath.Base(filePath))
if err != nil {
return b, "", err
}
f, err := os.Open(filePath)
if err != nil {
return b, "", err
}
if _, err = io.Copy(fw, f); err != nil {
return b, "", err
}
}
err := w.Close()
return b, w.FormDataContentType(), err
}

View file

@ -6,7 +6,9 @@ import (
"html/template"
"net/http"
"net/http/httptest"
"net/url"
"os"
"strings"
"testing"
"github.com/drakkan/sftpgo/dataprovider"
@ -280,6 +282,38 @@ func TestCompareUserFsConfig(t *testing.T) {
if err == nil {
t.Errorf("S3 key prefix does not match")
}
expected.FsConfig.S3Config.KeyPrefix = ""
expected.FsConfig.GCSConfig.KeyPrefix = "somedir/subdir"
err = compareUserFsConfig(expected, actual)
if err == nil {
t.Errorf("GCS key prefix does not match")
}
expected.FsConfig.GCSConfig.KeyPrefix = ""
expected.FsConfig.GCSConfig.Bucket = "bucket"
err = compareUserFsConfig(expected, actual)
if err == nil {
t.Errorf("GCS bucket does not match")
}
expected.FsConfig.GCSConfig.Bucket = ""
expected.FsConfig.GCSConfig.StorageClass = "Standard"
err = compareUserFsConfig(expected, actual)
if err == nil {
t.Errorf("GCS storage class does not match")
}
expected.FsConfig.GCSConfig.StorageClass = ""
}
func TestGCSWebInvalidFormFile(t *testing.T) {
form := make(url.Values)
form.Set("username", "test_username")
form.Set("fs_provider", "2")
req, _ := http.NewRequest(http.MethodPost, webUserPath, strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req.ParseForm()
_, err := getFsConfigFromUserPostFields(req)
if err != http.ErrNotMultipart {
t.Errorf("unexpected error: %v", err)
}
}
func TestApiCallsWithBadURL(t *testing.T) {

View file

@ -2,7 +2,7 @@ openapi: 3.0.1
info:
title: SFTPGo
description: 'SFTPGo REST API'
version: 1.5.0
version: 1.6.0
servers:
- url: /api/v1
@ -743,6 +743,26 @@ components:
- access_secret
nullable: true
description: S3 Compatible Object Storage configuration details
GCSConfig:
type: object
properties:
bucket:
type: string
minLength: 1
credentials:
type: string
format: byte
description: Google Cloud Storage JSON credentials base64 encoded. This field must be populated only when adding/updating an user. It will be always omitted, since there are sensitive data, when you search/get users. The credentials will be stored in the configured "credentials_path"
storage_class:
type: string
key_prefix:
type: string
description: key_prefix is similar to a chroot directory for a local filesystem. If specified the SFTP user will only see contents that starts with this prefix and so you can restrict access to a specific virtual folder. The prefix, if not empty, must not start with "/" and must end with "/". If empty the whole bucket contents will be available
example: folder/subfolder/
required:
- bucket
nullable: true
description: Google Cloud Storage configuration details
FilesystemConfig:
type: object
properties:
@ -751,12 +771,16 @@ components:
enum:
- 0
- 1
- 2
description: >
Providers:
* `0` - local filesystem
* `1` - S3 Compatible Object Storage
* `2` - Google Cloud Storage
s3config:
$ref: '#/components/schemas/S3Config'
gcsconfig:
$ref: '#/components/schemas/GCSConfig'
description: Storage filesystem details
User:
type: object

View file

@ -1,8 +1,11 @@
package httpd
import (
"encoding/base64"
"errors"
"fmt"
"html/template"
"io/ioutil"
"net/http"
"path/filepath"
"strconv"
@ -224,7 +227,7 @@ func getFiltersFromUserPostFields(r *http.Request) dataprovider.UserFilters {
return filters
}
func getFsConfigFromUserPostFields(r *http.Request) dataprovider.Filesystem {
func getFsConfigFromUserPostFields(r *http.Request) (dataprovider.Filesystem, error) {
var fs dataprovider.Filesystem
provider, err := strconv.Atoi(r.Form.Get("fs_provider"))
if err != nil {
@ -239,13 +242,33 @@ func getFsConfigFromUserPostFields(r *http.Request) dataprovider.Filesystem {
fs.S3Config.Endpoint = r.Form.Get("s3_endpoint")
fs.S3Config.StorageClass = r.Form.Get("s3_storage_class")
fs.S3Config.KeyPrefix = r.Form.Get("s3_key_prefix")
} else if fs.Provider == 2 {
fs.GCSConfig.Bucket = r.Form.Get("gcs_bucket")
fs.GCSConfig.StorageClass = r.Form.Get("gcs_storage_class")
fs.GCSConfig.KeyPrefix = r.Form.Get("gcs_key_prefix")
credentials, _, err := r.FormFile("gcs_credential_file")
if err == http.ErrMissingFile {
return fs, nil
}
if err != nil {
return fs, err
}
defer credentials.Close()
fileBytes, err := ioutil.ReadAll(credentials)
if err != nil || len(fileBytes) == 0 {
if len(fileBytes) == 0 {
err = errors.New("credentials file size must be greater than 0")
}
return fs, err
}
fs.GCSConfig.Credentials = base64.StdEncoding.EncodeToString(fileBytes)
}
return fs
return fs, nil
}
func getUserFromPostFields(r *http.Request) (dataprovider.User, error) {
var user dataprovider.User
err := r.ParseForm()
err := r.ParseMultipartForm(maxRequestSize)
if err != nil {
return user, err
}
@ -292,6 +315,10 @@ func getUserFromPostFields(r *http.Request) (dataprovider.User, error) {
}
expirationDateMillis = utils.GetTimeAsMsSinceEpoch(expirationDate)
}
fsConfig, err := getFsConfigFromUserPostFields(r)
if err != nil {
return user, err
}
user = dataprovider.User{
Username: r.Form.Get("username"),
Password: r.Form.Get("password"),
@ -308,7 +335,7 @@ func getUserFromPostFields(r *http.Request) (dataprovider.User, error) {
Status: status,
ExpirationDate: expirationDateMillis,
Filters: getFiltersFromUserPostFields(r),
FsConfig: getFsConfigFromUserPostFields(r),
FsConfig: fsConfig,
}
return user, err
}
@ -365,6 +392,7 @@ func handleWebUpdateUserGet(userID string, w http.ResponseWriter, r *http.Reques
}
func handleWebAddUserPost(w http.ResponseWriter, r *http.Request) {
r.Body = http.MaxBytesReader(w, r.Body, maxRequestSize)
user, err := getUserFromPostFields(r)
if err != nil {
renderAddUserPage(w, user, err.Error())
@ -379,6 +407,7 @@ func handleWebAddUserPost(w http.ResponseWriter, r *http.Request) {
}
func handleWebUpdateUserPost(userID string, w http.ResponseWriter, r *http.Request) {
r.Body = http.MaxBytesReader(w, r.Body, maxRequestSize)
id, err := strconv.ParseInt(userID, 10, 64)
if err != nil {
renderBadRequestPage(w, err)

View file

@ -246,22 +246,94 @@ var (
Help: "The total number of successful S3 head bucket requests",
})
// totalS3CreateBucket is the metric that reports the total successful S3 create bucket requests
totalS3CreateBucket = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_s3_create_bucket",
Help: "The total number of successful S3 create bucket requests",
})
// totalS3HeadBucketErrors is the metric that reports the total S3 head bucket errors
totalS3HeadBucketErrors = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_s3_head_bucket_errors",
Help: "The total number of S3 head bucket errors",
})
// totalS3CreateBucketErrors is the metric that reports the total S3 create bucket errors
totalS3CreateBucketErrors = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_s3_create_bucket_errors",
Help: "The total number of S3 create bucket errors",
// totalGCSUploads is the metric that reports the total number of successful GCS uploads
totalGCSUploads = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_gcs_uploads_total",
Help: "The total number of successful GCS uploads",
})
// totalGCSDownloads is the metric that reports the total number of successful GCS downloads
totalGCSDownloads = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_gcs_downloads_total",
Help: "The total number of successful GCS downloads",
})
// totalGCSUploadErrors is the metric that reports the total number of GCS upload errors
totalGCSUploadErrors = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_gcs_upload_errors_total",
Help: "The total number of GCS upload errors",
})
// totalGCSDownloadErrors is the metric that reports the total number of GCS download errors
totalGCSDownloadErrors = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_gcs_download_errors_total",
Help: "The total number of GCS download errors",
})
// totalGCSUploadSize is the metric that reports the total GCS uploads size as bytes
totalGCSUploadSize = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_gcs_upload_size",
Help: "The total GCS upload size as bytes, partial uploads are included",
})
// totalGCSDownloadSize is the metric that reports the total GCS downloads size as bytes
totalGCSDownloadSize = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_gcs_download_size",
Help: "The total GCS download size as bytes, partial downloads are included",
})
// totalS3ListObjects is the metric that reports the total successful GCS list objects requests
totalGCSListObjects = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_gcs_list_objects",
Help: "The total number of successful GCS list objects requests",
})
// totalGCSCopyObject is the metric that reports the total successful GCS copy object requests
totalGCSCopyObject = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_gcs_copy_object",
Help: "The total number of successful GCS copy object requests",
})
// totalGCSDeleteObject is the metric that reports the total successful S3 delete object requests
totalGCSDeleteObject = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_gcs_delete_object",
Help: "The total number of successful GCS delete object requests",
})
// totalGCSListObjectsError is the metric that reports the total GCS list objects errors
totalGCSListObjectsErrors = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_gcs_list_objects_errors",
Help: "The total number of GCS list objects errors",
})
// totalGCSCopyObjectErrors is the metric that reports the total GCS copy object errors
totalGCSCopyObjectErrors = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_gcs_copy_object_errors",
Help: "The total number of GCS copy object errors",
})
// totalGCSDeleteObjectErrors is the metric that reports the total GCS delete object errors
totalGCSDeleteObjectErrors = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_gcs_delete_object_errors",
Help: "The total number of GCS delete object errors",
})
// totalGCSHeadBucket is the metric that reports the total successful GCS head bucket requests
totalGCSHeadBucket = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_gcs_head_bucket",
Help: "The total number of successful GCS head bucket requests",
})
// totalGCSHeadBucketErrors is the metric that reports the total GCS head bucket errors
totalGCSHeadBucketErrors = promauto.NewCounter(prometheus.CounterOpts{
Name: "sftpgo_gcs_head_bucket_errors",
Help: "The total number of GCS head bucket errors",
})
)
@ -343,12 +415,60 @@ func S3HeadBucketCompleted(err error) {
}
}
// S3CreateBucketCompleted updates metrics after an S3 create bucket request terminates
func S3CreateBucketCompleted(err error) {
if err == nil {
totalS3CreateBucket.Inc()
// GCSTransferCompleted updates metrics after a GCS upload or a download
func GCSTransferCompleted(bytes int64, transferKind int, err error) {
if transferKind == 0 {
// upload
if err == nil {
totalGCSUploads.Inc()
} else {
totalGCSUploadErrors.Inc()
}
totalGCSUploadSize.Add(float64(bytes))
} else {
totalS3CreateBucketErrors.Inc()
// download
if err == nil {
totalGCSDownloads.Inc()
} else {
totalGCSDownloadErrors.Inc()
}
totalGCSDownloadSize.Add(float64(bytes))
}
}
// GCSListObjectsCompleted updates metrics after a GCS list objects request terminates
func GCSListObjectsCompleted(err error) {
if err == nil {
totalGCSListObjects.Inc()
} else {
totalGCSListObjectsErrors.Inc()
}
}
// GCSCopyObjectCompleted updates metrics after a GCS copy object request terminates
func GCSCopyObjectCompleted(err error) {
if err == nil {
totalGCSCopyObject.Inc()
} else {
totalGCSCopyObjectErrors.Inc()
}
}
// GCSDeleteObjectCompleted updates metrics after a GCS delete object request terminates
func GCSDeleteObjectCompleted(err error) {
if err == nil {
totalGCSDeleteObject.Inc()
} else {
totalGCSDeleteObjectErrors.Inc()
}
}
// GCSHeadBucketCompleted updates metrics after a GCS head bucket request terminates
func GCSHeadBucketCompleted(err error) {
if err == nil {
totalGCSHeadBucket.Inc()
} else {
totalGCSHeadBucketErrors.Inc()
}
}

View file

@ -54,6 +54,7 @@ Output:
"download_bandwidth": 60,
"expiration_date": 1546297200000,
"filesystem": {
"gcsconfig": {},
"provider": 1,
"s3config": {
"access_key": "accesskey",
@ -139,6 +140,7 @@ Output:
"download_bandwidth": 80,
"expiration_date": 0,
"filesystem": {
"gcsconfig": {},
"provider": 0,
"s3config": {}
},
@ -191,6 +193,7 @@ Output:
"download_bandwidth": 80,
"expiration_date": 0,
"filesystem": {
"gcsconfig": {},
"provider": 0,
"s3config": {}
},

View file

@ -1,5 +1,6 @@
#!/usr/bin/env python
import argparse
import base64
from datetime import datetime
import json
import platform
@ -74,7 +75,7 @@ class SFTPGoApiRequests:
max_sessions=0, quota_size=0, quota_files=0, permissions={}, upload_bandwidth=0, download_bandwidth=0,
status=1, expiration_date=0, allowed_ip=[], denied_ip=[], fs_provider='local', s3_bucket='',
s3_region='', s3_access_key='', s3_access_secret='', s3_endpoint='', s3_storage_class='',
s3_key_prefix=''):
s3_key_prefix='', gcs_bucket='', gcs_key_prefix='', gcs_storage_class='', gcs_credentials_file=''):
user = {"id":user_id, "username":username, "uid":uid, "gid":gid,
"max_sessions":max_sessions, "quota_size":quota_size, "quota_files":quota_files,
"upload_bandwidth":upload_bandwidth, "download_bandwidth":download_bandwidth,
@ -92,9 +93,9 @@ class SFTPGoApiRequests:
user.update({"permissions":permissions})
if allowed_ip or denied_ip:
user.update({"filters":self.buildFilters(allowed_ip, denied_ip)})
user.update({"filesystem":self.buildFsConfig(fs_provider, s3_bucket, s3_region, s3_access_key,
s3_access_secret, s3_endpoint, s3_storage_class,
s3_key_prefix)})
user.update({"filesystem":self.buildFsConfig(fs_provider, s3_bucket, s3_region, s3_access_key, s3_access_secret,
s3_endpoint, s3_storage_class, s3_key_prefix, gcs_bucket,
gcs_key_prefix, gcs_storage_class, gcs_credentials_file)})
return user
def buildPermissions(self, root_perms, subdirs_perms):
@ -129,13 +130,19 @@ class SFTPGoApiRequests:
return filters
def buildFsConfig(self, fs_provider, s3_bucket, s3_region, s3_access_key, s3_access_secret, s3_endpoint,
s3_storage_class, s3_key_prefix):
s3_storage_class, s3_key_prefix, gcs_bucket, gcs_key_prefix, gcs_storage_class, gcs_credentials_file):
fs_config = {'provider':0}
if fs_provider == 'S3':
s3config = {'bucket':s3_bucket, 'region':s3_region, 'access_key':s3_access_key, 'access_secret':
s3_access_secret, 'endpoint':s3_endpoint, 'storage_class':s3_storage_class, 'key_prefix':
s3_key_prefix}
fs_config.update({'provider':1, 's3config':s3config})
elif fs_provider == 'GCS':
gcsconfig = {'bucket':gcs_bucket, 'key_prefix':gcs_key_prefix, 'storage_class':gcs_storage_class}
if gcs_credentials_file:
with open(gcs_credentials_file) as creds:
gcsconfig.update({'credentials':base64.b64encode(creds.read().encode('UTF-8')).decode('UTF-8')})
fs_config.update({'provider':2, 'gcsconfig':gcsconfig})
return fs_config
def getUsers(self, limit=100, offset=0, order="ASC", username=""):
@ -150,11 +157,13 @@ class SFTPGoApiRequests:
def addUser(self, username="", password="", public_keys="", home_dir="", uid=0, gid=0, max_sessions=0, quota_size=0,
quota_files=0, perms=[], upload_bandwidth=0, download_bandwidth=0, status=1, expiration_date=0,
subdirs_permissions=[], allowed_ip=[], denied_ip=[], fs_provider='local', s3_bucket='', s3_region='',
s3_access_key='', s3_access_secret='', s3_endpoint='', s3_storage_class='', s3_key_prefix=''):
s3_access_key='', s3_access_secret='', s3_endpoint='', s3_storage_class='', s3_key_prefix='', gcs_bucket='',
gcs_key_prefix='', gcs_storage_class='', gcs_credentials_file=''):
u = self.buildUserObject(0, username, password, public_keys, home_dir, uid, gid, max_sessions,
quota_size, quota_files, self.buildPermissions(perms, subdirs_permissions), upload_bandwidth, download_bandwidth,
status, expiration_date, allowed_ip, denied_ip, fs_provider, s3_bucket, s3_region,
s3_access_key, s3_access_secret, s3_endpoint, s3_storage_class, s3_key_prefix)
status, expiration_date, allowed_ip, denied_ip, fs_provider, s3_bucket, s3_region, s3_access_key,
s3_access_secret, s3_endpoint, s3_storage_class, s3_key_prefix, gcs_bucket, gcs_key_prefix, gcs_storage_class,
gcs_credentials_file)
r = requests.post(self.userPath, json=u, auth=self.auth, verify=self.verify)
self.printResponse(r)
@ -162,11 +171,12 @@ class SFTPGoApiRequests:
quota_size=0, quota_files=0, perms=[], upload_bandwidth=0, download_bandwidth=0, status=1,
expiration_date=0, subdirs_permissions=[], allowed_ip=[], denied_ip=[], fs_provider='local',
s3_bucket='', s3_region='', s3_access_key='', s3_access_secret='', s3_endpoint='', s3_storage_class='',
s3_key_prefix=''):
s3_key_prefix='', gcs_bucket='', gcs_key_prefix='', gcs_storage_class='', gcs_credentials_file=''):
u = self.buildUserObject(user_id, username, password, public_keys, home_dir, uid, gid, max_sessions,
quota_size, quota_files, self.buildPermissions(perms, subdirs_permissions), upload_bandwidth, download_bandwidth,
status, expiration_date, allowed_ip, denied_ip, fs_provider, s3_bucket, s3_region, s3_access_key,
s3_access_secret, s3_endpoint, s3_storage_class, s3_key_prefix)
s3_access_secret, s3_endpoint, s3_storage_class, s3_key_prefix, gcs_bucket, gcs_key_prefix, gcs_storage_class,
gcs_credentials_file)
r = requests.put(urlparse.urljoin(self.userPath, "user/" + str(user_id)), json=u, auth=self.auth, verify=self.verify)
self.printResponse(r)
@ -420,7 +430,7 @@ def addCommonUserArguments(parser):
help='Allowed IP/Mask in CIDR notation. For example "192.168.2.0/24" or "2001:db8::/32". Default: %(default)s')
parser.add_argument('-N', '--denied-ip', type=str, nargs='+', default=[],
help='Denied IP/Mask in CIDR notation. For example "192.168.2.0/24" or "2001:db8::/32". Default: %(default)s')
parser.add_argument('--fs', type=str, default='local', choices=['local', 'S3'],
parser.add_argument('--fs', type=str, default='local', choices=['local', 'S3', 'GCS'],
help='Filesystem provider. Default: %(default)s')
parser.add_argument('--s3-bucket', type=str, default='', help='Default: %(default)s')
parser.add_argument('--s3-key-prefix', type=str, default='', help='Virtual root directory. If non empty only this ' +
@ -431,6 +441,12 @@ def addCommonUserArguments(parser):
parser.add_argument('--s3-access-secret', type=str, default='', help='Default: %(default)s')
parser.add_argument('--s3-endpoint', type=str, default='', help='Default: %(default)s')
parser.add_argument('--s3-storage-class', type=str, default='', help='Default: %(default)s')
parser.add_argument('--gcs-bucket', type=str, default='', help='Default: %(default)s')
parser.add_argument('--gcs-key-prefix', type=str, default='', help='Virtual root directory. If non empty only this ' +
'directory and its contents will be available. Cannot start with "/". For example "folder/subfolder/".' +
' Default: %(default)s')
parser.add_argument('--gcs-storage-class', type=str, default='', help='Default: %(default)s')
parser.add_argument('--gcs-credentials-file', type=str, default='', help='Default: %(default)s')
if __name__ == '__main__':
@ -534,14 +550,16 @@ if __name__ == '__main__':
args.quota_size, args.quota_files, args.permissions, args.upload_bandwidth, args.download_bandwidth,
args.status, getDatetimeAsMillisSinceEpoch(args.expiration_date), args.subdirs_permissions, args.allowed_ip,
args.denied_ip, args.fs, args.s3_bucket, args.s3_region, args.s3_access_key, args.s3_access_secret,
args.s3_endpoint, args.s3_storage_class, args.s3_key_prefix)
args.s3_endpoint, args.s3_storage_class, args.s3_key_prefix, args.gcs_bucket, args.gcs_key_prefix,
args.gcs_storage_class, args.gcs_credentials_file)
elif args.command == 'update-user':
api.updateUser(args.id, args.username, args.password, args.public_keys, args.home_dir, args.uid, args.gid,
args.max_sessions, args.quota_size, args.quota_files, args.permissions, args.upload_bandwidth,
args.download_bandwidth, args.status, getDatetimeAsMillisSinceEpoch(args.expiration_date),
args.subdirs_permissions, args.allowed_ip, args.denied_ip, args.fs, args.s3_bucket, args.s3_region,
args.s3_access_key, args.s3_access_secret, args.s3_endpoint, args.s3_storage_class,
args.s3_key_prefix)
args.s3_key_prefix, args.gcs_bucket, args.gcs_key_prefix, args.gcs_storage_class,
args.gcs_credentials_file)
elif args.command == 'delete-user':
api.deleteUser(args.id)
elif args.command == 'get-users':

View file

@ -6,6 +6,7 @@ import (
"math/rand"
"os"
"os/signal"
"path/filepath"
"strings"
"syscall"
"time"
@ -146,6 +147,7 @@ func (s *Service) StartPortableMode(sftpdPort int, enabledSSHCommands []string,
}
dataProviderConf := config.GetProviderConf()
dataProviderConf.Driver = dataprovider.MemoryDataProviderName
dataProviderConf.CredentialsPath = filepath.Join(os.TempDir(), "credentials")
config.SetProviderConf(dataProviderConf)
httpdConf := config.GetHTTPDConfig()
httpdConf.BindPort = 0

View file

@ -1053,7 +1053,7 @@ func TestLoginInvalidFs(t *testing.T) {
t.Errorf("unable to add user: %v", err)
}
// we update the database using sqlite3 CLI since we cannot add an user with an invalid config
time.Sleep(150 * time.Millisecond)
time.Sleep(200 * time.Millisecond)
updateUserQuery := fmt.Sprintf("UPDATE users SET filesystem='{\"provider\":1}' WHERE id=%v", user.ID)
cmd := exec.Command("sqlite3", dbPath, updateUserQuery)
out, err := cmd.CombinedOutput()
@ -3039,7 +3039,11 @@ func TestRelativePaths(t *testing.T) {
KeyPrefix: strings.TrimPrefix(user.GetHomeDir(), "/") + "/",
}
s3fs, _ := vfs.NewS3Fs("", user.GetHomeDir(), s3config)
filesystems = append(filesystems, s3fs)
gcsConfig := vfs.GCSFsConfig{
KeyPrefix: strings.TrimPrefix(user.GetHomeDir(), "/") + "/",
}
gcsfs, _ := vfs.NewGCSFs("", user.GetHomeDir(), gcsConfig)
filesystems = append(filesystems, s3fs, gcsfs)
for _, fs := range filesystems {
path = filepath.Join(user.HomeDir, "/")
rel = fs.GetRelativePath(path)
@ -3104,7 +3108,11 @@ func TestResolvePaths(t *testing.T) {
}
os.MkdirAll(user.GetHomeDir(), 0777)
s3fs, _ := vfs.NewS3Fs("", user.GetHomeDir(), s3config)
filesystems = append(filesystems, s3fs)
gcsConfig := vfs.GCSFsConfig{
KeyPrefix: strings.TrimPrefix(user.GetHomeDir(), "/") + "/",
}
gcsfs, _ := vfs.NewGCSFs("", user.GetHomeDir(), gcsConfig)
filesystems = append(filesystems, s3fs, gcsfs)
for _, fs := range filesystems {
path = "/"
resolved, _ = fs.ResolvePath(filepath.ToSlash(path))
@ -3509,7 +3517,8 @@ func TestSCPBasicHandling(t *testing.T) {
t.Errorf("stat for the downloaded file must succeed")
} else {
if fi.Size() != testFileSize {
t.Errorf("size of the file downloaded via SCP does not match the expected one")
t.Errorf("size of the file downloaded via SCP does not match the expected one: %v/%v",
fi.Size(), testFileSize)
}
}
os.Remove(localPath)
@ -3582,7 +3591,8 @@ func TestSCPUploadFileOverwrite(t *testing.T) {
t.Errorf("stat for the downloaded file must succeed")
} else {
if fi.Size() != testFileSize {
t.Errorf("size of the file downloaded via SCP does not match the expected one")
t.Errorf("size of the file downloaded via SCP does not match the expected one: %v/%v",
fi.Size(), testFileSize)
}
}
os.Remove(localPath)

View file

@ -42,7 +42,8 @@
"http_notification_url": ""
},
"external_auth_program": "",
"external_auth_scope": 0
"external_auth_scope": 0,
"credentials_path": "credentials"
},
"httpd": {
"bind_port": 8080,

View file

@ -15,7 +15,7 @@
<div class="card-body text-form-error">{{.Error}}</div>
</div>
{{end}}
<form id="user_form" action="{{.CurrentURL}}" method="POST" autocomplete="off">
<form id="user_form" enctype="multipart/form-data" action="{{.CurrentURL}}" method="POST" autocomplete="off">
<div class="form-group row">
<label for="idUsername" class="col-sm-2 col-form-label">Username</label>
<div class="col-sm-10">
@ -194,14 +194,15 @@
<div class="form-group row">
<label for="idFilesystem" class="col-sm-2 col-form-label">Storage</label>
<div class="col-sm-10">
<select class="form-control" id="idFilesystem" name="fs_provider">
<select class="form-control" id="idFilesystem" name="fs_provider" onchange="onFilesystemChanged(this.value)">
<option value="0" {{if eq .User.FsConfig.Provider 0 }}selected{{end}}>local</option>
<option value="1" {{if eq .User.FsConfig.Provider 1 }}selected{{end}}>S3</option>
<option value="1" {{if eq .User.FsConfig.Provider 1 }}selected{{end}}>Amazon S3 (Compatible)</option>
<option value="2" {{if eq .User.FsConfig.Provider 2 }}selected{{end}}>Google Cloud Storage</option>
</select>
</div>
</div>
<div class="form-group row">
<div class="form-group row s3">
<label for="idS3Bucket" class="col-sm-2 col-form-label">S3 Bucket</label>
<div class="col-sm-3">
<input type="text" class="form-control" id="idS3Bucket" name="s3_bucket" placeholder=""
@ -215,7 +216,7 @@
</div>
</div>
<div class="form-group row">
<div class="form-group row s3">
<label for="idS3AccessKey" class="col-sm-2 col-form-label">S3 Access Key</label>
<div class="col-sm-3">
<input type="text" class="form-control" id="idS3AccessKey" name="s3_access_key" placeholder=""
@ -229,7 +230,7 @@
</div>
</div>
<div class="form-group row">
<div class="form-group row s3">
<label for="idS3StorageClass" class="col-sm-2 col-form-label">S3 Storage Class</label>
<div class="col-sm-3">
<input type="text" class="form-control" id="idS3StorageClass" name="s3_storage_class" placeholder=""
@ -243,7 +244,7 @@
</div>
</div>
<div class="form-group row">
<div class="form-group row s3">
<label for="idS3KeyPrefix" class="col-sm-2 col-form-label">S3 Key Prefix</label>
<div class="col-sm-10">
<input type="text" class="form-control" id="idS3KeyPrefix" name="s3_key_prefix" placeholder=""
@ -254,6 +255,43 @@
</div>
</div>
<div class="form-group row gcs">
<label for="idGCSBucket" class="col-sm-2 col-form-label">GCS Bucket</label>
<div class="col-sm-10">
<input type="text" class="form-control" id="idGCSBucket" name="gcs_bucket" placeholder=""
value="{{.User.FsConfig.GCSConfig.Bucket}}" maxlength="255">
</div>
</div>
<div class="form-group row gcs">
<label for="idGCSCredentialFile" class="col-sm-2 col-form-label">GCS Credential file</label>
<div class="col-sm-4">
<input type="file" class="form-control-file" id="idGCSCredentialFile" name="gcs_credential_file"
aria-describedby="GCSCredentialsHelpBlock">
<small id="GCSCredentialsHelpBlock" class="form-text text-muted">
Add or update credentials from a JSON file
</small>
</div>
<div class="col-sm-1"></div>
<label for="idGCSStorageClass" class="col-sm-2 col-form-label">GCS Storage Class</label>
<div class="col-sm-3">
<input type="text" class="form-control" id="idGCSStorageClass" name="gcs_storage_class" placeholder=""
value="{{.User.FsConfig.GCSConfig.StorageClass}}" maxlength="255">
</div>
</div>
<div class="form-group row gcs">
<label for="idGCSKeyPrefix" class="col-sm-2 col-form-label">GCS Key Prefix</label>
<div class="col-sm-10">
<input type="text" class="form-control" id="idGCSKeyPrefix" name="gcs_key_prefix" placeholder=""
value="{{.User.FsConfig.GCSConfig.KeyPrefix}}" maxlength="255" aria-describedby="GCSKeyPrefixHelpBlock">
<small id="GCSKeyPrefixHelpBlock" class="form-text text-muted">
Similar to a chroot for local filesystem. Cannot start with "/". Example: "somedir/subdir/".
</small>
</div>
</div>
<input type="hidden" name="expiration_date" id="hidden_start_datetime" value="">
<button type="submit" class="btn btn-primary float-right mt-3 mb-5 px-5 px-3">Submit</button>
</form>
@ -295,6 +333,22 @@
}
return true;
});
onFilesystemChanged('{{.User.FsConfig.Provider}}');
});
function onFilesystemChanged(val){
if (val == '1'){
$('.form-group.row.gcs').hide();
$('.form-group.row.s3').show();
} else if (val == '2'){
$('.form-group.row.gcs').show();
$('.form-group.row.s3').hide();
} else {
$('.form-group.row.gcs').hide();
$('.form-group.row.s3').hide();
}
}
</script>
{{end}}

View file

@ -5,8 +5,8 @@ import (
"time"
)
// S3FileInfo implements os.FileInfo for a file in S3.
type S3FileInfo struct {
// FileInfo implements os.FileInfo for a file in S3.
type FileInfo struct {
name string
sizeInBytes int64
modTime time.Time
@ -14,14 +14,14 @@ type S3FileInfo struct {
sys interface{}
}
// NewS3FileInfo creates file info.
func NewS3FileInfo(name string, isDirectory bool, sizeInBytes int64, modTime time.Time) S3FileInfo {
// NewFileInfo creates file info.
func NewFileInfo(name string, isDirectory bool, sizeInBytes int64, modTime time.Time) FileInfo {
mode := os.FileMode(0644)
if isDirectory {
mode = os.FileMode(0755) | os.ModeDir
}
return S3FileInfo{
return FileInfo{
name: name,
sizeInBytes: sizeInBytes,
modTime: modTime,
@ -30,31 +30,31 @@ func NewS3FileInfo(name string, isDirectory bool, sizeInBytes int64, modTime tim
}
// Name provides the base name of the file.
func (fi S3FileInfo) Name() string {
func (fi FileInfo) Name() string {
return fi.name
}
// Size provides the length in bytes for a file.
func (fi S3FileInfo) Size() int64 {
func (fi FileInfo) Size() int64 {
return fi.sizeInBytes
}
// Mode provides the file mode bits
func (fi S3FileInfo) Mode() os.FileMode {
func (fi FileInfo) Mode() os.FileMode {
return fi.mode
}
// ModTime provides the last modification time.
func (fi S3FileInfo) ModTime() time.Time {
func (fi FileInfo) ModTime() time.Time {
return fi.modTime
}
// IsDir provides the abbreviation for Mode().IsDir()
func (fi S3FileInfo) IsDir() bool {
func (fi FileInfo) IsDir() bool {
return fi.mode&os.ModeDir != 0
}
// Sys provides the underlying data source (can return nil)
func (fi S3FileInfo) Sys() interface{} {
func (fi FileInfo) Sys() interface{} {
return fi.getFileInfoSys()
}

View file

@ -21,7 +21,7 @@ func init() {
}
}
func (fi S3FileInfo) getFileInfoSys() interface{} {
func (fi FileInfo) getFileInfoSys() interface{} {
return &syscall.Stat_t{
Uid: uint32(defaultUID),
Gid: uint32(defaultGID)}

View file

@ -2,6 +2,6 @@ package vfs
import "syscall"
func (fi S3FileInfo) getFileInfoSys() interface{} {
func (fi FileInfo) getFileInfoSys() interface{} {
return syscall.Win32FileAttributeData{}
}

511
vfs/gcsfs.go Normal file
View file

@ -0,0 +1,511 @@
package vfs
import (
"context"
"errors"
"fmt"
"io"
"net/http"
"os"
"path"
"strings"
"time"
"cloud.google.com/go/storage"
"github.com/drakkan/sftpgo/logger"
"github.com/drakkan/sftpgo/metrics"
"github.com/eikenb/pipeat"
"google.golang.org/api/googleapi"
"google.golang.org/api/iterator"
"google.golang.org/api/option"
)
var (
// we cannot use attrs selection until this bug is fixed:
//
// https://github.com/googleapis/google-cloud-go/issues/1763
//
gcsDefaultFieldsSelection = []string{"Name", "Size", "Deleted", "Updated"}
)
// GCSFsConfig defines the configuration for Google Cloud Storage based filesystem
type GCSFsConfig struct {
Bucket string `json:"bucket,omitempty"`
// KeyPrefix is similar to a chroot directory for local filesystem.
// If specified the SFTP user will only see objects that starts with
// this prefix and so you can restrict access to a specific virtual
// folder. The prefix, if not empty, must not start with "/" and must
// end with "/".
// If empty the whole bucket contents will be available
KeyPrefix string `json:"key_prefix,omitempty"`
CredentialFile string `json:"-"`
Credentials string `json:"credentials,omitempty"`
StorageClass string `json:"storage_class,omitempty"`
}
// GCSFs is a Fs implementation for Google Cloud Storage.
type GCSFs struct {
connectionID string
localTempDir string
config GCSFsConfig
svc *storage.Client
ctxTimeout time.Duration
ctxLongTimeout time.Duration
}
// NewGCSFs returns an GCSFs object that allows to interact with Google Cloud Storage
func NewGCSFs(connectionID, localTempDir string, config GCSFsConfig) (Fs, error) {
var err error
fs := GCSFs{
connectionID: connectionID,
localTempDir: localTempDir,
config: config,
ctxTimeout: 30 * time.Second,
ctxLongTimeout: 300 * time.Second,
}
if err = ValidateGCSFsConfig(&fs.config, fs.config.CredentialFile); err != nil {
return fs, err
}
ctx := context.Background()
fs.svc, err = storage.NewClient(ctx, option.WithCredentialsFile(fs.config.CredentialFile))
return fs, err
}
// Name returns the name for the Fs implementation
func (fs GCSFs) Name() string {
return fmt.Sprintf("GCSFs bucket: %#v", fs.config.Bucket)
}
// ConnectionID returns the SSH connection ID associated to this Fs implementation
func (fs GCSFs) ConnectionID() string {
return fs.connectionID
}
// Stat returns a FileInfo describing the named file
func (fs GCSFs) Stat(name string) (os.FileInfo, error) {
var result FileInfo
var err error
if len(name) == 0 || name == "." {
err := fs.checkIfBucketExists()
if err != nil {
return result, err
}
return NewFileInfo(name, true, 0, time.Time{}), nil
}
if fs.config.KeyPrefix == name+"/" {
return NewFileInfo(name, true, 0, time.Time{}), nil
}
prefix := fs.getPrefixForStat(name)
query := &storage.Query{Prefix: prefix, Delimiter: "/"}
/*err = query.SetAttrSelection(gcsDefaultFieldsSelection)
if err != nil {
return result, err
}*/
ctx, cancelFn := context.WithDeadline(context.Background(), time.Now().Add(fs.ctxTimeout))
defer cancelFn()
bkt := fs.svc.Bucket(fs.config.Bucket)
it := bkt.Objects(ctx, query)
for {
attrs, err := it.Next()
if err == iterator.Done {
break
}
if err != nil {
metrics.GCSListObjectsCompleted(err)
return result, err
}
if len(attrs.Prefix) > 0 {
if fs.isEqual(attrs.Prefix, name) {
result = NewFileInfo(name, true, 0, time.Time{})
}
} else {
if !attrs.Deleted.IsZero() {
continue
}
if fs.isEqual(attrs.Name, name) {
isDir := strings.HasSuffix(attrs.Name, "/")
result = NewFileInfo(name, isDir, attrs.Size, attrs.Updated)
}
}
}
metrics.GCSListObjectsCompleted(nil)
if len(result.Name()) == 0 {
err = errors.New("404 no such file or directory")
}
return result, err
}
// Lstat returns a FileInfo describing the named file
func (fs GCSFs) Lstat(name string) (os.FileInfo, error) {
return fs.Stat(name)
}
// Open opens the named file for reading
func (fs GCSFs) Open(name string) (*os.File, *pipeat.PipeReaderAt, func(), error) {
r, w, err := pipeat.AsyncWriterPipeInDir(fs.localTempDir)
if err != nil {
return nil, nil, nil, err
}
bkt := fs.svc.Bucket(fs.config.Bucket)
obj := bkt.Object(name)
ctx, cancelFn := context.WithCancel(context.Background())
objectReader, err := obj.NewReader(ctx)
if err != nil {
r.Close()
w.Close()
cancelFn()
return nil, nil, nil, err
}
go func() {
defer cancelFn()
defer objectReader.Close()
n, err := io.Copy(w, objectReader)
w.CloseWithError(err)
fsLog(fs, logger.LevelDebug, "download completed, path: %#v size: %v, err: %v", name, n, err)
metrics.GCSTransferCompleted(n, 1, err)
}()
return nil, r, cancelFn, nil
}
// Create creates or opens the named file for writing
func (fs GCSFs) Create(name string, flag int) (*os.File, *pipeat.PipeWriterAt, func(), error) {
r, w, err := pipeat.PipeInDir(fs.localTempDir)
if err != nil {
return nil, nil, nil, err
}
bkt := fs.svc.Bucket(fs.config.Bucket)
obj := bkt.Object(name)
ctx, cancelFn := context.WithCancel(context.Background())
objectWriter := obj.NewWriter(ctx)
if len(fs.config.StorageClass) > 0 {
objectWriter.ObjectAttrs.StorageClass = fs.config.StorageClass
}
go func() {
defer cancelFn()
defer objectWriter.Close()
n, err := io.Copy(objectWriter, r)
r.CloseWithError(err)
fsLog(fs, logger.LevelDebug, "upload completed, path: %#v, readed bytes: %v, err: %v", name, n, err)
metrics.GCSTransferCompleted(n, 0, err)
}()
return nil, w, cancelFn, nil
}
// Rename renames (moves) source to target.
// We don't support renaming non empty directories since we should
// rename all the contents too and this could take long time: think
// about directories with thousands of files, for each file we should
// execute a CopyObject call.
func (fs GCSFs) Rename(source, target string) error {
if source == target {
return nil
}
fi, err := fs.Stat(source)
if err != nil {
return err
}
if fi.IsDir() {
contents, err := fs.ReadDir(source)
if err != nil {
return err
}
if len(contents) > 0 {
return fmt.Errorf("Cannot rename non empty directory: %#v", source)
}
if !strings.HasSuffix(source, "/") {
source += "/"
}
if !strings.HasSuffix(target, "/") {
target += "/"
}
}
src := fs.svc.Bucket(fs.config.Bucket).Object(source)
dst := fs.svc.Bucket(fs.config.Bucket).Object(target)
ctx, cancelFn := context.WithDeadline(context.Background(), time.Now().Add(fs.ctxTimeout))
defer cancelFn()
copier := dst.CopierFrom(src)
if len(fs.config.StorageClass) > 0 {
copier.StorageClass = fs.config.StorageClass
}
_, err = copier.Run(ctx)
metrics.GCSCopyObjectCompleted(err)
if err != nil {
return err
}
return fs.Remove(source, fi.IsDir())
}
// Remove removes the named file or (empty) directory.
func (fs GCSFs) Remove(name string, isDir bool) error {
if isDir {
contents, err := fs.ReadDir(name)
if err != nil {
return err
}
if len(contents) > 0 {
return fmt.Errorf("Cannot remove non empty directory: %#v", name)
}
if !strings.HasSuffix(name, "/") {
name += "/"
}
}
ctx, cancelFn := context.WithDeadline(context.Background(), time.Now().Add(fs.ctxTimeout))
defer cancelFn()
err := fs.svc.Bucket(fs.config.Bucket).Object(name).Delete(ctx)
metrics.GCSDeleteObjectCompleted(err)
return err
}
// Mkdir creates a new directory with the specified name and default permissions
func (fs GCSFs) Mkdir(name string) error {
_, err := fs.Stat(name)
if !fs.IsNotExist(err) {
return err
}
if !strings.HasSuffix(name, "/") {
name += "/"
}
_, w, _, err := fs.Create(name, 0)
if err != nil {
return err
}
return w.Close()
}
// Symlink creates source as a symbolic link to target.
func (GCSFs) Symlink(source, target string) error {
return errors.New("403 symlinks are not supported")
}
// Chown changes the numeric uid and gid of the named file.
// Silently ignored.
func (GCSFs) Chown(name string, uid int, gid int) error {
return nil
}
// Chmod changes the mode of the named file to mode.
// Silently ignored.
func (GCSFs) Chmod(name string, mode os.FileMode) error {
return nil
}
// Chtimes changes the access and modification times of the named file.
// Silently ignored.
func (GCSFs) Chtimes(name string, atime, mtime time.Time) error {
return errors.New("403 chtimes is not supported")
}
// ReadDir reads the directory named by dirname and returns
// a list of directory entries.
func (fs GCSFs) ReadDir(dirname string) ([]os.FileInfo, error) {
var result []os.FileInfo
// dirname deve essere già cleaned
prefix := ""
if len(dirname) > 0 && dirname != "." {
prefix = strings.TrimPrefix(dirname, "/")
if !strings.HasSuffix(prefix, "/") {
prefix += "/"
}
}
query := &storage.Query{Prefix: prefix, Delimiter: "/"}
/*err := query.SetAttrSelection(gcsDefaultFieldsSelection)
if err != nil {
return result, err
}*/
ctx, cancelFn := context.WithDeadline(context.Background(), time.Now().Add(fs.ctxTimeout))
defer cancelFn()
bkt := fs.svc.Bucket(fs.config.Bucket)
it := bkt.Objects(ctx, query)
for {
attrs, err := it.Next()
if err == iterator.Done {
break
}
if err != nil {
metrics.GCSListObjectsCompleted(err)
return result, err
}
if len(attrs.Prefix) > 0 {
name, _ := fs.resolve(attrs.Prefix, prefix)
result = append(result, NewFileInfo(name, true, 0, time.Time{}))
} else {
name, isDir := fs.resolve(attrs.Name, prefix)
if len(name) == 0 {
continue
}
if !attrs.Deleted.IsZero() {
continue
}
result = append(result, NewFileInfo(name, isDir, attrs.Size, attrs.Updated))
}
}
metrics.GCSListObjectsCompleted(nil)
return result, nil
}
// IsUploadResumeSupported returns true if upload resume is supported.
// SFTP Resume is not supported on S3
func (GCSFs) IsUploadResumeSupported() bool {
return false
}
// IsAtomicUploadSupported returns true if atomic upload is supported.
// S3 uploads are already atomic, we don't need to upload to a temporary
// file
func (GCSFs) IsAtomicUploadSupported() bool {
return false
}
// IsNotExist returns a boolean indicating whether the error is known to
// report that a file or directory does not exist
func (GCSFs) IsNotExist(err error) bool {
if err == nil {
return false
}
if err == storage.ErrObjectNotExist || err == storage.ErrBucketNotExist {
return true
}
if e, ok := err.(*googleapi.Error); ok {
if e.Code == http.StatusNotFound {
return true
}
}
return strings.Contains(err.Error(), "404")
}
// IsPermission returns a boolean indicating whether the error is known to
// report that permission is denied.
func (GCSFs) IsPermission(err error) bool {
if err == nil {
return false
}
if e, ok := err.(*googleapi.Error); ok {
if e.Code == http.StatusForbidden || e.Code == http.StatusUnauthorized {
return true
}
}
return strings.Contains(err.Error(), "403")
}
// CheckRootPath creates the specified root directory if it does not exists
func (fs GCSFs) CheckRootPath(username string, uid int, gid int) bool {
// we need a local directory for temporary files
osFs := NewOsFs(fs.ConnectionID(), fs.localTempDir)
osFs.CheckRootPath(username, uid, gid)
return fs.checkIfBucketExists() != nil
}
// ScanRootDirContents returns the number of files contained in the bucket,
// and their size
func (fs GCSFs) ScanRootDirContents() (int, int64, error) {
numFiles := 0
size := int64(0)
query := &storage.Query{Prefix: fs.config.KeyPrefix}
err := query.SetAttrSelection(gcsDefaultFieldsSelection)
if err != nil {
return numFiles, size, err
}
ctx, cancelFn := context.WithDeadline(context.Background(), time.Now().Add(fs.ctxLongTimeout))
defer cancelFn()
bkt := fs.svc.Bucket(fs.config.Bucket)
it := bkt.Objects(ctx, query)
for {
attrs, err := it.Next()
if err == iterator.Done {
break
}
if err != nil {
metrics.GCSListObjectsCompleted(err)
return numFiles, size, err
}
if !attrs.Deleted.IsZero() {
continue
}
numFiles++
size += attrs.Size
}
metrics.GCSListObjectsCompleted(nil)
return numFiles, size, err
}
// GetAtomicUploadPath returns the path to use for an atomic upload.
// S3 uploads are already atomic, we never call this method for S3
func (GCSFs) GetAtomicUploadPath(name string) string {
return ""
}
// GetRelativePath returns the path for a file relative to the user's home dir.
// This is the path as seen by SFTP users
func (fs GCSFs) GetRelativePath(name string) string {
rel := path.Clean(name)
if rel == "." {
rel = ""
}
if !path.IsAbs(rel) {
rel = "/" + rel
}
if len(fs.config.KeyPrefix) > 0 {
if !strings.HasPrefix(rel, "/"+fs.config.KeyPrefix) {
rel = "/"
}
rel = path.Clean("/" + strings.TrimPrefix(rel, "/"+fs.config.KeyPrefix))
}
return rel
}
// Join joins any number of path elements into a single path
func (GCSFs) Join(elem ...string) string {
return strings.TrimPrefix(path.Join(elem...), "/")
}
// ResolvePath returns the matching filesystem path for the specified sftp path
func (fs GCSFs) ResolvePath(sftpPath string) (string, error) {
if !path.IsAbs(sftpPath) {
sftpPath = path.Clean("/" + sftpPath)
}
return fs.Join(fs.config.KeyPrefix, strings.TrimPrefix(sftpPath, "/")), nil
}
func (fs *GCSFs) resolve(name string, prefix string) (string, bool) {
result := strings.TrimPrefix(name, prefix)
isDir := strings.HasSuffix(result, "/")
if isDir {
result = strings.TrimSuffix(result, "/")
}
return result, isDir
}
func (fs *GCSFs) isEqual(key string, sftpName string) bool {
if key == sftpName {
return true
}
if key == sftpName+"/" {
return true
}
if key+"/" == sftpName {
return true
}
return false
}
func (fs *GCSFs) checkIfBucketExists() error {
ctx, cancelFn := context.WithDeadline(context.Background(), time.Now().Add(fs.ctxTimeout))
defer cancelFn()
bkt := fs.svc.Bucket(fs.config.Bucket)
_, err := bkt.Attrs(ctx)
metrics.GCSHeadBucketCompleted(err)
return err
}
func (fs *GCSFs) getPrefixForStat(name string) string {
prefix := path.Dir(name)
if prefix == "/" || prefix == "." || len(prefix) == 0 {
prefix = ""
} else {
prefix = strings.TrimPrefix(prefix, "/")
if !strings.HasSuffix(prefix, "/") {
prefix += "/"
}
}
return prefix
}

View file

@ -21,15 +21,15 @@ import (
"github.com/eikenb/pipeat"
)
// S3FsConfig defines the configuration for S3fs
// S3FsConfig defines the configuration for S3 based filesystem
type S3FsConfig struct {
Bucket string `json:"bucket,omitempty"`
// KeyPrefix is similar to a chroot directory for local filesystem.
// If specified the SFTP user will only see contents that starts with
// If specified the SFTP user will only see objects that starts with
// this prefix and so you can restrict access to a specific virtual
// folder. The prefix, if not empty, must not start with "/" and must
// end with "/".
//If empty the whole bucket contents will be available
// If empty the whole bucket contents will be available
KeyPrefix string `json:"key_prefix,omitempty"`
Region string `json:"region,omitempty"`
AccessKey string `json:"access_key,omitempty"`
@ -70,7 +70,6 @@ func NewS3Fs(connectionID, localTempDir string, config S3FsConfig) (Fs, error) {
Region: aws.String(fs.config.Region),
Credentials: credentials.NewStaticCredentials(fs.config.AccessKey, fs.config.AccessSecret, ""),
}
//config.WithLogLevel(aws.LogDebugWithHTTPBody)
if len(fs.config.Endpoint) > 0 {
awsConfig.Endpoint = aws.String(fs.config.Endpoint)
awsConfig.S3ForcePathStyle = aws.Bool(true)
@ -95,16 +94,16 @@ func (fs S3Fs) ConnectionID() string {
// Stat returns a FileInfo describing the named file
func (fs S3Fs) Stat(name string) (os.FileInfo, error) {
var result S3FileInfo
var result FileInfo
if name == "/" || name == "." {
err := fs.checkIfBucketExists()
if err != nil {
return result, err
}
return NewS3FileInfo(name, true, 0, time.Time{}), nil
return NewFileInfo(name, true, 0, time.Time{}), nil
}
if "/"+fs.config.KeyPrefix == name+"/" {
return NewS3FileInfo(name, true, 0, time.Time{}), nil
return NewFileInfo(name, true, 0, time.Time{}), nil
}
prefix := path.Dir(name)
if prefix == "/" || prefix == "." {
@ -124,7 +123,7 @@ func (fs S3Fs) Stat(name string) (os.FileInfo, error) {
}, func(page *s3.ListObjectsV2Output, lastPage bool) bool {
for _, p := range page.CommonPrefixes {
if fs.isEqual(p.Prefix, name) {
result = NewS3FileInfo(name, true, 0, time.Time{})
result = NewFileInfo(name, true, 0, time.Time{})
return false
}
}
@ -133,7 +132,7 @@ func (fs S3Fs) Stat(name string) (os.FileInfo, error) {
objectSize := *fileObject.Size
objectModTime := *fileObject.LastModified
isDir := strings.HasSuffix(*fileObject.Key, "/")
result = NewS3FileInfo(name, isDir, objectSize, objectModTime)
result = NewFileInfo(name, isDir, objectSize, objectModTime)
return false
}
}
@ -325,7 +324,7 @@ func (fs S3Fs) ReadDir(dirname string) ([]os.FileInfo, error) {
}, func(page *s3.ListObjectsV2Output, lastPage bool) bool {
for _, p := range page.CommonPrefixes {
name, isDir := fs.resolve(p.Prefix, prefix)
result = append(result, NewS3FileInfo(name, isDir, 0, time.Time{}))
result = append(result, NewFileInfo(name, isDir, 0, time.Time{}))
}
for _, fileObject := range page.Contents {
objectSize := *fileObject.Size
@ -334,7 +333,7 @@ func (fs S3Fs) ReadDir(dirname string) ([]os.FileInfo, error) {
if len(name) == 0 {
continue
}
result = append(result, NewS3FileInfo(name, isDir, objectSize, objectModTime))
result = append(result, NewFileInfo(name, isDir, objectSize, objectModTime))
}
return true
})
@ -394,23 +393,7 @@ func (fs S3Fs) CheckRootPath(username string, uid int, gid int) bool {
// we need a local directory for temporary files
osFs := NewOsFs(fs.ConnectionID(), fs.localTempDir)
osFs.CheckRootPath(username, uid, gid)
err := fs.checkIfBucketExists()
if err == nil {
return true
}
if !fs.IsNotExist(err) {
return false
}
ctx, cancelFn := context.WithDeadline(context.Background(), time.Now().Add(fs.ctxTimeout))
defer cancelFn()
input := &s3.CreateBucketInput{
Bucket: aws.String(fs.config.Bucket),
}
_, err = fs.svc.CreateBucketWithContext(ctx, input)
fsLog(fs, logger.LevelDebug, "bucket %#v for user %#v does not exists, try to create, error: %v",
fs.config.Bucket, username, err)
metrics.S3CreateBucketCompleted(err)
return err == nil
return fs.checkIfBucketExists() != nil
}
// ScanRootDirContents returns the number of files contained in the bucket,

View file

@ -3,6 +3,7 @@ package vfs
import (
"errors"
"fmt"
"os"
"path"
"runtime"
@ -14,7 +15,7 @@ import (
"github.com/pkg/sftp"
)
// Fs defines the interface for filesystems backends
// Fs defines the interface for filesystem backends
type Fs interface {
Name() string
ConnectionID() string
@ -94,6 +95,32 @@ func ValidateS3FsConfig(config *S3FsConfig) error {
return nil
}
// ValidateGCSFsConfig returns nil if the specified GCS config is valid, otherwise an error
func ValidateGCSFsConfig(config *GCSFsConfig, credentialsFilePath string) error {
if len(config.Bucket) == 0 {
return errors.New("bucket cannot be empty")
}
if len(config.KeyPrefix) > 0 {
if strings.HasPrefix(config.KeyPrefix, "/") {
return errors.New("key_prefix cannot start with /")
}
config.KeyPrefix = path.Clean(config.KeyPrefix)
if !strings.HasSuffix(config.KeyPrefix, "/") {
config.KeyPrefix += "/"
}
}
if len(config.Credentials) == 0 {
fi, err := os.Stat(credentialsFilePath)
if err != nil {
return fmt.Errorf("invalid credentials %v", err)
}
if fi.Size() == 0 {
return errors.New("credentials cannot be empty")
}
}
return nil
}
// SetPathPermissions calls fs.Chown.
// It does nothing for local filesystem on windows
func SetPathPermissions(fs Fs, path string, uid int, gid int) {