Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: johannesboyne/gofakes3
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: master
Choose a base ref
...
head repository: JankariTech/gofakes3
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: S3toWebdavProxy
Choose a head ref
Can’t automatically merge. Don’t worry, you can still create the pull request.

Commits on Sep 18, 2022

  1. Copy the full SHA
    7bc6e24 View commit details
  2. feat: remove unused files

    Mikubill committed Sep 18, 2022
    Copy the full SHA
    548453f View commit details
  3. Copy the full SHA
    d1bf540 View commit details

Commits on Sep 21, 2022

  1. Copy the full SHA
    99328eb View commit details
  2. docs: update README.md

    Mikubill committed Sep 21, 2022
    Copy the full SHA
    bee10b7 View commit details
  3. Copy the full SHA
    897c7b3 View commit details
  4. Copy the full SHA
    3b04724 View commit details

Commits on Sep 22, 2022

  1. fix: add more metadata field

    Mikubill committed Sep 22, 2022
    Copy the full SHA
    dd6e44f View commit details

Commits on Oct 3, 2022

  1. Copy the full SHA
    1fe0571 View commit details
  2. docs: update readme

    Mikubill committed Oct 3, 2022
    Copy the full SHA
    cefe822 View commit details
  3. fix: minor fix, format code

    Mikubill committed Oct 3, 2022
    Copy the full SHA
    2a2d530 View commit details

Commits on Oct 22, 2022

  1. Copy the full SHA
    6e93ea4 View commit details
  2. docs: update docs

    Mikubill committed Oct 22, 2022
    Copy the full SHA
    4a3b2e8 View commit details

Commits on Oct 27, 2022

  1. add more func; dep update

    Mikubill committed Oct 27, 2022
    Copy the full SHA
    c88933c View commit details
  2. fix: add urlencode function

    Mikubill committed Oct 27, 2022
    Copy the full SHA
    137f425 View commit details

Commits on Oct 29, 2022

  1. fix: auth error

    Mikubill committed Oct 29, 2022
    Copy the full SHA
    68b260c View commit details
  2. fix: invalid auth provider

    Mikubill committed Oct 29, 2022
    Copy the full SHA
    65b92b1 View commit details

Commits on Oct 30, 2022

  1. Copy the full SHA
    725f2cf View commit details

Commits on Mar 12, 2023

  1. Fix chunkedReader.Read not handling early inner.Read return correctly

    The previous implementation expected inner.Read to always read until the end of the provided slice.
    oniumy committed Mar 12, 2023
    Copy the full SHA
    2077be7 View commit details
  2. Add support for chunked reader in multipart upload

    Multipart uploads can include chunked readers in the part upload as well.
    oniumy committed Mar 12, 2023
    Copy the full SHA
    7e26e73 View commit details

Commits on Mar 16, 2023

  1. Merge pull request #1 from oniumy/master

    Fix issues with chunked multipart uploads
    Mikubill authored Mar 16, 2023
    Copy the full SHA
    723f185 View commit details

Commits on Jun 20, 2023

  1. also URL encode the prefix

    without this the prefix in the response would not be URL encoded
    
    before:
    ```
    <Prefix>
    sub-folder &lt;&gt;/sub2/
    </Prefix>
    ```
    
    after
    ```
    <Prefix>
    sub-folder+%3C%3E/sub2/
    </Prefix>
    ```
    individual-it committed Jun 20, 2023
    Copy the full SHA
    f4d6b12 View commit details

Commits on Jun 21, 2023

  1. set encoding type to "url"

    at least for the `mc` client of minio this makes things clearer
    having a folder/file name in the backend that contains unusual characters e.g. `bucket2/sub-folder <>/a space $%&#.txt`
    results without this change in this output when running `mc ls`
    `[2023-06-16 17:19:10 +0545]    11B STANDARD /bucket2/sub-folder+%3C%3E/sub2/a+space+%24%25%26%23.txt`
    also such a folder/bucket cannot be downloaded using `mc cp`
    
    after the change the output of `mc ls` is correct and same as if it connects to AWS S3
    `[2023-06-16 17:19:10 +0545]    11B STANDARD a space $%&#.txt`
    
    and also the `mc tree -f` output looks correct
    ```
    proxy/bucket2/
    └─ sub-folder <>
       ├─ a space $%&#.txt
       └─ sub2
          └─ a space $%&#.txt
    ```
    individual-it committed Jun 21, 2023
    Copy the full SHA
    7338612 View commit details

Commits on Jun 22, 2023

  1. Copy the full SHA
    522293e View commit details
  2. set module path to JankariTech

    individual-it authored and saw-jan committed Jun 22, 2023
    Copy the full SHA
    87b86b5 View commit details
  3. Copy the full SHA
    4878bd6 View commit details
  4. remove unused fields

    saw-jan committed Jun 22, 2023
    Copy the full SHA
    4d873ba View commit details
  5. Merge pull request #1 from JankariTech/pass-accesskey

    Parse accessKey from request and pass to the Backend methods
    individual-it authored Jun 22, 2023
    Copy the full SHA
    4b38c65 View commit details
  6. fix tests

    saw-jan committed Jun 22, 2023
    Copy the full SHA
    cf1447d View commit details
  7. Merge pull request #2 from JankariTech/fix-tests-implementation

    Fix Backend implementation and tests
    individual-it authored Jun 22, 2023
    Copy the full SHA
    1b69ffd View commit details
Showing with 10,822 additions and 2,762 deletions.
  1. +0 −17 .circleci/config.yml
  2. +1 −0 .gitignore
  3. +6 −198 README.md
  4. +1 −1 awscli_test.go
  5. +13 −13 backend.go
  6. +0 −319 backend/s3afero/backend_test.go
  7. +0 −18 backend/s3afero/doc.go
  8. +0 −21 backend/s3afero/hash.go
  9. +0 −131 backend/s3afero/meta.go
  10. +0 −92 backend/s3afero/modres.go
  11. +0 −514 backend/s3afero/multi.go
  12. +0 −19 backend/s3afero/option.go
  13. +0 −426 backend/s3afero/single.go
  14. +0 −72 backend/s3afero/util.go
  15. +0 −377 backend/s3bolt/backend.go
  16. +0 −94 backend/s3bolt/schema.go
  17. 0 build/.gitdir
  18. +7 −14 chunk.go
  19. +0 −271 cmd/gofakes3/main.go
  20. +8 −10 error.go
  21. +2 −1 error_test.go
  22. +14 −11 go.mod
  23. +57 −25 go.sum
  24. +158 −69 gofakes3.go
  25. +2 −1 gofakes3_internal_test.go
  26. +5 −4 gofakes3_test.go
  27. +11 −11 init_test.go
  28. +3 −2 makefile.go
  29. +6 −4 messages.go
  30. +2 −1 messages_test.go
  31. +4 −0 option.go
  32. +1 −1 option_autobucket_test.go
  33. +38 −12 {backend → }/s3mem/backend.go
  34. +2 −2 {backend → }/s3mem/bucket.go
  35. +1 −1 {backend → }/s3mem/versionid.go
  36. +1 −1 {backend → }/s3mem/versionid_test.go
  37. +53 −0 signature/credentials.go
  38. +146 −0 signature/signature-errors.go
  39. +135 −0 signature/signature-v4-parser.go
  40. +89 −0 signature/signature-v4-utils.go
  41. +194 −0 signature/signature-v4.go
  42. +64 −0 signature/signature-v4_test.go
  43. +109 −0 signature/utils.go
  44. +8 −8 uploader.go
  45. +1 −1 uploader_test.go
  46. +77 −0 urlencoder.go
  47. +56 −0 xml/atom_test.go
  48. +84 −0 xml/example_marshaling_test.go
  49. +151 −0 xml/example_test.go
  50. +79 −0 xml/example_test_marshaling_test.go
  51. +1,062 −0 xml/marshal.go
  52. +2,532 −0 xml/marshal_test.go
  53. +771 −0 xml/read.go
  54. +1,128 −0 xml/read_test.go
  55. +374 −0 xml/typeinfo.go
  56. +2,100 −0 xml/xml.go
  57. +1,266 −0 xml/xml_test.go
17 changes: 0 additions & 17 deletions .circleci/config.yml

This file was deleted.

1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -6,4 +6,5 @@ logs/
s3f_*
*.test
*.out
.devcontainer/
build/
204 changes: 6 additions & 198 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,203 +1,11 @@
[![CircleCI](https://circleci.com/gh/johannesboyne/gofakes3.svg?style=svg)](https://circleci.com/gh/johannesboyne/gofakes3)
[![Codecov](https://codecov.io/gh/johannesboyne/gofakes3/branch/master/graph/badge.svg)](https://codecov.io/gh/johannesboyne/gofakes3)

![Logo](/GoFakeS3.png)
# AWS (GOFAKE)S3

AWS S3 fake server and testing library for extensive S3 test integrations.
Either by running a test-server, e.g. for testing of AWS Lambda functions
accessing S3. Or, to have a simple and convencience S3 mock- and test-server.
This is a fork of [johannesboyne/gofakes3](https://github.com/johannesboyne/gofakes3).

## What to use it for?
Notable differences:

We're using it for the local development of S3 dependent Lambda functions,
to test AWS S3 golang implementations and access, and
to test browser based direct uploads to S3 locally.


## What not to use it for?

Please don't use gofakes3 as a production service. The intended use case for
gofakes3 is currently to facilitate testing. It's not meant to be used for
safe, persistent access to production data at the moment.

There's no reason we couldn't set that as a stretch goal at a later date, but
it's a long way down the road, especially while we have so much of the API left
to implement; breaking changes are inevitable.

In the meantime, there are more battle-hardened solutions for production
workloads out there, some of which are listed in the "Similar Notable Projects"
section below.


## How to use it?

### Example (aws-sdk-go version 1)

```golang
// fake s3
backend := s3mem.New()
faker := gofakes3.New(backend)
ts := httptest.NewServer(faker.Server())
defer ts.Close()

// configure S3 client
s3Config := &aws.Config{
Credentials: credentials.NewStaticCredentials("YOUR-ACCESSKEYID", "YOUR-SECRETACCESSKEY", ""),
Endpoint: aws.String(ts.URL),
Region: aws.String("eu-central-1"),
DisableSSL: aws.Bool(true),
S3ForcePathStyle: aws.Bool(true),
}
newSession := session.New(s3Config)

s3Client := s3.New(newSession)
cparams := &s3.CreateBucketInput{
Bucket: aws.String("newbucket"),
}

// Create a new bucket using the CreateBucket call.
_, err := s3Client.CreateBucket(cparams)
if err != nil {
// Message from an error.
fmt.Println(err.Error())
return
}

// Upload a new object "testobject" with the string "Hello World!" to our "newbucket".
_, err = s3Client.PutObject(&s3.PutObjectInput{
Body: strings.NewReader(`{"configuration": {"main_color": "#333"}, "screens": []}`),
Bucket: aws.String("newbucket"),
Key: aws.String("test.txt"),
})

// ... accessing of test.txt through any S3 client would now be possible
```

### Example for V2 (aws-sdk-go-v2)

```golang
// ... same as above
ts := httptest.NewServer(faker.Server())
defer ts.Close()

// Difference in configuring the client

// Setup a new config
cfg, _ := config.LoadDefaultConfig(
context.TODO(),
config.WithSharedConfigProfile("test"),
config.WithHTTPClient(
&http.Client{
Transport: &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
},
},
),
config.WithEndpointResolver(
aws.EndpointResolverFunc(func(service, region string) (aws.Endpoint, error) {
return aws.Endpoint{URL: ts.URL}, nil
}),
),
)

// Create an Amazon S3 v2 client, important to use o.UsePathStyle
// alternatively change local DNS settings, e.g., in /etc/hosts
// to support requests to http://<bucketname>.127.0.0.1:32947/...
client := s3.NewFromConfig(cfg, func(o *s3.Options) {
o.UsePathStyle = true
})

```


Please feel free to check it out and to provide useful feedback (using github
issues), but be aware, this software is used internally and for the local
development only. Thus, it has no demand for correctness, performance or
security.

There are two ways to run locally: using DNS, or using S3 path mode.

S3 path mode is the most flexible and least restrictive, but it does require that you
are able to modify your client code.In Go, the modification would look like so:

config := aws.Config{}
config.WithS3ForcePathStyle(true)

S3 path mode works over the network by default for all bucket names.

If you are unable to modify the code, DNS mode can be used, but it comes with further
restrictions and requires you to be able to modify your local DNS resolution.

If using `localhost` as your endpoint, you will need to add the following to
`/etc/hosts` for *every bucket you want to fake*:

127.0.0.1 <bucket-name>.localhost

It is trickier if you want other machines to be able to use your fake S3 server
as you need to be able to modify their DNS resolution as well.


## Exemplary usage

### Lambda Example

```javascript
var AWS = require('aws-sdk')

var ep = new AWS.Endpoint('http://localhost:9000');
var s3 = new AWS.S3({endpoint: ep});

exports.handle = function (e, ctx) {
s3.createBucket({
Bucket: '<bucket-name>',
}, function(err, data) {
if (err) return console.log(err, err.stack);
ctx.succeed(data)
});
}
```

### Upload Example

```html
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body>

<form action="http://localhost:9000/<bucket-name>/" method="post" enctype="multipart/form-data">
Key to upload:
<input type="input" name="key" value="user/user1/test/<filename>" /><br />
<input type="hidden" name="acl" value="public-read" />
<input type="hidden" name="x-amz-meta-uuid" value="14365123651274" />
<input type="hidden" name="x-amz-server-side-encryption" value="AES256" />
<input type="text" name="X-Amz-Credential" value="AKIAIOSFODNN7EXAMPLE/20151229/us-east-1/s3/aws4_request" />
<input type="text" name="X-Amz-Algorithm" value="AWS4-HMAC-SHA256" />
<input type="text" name="X-Amz-Date" value="20151229T000000Z" />

Tags for File:
<input type="input" name="x-amz-meta-tag" value="" /><br />
<input type="hidden" name="Policy" value='<Base64-encoded policy string>' />
<input type="hidden" name="X-Amz-Signature" value="<signature-value>" />
File:
<input type="file" name="file" /> <br />
<!-- The elements after this will be ignored -->
<input type="submit" name="submit" value="Upload to Amazon S3" />
</form>
</html>
```

## Similar notable projects

- https://github.com/minio/minio **not similar but powerfull ;-)**
- https://github.com/andrewgaul/s3proxy by @andrewgaul

## Contributors

A big thank you to all the [contributors](https://github.com/johannesboyne/gofakes3/graphs/contributors),
especially [Blake @shabbyrobe](https://github.com/shabbyrobe) who pushed this
little project to the next level!

**Help wanred**
* Use modified xml library to handle more control chars
* Func `getVersioningConfiguration` will return empty when unversioned
* New func in `backend` interface: `CopyObjcet`
* Support authentication with AWS Signature V4
2 changes: 1 addition & 1 deletion awscli_test.go
Original file line number Diff line number Diff line change
@@ -14,7 +14,7 @@ import (
"testing"
"time"

"github.com/johannesboyne/gofakes3"
"github.com/JankariTech/gofakes3"
)

func TestCLILsBuckets(t *testing.T) {
26 changes: 13 additions & 13 deletions backend.go
Original file line number Diff line number Diff line change
@@ -130,12 +130,11 @@ type PutObjectResult struct {
//
// The Backend API is not yet stable; if you create your own Backend, breakage
// is likely until this notice is removed.
//
type Backend interface {
// ListBuckets returns a list of all buckets owned by the authenticated
// sender of the request.
// https://docs.aws.amazon.com/AmazonS3/latest/API/RESTServiceGET.html
ListBuckets() ([]BucketInfo, error)
ListBuckets(accessKey string) ([]BucketInfo, error)

// ListBucket returns the contents of a bucket. Backends should use the
// supplied prefix to limit the contents of the bucket and to sort the
@@ -156,18 +155,18 @@ type Backend interface {
// work fine if you ignore the pagination request, but this may not suit
// your application. Not all backends bundled with gofakes3 correctly
// support this pagination yet, but that will change.
ListBucket(name string, prefix *Prefix, page ListBucketPage) (*ObjectList, error)
ListBucket(accessKey string, name string, prefix *Prefix, page ListBucketPage) (*ObjectList, error)

// CreateBucket creates the bucket if it does not already exist. The name
// should be assumed to be a valid name.
//
// If the bucket already exists, a gofakes3.ResourceError with
// gofakes3.ErrBucketAlreadyExists MUST be returned.
CreateBucket(name string) error
CreateBucket(accessKey string, name string) error

// BucketExists should return a boolean indicating the bucket existence, or
// an error if the backend was unable to determine existence.
BucketExists(name string) (exists bool, err error)
BucketExists(accessKey string, name string) (exists bool, err error)

// DeleteBucket deletes a bucket if and only if it is empty.
//
@@ -177,7 +176,7 @@ type Backend interface {
// If the bucket does not exist, gofakes3.ErrNoSuchBucket MUST be returned.
//
// AWS does not validate the bucket's name for anything other than existence.
DeleteBucket(name string) error
DeleteBucket(accessKey string, name string) error

// GetObject must return a gofakes3.ErrNoSuchKey error if the object does
// not exist. See gofakes3.KeyNotFound() for a convenient way to create
@@ -192,7 +191,7 @@ type Backend interface {
// implementers MUST return ErrNotImplemented.
//
// If the backend is a VersionedBackend, GetObject retrieves the latest version.
GetObject(bucketName, objectName string, rangeRequest *ObjectRangeRequest) (*Object, error)
GetObject(accessKey string, bucketName, objectName string, rangeRequest *ObjectRangeRequest) (*Object, error)

// HeadObject fetches the Object from the backend, but reading the Contents
// will return io.EOF immediately.
@@ -203,7 +202,7 @@ type Backend interface {
//
// HeadObject should return a NotFound() error if the object does not
// exist.
HeadObject(bucketName, objectName string) (*Object, error)
HeadObject(accessKey string, bucketName, objectName string) (*Object, error)

// DeleteObject deletes an object from the bucket.
//
@@ -221,16 +220,18 @@ type Backend interface {
// delete marker, which becomes the latest version of the object. If there
// isn't a null version, Amazon S3 does not remove any objects.
//
DeleteObject(bucketName, objectName string) (ObjectDeleteResult, error)
DeleteObject(accessKey string, bucketName, objectName string) (ObjectDeleteResult, error)

// PutObject should assume that the key is valid. The map containing meta
// may be nil.
//
// The size can be used if the backend needs to read the whole reader; use
// gofakes3.ReadAll() for this job rather than ioutil.ReadAll().
PutObject(bucketName, key string, meta map[string]string, input io.Reader, size int64) (PutObjectResult, error)
PutObject(accessKey string, bucketName, key string, meta map[string]string, input io.Reader, size int64) (PutObjectResult, error)

DeleteMulti(accessKey string, bucketName string, objects ...string) (MultiDeleteResult, error)

DeleteMulti(bucketName string, objects ...string) (MultiDeleteResult, error)
CopyObject(accessKey string, srcBucket, srcKey, dstBucket, dstKey string, meta map[string]string) (CopyObjectResult, error)
}

// VersionedBackend may be optionally implemented by a Backend in order to support
@@ -239,7 +240,6 @@ type Backend interface {
// If you don't implement VersionedBackend, requests to GoFakeS3 that attempt to
// make use of versions will return ErrNotImplemented if GoFakesS3 is unable to
// find another way to satisfy the request.
//
type VersionedBackend interface {
// VersioningConfiguration must return a gofakes3.ErrNoSuchBucket error if the bucket
// does not exist. See gofakes3.BucketNotFound() for a convenient way to create one.
@@ -310,7 +310,7 @@ type VersionedBackend interface {

func MergeMetadata(db Backend, bucketName string, objectName string, meta map[string]string) error {
// get potential existing object to potentially carry metadata over
existingObj, err := db.GetObject(bucketName, objectName, nil)
existingObj, err := db.GetObject("", bucketName, objectName, nil)
if err != nil {
if awsErr, ok := err.(awserr.Error); ok && awsErr.Code() != string(ErrNoSuchKey) {
return err
Loading