Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Blob Copy & Page Blob support to by SQL based metadata implementation in Blob API and Sync Loki and SQL metadata stores #2225

Open
wants to merge 17 commits into
base: main
Choose a base branch
from
Open

Add Blob Copy & Page Blob support to by SQL based metadata implementation in Blob API and Sync Loki and SQL metadata stores #2225

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
19cf172
Changed properties from separate fields into a single one to allow se…
mahmoudbahaa Oct 14, 2023
ef2896a
Added missing await from some async functions in SqlBlobMetadataStore
mahmoudbahaa Oct 14, 2023
0abed5f
Ported copyFromURL from Loki to SQL blobMeatadataStore
mahmoudbahaa Oct 14, 2023
fb46319
Ported appendBlock from Loki to SQL blobMeatadataStore
mahmoudbahaa Oct 14, 2023
787be28
Ported uploadPages from Loki to SQL blobMeatadataStore
mahmoudbahaa Oct 14, 2023
2732b26
Ported clearRange from Loki to SQL blobMeatadataStore
mahmoudbahaa Oct 14, 2023
4e8442d
Ported getPageRanges from Loki to SQL blobMeatadataStore
mahmoudbahaa Oct 14, 2023
390bbad
Ported resizePageBlob from Loki to SQL blobMeatadataStore
mahmoudbahaa Oct 14, 2023
48a79f6
Ported updateSequenceNumber from Loki to SQL blobMeatadataStore
mahmoudbahaa Oct 14, 2023
54fe209
Return blobCommittedBlockCount for append blob in getBlobProperties i…
mahmoudbahaa Oct 14, 2023
99963a4
Check for CopyIfExists condition in startCopyFromURL in SQL blobMetad…
mahmoudbahaa Oct 14, 2023
50df667
Add @sql to all @loki tests
mahmoudbahaa Oct 14, 2023
596fecf
Ignore generateAccountSASQueryParameters in sas blob tests as it depe…
mahmoudbahaa Oct 14, 2023
8fa4e29
skip blob production style URL test as it fails in new sdk
mahmoudbahaa Oct 14, 2023
858a6ef
Remove skipping of "list uncimmited blob from container" in blob api…
mahmoudbahaa Oct 14, 2023
bac8df7
Added await to 2 async functions in 2 tests that was causing these 2 …
mahmoudbahaa Oct 14, 2023
33d89d4
change readme and changelog to reflect the sycn of SQL blobmetadaStor…
mahmoudbahaa Oct 14, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,5 @@ temp
azurite.exe
.pkg-cache
release
querydb*.json
querydb*.json
.idea
1 change: 1 addition & 0 deletions ChangeLog.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ Blob:
- Fixed startCopyFromURL, copyFromURL API to return 400 (InvalidHeaderValue) when copy source has invalid format. (issue #1954)
- Fixed CommitBlockList API to return 400 (InvalidXmlDocument) when the request is sent with JSON body. (issue #1955)
- Added "x-ms-is-hns-enabled" header in x-ms-is-hns-enabled API responds (issue #1810)
- Blob Copy & Page Blob are now supported by SQL based metadata implementation. This making the SQL based metadata implementation in sync with File based one. (issue #2224)

Queue:

Expand Down
2 changes: 0 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -488,8 +488,6 @@ This feature is in preview, when Azurite changes database table schema, you need

> Note. Need to manually create database before starting Azurite instance.

> Note. Blob Copy & Page Blob are not supported by SQL based metadata implementation.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The support matrix section in ReadMe.md should also be updated.

Azurite/README.md

Line 1011 in 9208664

- Copy Blob From URL (Only supports copy within same Azurite instance, only on Loki)

> Tips. Create database instance quickly with docker, for example `docker run --name mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql:latest`. Grant external access and create database `azurite_blob` using `docker exec mysql mysql -u root -pmy-secret-pw -e "GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' WITH GRANT OPTION; FLUSH PRIVILEGES; create database azurite_blob;"`. Notice that, above commands are examples, you need to carefully define the access permissions in your production environment.

## HTTPS Setup
Expand Down
812 changes: 612 additions & 200 deletions src/blob/persistence/SqlBlobMetadataStore.ts
View file
Open in desktop

Large diffs are not rendered by default.

38 changes: 19 additions & 19 deletions tests/blob/apis/appendblob.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ describe("AppendBlobAPIs", () => {
await containerClient.delete();
});

it("Create append blob should work @loki", async () => {
it("Create append blob should work @loki @sql", async () => {
await appendBlobClient.create();
const properties = await appendBlobClient.getProperties();
assert.deepStrictEqual(properties.blobType, "AppendBlob");
Expand All @@ -85,7 +85,7 @@ describe("AppendBlobAPIs", () => {
assert.deepStrictEqual(properties.blobCommittedBlockCount, 0);
});

it("Create append blob override existing pageblob @loki", async () => {
it("Create append blob override existing pageblob @loki @sql", async () => {
const pageBlobClient = blobClient.getPageBlobClient();
await pageBlobClient.create(512);

Expand Down Expand Up @@ -133,12 +133,12 @@ describe("AppendBlobAPIs", () => {
assert.deepStrictEqual(properties.blobCommittedBlockCount, 0);
});

it("Delete append blob should work @loki", async () => {
it("Delete append blob should work @loki @sql", async () => {
await appendBlobClient.create();
await appendBlobClient.delete();
});

it("Create append blob snapshot should work @loki", async () => {
it("Create append blob snapshot should work @loki @sql", async () => {
await appendBlobClient.create();
const response = await appendBlobClient.createSnapshot();
const appendBlobSnapshotClient = appendBlobClient.withSnapshot(
Expand Down Expand Up @@ -176,7 +176,7 @@ describe("AppendBlobAPIs", () => {
assert.deepStrictEqual(properties.blobCommittedBlockCount, 0);
});

it("Copy append blob snapshot should work @loki", async () => {
it("Copy append blob snapshot should work @loki @sql", async () => {
await appendBlobClient.create();
await appendBlobClient.appendBlock("hello", 5);

Expand Down Expand Up @@ -212,7 +212,7 @@ describe("AppendBlobAPIs", () => {
assert.deepStrictEqual(properties.copyStatus, "success");
});

it("Synchronized copy append blob snapshot should work @loki", async () => {
it("Synchronized copy append blob snapshot should work @loki @sql", async () => {
await appendBlobClient.create();
await appendBlobClient.appendBlock("hello", 5);

Expand Down Expand Up @@ -247,7 +247,7 @@ describe("AppendBlobAPIs", () => {
assert.deepStrictEqual(properties.copySource, appendBlobSnapshotClient.url);
});

it("Set append blob metadata should work @loki", async () => {
it("Set append blob metadata should work @loki @sql", async () => {
await appendBlobClient.create();

const metadata = {
Expand All @@ -260,7 +260,7 @@ describe("AppendBlobAPIs", () => {
assert.deepStrictEqual(properties.metadata, metadata);
});

it("Set append blob HTTP headers should work @loki", async () => {
it("Set append blob HTTP headers should work @loki @sql", async () => {
await appendBlobClient.create();

const md5 = new Uint8Array([1, 2, 3, 4, 5]);
Expand Down Expand Up @@ -292,7 +292,7 @@ describe("AppendBlobAPIs", () => {
);
});

it("Set tier should not work for append blob @loki", async function () {
it("Set tier should not work for append blob @loki @sql", async function () {
await appendBlobClient.create();
try {
await blobClient.setAccessTier("hot");
Expand All @@ -302,7 +302,7 @@ describe("AppendBlobAPIs", () => {
assert.fail();
});

it("Append block should work @loki", async () => {
it("Append block should work @loki @sql", async () => {
await appendBlobClient.create();
let appendBlockResponse = await appendBlobClient.appendBlock("abcdef", 6);
assert.deepStrictEqual(appendBlockResponse.blobAppendOffset, "0");
Expand Down Expand Up @@ -357,7 +357,7 @@ describe("AppendBlobAPIs", () => {
assert.deepStrictEqual(string, "abcdef123456T@");
});

it("Download append blob should work @loki", async () => {
it("Download append blob should work @loki @sql", async () => {
await appendBlobClient.create();
await appendBlobClient.appendBlock("abcdef", 6);
await appendBlobClient.appendBlock("123456", 6);
Expand All @@ -374,7 +374,7 @@ describe("AppendBlobAPIs", () => {
assert.deepStrictEqual(response.contentRange, "bytes 5-12/14");
});

it("Download append blob should work for snapshot @loki", async () => {
it("Download append blob should work for snapshot @loki @sql", async () => {
await appendBlobClient.create();
await appendBlobClient.appendBlock("abcdef", 6);

Expand All @@ -393,7 +393,7 @@ describe("AppendBlobAPIs", () => {
assert.deepEqual(response.contentMD5, await getMD5FromString("def"));
});

it("Download append blob should work for copied blob @loki", async () => {
it("Download append blob should work for copied blob @loki @sql", async () => {
await appendBlobClient.create();
await appendBlobClient.appendBlock("abcdef", 6);

Expand All @@ -410,7 +410,7 @@ describe("AppendBlobAPIs", () => {
assert.deepEqual(response.contentMD5, await getMD5FromString("def"));
});

it("Append block with invalid blob type should not work @loki", async () => {
it("Append block with invalid blob type should not work @loki @sql", async () => {
const pageBlobClient = appendBlobClient.getPageBlobClient();
await pageBlobClient.create(512);

Expand All @@ -423,7 +423,7 @@ describe("AppendBlobAPIs", () => {
assert.fail();
});

it("Append block with content length 0 should not work @loki", async () => {
it("Append block with content length 0 should not work @loki @sql", async () => {
await appendBlobClient.create();

try {
Expand All @@ -435,7 +435,7 @@ describe("AppendBlobAPIs", () => {
assert.fail();
});

it("Append block append position access condition should work @loki", async () => {
it("Append block append position access condition should work @loki @sql", async () => {
await appendBlobClient.create();
await appendBlobClient.appendBlock("a", 1, {
conditions: {
Expand Down Expand Up @@ -480,7 +480,7 @@ describe("AppendBlobAPIs", () => {
assert.fail();
});

it("Append block md5 validation should work @loki", async () => {
it("Append block md5 validation should work @loki @sql", async () => {
await appendBlobClient.create();
await appendBlobClient.appendBlock("aEf", 1, {
transactionalContentMD5: await getMD5FromString("aEf")
Expand All @@ -498,7 +498,7 @@ describe("AppendBlobAPIs", () => {
assert.fail();
});

it("Append block access condition should work @loki", async () => {
it("Append block access condition should work @loki @sql", async () => {
let response = await appendBlobClient.create();
response = await appendBlobClient.appendBlock("a", 1, {
conditions: {
Expand Down Expand Up @@ -538,7 +538,7 @@ describe("AppendBlobAPIs", () => {
assert.fail();
});

it("Append block lease condition should work @loki", async () => {
it("Append block lease condition should work @loki @sql", async () => {
await appendBlobClient.create();

const leaseId = "abcdefg";
Expand Down
28 changes: 14 additions & 14 deletions tests/blob/apis/blob.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -840,7 +840,7 @@ describe("BlobAPIs", () => {
}
});

it("Upload blob with accesstier should get accessTierInferred as false @loki", async () => {
it("Upload blob with accesstier should get accessTierInferred as false @loki @sql", async () => {
const blobName = getUniqueName("blob");

const blobClient = containerClient.getBlockBlobClient(blobName);
Expand Down Expand Up @@ -896,7 +896,7 @@ describe("BlobAPIs", () => {
);
});

it("Copy blob should work @loki", async () => {
it("Copy blob should work @loki @sql", async () => {
const sourceBlob = getUniqueName("blob");
const destBlob = getUniqueName("blob");

Expand Down Expand Up @@ -955,7 +955,7 @@ describe("BlobAPIs", () => {
);
});

it("Copy blob should work to override metadata @loki", async () => {
it("Copy blob should work to override metadata @loki @sql", async () => {
const sourceBlob = getUniqueName("blob");
const destBlob = getUniqueName("blob");

Expand All @@ -979,7 +979,7 @@ describe("BlobAPIs", () => {
assert.deepStrictEqual(result.metadata, metadata2);
});

it("Copy blob should work with source archive blob and accesstier header @loki, @sql", async () => {
it("Copy blob should work with source archive blob and accesstier header @loki @sql", async () => {
const sourceBlob = getUniqueName("blob");
const destBlob = getUniqueName("blob");

Expand Down Expand Up @@ -1014,7 +1014,7 @@ describe("BlobAPIs", () => {
assert.deepStrictEqual(result.accessTier, "Hot");
});

it("Copy blob should not override destination Lease status @loki", async () => {
it("Copy blob should not override destination Lease status @loki @sql", async () => {
const sourceBlob = getUniqueName("blob");
const destBlob = getUniqueName("blob");

Expand Down Expand Up @@ -1049,7 +1049,7 @@ describe("BlobAPIs", () => {
await destLeaseClient.releaseLease();
});

it("Copy blob should work for page blob @loki", async () => {
it("Copy blob should work for page blob @loki @sql", async () => {
const sourceBlob = getUniqueName("blob");
const destBlob = getUniqueName("blob");

Expand Down Expand Up @@ -1108,7 +1108,7 @@ describe("BlobAPIs", () => {
);
});

it("Copy blob should not work for page blob and set tier @loki", async () => {
it("Copy blob should not work for page blob and set tier @loki @sql", async () => {
const sourceBlob = getUniqueName("blob");
const destBlob = getUniqueName("blob");

Expand Down Expand Up @@ -1146,7 +1146,7 @@ describe("BlobAPIs", () => {
assert.deepStrictEqual(err.statusCode, 400);
});

it("Copy blob should fail with 400 when copy source is invalid @loki", async () => {
it("Copy blob should fail with 400 when copy source is invalid @loki @sql", async () => {
const destBlob = getUniqueName("blob");

const destBlobClient = containerClient.getBlockBlobClient(destBlob);
Expand All @@ -1163,7 +1163,7 @@ describe("BlobAPIs", () => {
assert.fail();
});

it("Copy blob should not work with ifNoneMatch * when dest exist @loki", async () => {
it("Copy blob should not work with ifNoneMatch * when dest exist @loki @sql", async () => {
const sourceBlob = getUniqueName("blob");
const destBlob = getUniqueName("blob");

Expand Down Expand Up @@ -1234,7 +1234,7 @@ describe("BlobAPIs", () => {
assert.fail();
});

it("Synchronized copy blob should work @loki", async () => {
it("Synchronized copy blob should work @loki @sql", async () => {
const sourceBlob = getUniqueName("blob");
const destBlob = getUniqueName("blob");

Expand Down Expand Up @@ -1292,7 +1292,7 @@ describe("BlobAPIs", () => {
);
});

it("Synchronized copy blob should work to override metadata @loki", async () => {
it("Synchronized copy blob should work to override metadata @loki @sql", async () => {
const sourceBlob = getUniqueName("blob");
const destBlob = getUniqueName("blob");

Expand All @@ -1316,7 +1316,7 @@ describe("BlobAPIs", () => {
assert.deepStrictEqual(result.metadata, metadata2);
});

it("Synchronized copy blob should not override destination Lease status @loki", async () => {
it("Synchronized copy blob should not override destination Lease status @loki @sql", async () => {
const sourceBlob = getUniqueName("blob");
const destBlob = getUniqueName("blob");

Expand Down Expand Up @@ -1351,7 +1351,7 @@ describe("BlobAPIs", () => {
await destLeaseClient.releaseLease();
});

it("Synchronized copy blob should work for page blob @loki", async () => {
it("Synchronized copy blob should work for page blob @loki @sql", async () => {
const sourceBlob = getUniqueName("blob");
const destBlob = getUniqueName("blob");

Expand Down Expand Up @@ -1590,7 +1590,7 @@ describe("BlobAPIs", () => {
blockBlobClient2.delete();
});

it("set blob tag should work in create page/append blob, copyFromURL. @loki", async () => {
it("set blob tag should work in create page/append blob, copyFromURL. @loki @sql", async () => {
const tags = {
tag1: "val1",
tag2: "val2",
Expand Down
5 changes: 2 additions & 3 deletions tests/blob/apis/container.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -563,8 +563,7 @@ describe("ContainerAPIs", () => {
}
});

// TODO: azure/storage-blob 12.9.0 will fail on list uncimmited blob from container, will skip the case until this is fix in SDK or Azurite
it.skip("should only show uncommitted blobs in listBlobFlatSegment with uncommittedblobs option @loki @sql", async () => {
it("should only show uncommitted blobs in listBlobFlatSegment with uncommittedblobs option @loki @sql", async () => {
const blobClient = containerClient.getBlobClient(
getUniqueName("uncommittedblob")
);
Expand Down Expand Up @@ -1043,7 +1042,7 @@ describe("ContainerAPIs", () => {
await blockBlobClient.upload("", 0);
blobClients.push(blobClient);
}
blobClients[0].createSnapshot();
await blobClients[0].createSnapshot();

// create account sas
const storageSharedKeyCredential = new StorageSharedKeyCredential(
Expand Down
Loading