Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(log-viewer-webui): Refactor S3Manager into a Fastify plugin. #689

Merged
merged 6 commits into from
Jan 31, 2025
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 11 additions & 4 deletions components/log-viewer-webui/server/src/S3Manager.js
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
import fastifyPlugin from "fastify-plugin";

import {
GetObjectCommand,
S3Client,
Expand All @@ -19,9 +21,7 @@ class S3Manager {
/**
* @param {string} region
*/
constructor (
region,
) {
constructor (region) {
haiqi96 marked this conversation as resolved.
Show resolved Hide resolved
this.#s3Client = new S3Client({
region: region,
});
Expand Down Expand Up @@ -55,4 +55,11 @@ class S3Manager {
}
}

export default S3Manager;
export default fastifyPlugin(async (app, options) => {
haiqi96 marked this conversation as resolved.
Show resolved Hide resolved
const {region} = options;
if (null === region) {
return;
}
console.log(`s3Manager initialized with region ${region}`);
haiqi96 marked this conversation as resolved.
Show resolved Hide resolved
await app.decorate("s3Manager", new S3Manager(region));
haiqi96 marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add another log after the decorate is done? Just for debugging purpose.
@junhaoliao

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think for now it's fine not to log after init completion, since we don't expect constructor to hang. In cases where the constructor throws, we shall be notified for any exception messages.

});
2 changes: 2 additions & 0 deletions components/log-viewer-webui/server/src/app.js
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import DbManager from "./DbManager.js";
import exampleRoutes from "./routes/example.js";
import queryRoutes from "./routes/query.js";
import staticRoutes from "./routes/static.js";
import S3Manager from "./S3Manager.js";


/**
Expand Down Expand Up @@ -41,6 +42,7 @@ const app = async ({
port: settings.MongoDbPort,
},
});
await server.register(S3Manager, {region: settings.StreamFilesS3Region});
}

await server.register(staticRoutes);
Expand Down
25 changes: 10 additions & 15 deletions components/log-viewer-webui/server/src/routes/query.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,22 +2,15 @@ import {StatusCodes} from "http-status-codes";

import settings from "../../settings.json" with {type: "json"};
import {EXTRACT_JOB_TYPES} from "../DbManager.js";
import S3Manager from "../S3Manager.js";


const S3_MANAGER = (
null === settings.StreamFilesS3PathPrefix ||
null === settings.StreamFilesS3Region
) ?
null :
new S3Manager(settings.StreamFilesS3Region);


/**
* Submits a stream extraction job and returns the metadata of the extracted stream.
*
* @param {object} props
* @param {import("fastify").FastifyInstance | {dbManager: DbManager}} props.fastify
* @param {import("fastify").FastifyInstance |
* {dbManager: DbManager} |
* {s3Manager: S3Manager}} props.fastify
* @param {EXTRACT_JOB_TYPES} props.jobType
* @param {number} props.logEventIdx
* @param {string} props.streamId
Expand Down Expand Up @@ -63,7 +56,9 @@ const extractStreamAndGetMetadata = async ({
/**
* Creates query routes.
*
* @param {import("fastify").FastifyInstance | {dbManager: DbManager}} fastify
* @param {import("fastify").FastifyInstance |
* {dbManager: DbManager} |
* {s3Manager: S3Manager}} fastify
* @param {import("fastify").FastifyPluginOptions} options
* @return {Promise<void>}
*/
Expand Down Expand Up @@ -96,12 +91,12 @@ const routes = async (fastify, options) => {
});
}

if (null === S3_MANAGER) {
streamMetadata.path = `/streams/${streamMetadata.path}`;
} else {
streamMetadata.path = await S3_MANAGER.getPreSignedUrl(
if (fastify.hasDecorator("s3Manager")) {
streamMetadata.path = await fastify.s3Manager.getPreSignedUrl(
`s3://${settings.StreamFilesS3PathPrefix}${streamMetadata.path}`
);
} else {
streamMetadata.path = `/streams/${streamMetadata.path}`;
}

return streamMetadata;
Expand Down