From 7eb221c1a81d8e373af1caa6dbd4a49b3e610b53 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Thu, 11 Jul 2024 12:24:18 +0000 Subject: [PATCH] deploy: dcb6dc8ea02c5cdf574405dc717b66384e26e0d2 --- 404.html | 4 ++-- FAQ/index.html | 4 ++-- assets/js/0734eb39.7fd12fdb.js | 1 - assets/js/0734eb39.d090e628.js | 1 + assets/js/3dce4fd2.169f590e.js | 1 + assets/js/3dce4fd2.c0aed4f8.js | 1 - assets/js/68ffc9f1.3aac1eaa.js | 1 - assets/js/68ffc9f1.d6e6b40c.js | 1 + assets/js/935f2afb.59ef3ce8.js | 1 + assets/js/935f2afb.74bbde3d.js | 1 - assets/js/runtime~main.2bcaa095.js | 1 - assets/js/runtime~main.daa5aa02.js | 1 + developers/how-to-build-plugins/index.html | 4 ++-- developers/how-to-create-exhaust-script/index.html | 4 ++-- developers/how-to-refine-plugins/index.html | 4 ++-- developers/how-to-submit-plugins/index.html | 4 ++-- developers/how-to-write-unit-tests/index.html | 4 ++-- developers/index.html | 4 ++-- index.html | 4 ++-- intro/index.html | 4 ++-- major-concepts/aggregation/index.html | 4 ++-- major-concepts/design-philosophy/index.html | 4 ++-- major-concepts/exhaust-script/index.html | 4 ++-- major-concepts/groupby/index.html | 4 ++-- major-concepts/if/index.html | 4 ++-- major-concepts/index.html | 4 ++-- major-concepts/manifest-file/index.html | 4 ++-- major-concepts/parameters/index.html | 4 ++-- major-concepts/pipelines/index.html | 4 ++-- major-concepts/plugins/index.html | 4 ++-- major-concepts/time/index.html | 4 ++-- pipelines/index.html | 4 ++-- pipelines/instance-metadata/index.html | 4 ++-- pipelines/sci/index.html | 4 ++-- pipelines/teads/index.html | 4 ++-- reference/cli/index.html | 4 ++-- reference/errors/index.html | 4 ++-- reference/index.html | 4 ++-- reference/plugins/index.html | 4 ++-- users/how-to-compare-files-with-if-diff/index.html | 6 +++--- users/how-to-export-csv-file-with-if-csv/index.html | 8 ++++---- users/how-to-import-plugins/index.html | 6 +++--- users/how-to-install-if/index.html | 6 +++--- users/how-to-verify-files-with-if-check/index.html | 6 +++--- users/how-to-write-manifests/index.html | 8 ++++---- users/index.html | 6 +++--- users/quick-start/index.html | 6 +++--- 47 files changed, 89 insertions(+), 89 deletions(-) delete mode 100644 assets/js/0734eb39.7fd12fdb.js create mode 100644 assets/js/0734eb39.d090e628.js create mode 100644 assets/js/3dce4fd2.169f590e.js delete mode 100644 assets/js/3dce4fd2.c0aed4f8.js delete mode 100644 assets/js/68ffc9f1.3aac1eaa.js create mode 100644 assets/js/68ffc9f1.d6e6b40c.js create mode 100644 assets/js/935f2afb.59ef3ce8.js delete mode 100644 assets/js/935f2afb.74bbde3d.js delete mode 100644 assets/js/runtime~main.2bcaa095.js create mode 100644 assets/js/runtime~main.daa5aa02.js diff --git a/404.html b/404.html index caa8f39a..13fbd1fe 100644 --- a/404.html +++ b/404.html @@ -8,13 +8,13 @@ Page Not Found | Impact Framework - +
Skip to main content
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

Page Not Found

We could not find what you were looking for.

Please contact the owner of the site that linked you to the original URL and let them know their link is broken.

- + \ No newline at end of file diff --git a/FAQ/index.html b/FAQ/index.html index a1102112..c31f0af1 100644 --- a/FAQ/index.html +++ b/FAQ/index.html @@ -8,13 +8,13 @@ FAQs | Impact Framework - +
Skip to main content
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

FAQs

How can I contribute to Impact framework?​

We welcome all kinds of contributions to Impact Framework, from bug reports to work on core features or documentation. For detailed instructions on how to start and what to expect when you contribute, please visit our contributions guidelines. There you will find guidance on how to raise bug reports, how to contribute code and what processes we have in place for handling your issues and PRs.

To contribute to these docs, you can raise pull requests against our Github repository.

In general, you can consider any ticket on our issue board open for community contributions if it does not have the core-only tag. We tag suitable introductory issues as good-first-issue - these are great places to get started.

Is there any way to auto-generate manifest files?​

Not really. We provide a command line tool called if-env which will generate a template manifest for you if you run it in an empty folder. We also provide a built-in feature called mockObservations that will autogenerate some input data for you (designed to generate dummy data for experimenting or scenario testing). There are also importer plugins that will pull data from files or APIs into IF manifests. However, we do not currently provide any tooling that will generate a full manifest file for you - we think humans in the loop are important for ensuring the nuances of your system are captured in the manifest and in general, we'd prefer not to automate that away. That said, there is potential for tooling that helps abstract out the more repetitive tasks associated with building large manifest files.

Where do I get the data for my manifest file?​

This is up to you! One of the strengths of the IF is that you can build manifests your way. We expect input data to come from an external file or an API in most cases, although you can manually add input data into your manifest too. People have built importer plugins for Azure's monitor API, Prometheus databases, CSV files, Datadog, and others. These importer plugins can be found on the Explorer website.

What is your vision for the IF?​

The vision is to build a protocol that enables you to calculate the appropriate environmental impact for your application, whatever it is. We want to be an open communication standard - our dream is that an IF manifest is the primary way that people share the environmental impacts of their systems.

Right now we consider our core functionality to be calculating software carbon intensity scores for software applications running in the cloud, but we have people using IF on-premise and even for supply chain modeling and other non-software applications. The idea is that we provide the minimal infrastructure required for you to build up different use cases using plugins.

Is there a way to generate an audit/report along with a csv/yaml output?​

No - we see the manifest file and associated manifest as a new type of "executable audit". The IF itself does not generate any other form of report, although there is nothing stopping others from building out this functionality on top of IF.

IF makes things transparent, but how can you fight against users inputting incorrect or misleading data?​

We can't really stop people providing fake data into a manifest file. We're very interested in ways we can verify that the computation was done correctly and provide public proofs, but without direct access to a user's systems we can never guarantee they are providing truthful data. This is not unique to IF - fraud is a problem across all industries.

Is it planned to make the IF more user-friendly or also more usable for less technical people?​

IF is a low-level infrastructure project. The core team focuses on building out solid foundations for others to build UIs, apps etc on top. There is no official IF UI.

Does it only calculate emissions for CPU usage or does it also work with other meters like storage/bandwidth?​

We do have basic models for memory and network usage in addition to CPU. You can create plugins for anything you can observe and model.

Have you compared the results to the carbon reports that cloud providers are producing for their customers?​

No we haven't - we've been focussing on capturing impacts of individual applications over relatively short periods so far. It's difficult to find the underlying data to build the comparisons.

Are you planning to support e.g. for PaaS, IaaS, alternative cloud providers...etc?​

We are focused on building out the low-level core infrastructure so that users can apply IF to any use case. We do not currently intend to build out support for any specific service. We want users to build plugins and publish them to the Explorer and share manifests to support individual use cases.

What is on your roadmap?​

You can check what's coming up for IF core development by checking our Github discussion board. We publish our plans in advance of incorporating them into our development sprints so that the community can comment, ask for more information and report any potential impact for their specific use cases.

- + \ No newline at end of file diff --git a/assets/js/0734eb39.7fd12fdb.js b/assets/js/0734eb39.7fd12fdb.js deleted file mode 100644 index 69d33d92..00000000 --- a/assets/js/0734eb39.7fd12fdb.js +++ /dev/null @@ -1 +0,0 @@ -"use strict";(self.webpackChunkgreen_software_training=self.webpackChunkgreen_software_training||[]).push([[939],{4137:(e,t,n)=>{n.d(t,{Zo:()=>f,kt:()=>d});var i=n(7294);function r(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function o(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);t&&(i=i.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,i)}return n}function a(e){for(var t=1;t=0||(r[n]=e[n]);return r}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(i=0;i=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(r[n]=e[n])}return r}var l=i.createContext({}),u=function(e){var t=i.useContext(l),n=t;return e&&(n="function"==typeof e?e(t):a(a({},t),e)),n},f=function(e){var t=u(e.components);return i.createElement(l.Provider,{value:t},e.children)},p={inlineCode:"code",wrapper:function(e){var t=e.children;return i.createElement(i.Fragment,{},t)}},c=i.forwardRef((function(e,t){var n=e.components,r=e.mdxType,o=e.originalType,l=e.parentName,f=s(e,["components","mdxType","originalType","parentName"]),c=u(n),d=r,m=c["".concat(l,".").concat(d)]||c[d]||p[d]||o;return n?i.createElement(m,a(a({ref:t},f),{},{components:n})):i.createElement(m,a({ref:t},f))}));function d(e,t){var n=arguments,r=t&&t.mdxType;if("string"==typeof e||r){var o=n.length,a=new Array(o);a[0]=c;var s={};for(var l in t)hasOwnProperty.call(t,l)&&(s[l]=t[l]);s.originalType=e,s.mdxType="string"==typeof e?e:r,a[1]=s;for(var u=2;u{n.r(t),n.d(t,{assets:()=>l,contentTitle:()=>a,default:()=>p,frontMatter:()=>o,metadata:()=>s,toc:()=>u});var i=n(7462),r=(n(7294),n(4137));const o={sidebar_position:6},a="How to compare files with `if-diff`",s={unversionedId:"users/how-to-compare-files-with-if-diff",id:"users/how-to-compare-files-with-if-diff",title:"How to compare files with `if-diff`",description:"if-diff is a command line tool that allows you to compare two if-run output files. They either match according to if-diff's matching rules, or they don't. If they match, then if-diff returns a simple success response. If the differ, then if-diff returns a report of the differences it finds.",source:"@site/docs/users/how-to-compare-files-with-if-diff.md",sourceDirName:"users",slug:"/users/how-to-compare-files-with-if-diff",permalink:"/users/how-to-compare-files-with-if-diff",draft:!1,editUrl:"https://github.com/Green-Software-Foundation/if-docs/edit/master/docs/users/how-to-compare-files-with-if-diff.md",tags:[],version:"current",sidebarPosition:6,frontMatter:{sidebar_position:6},sidebar:"tutorialSidebar",previous:{title:"how-to-export-csv-file-with-if-csv",permalink:"/users/how-to-export-csv-file-with-if-csv"},next:{title:"Verifying IF outputs with `if-check`",permalink:"/users/how-to-verify-files-with-if-check"}},l={},u=[{value:"Why is this useful?",id:"why-is-this-useful",level:2},{value:"Example: output verification",id:"example-output-verification",level:2},{value:"Example: debugging",id:"example-debugging",level:2}],f={toc:u};function p(e){let{components:t,...n}=e;return(0,r.kt)("wrapper",(0,i.Z)({},f,n,{components:t,mdxType:"MDXLayout"}),(0,r.kt)("h1",{id:"how-to-compare-files-with-if-diff"},"How to compare files with ",(0,r.kt)("inlineCode",{parentName:"h1"},"if-diff")),(0,r.kt)("p",null,(0,r.kt)("inlineCode",{parentName:"p"},"if-diff")," is a command line tool that allows you to compare two ",(0,r.kt)("inlineCode",{parentName:"p"},"if-run")," output files. They either match according to ",(0,r.kt)("inlineCode",{parentName:"p"},"if-diff"),"'s matching rules, or they don't. If they match, then ",(0,r.kt)("inlineCode",{parentName:"p"},"if-diff")," returns a simple success response. If the differ, then ",(0,r.kt)("inlineCode",{parentName:"p"},"if-diff")," returns a report of the differences it finds."),(0,r.kt)("h2",{id:"why-is-this-useful"},"Why is this useful?"),(0,r.kt)("p",null,(0,r.kt)("inlineCode",{parentName:"p"},"if-diff")," can be used to verify that a given output file was correctly executed and that it was not tampered with after it was computed. Imagine you received an output file from someone, reporting their carbon expenditure. It is better to verify than trust this person's report, so you simply delete the ",(0,r.kt)("inlineCode",{parentName:"p"},"outputs")," block from the file (creating a manifest), run it through ",(0,r.kt)("inlineCode",{parentName:"p"},"if-run")," and compare the output to the original file you received. All being well, the two files are identical. if not, you can see exactly where the differences are."),(0,r.kt)("p",null,(0,r.kt)("inlineCode",{parentName:"p"},"if-diff")," can also be used for debugging your own files. Maybe you have some large manifest files in development and have accidentally introduced some changes that you are now struggling to identify, but are leading to different ",(0,r.kt)("inlineCode",{parentName:"p"},"if-run")," outcomes. ",(0,r.kt)("inlineCode",{parentName:"p"},"if-diff")," can quickly scan your two files and tell you where the differences are so you can get back in sync."),(0,r.kt)("h2",{id:"example-output-verification"},"Example: output verification"),(0,r.kt)("p",null,"Let's say someone provides you with this output file, ",(0,r.kt)("inlineCode",{parentName:"p"},"given-output-file.yml"),":"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-yaml"},"name: sum\ndescription: successful path\ntags:\ninitialize:\n plugins:\n sum:\n method: Sum\n path: 'builtin'\n global-config:\n input-parameters: ['cpu/energy', 'network/energy']\n output-parameter: 'energy'\ntree:\n children:\n child:\n pipeline:\n - sum\n config:\n sum:\n inputs:\n - timestamp: 2023-08-06T00:00\n duration: 3600\n cpu/energy: 0.001\n network/energy: 0.001\n outputs:\n - timestamp: 2023-08-06T00:00\n duration: 3600\n cpu/energy: 0.001\n network/energy: 0.001\n energy: 0.0005\n")),(0,r.kt)("p",null,"This manifest simply sums two components, ",(0,r.kt)("inlineCode",{parentName:"p"},"cpu/energy")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"network/energy")," and assigns the result to ",(0,r.kt)("inlineCode",{parentName:"p"},"energy")," in the outputs array. You receive this file and feel like something's not quite right. So you delete the outputs block to create ",(0,r.kt)("inlineCode",{parentName:"p"},"test-manifest.yml"),":"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-yaml"},"name: sum\ndescription: successful path\ntags:\ninitialize:\n plugins:\n sum:\n method: Sum\n path: \"builtin\"\n global-config:\n input-parameters: ['cpu/energy', 'network/energy']\n output-parameter: 'energy'\ntree:\n children:\n child:\n pipeline:\n - sum\n config:\n sum:\n inputs:\n - timestamp: 2023-08-06T00:00\n duration: 3600\n cpu/energy: 0.001\n network/energy: 0.001\n")),(0,r.kt)("p",null,"Now you want to ",(0,r.kt)("em",{parentName:"p"},"run")," the manifest through ",(0,r.kt)("inlineCode",{parentName:"p"},"if-run")," and compare the result to the given output file. You can do this by piping the result of ",(0,r.kt)("inlineCode",{parentName:"p"},"if-run")," directly into ",(0,r.kt)("inlineCode",{parentName:"p"},"if-diff")," as follows:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-bash"},"if-run -m test-manifest.yml | if-diff --target given-output-file.yml\n")),(0,r.kt)("p",null,"The result is:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-sh"},"Files do not match!\ntree.children.child[0].energy\nsource: 0.002\ntarget: 0.0005\n")),(0,r.kt)("p",null,"Uh oh. It seems there has been some mistake or tampering with the outputs in ",(0,r.kt)("inlineCode",{parentName:"p"},"given-output-file.yml"),". The right result of summing ",(0,r.kt)("inlineCode",{parentName:"p"},"cpu/energy")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"network/energy")," is 0.002, but they reported 0.0005. You can now query that result with the sender and ask them to fix it."),(0,r.kt)("p",null,"Obviously, this is an arbitrary, simplified example, but ",(0,r.kt)("inlineCode",{parentName:"p"},"if-diff")," enables you to do this kind of output verification on very complex manifests where errors are harder to spot by eye and to do it programmatically over large numbers of files."),(0,r.kt)("h2",{id:"example-debugging"},"Example: debugging"),(0,r.kt)("p",null,"Imagine you developed a manifest that was giving you a consistent result, but now when you run it your result is different and you are not sure why. Maybe one of your colleagues changed something and forgot to tell you, maybe you accidentally inserted or removed something while you were working."),(0,r.kt)("p",null,"You could revert to an archived version, but you moved all the components around into a structure you prefer! ",(0,r.kt)("inlineCode",{parentName:"p"},"if-diff")," has you covered. It will step through the files identifying all the functional differences between the files to help you identify the problematic one(s)."),(0,r.kt)("p",null,"You have ",(0,r.kt)("inlineCode",{parentName:"p"},"original-manifest.yml")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"new-manifest.yml"),". Pass them to ",(0,r.kt)("inlineCode",{parentName:"p"},"if-diff")," as follows:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-sh"},"if-diff --source original-manifest.yml --target new-manifest.yml\n")),(0,r.kt)("p",null,(0,r.kt)("inlineCode",{parentName:"p"},"if-diff")," will report each difference it finds. You can fix the difference and run ",(0,r.kt)("inlineCode",{parentName:"p"},"if-diff")," again until you get a success response - since ",(0,r.kt)("inlineCode",{parentName:"p"},"if-diff")," ignores positional differences and only considers differences in ",(0,r.kt)("inlineCode",{parentName:"p"},"context")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"tree")," keys and values, your two manifests will run identically even though you persist your tree reorganization."))}p.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/assets/js/0734eb39.d090e628.js b/assets/js/0734eb39.d090e628.js new file mode 100644 index 00000000..5ba7cf6d --- /dev/null +++ b/assets/js/0734eb39.d090e628.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkgreen_software_training=self.webpackChunkgreen_software_training||[]).push([[939],{4137:(e,t,n)=>{n.d(t,{Zo:()=>f,kt:()=>d});var i=n(7294);function r(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function o(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);t&&(i=i.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,i)}return n}function a(e){for(var t=1;t=0||(r[n]=e[n]);return r}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(i=0;i=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(r[n]=e[n])}return r}var s=i.createContext({}),u=function(e){var t=i.useContext(s),n=t;return e&&(n="function"==typeof e?e(t):a(a({},t),e)),n},f=function(e){var t=u(e.components);return i.createElement(s.Provider,{value:t},e.children)},p={inlineCode:"code",wrapper:function(e){var t=e.children;return i.createElement(i.Fragment,{},t)}},c=i.forwardRef((function(e,t){var n=e.components,r=e.mdxType,o=e.originalType,s=e.parentName,f=l(e,["components","mdxType","originalType","parentName"]),c=u(n),d=r,m=c["".concat(s,".").concat(d)]||c[d]||p[d]||o;return n?i.createElement(m,a(a({ref:t},f),{},{components:n})):i.createElement(m,a({ref:t},f))}));function d(e,t){var n=arguments,r=t&&t.mdxType;if("string"==typeof e||r){var o=n.length,a=new Array(o);a[0]=c;var l={};for(var s in t)hasOwnProperty.call(t,s)&&(l[s]=t[s]);l.originalType=e,l.mdxType="string"==typeof e?e:r,a[1]=l;for(var u=2;u{n.r(t),n.d(t,{assets:()=>s,contentTitle:()=>a,default:()=>p,frontMatter:()=>o,metadata:()=>l,toc:()=>u});var i=n(7462),r=(n(7294),n(4137));const o={sidebar_position:6},a="How to compare files with `if-diff`",l={unversionedId:"users/how-to-compare-files-with-if-diff",id:"users/how-to-compare-files-with-if-diff",title:"How to compare files with `if-diff`",description:"if-diff is a command line tool that allows you to compare two if-run output files. They either match according to if-diff's matching rules, or they don't. If they match, then if-diff returns a simple success response. If the differ, then if-diff returns a report of the differences it finds.",source:"@site/docs/users/how-to-compare-files-with-if-diff.md",sourceDirName:"users",slug:"/users/how-to-compare-files-with-if-diff",permalink:"/users/how-to-compare-files-with-if-diff",draft:!1,editUrl:"https://github.com/Green-Software-Foundation/if-docs/edit/master/docs/users/how-to-compare-files-with-if-diff.md",tags:[],version:"current",sidebarPosition:6,frontMatter:{sidebar_position:6},sidebar:"tutorialSidebar",previous:{title:"Exporting CSV file with `if-csv`",permalink:"/users/how-to-export-csv-file-with-if-csv"},next:{title:"Verifying IF outputs with `if-check`",permalink:"/users/how-to-verify-files-with-if-check"}},s={},u=[{value:"Why is this useful?",id:"why-is-this-useful",level:2},{value:"Example: output verification",id:"example-output-verification",level:2},{value:"Example: debugging",id:"example-debugging",level:2}],f={toc:u};function p(e){let{components:t,...n}=e;return(0,r.kt)("wrapper",(0,i.Z)({},f,n,{components:t,mdxType:"MDXLayout"}),(0,r.kt)("h1",{id:"how-to-compare-files-with-if-diff"},"How to compare files with ",(0,r.kt)("inlineCode",{parentName:"h1"},"if-diff")),(0,r.kt)("p",null,(0,r.kt)("inlineCode",{parentName:"p"},"if-diff")," is a command line tool that allows you to compare two ",(0,r.kt)("inlineCode",{parentName:"p"},"if-run")," output files. They either match according to ",(0,r.kt)("inlineCode",{parentName:"p"},"if-diff"),"'s matching rules, or they don't. If they match, then ",(0,r.kt)("inlineCode",{parentName:"p"},"if-diff")," returns a simple success response. If the differ, then ",(0,r.kt)("inlineCode",{parentName:"p"},"if-diff")," returns a report of the differences it finds."),(0,r.kt)("h2",{id:"why-is-this-useful"},"Why is this useful?"),(0,r.kt)("p",null,(0,r.kt)("inlineCode",{parentName:"p"},"if-diff")," can be used to verify that a given output file was correctly executed and that it was not tampered with after it was computed. Imagine you received an output file from someone, reporting their carbon expenditure. It is better to verify than trust this person's report, so you simply delete the ",(0,r.kt)("inlineCode",{parentName:"p"},"outputs")," block from the file (creating a manifest), run it through ",(0,r.kt)("inlineCode",{parentName:"p"},"if-run")," and compare the output to the original file you received. All being well, the two files are identical. if not, you can see exactly where the differences are."),(0,r.kt)("p",null,(0,r.kt)("inlineCode",{parentName:"p"},"if-diff")," can also be used for debugging your own files. Maybe you have some large manifest files in development and have accidentally introduced some changes that you are now struggling to identify, but are leading to different ",(0,r.kt)("inlineCode",{parentName:"p"},"if-run")," outcomes. ",(0,r.kt)("inlineCode",{parentName:"p"},"if-diff")," can quickly scan your two files and tell you where the differences are so you can get back in sync."),(0,r.kt)("h2",{id:"example-output-verification"},"Example: output verification"),(0,r.kt)("p",null,"Let's say someone provides you with this output file, ",(0,r.kt)("inlineCode",{parentName:"p"},"given-output-file.yml"),":"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-yaml"},"name: sum\ndescription: successful path\ntags:\ninitialize:\n plugins:\n sum:\n method: Sum\n path: 'builtin'\n global-config:\n input-parameters: ['cpu/energy', 'network/energy']\n output-parameter: 'energy'\ntree:\n children:\n child:\n pipeline:\n - sum\n config:\n sum:\n inputs:\n - timestamp: 2023-08-06T00:00\n duration: 3600\n cpu/energy: 0.001\n network/energy: 0.001\n outputs:\n - timestamp: 2023-08-06T00:00\n duration: 3600\n cpu/energy: 0.001\n network/energy: 0.001\n energy: 0.0005\n")),(0,r.kt)("p",null,"This manifest simply sums two components, ",(0,r.kt)("inlineCode",{parentName:"p"},"cpu/energy")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"network/energy")," and assigns the result to ",(0,r.kt)("inlineCode",{parentName:"p"},"energy")," in the outputs array. You receive this file and feel like something's not quite right. So you delete the outputs block to create ",(0,r.kt)("inlineCode",{parentName:"p"},"test-manifest.yml"),":"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-yaml"},"name: sum\ndescription: successful path\ntags:\ninitialize:\n plugins:\n sum:\n method: Sum\n path: \"builtin\"\n global-config:\n input-parameters: ['cpu/energy', 'network/energy']\n output-parameter: 'energy'\ntree:\n children:\n child:\n pipeline:\n - sum\n config:\n sum:\n inputs:\n - timestamp: 2023-08-06T00:00\n duration: 3600\n cpu/energy: 0.001\n network/energy: 0.001\n")),(0,r.kt)("p",null,"Now you want to ",(0,r.kt)("em",{parentName:"p"},"run")," the manifest through ",(0,r.kt)("inlineCode",{parentName:"p"},"if-run")," and compare the result to the given output file. You can do this by piping the result of ",(0,r.kt)("inlineCode",{parentName:"p"},"if-run")," directly into ",(0,r.kt)("inlineCode",{parentName:"p"},"if-diff")," as follows:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-bash"},"if-run -m test-manifest.yml | if-diff --target given-output-file.yml\n")),(0,r.kt)("p",null,"The result is:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-sh"},"Files do not match!\ntree.children.child[0].energy\nsource: 0.002\ntarget: 0.0005\n")),(0,r.kt)("p",null,"Uh oh. It seems there has been some mistake or tampering with the outputs in ",(0,r.kt)("inlineCode",{parentName:"p"},"given-output-file.yml"),". The right result of summing ",(0,r.kt)("inlineCode",{parentName:"p"},"cpu/energy")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"network/energy")," is 0.002, but they reported 0.0005. You can now query that result with the sender and ask them to fix it."),(0,r.kt)("p",null,"Obviously, this is an arbitrary, simplified example, but ",(0,r.kt)("inlineCode",{parentName:"p"},"if-diff")," enables you to do this kind of output verification on very complex manifests where errors are harder to spot by eye and to do it programmatically over large numbers of files."),(0,r.kt)("h2",{id:"example-debugging"},"Example: debugging"),(0,r.kt)("p",null,"Imagine you developed a manifest that was giving you a consistent result, but now when you run it your result is different and you are not sure why. Maybe one of your colleagues changed something and forgot to tell you, maybe you accidentally inserted or removed something while you were working."),(0,r.kt)("p",null,"You could revert to an archived version, but you moved all the components around into a structure you prefer! ",(0,r.kt)("inlineCode",{parentName:"p"},"if-diff")," has you covered. It will step through the files identifying all the functional differences between the files to help you identify the problematic one(s)."),(0,r.kt)("p",null,"You have ",(0,r.kt)("inlineCode",{parentName:"p"},"original-manifest.yml")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"new-manifest.yml"),". Pass them to ",(0,r.kt)("inlineCode",{parentName:"p"},"if-diff")," as follows:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-sh"},"if-diff --source original-manifest.yml --target new-manifest.yml\n")),(0,r.kt)("p",null,(0,r.kt)("inlineCode",{parentName:"p"},"if-diff")," will report each difference it finds. You can fix the difference and run ",(0,r.kt)("inlineCode",{parentName:"p"},"if-diff")," again until you get a success response - since ",(0,r.kt)("inlineCode",{parentName:"p"},"if-diff")," ignores positional differences and only considers differences in ",(0,r.kt)("inlineCode",{parentName:"p"},"context")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"tree")," keys and values, your two manifests will run identically even though you persist your tree reorganization."))}p.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/assets/js/3dce4fd2.169f590e.js b/assets/js/3dce4fd2.169f590e.js new file mode 100644 index 00000000..d33e6bf8 --- /dev/null +++ b/assets/js/3dce4fd2.169f590e.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkgreen_software_training=self.webpackChunkgreen_software_training||[]).push([[278],{4137:(e,n,t)=>{t.d(n,{Zo:()=>p,kt:()=>m});var r=t(7294);function i(e,n,t){return n in e?Object.defineProperty(e,n,{value:t,enumerable:!0,configurable:!0,writable:!0}):e[n]=t,e}function s(e,n){var t=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);n&&(r=r.filter((function(n){return Object.getOwnPropertyDescriptor(e,n).enumerable}))),t.push.apply(t,r)}return t}function o(e){for(var n=1;n=0||(i[t]=e[t]);return i}(e,n);if(Object.getOwnPropertySymbols){var s=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,t)&&(i[t]=e[t])}return i}var l=r.createContext({}),c=function(e){var n=r.useContext(l),t=n;return e&&(t="function"==typeof e?e(n):o(o({},n),e)),t},p=function(e){var n=c(e.components);return r.createElement(l.Provider,{value:n},e.children)},u={inlineCode:"code",wrapper:function(e){var n=e.children;return r.createElement(r.Fragment,{},n)}},f=r.forwardRef((function(e,n){var t=e.components,i=e.mdxType,s=e.originalType,l=e.parentName,p=a(e,["components","mdxType","originalType","parentName"]),f=c(t),m=i,d=f["".concat(l,".").concat(m)]||f[m]||u[m]||s;return t?r.createElement(d,o(o({ref:n},p),{},{components:t})):r.createElement(d,o({ref:n},p))}));function m(e,n){var t=arguments,i=n&&n.mdxType;if("string"==typeof e||i){var s=t.length,o=new Array(s);o[0]=f;var a={};for(var l in n)hasOwnProperty.call(n,l)&&(a[l]=n[l]);a.originalType=e,a.mdxType="string"==typeof e?e:i,o[1]=a;for(var c=2;c{t.r(n),t.d(n,{assets:()=>l,contentTitle:()=>o,default:()=>u,frontMatter:()=>s,metadata:()=>a,toc:()=>c});var r=t(7462),i=(t(7294),t(4137));const s={sidebar_position:5},o="Exporting CSV file with `if-csv`",a={unversionedId:"users/how-to-export-csv-file-with-if-csv",id:"users/how-to-export-csv-file-with-if-csv",title:"Exporting CSV file with `if-csv`",description:"IF includes a command line tool called if-csv which is designed to export CSV files based on a specified manifest file and metric.",source:"@site/docs/users/how-to-export-csv-file-with-if-csv.md",sourceDirName:"users",slug:"/users/how-to-export-csv-file-with-if-csv",permalink:"/users/how-to-export-csv-file-with-if-csv",draft:!1,editUrl:"https://github.com/Green-Software-Foundation/if-docs/edit/master/docs/users/how-to-export-csv-file-with-if-csv.md",tags:[],version:"current",sidebarPosition:5,frontMatter:{sidebar_position:5},sidebar:"tutorialSidebar",previous:{title:"How to write a manifest file",permalink:"/users/how-to-write-manifests"},next:{title:"How to compare files with `if-diff`",permalink:"/users/how-to-compare-files-with-if-diff"}},l={},c=[{value:"Example:",id:"example",level:2}],p={toc:c};function u(e){let{components:n,...t}=e;return(0,i.kt)("wrapper",(0,r.Z)({},p,t,{components:n,mdxType:"MDXLayout"}),(0,i.kt)("h1",{id:"exporting-csv-file-with-if-csv"},"Exporting CSV file with ",(0,i.kt)("inlineCode",{parentName:"h1"},"if-csv")),(0,i.kt)("p",null,"IF includes a command line tool called ",(0,i.kt)("inlineCode",{parentName:"p"},"if-csv")," which is designed to export CSV files based on a specified manifest file and metric."),(0,i.kt)("h2",{id:"example"},"Example:"),(0,i.kt)("p",null,"Let's execute this manifest file. This manifest simply sums two components, ",(0,i.kt)("inlineCode",{parentName:"p"},"cpu/energy")," and ",(0,i.kt)("inlineCode",{parentName:"p"},"network/energy")," and assigns the result to ",(0,i.kt)("inlineCode",{parentName:"p"},"energy")," in the outputs array."),(0,i.kt)("pre",null,(0,i.kt)("code",{parentName:"pre",className:"language-yaml"},"name: sum\ndescription: successful path\ntags: null\ninitialize:\n plugins:\n sum:\n path: builtin\n method: Sum\n global-config:\n input-parameters:\n - cpu/energy\n - network/energy\n output-parameter: energy\n outputs:\n - yaml\nexecution:\n command: >-\n /Users/manushak/.npm/_npx/1bf7c3c15bf47d04/node_modules/.bin/ts-node\n /Users/manushak/Documents/Projects/Green-Software/if/src/if-run/index.ts -m\n ./manifests/test.yaml -o ./manifests/re-test\n environment:\n if-version: 0.5.0\n os: macOS\n os-version: 13.6.7\n node-version: 18.20.0\n date-time: 2024-07-09T16:00:58.218Z (UTC)\n dependencies:\n - '@babel/core@7.22.10'\n - '@babel/preset-typescript@7.23.3'\n - '@commitlint/cli@18.6.0'\n - '@commitlint/config-conventional@18.6.0'\n - '@grnsft/if-core@0.0.10'\n - '@grnsft/if-plugins@v0.3.2 extraneous -> file:../../../if-models'\n - >-\n @grnsft/if-unofficial-plugins@v0.3.0 extraneous ->\n file:../../../if-unofficial-models\n - '@jest/globals@29.7.0'\n - '@types/jest@29.5.8'\n - '@types/js-yaml@4.0.9'\n - '@types/luxon@3.4.2'\n - '@types/node@20.9.0'\n - axios-mock-adapter@1.22.0\n - axios@1.7.2\n - cross-env@7.0.3\n - csv-parse@5.5.6\n - csv-stringify@6.4.6\n - fixpack@4.0.0\n - gts@5.2.0\n - husky@8.0.3\n - jest@29.7.0\n - js-yaml@4.1.0\n - lint-staged@15.2.2\n - luxon@3.4.4\n - release-it@16.3.0\n - rimraf@5.0.5\n - ts-command-line-args@2.5.1\n - ts-jest@29.1.1\n - typescript-cubic-spline@1.0.1\n - typescript@5.2.2\n - winston@3.11.0\n - zod@3.22.4\n status: success\ntree:\n children:\n child:\n pipeline:\n - sum\n config:\n sum: null\n inputs:\n - timestamp: 2023-08-06T00:00\n duration: 3600\n cpu/energy: 0.001\n network/energy: 0.001\n outputs:\n - timestamp: 2023-08-06T00:00\n duration: 3600\n cpu/energy: 0.001\n network/energy: 0.001\n energy: 0.002\n")),(0,i.kt)("p",null,"To generate a CSV file in the provided path, run the following command:"),(0,i.kt)("pre",null,(0,i.kt)("code",{parentName:"pre",className:"language-sh"},"if-csv -m ./sum.yaml -p energy -o ./output-sum\n")),(0,i.kt)("p",null,"To print data in the console, you need to run:"),(0,i.kt)("pre",null,(0,i.kt)("code",{parentName:"pre",className:"language-sh"},"if-csv -m ./sum.yaml -p energy\n")),(0,i.kt)("p",null,"The output will be:"),(0,i.kt)("pre",null,(0,i.kt)("code",{parentName:"pre",className:"language-sh"},"Path,2023-08-06T00:00\ntree.children.child.energy,0.002\n")),(0,i.kt)("p",null,"Alternatively, you can pipe the result from ",(0,i.kt)("inlineCode",{parentName:"p"},"if-run"),"."),(0,i.kt)("pre",null,(0,i.kt)("code",{parentName:"pre",className:"language-sh"},"if-run -m ./sum.yaml | if-csv -p energy\n")))}u.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/assets/js/3dce4fd2.c0aed4f8.js b/assets/js/3dce4fd2.c0aed4f8.js deleted file mode 100644 index f39979d5..00000000 --- a/assets/js/3dce4fd2.c0aed4f8.js +++ /dev/null @@ -1 +0,0 @@ -"use strict";(self.webpackChunkgreen_software_training=self.webpackChunkgreen_software_training||[]).push([[278],{4137:(e,n,t)=>{t.d(n,{Zo:()=>p,kt:()=>m});var r=t(7294);function i(e,n,t){return n in e?Object.defineProperty(e,n,{value:t,enumerable:!0,configurable:!0,writable:!0}):e[n]=t,e}function o(e,n){var t=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);n&&(r=r.filter((function(n){return Object.getOwnPropertyDescriptor(e,n).enumerable}))),t.push.apply(t,r)}return t}function s(e){for(var n=1;n=0||(i[t]=e[t]);return i}(e,n);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,t)&&(i[t]=e[t])}return i}var l=r.createContext({}),c=function(e){var n=r.useContext(l),t=n;return e&&(t="function"==typeof e?e(n):s(s({},n),e)),t},p=function(e){var n=c(e.components);return r.createElement(l.Provider,{value:n},e.children)},u={inlineCode:"code",wrapper:function(e){var n=e.children;return r.createElement(r.Fragment,{},n)}},f=r.forwardRef((function(e,n){var t=e.components,i=e.mdxType,o=e.originalType,l=e.parentName,p=a(e,["components","mdxType","originalType","parentName"]),f=c(t),m=i,d=f["".concat(l,".").concat(m)]||f[m]||u[m]||o;return t?r.createElement(d,s(s({ref:n},p),{},{components:t})):r.createElement(d,s({ref:n},p))}));function m(e,n){var t=arguments,i=n&&n.mdxType;if("string"==typeof e||i){var o=t.length,s=new Array(o);s[0]=f;var a={};for(var l in n)hasOwnProperty.call(n,l)&&(a[l]=n[l]);a.originalType=e,a.mdxType="string"==typeof e?e:i,s[1]=a;for(var c=2;c{t.r(n),t.d(n,{assets:()=>l,contentTitle:()=>s,default:()=>u,frontMatter:()=>o,metadata:()=>a,toc:()=>c});var r=t(7462),i=(t(7294),t(4137));const o={sidebar_position:5},s=void 0,a={unversionedId:"users/how-to-export-csv-file-with-if-csv",id:"users/how-to-export-csv-file-with-if-csv",title:"how-to-export-csv-file-with-if-csv",description:"Exporting CSV file with if-csv",source:"@site/docs/users/how-to-export-csv-file-with-if-csv.md",sourceDirName:"users",slug:"/users/how-to-export-csv-file-with-if-csv",permalink:"/users/how-to-export-csv-file-with-if-csv",draft:!1,editUrl:"https://github.com/Green-Software-Foundation/if-docs/edit/master/docs/users/how-to-export-csv-file-with-if-csv.md",tags:[],version:"current",sidebarPosition:5,frontMatter:{sidebar_position:5},sidebar:"tutorialSidebar",previous:{title:"How to write a manifest file",permalink:"/users/how-to-write-manifests"},next:{title:"How to compare files with `if-diff`",permalink:"/users/how-to-compare-files-with-if-diff"}},l={},c=[{value:"Exporting CSV file with if-csv",id:"exporting-csv-file-with-if-csv",level:2},{value:"Example:",id:"example",level:2}],p={toc:c};function u(e){let{components:n,...t}=e;return(0,i.kt)("wrapper",(0,r.Z)({},p,t,{components:n,mdxType:"MDXLayout"}),(0,i.kt)("h2",{id:"exporting-csv-file-with-if-csv"},"Exporting CSV file with ",(0,i.kt)("inlineCode",{parentName:"h2"},"if-csv")),(0,i.kt)("p",null,"IF includes a command line tool called ",(0,i.kt)("inlineCode",{parentName:"p"},"if-csv")," which is designed to export CSV files based on a specified manifest file and metric."),(0,i.kt)("h2",{id:"example"},"Example:"),(0,i.kt)("p",null,"Let's execute this manifest file. This manifest simply sums two components, ",(0,i.kt)("inlineCode",{parentName:"p"},"cpu/energy")," and ",(0,i.kt)("inlineCode",{parentName:"p"},"network/energy")," and assigns the result to ",(0,i.kt)("inlineCode",{parentName:"p"},"energy")," in the outputs array."),(0,i.kt)("pre",null,(0,i.kt)("code",{parentName:"pre",className:"language-yaml"},"name: sum\ndescription: successful path\ntags: null\ninitialize:\n plugins:\n sum:\n path: builtin\n method: Sum\n global-config:\n input-parameters:\n - cpu/energy\n - network/energy\n output-parameter: energy\n outputs:\n - yaml\nexecution:\n command: >-\n /Users/manushak/.npm/_npx/1bf7c3c15bf47d04/node_modules/.bin/ts-node\n /Users/manushak/Documents/Projects/Green-Software/if/src/if-run/index.ts -m\n ./manifests/test.yaml -o ./manifests/re-test\n environment:\n if-version: 0.5.0\n os: macOS\n os-version: 13.6.7\n node-version: 18.20.0\n date-time: 2024-07-09T16:00:58.218Z (UTC)\n dependencies:\n - '@babel/core@7.22.10'\n - '@babel/preset-typescript@7.23.3'\n - '@commitlint/cli@18.6.0'\n - '@commitlint/config-conventional@18.6.0'\n - '@grnsft/if-core@0.0.10'\n - '@grnsft/if-plugins@v0.3.2 extraneous -> file:../../../if-models'\n - >-\n @grnsft/if-unofficial-plugins@v0.3.0 extraneous ->\n file:../../../if-unofficial-models\n - '@jest/globals@29.7.0'\n - '@types/jest@29.5.8'\n - '@types/js-yaml@4.0.9'\n - '@types/luxon@3.4.2'\n - '@types/node@20.9.0'\n - axios-mock-adapter@1.22.0\n - axios@1.7.2\n - cross-env@7.0.3\n - csv-parse@5.5.6\n - csv-stringify@6.4.6\n - fixpack@4.0.0\n - gts@5.2.0\n - husky@8.0.3\n - jest@29.7.0\n - js-yaml@4.1.0\n - lint-staged@15.2.2\n - luxon@3.4.4\n - release-it@16.3.0\n - rimraf@5.0.5\n - ts-command-line-args@2.5.1\n - ts-jest@29.1.1\n - typescript-cubic-spline@1.0.1\n - typescript@5.2.2\n - winston@3.11.0\n - zod@3.22.4\n status: success\ntree:\n children:\n child:\n pipeline:\n - sum\n config:\n sum: null\n inputs:\n - timestamp: 2023-08-06T00:00\n duration: 3600\n cpu/energy: 0.001\n network/energy: 0.001\n outputs:\n - timestamp: 2023-08-06T00:00\n duration: 3600\n cpu/energy: 0.001\n network/energy: 0.001\n energy: 0.002\n")),(0,i.kt)("p",null,"To generate a CSV file in the provided path, run the following command:"),(0,i.kt)("pre",null,(0,i.kt)("code",{parentName:"pre",className:"language-sh"},"if-csv -m ./sum.yaml -p energy -o ./output-sum\n")),(0,i.kt)("p",null,"To print data in the console, you need to run:"),(0,i.kt)("pre",null,(0,i.kt)("code",{parentName:"pre",className:"language-sh"},"if-csv -m ./sum.yaml -p energy\n")),(0,i.kt)("p",null,"The output will be:"),(0,i.kt)("pre",null,(0,i.kt)("code",{parentName:"pre",className:"language-sh"},"Path,2023-08-06T00:00\ntree.children.child.energy,0.002\n")),(0,i.kt)("p",null,"Alternatively, you can pipe the result from ",(0,i.kt)("inlineCode",{parentName:"p"},"if-run"),"."),(0,i.kt)("pre",null,(0,i.kt)("code",{parentName:"pre",className:"language-sh"},"if-run -m ./sum.yaml | if-csv -p energy\n")))}u.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/assets/js/68ffc9f1.3aac1eaa.js b/assets/js/68ffc9f1.3aac1eaa.js deleted file mode 100644 index 9ee27f4a..00000000 --- a/assets/js/68ffc9f1.3aac1eaa.js +++ /dev/null @@ -1 +0,0 @@ -"use strict";(self.webpackChunkgreen_software_training=self.webpackChunkgreen_software_training||[]).push([[644],{4137:(e,n,t)=>{t.d(n,{Zo:()=>u,kt:()=>d});var i=t(7294);function a(e,n,t){return n in e?Object.defineProperty(e,n,{value:t,enumerable:!0,configurable:!0,writable:!0}):e[n]=t,e}function o(e,n){var t=Object.keys(e);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);n&&(i=i.filter((function(n){return Object.getOwnPropertyDescriptor(e,n).enumerable}))),t.push.apply(t,i)}return t}function r(e){for(var n=1;n=0||(a[t]=e[t]);return a}(e,n);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(i=0;i=0||Object.prototype.propertyIsEnumerable.call(e,t)&&(a[t]=e[t])}return a}var p=i.createContext({}),s=function(e){var n=i.useContext(p),t=n;return e&&(t="function"==typeof e?e(n):r(r({},n),e)),t},u=function(e){var n=s(e.components);return i.createElement(p.Provider,{value:n},e.children)},c={inlineCode:"code",wrapper:function(e){var n=e.children;return i.createElement(i.Fragment,{},n)}},m=i.forwardRef((function(e,n){var t=e.components,a=e.mdxType,o=e.originalType,p=e.parentName,u=l(e,["components","mdxType","originalType","parentName"]),m=s(t),d=a,h=m["".concat(p,".").concat(d)]||m[d]||c[d]||o;return t?i.createElement(h,r(r({ref:n},u),{},{components:t})):i.createElement(h,r({ref:n},u))}));function d(e,n){var t=arguments,a=n&&n.mdxType;if("string"==typeof e||a){var o=t.length,r=new Array(o);r[0]=m;var l={};for(var p in n)hasOwnProperty.call(n,p)&&(l[p]=n[p]);l.originalType=e,l.mdxType="string"==typeof e?e:a,r[1]=l;for(var s=2;s{t.r(n),t.d(n,{assets:()=>p,contentTitle:()=>r,default:()=>c,frontMatter:()=>o,metadata:()=>l,toc:()=>s});var i=t(7462),a=(t(7294),t(4137));const o={sidebar_position:4},r="How to write a manifest file",l={unversionedId:"users/how-to-write-manifests",id:"users/how-to-write-manifests",title:"How to write a manifest file",description:"The Impact Framework receives all its configuration and input data in the form of a manifest file known as an manifest. To use the framework, you will need to write a manifest file and pass its path to the command line tool. This guide will help you to understand how to construct one of these files and use it to measure the energy and carbon usage of your app.",source:"@site/docs/users/how-to-write-manifests.md",sourceDirName:"users",slug:"/users/how-to-write-manifests",permalink:"/users/how-to-write-manifests",draft:!1,editUrl:"https://github.com/Green-Software-Foundation/if-docs/edit/master/docs/users/how-to-write-manifests.md",tags:[],version:"current",sidebarPosition:4,frontMatter:{sidebar_position:4},sidebar:"tutorialSidebar",previous:{title:"How to load plugins",permalink:"/users/how-to-import-plugins"},next:{title:"how-to-export-csv-file-with-if-csv",permalink:"/users/how-to-export-csv-file-with-if-csv"}},p={},s=[{value:"Structure of a manifest",id:"structure-of-a-manifest",level:2},{value:"Project metadata",id:"project-metadata",level:3},{value:"Initialize",id:"initialize",level:3},{value:"Tree",id:"tree",level:3},{value:"Inputs",id:"inputs",level:3},{value:"More complex manifests",id:"more-complex-manifests",level:2},{value:"Complex pipelines",id:"complex-pipelines",level:3},{value:"Complex applications",id:"complex-applications",level:3},{value:"Choosing which plugins to run",id:"choosing-which-plugins-to-run",level:2},{value:"Adding real-life inputs",id:"adding-real-life-inputs",level:2},{value:"Running a manifest",id:"running-a-manifest",level:2}],u={toc:s};function c(e){let{components:n,...t}=e;return(0,a.kt)("wrapper",(0,i.Z)({},u,t,{components:n,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"how-to-write-a-manifest-file"},"How to write a manifest file"),(0,a.kt)("p",null,"The Impact Framework receives all its configuration and input data in the form of a manifest file known as an manifest. To use the framework, you will need to write a manifest file and pass its path to the command line tool. This guide will help you to understand how to construct one of these files and use it to measure the energy and carbon usage of your app."),(0,a.kt)("h2",{id:"structure-of-a-manifest"},"Structure of a manifest"),(0,a.kt)("p",null,"The basic structure of a manifest is as follows: "),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-yaml"},"name: \ndescription: \ntags: \ninitialize:\n plugins:\n : \n method:\n path: \ntree:\n children:\n child:\n pipeline:\n - \n config:\n defaults:\n inputs:\n\n")),(0,a.kt)("h3",{id:"project-metadata"},"Project metadata"),(0,a.kt)("p",null,"The file starts with some metadata about the project. There are no strict specifications for what to put in these fields, they are for you to keep track of your manifest files and to help other users to understand your use case."),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-yaml"},"name:\ndescription:\ntags:\n")),(0,a.kt)("h3",{id:"initialize"},"Initialize"),(0,a.kt)("p",null,"The ",(0,a.kt)("inlineCode",{parentName:"p"},"initialize")," fields are where you specify each individual plugin that will be initialized in your pipeline. The plugins can be initialized in any order, but can only be invoked elsewhere in the manifest if they have been initialized first here. In each case, you will need to provide the ",(0,a.kt)("inlineCode",{parentName:"p"},"name"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"path")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"method")," (and ",(0,a.kt)("inlineCode",{parentName:"p"},"global-config")," if your plugin requires it):"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-yaml"},"initialize:\n plugins:\n sci-m:\n path: ''\n method:\n")),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"The ",(0,a.kt)("inlineCode",{parentName:"li"},"name")," is the name you want this plugin instance to be recognized as by Impact Framework."),(0,a.kt)("li",{parentName:"ul"},"The ",(0,a.kt)("inlineCode",{parentName:"li"},"path")," defines where IF should look for the installed plugin. For example, for our standard library of plugins you would specify ",(0,a.kt)("inlineCode",{parentName:"li"},"builtin"),", for other installed plugins you use the name of the directory they are installed into in ",(0,a.kt)("inlineCode",{parentName:"li"},"node_modules"),"."),(0,a.kt)("li",{parentName:"ul"},"For the ",(0,a.kt)("inlineCode",{parentName:"li"},"method")," field, you should provide the name of the function exported by your plugin. For example, for the ",(0,a.kt)("inlineCode",{parentName:"li"},"sum")," plugin, the correct value is ",(0,a.kt)("inlineCode",{parentName:"li"},"Sum"),".")),(0,a.kt)("h3",{id:"tree"},"Tree"),(0,a.kt)("p",null,"The ",(0,a.kt)("inlineCode",{parentName:"p"},"tree")," fields are where you define the various components of your application. Each component is defined as ",(0,a.kt)("inlineCode",{parentName:"p"},"children"),", where each ",(0,a.kt)("inlineCode",{parentName:"p"},"child"),"'s output is summed to give the overall impact. Each ",(0,a.kt)("inlineCode",{parentName:"p"},"child")," can have its own plugin pipeline and its own configuration, but when none is provided, it is inherited from the tree-level configuration."),(0,a.kt)("p",null,"In the following example, there is only one component but the plugin pipeline contains two plugins; ",(0,a.kt)("inlineCode",{parentName:"p"},"teads-curve")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"sci-m"),". Neither requires any ",(0,a.kt)("inlineCode",{parentName:"p"},"config")," data, but certain information is required in ",(0,a.kt)("inlineCode",{parentName:"p"},"inputs"),"."),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-yaml"},"tree:\n children:\n child:\n pipeline:\n - teads-curve\n - sci-m\n config:\n defaults:\n inputs:\n - timestamp: '2023-11-02T10:35:31.820Z'\n duration: 3600\n total-embodied-emissions: 1533.12\n time-reserved: 1\n expected-lifespan: 3\n resources-reserved: 1\n total-resources: 8\n\n")),(0,a.kt)("h3",{id:"inputs"},"Inputs"),(0,a.kt)("p",null,"The most granular level of the manifest file are the ",(0,a.kt)("inlineCode",{parentName:"p"},"inputs"),". This is where you can add specific data for each ",(0,a.kt)("inlineCode",{parentName:"p"},"child"),". Inputs must always include a ",(0,a.kt)("inlineCode",{parentName:"p"},"timestamp")," and a ",(0,a.kt)("inlineCode",{parentName:"p"},"duration"),"."),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-yaml"},"inputs:\n - timestamp: 2023-07-06T00:00\n duration: 3600\n cpu-util: 45\n")),(0,a.kt)("p",null,"You now have a simple manifest file that will use the plugin config and input data to run the ",(0,a.kt)("inlineCode",{parentName:"p"},"teads-curve")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"sci-m")," plugins. The output data will be appended to the manifest under a new ",(0,a.kt)("inlineCode",{parentName:"p"},"outputs")," field and saved as an output file."),(0,a.kt)("h2",{id:"more-complex-manifests"},"More complex manifests"),(0,a.kt)("h3",{id:"complex-pipelines"},"Complex pipelines"),(0,a.kt)("p",null,"Whilst the manifest file we looked at above works perfectly well, it will only return the most basic output data. Most users will want to calculate an SCI score, which implies a number of additional steps:"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("inlineCode",{parentName:"li"},"operational-carbon")," and ",(0,a.kt)("inlineCode",{parentName:"li"},"embodied-carbon")," must appear as inputs."),(0,a.kt)("li",{parentName:"ul"},"This means that ",(0,a.kt)("inlineCode",{parentName:"li"},"sci")," will need to be preceded by ",(0,a.kt)("inlineCode",{parentName:"li"},"sci-m")," and ",(0,a.kt)("inlineCode",{parentName:"li"},"sci-o")," in the plugin pipeline."),(0,a.kt)("li",{parentName:"ul"},"In most cases, ",(0,a.kt)("inlineCode",{parentName:"li"},"sci-o")," will have to be preceded by ",(0,a.kt)("inlineCode",{parentName:"li"},"sci-e")," to ensure ",(0,a.kt)("inlineCode",{parentName:"li"},"energy")," is available to be piped to ",(0,a.kt)("inlineCode",{parentName:"li"},"sci-o"),"."),(0,a.kt)("li",{parentName:"ul"},"The inputs to ",(0,a.kt)("inlineCode",{parentName:"li"},"sci-e")," will most likely be coming from a plugin such as ",(0,a.kt)("inlineCode",{parentName:"li"},"teads-curve")," or ",(0,a.kt)("inlineCode",{parentName:"li"},"boavizta"),"."),(0,a.kt)("li",{parentName:"ul"},"The ",(0,a.kt)("inlineCode",{parentName:"li"},"sci")," plugin also requires ",(0,a.kt)("inlineCode",{parentName:"li"},"functional-unit")," information so it can convert the estimated ",(0,a.kt)("inlineCode",{parentName:"li"},"carbon")," into a useful unit."),(0,a.kt)("li",{parentName:"ul"},"You may also wish to grab your ",(0,a.kt)("inlineCode",{parentName:"li"},"input")," data by querying a metrics API on a virtual machine. ")),(0,a.kt)("p",null,"The example below gives you the full pipeline implemented in a manifest. There are also several other executable example manifests in ",(0,a.kt)("inlineCode",{parentName:"p"},"if/manifests/examples")," that you can run for yourself."),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-yaml"},'name: pipeline-with-aggregate\ndescription: a full pipeline with the aggregate feature enabled\ntags:\naggregation:\n metrics:\n - "carbon"\n type: "both"\ninitialize:\n plugins:\n "interpolate":\n method: Interpolation\n path: \'builtin\'\n global-config:\n method: linear\n x: [0, 10, 50, 100]\n y: [0.12, 0.32, 0.75, 1.02]\n input-parameter: \'cpu/utilization\'\n output-parameter: \'cpu-factor\'\n "cpu-factor-to-wattage":\n method: Multiply\n path: builtin\n global-config:\n input-parameters: ["cpu-factor", "cpu/thermal-design-power"]\n output-parameter: "cpu-wattage"\n "wattage-times-duration":\n method: Multiply\n path: builtin\n global-config:\n input-parameters: ["cpu-wattage", "duration"]\n output-parameter: "cpu-wattage-times-duration"\n "wattage-to-energy-kwh":\n method: Divide\n path: "builtin"\n global-config:\n numerator: cpu-wattage-times-duration\n denominator: 3600000\n output: cpu-energy-raw\n "calculate-vcpu-ratio":\n method: Divide\n path: "builtin"\n global-config:\n numerator: vcpus-total\n denominator: vcpus-allocated\n output: vcpu-ratio\n "correct-cpu-energy-for-vcpu-ratio":\n method: Divide\n path: "builtin"\n global-config:\n numerator: cpu-energy-raw\n denominator: vcpu-ratio\n output: cpu-energy-kwh\n "sci-embodied":\n path: "builtin"\n method: SciEmbodied\n "operational-carbon":\n method: Multiply\n path: builtin\n global-config:\n input-parameters: ["cpu-energy-kwh", "grid/carbon-intensity"]\n output-parameter: "carbon-operational"\n "sci":\n path: "builtin"\n method: Sci\n global-config:\n functional-unit-time: 1 sec\n functional-unit: requests # factor to convert per time to per f.unit\n "sum-carbon":\n path: "builtin"\n method: Sum\n global-config:\n input-parameters:\n - carbon-operational\n - carbon-embodied\n output-parameter: carbon\n "time-sync":\n method: TimeSync\n path: "builtin"\n global-config:\n start-time: "2023-12-12T00:00:00.000Z"\n end-time: "2023-12-12T00:01:00.000Z"\n interval: 5\n allow-padding: true\ntree:\n children:\n child-1:\n pipeline:\n - interpolate\n - cpu-factor-to-wattage\n - wattage-times-duration\n - wattage-to-energy-kwh\n - calculate-vcpu-ratio\n - correct-cpu-energy-for-vcpu-ratio\n - sci-embodied\n - operational-carbon\n - sum-carbon\n - time-sync\n # - sci\n config:\n group-by:\n group:\n - cloud/region\n - cloud/instance-type\n defaults:\n cpu/thermal-design-power: 100\n grid/carbon-intensity: 800\n device/emissions-embodied: 1533.120 # gCO2eq\n time-reserved: 3600 # 1hr in seconds\n device/expected-lifespan: 94608000 # 3 years in seconds\n vcpus-total: 8\n vcpus-allocated: 1\n inputs:\n - timestamp: "2023-12-12T00:00:00.000Z"\n cloud/instance-type: A1\n cloud/region: uk-west\n duration: 1\n cpu/utilization: 10\n requests: 10\n - timestamp: "2023-12-12T00:00:01.000Z"\n duration: 5\n cpu/utilization: 20\n cloud/instance-type: A1\n cloud/region: uk-west\n requests: 5\n - timestamp: "2023-12-12T00:00:06.000Z"\n duration: 7\n cpu/utilization: 15\n cloud/instance-type: A1\n cloud/region: uk-west\n requests: 15\n - timestamp: "2023-12-12T00:00:13.000Z"\n duration: 30\n cloud/instance-type: A1\n cloud/region: uk-west\n cpu/utilization: 15\n requests: 30\n\n')),(0,a.kt)("h3",{id:"complex-applications"},"Complex applications"),(0,a.kt)("p",null,"The manifest examples provided so far have only had a single component. However, Impact Framework can handle any number of nested ",(0,a.kt)("inlineCode",{parentName:"p"},"children"),"."),(0,a.kt)("p",null,"In this way, you can combine complex plugin pipelines and application architectures to calculate the energy and carbon outputs of complicated systems."),(0,a.kt)("h2",{id:"choosing-which-plugins-to-run"},"Choosing which plugins to run"),(0,a.kt)("p",null,"The plugins are designed to be composable, but they each have specific input requirements that must be met in order for the plugins to run correctly. For example, the ",(0,a.kt)("inlineCode",{parentName:"p"},"teads-curve")," plugin requires ",(0,a.kt)("inlineCode",{parentName:"p"},"cpu/thermal-design-power")," to be available in the manifest. If it is not there, the plugin cannot use it to calculate ",(0,a.kt)("inlineCode",{parentName:"p"},"cpu/energy"),"."),(0,a.kt)("p",null,"It is also possible to leapfrog some plugins if you have access to high-level data. For example, perhaps you already know the energy being used by your CPU. In this case, there is no need to run ",(0,a.kt)("inlineCode",{parentName:"p"},"teads-curve"),", you can simply provide ",(0,a.kt)("inlineCode",{parentName:"p"},"cpu/energy")," as an ",(0,a.kt)("inlineCode",{parentName:"p"},"input")," and omit ",(0,a.kt)("inlineCode",{parentName:"p"},"teads-curve")," from the plugin pipeline."),(0,a.kt)("p",null,"We have deliberately made the plugins modular and composable so that you can be creative in developing new plugins to replace those provided as part of IF."),(0,a.kt)("h2",{id:"adding-real-life-inputs"},"Adding real-life inputs"),(0,a.kt)("p",null,"The examples above already include inputs for the components. However, you may want to input real-life data into the manifest file."),(0,a.kt)("p",null,"There is no one-size-fits-all solution for getting data into the manifest file. This is because there are so many possible sources for your input data, all of which have their own particular requirements related to authorization, API request syntax and return types. Therefore, the approach taken by IF is to have specific plugins for specific services."),(0,a.kt)("p",null,"The recommended method for integrating data is to use the plugin system of the Impact Framework. You can either use an existing specific importer plugin or write your own."),(0,a.kt)("p",null,"There are already some community plugins available, including plugins for fetching data from Kubernetes, GCP, and third-party data aggregators like Datadog."),(0,a.kt)("p",null,"If there is no fitting plugin available yet, we encourage you to write and add one for your specific use case. See ",(0,a.kt)("a",{parentName:"p",href:"./developers/"},"developer documentation")," for more information on how to build a plugin. There is a ",(0,a.kt)("a",{parentName:"p",href:"https://github.com/Green-Software-Foundation/if-unofficial-plugins/blob/main/src/lib/azure-importer/README.md"},"Azure-Importer")," you can as a prototype and starting point for your own development.\nIf you already have external scripts you might have a look at the ",(0,a.kt)("a",{parentName:"p",href:"https://github.com/Green-Software-Foundation/if-plugins/blob/main/src/lib/shell/README.md"},"shell plugin")," to integrate them with the Impact Framework."),(0,a.kt)("p",null,"If you just need data for testing purposes, you can use the ",(0,a.kt)("a",{parentName:"p",href:"https://github.com/Green-Software-Foundation/if-plugins/blob/main/src/lib/mock-observations/README.md"},"mock-observation")," plugin."),(0,a.kt)("h2",{id:"running-a-manifest"},"Running a manifest"),(0,a.kt)("p",null,"You run a manifest by providing its path to our command line tool and a path to save the results file to. You can run a manifest named ",(0,a.kt)("inlineCode",{parentName:"p"},"my-manifest.yml")," using the following command:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-sh"},"if-run --manifest my-manifest.yml\n")))}c.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/assets/js/68ffc9f1.d6e6b40c.js b/assets/js/68ffc9f1.d6e6b40c.js new file mode 100644 index 00000000..97d55ee7 --- /dev/null +++ b/assets/js/68ffc9f1.d6e6b40c.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkgreen_software_training=self.webpackChunkgreen_software_training||[]).push([[644],{4137:(e,n,t)=>{t.d(n,{Zo:()=>u,kt:()=>d});var i=t(7294);function a(e,n,t){return n in e?Object.defineProperty(e,n,{value:t,enumerable:!0,configurable:!0,writable:!0}):e[n]=t,e}function o(e,n){var t=Object.keys(e);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);n&&(i=i.filter((function(n){return Object.getOwnPropertyDescriptor(e,n).enumerable}))),t.push.apply(t,i)}return t}function r(e){for(var n=1;n=0||(a[t]=e[t]);return a}(e,n);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(i=0;i=0||Object.prototype.propertyIsEnumerable.call(e,t)&&(a[t]=e[t])}return a}var p=i.createContext({}),s=function(e){var n=i.useContext(p),t=n;return e&&(t="function"==typeof e?e(n):r(r({},n),e)),t},u=function(e){var n=s(e.components);return i.createElement(p.Provider,{value:n},e.children)},c={inlineCode:"code",wrapper:function(e){var n=e.children;return i.createElement(i.Fragment,{},n)}},m=i.forwardRef((function(e,n){var t=e.components,a=e.mdxType,o=e.originalType,p=e.parentName,u=l(e,["components","mdxType","originalType","parentName"]),m=s(t),d=a,h=m["".concat(p,".").concat(d)]||m[d]||c[d]||o;return t?i.createElement(h,r(r({ref:n},u),{},{components:t})):i.createElement(h,r({ref:n},u))}));function d(e,n){var t=arguments,a=n&&n.mdxType;if("string"==typeof e||a){var o=t.length,r=new Array(o);r[0]=m;var l={};for(var p in n)hasOwnProperty.call(n,p)&&(l[p]=n[p]);l.originalType=e,l.mdxType="string"==typeof e?e:a,r[1]=l;for(var s=2;s{t.r(n),t.d(n,{assets:()=>p,contentTitle:()=>r,default:()=>c,frontMatter:()=>o,metadata:()=>l,toc:()=>s});var i=t(7462),a=(t(7294),t(4137));const o={sidebar_position:4},r="How to write a manifest file",l={unversionedId:"users/how-to-write-manifests",id:"users/how-to-write-manifests",title:"How to write a manifest file",description:"The Impact Framework receives all its configuration and input data in the form of a manifest file known as an manifest. To use the framework, you will need to write a manifest file and pass its path to the command line tool. This guide will help you to understand how to construct one of these files and use it to measure the energy and carbon usage of your app.",source:"@site/docs/users/how-to-write-manifests.md",sourceDirName:"users",slug:"/users/how-to-write-manifests",permalink:"/users/how-to-write-manifests",draft:!1,editUrl:"https://github.com/Green-Software-Foundation/if-docs/edit/master/docs/users/how-to-write-manifests.md",tags:[],version:"current",sidebarPosition:4,frontMatter:{sidebar_position:4},sidebar:"tutorialSidebar",previous:{title:"How to load plugins",permalink:"/users/how-to-import-plugins"},next:{title:"Exporting CSV file with `if-csv`",permalink:"/users/how-to-export-csv-file-with-if-csv"}},p={},s=[{value:"Structure of a manifest",id:"structure-of-a-manifest",level:2},{value:"Project metadata",id:"project-metadata",level:3},{value:"Initialize",id:"initialize",level:3},{value:"Tree",id:"tree",level:3},{value:"Inputs",id:"inputs",level:3},{value:"More complex manifests",id:"more-complex-manifests",level:2},{value:"Complex pipelines",id:"complex-pipelines",level:3},{value:"Complex applications",id:"complex-applications",level:3},{value:"Choosing which plugins to run",id:"choosing-which-plugins-to-run",level:2},{value:"Adding real-life inputs",id:"adding-real-life-inputs",level:2},{value:"Running a manifest",id:"running-a-manifest",level:2}],u={toc:s};function c(e){let{components:n,...t}=e;return(0,a.kt)("wrapper",(0,i.Z)({},u,t,{components:n,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"how-to-write-a-manifest-file"},"How to write a manifest file"),(0,a.kt)("p",null,"The Impact Framework receives all its configuration and input data in the form of a manifest file known as an manifest. To use the framework, you will need to write a manifest file and pass its path to the command line tool. This guide will help you to understand how to construct one of these files and use it to measure the energy and carbon usage of your app."),(0,a.kt)("h2",{id:"structure-of-a-manifest"},"Structure of a manifest"),(0,a.kt)("p",null,"The basic structure of a manifest is as follows: "),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-yaml"},"name: \ndescription: \ntags: \ninitialize:\n plugins:\n : \n method:\n path: \ntree:\n children:\n child:\n pipeline:\n - \n config:\n defaults:\n inputs:\n\n")),(0,a.kt)("h3",{id:"project-metadata"},"Project metadata"),(0,a.kt)("p",null,"The file starts with some metadata about the project. There are no strict specifications for what to put in these fields, they are for you to keep track of your manifest files and to help other users to understand your use case."),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-yaml"},"name:\ndescription:\ntags:\n")),(0,a.kt)("h3",{id:"initialize"},"Initialize"),(0,a.kt)("p",null,"The ",(0,a.kt)("inlineCode",{parentName:"p"},"initialize")," fields are where you specify each individual plugin that will be initialized in your pipeline. The plugins can be initialized in any order, but can only be invoked elsewhere in the manifest if they have been initialized first here. In each case, you will need to provide the ",(0,a.kt)("inlineCode",{parentName:"p"},"name"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"path")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"method")," (and ",(0,a.kt)("inlineCode",{parentName:"p"},"global-config")," if your plugin requires it):"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-yaml"},"initialize:\n plugins:\n sci-m:\n path: ''\n method:\n")),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"The ",(0,a.kt)("inlineCode",{parentName:"li"},"name")," is the name you want this plugin instance to be recognized as by Impact Framework."),(0,a.kt)("li",{parentName:"ul"},"The ",(0,a.kt)("inlineCode",{parentName:"li"},"path")," defines where IF should look for the installed plugin. For example, for our standard library of plugins you would specify ",(0,a.kt)("inlineCode",{parentName:"li"},"builtin"),", for other installed plugins you use the name of the directory they are installed into in ",(0,a.kt)("inlineCode",{parentName:"li"},"node_modules"),"."),(0,a.kt)("li",{parentName:"ul"},"For the ",(0,a.kt)("inlineCode",{parentName:"li"},"method")," field, you should provide the name of the function exported by your plugin. For example, for the ",(0,a.kt)("inlineCode",{parentName:"li"},"sum")," plugin, the correct value is ",(0,a.kt)("inlineCode",{parentName:"li"},"Sum"),".")),(0,a.kt)("h3",{id:"tree"},"Tree"),(0,a.kt)("p",null,"The ",(0,a.kt)("inlineCode",{parentName:"p"},"tree")," fields are where you define the various components of your application. Each component is defined as ",(0,a.kt)("inlineCode",{parentName:"p"},"children"),", where each ",(0,a.kt)("inlineCode",{parentName:"p"},"child"),"'s output is summed to give the overall impact. Each ",(0,a.kt)("inlineCode",{parentName:"p"},"child")," can have its own plugin pipeline and its own configuration, but when none is provided, it is inherited from the tree-level configuration."),(0,a.kt)("p",null,"In the following example, there is only one component but the plugin pipeline contains two plugins; ",(0,a.kt)("inlineCode",{parentName:"p"},"teads-curve")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"sci-m"),". Neither requires any ",(0,a.kt)("inlineCode",{parentName:"p"},"config")," data, but certain information is required in ",(0,a.kt)("inlineCode",{parentName:"p"},"inputs"),"."),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-yaml"},"tree:\n children:\n child:\n pipeline:\n - teads-curve\n - sci-m\n config:\n defaults:\n inputs:\n - timestamp: '2023-11-02T10:35:31.820Z'\n duration: 3600\n total-embodied-emissions: 1533.12\n time-reserved: 1\n expected-lifespan: 3\n resources-reserved: 1\n total-resources: 8\n\n")),(0,a.kt)("h3",{id:"inputs"},"Inputs"),(0,a.kt)("p",null,"The most granular level of the manifest file are the ",(0,a.kt)("inlineCode",{parentName:"p"},"inputs"),". This is where you can add specific data for each ",(0,a.kt)("inlineCode",{parentName:"p"},"child"),". Inputs must always include a ",(0,a.kt)("inlineCode",{parentName:"p"},"timestamp")," and a ",(0,a.kt)("inlineCode",{parentName:"p"},"duration"),"."),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-yaml"},"inputs:\n - timestamp: 2023-07-06T00:00\n duration: 3600\n cpu-util: 45\n")),(0,a.kt)("p",null,"You now have a simple manifest file that will use the plugin config and input data to run the ",(0,a.kt)("inlineCode",{parentName:"p"},"teads-curve")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"sci-m")," plugins. The output data will be appended to the manifest under a new ",(0,a.kt)("inlineCode",{parentName:"p"},"outputs")," field and saved as an output file."),(0,a.kt)("h2",{id:"more-complex-manifests"},"More complex manifests"),(0,a.kt)("h3",{id:"complex-pipelines"},"Complex pipelines"),(0,a.kt)("p",null,"Whilst the manifest file we looked at above works perfectly well, it will only return the most basic output data. Most users will want to calculate an SCI score, which implies a number of additional steps:"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("inlineCode",{parentName:"li"},"operational-carbon")," and ",(0,a.kt)("inlineCode",{parentName:"li"},"embodied-carbon")," must appear as inputs."),(0,a.kt)("li",{parentName:"ul"},"This means that ",(0,a.kt)("inlineCode",{parentName:"li"},"sci")," will need to be preceded by ",(0,a.kt)("inlineCode",{parentName:"li"},"sci-m")," and ",(0,a.kt)("inlineCode",{parentName:"li"},"sci-o")," in the plugin pipeline."),(0,a.kt)("li",{parentName:"ul"},"In most cases, ",(0,a.kt)("inlineCode",{parentName:"li"},"sci-o")," will have to be preceded by ",(0,a.kt)("inlineCode",{parentName:"li"},"sci-e")," to ensure ",(0,a.kt)("inlineCode",{parentName:"li"},"energy")," is available to be piped to ",(0,a.kt)("inlineCode",{parentName:"li"},"sci-o"),"."),(0,a.kt)("li",{parentName:"ul"},"The inputs to ",(0,a.kt)("inlineCode",{parentName:"li"},"sci-e")," will most likely be coming from a plugin such as ",(0,a.kt)("inlineCode",{parentName:"li"},"teads-curve")," or ",(0,a.kt)("inlineCode",{parentName:"li"},"boavizta"),"."),(0,a.kt)("li",{parentName:"ul"},"The ",(0,a.kt)("inlineCode",{parentName:"li"},"sci")," plugin also requires ",(0,a.kt)("inlineCode",{parentName:"li"},"functional-unit")," information so it can convert the estimated ",(0,a.kt)("inlineCode",{parentName:"li"},"carbon")," into a useful unit."),(0,a.kt)("li",{parentName:"ul"},"You may also wish to grab your ",(0,a.kt)("inlineCode",{parentName:"li"},"input")," data by querying a metrics API on a virtual machine. ")),(0,a.kt)("p",null,"The example below gives you the full pipeline implemented in a manifest. There are also several other executable example manifests in ",(0,a.kt)("inlineCode",{parentName:"p"},"if/manifests/examples")," that you can run for yourself."),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-yaml"},'name: pipeline-with-aggregate\ndescription: a full pipeline with the aggregate feature enabled\ntags:\naggregation:\n metrics:\n - "carbon"\n type: "both"\ninitialize:\n plugins:\n "interpolate":\n method: Interpolation\n path: \'builtin\'\n global-config:\n method: linear\n x: [0, 10, 50, 100]\n y: [0.12, 0.32, 0.75, 1.02]\n input-parameter: \'cpu/utilization\'\n output-parameter: \'cpu-factor\'\n "cpu-factor-to-wattage":\n method: Multiply\n path: builtin\n global-config:\n input-parameters: ["cpu-factor", "cpu/thermal-design-power"]\n output-parameter: "cpu-wattage"\n "wattage-times-duration":\n method: Multiply\n path: builtin\n global-config:\n input-parameters: ["cpu-wattage", "duration"]\n output-parameter: "cpu-wattage-times-duration"\n "wattage-to-energy-kwh":\n method: Divide\n path: "builtin"\n global-config:\n numerator: cpu-wattage-times-duration\n denominator: 3600000\n output: cpu-energy-raw\n "calculate-vcpu-ratio":\n method: Divide\n path: "builtin"\n global-config:\n numerator: vcpus-total\n denominator: vcpus-allocated\n output: vcpu-ratio\n "correct-cpu-energy-for-vcpu-ratio":\n method: Divide\n path: "builtin"\n global-config:\n numerator: cpu-energy-raw\n denominator: vcpu-ratio\n output: cpu-energy-kwh\n "sci-embodied":\n path: "builtin"\n method: SciEmbodied\n "operational-carbon":\n method: Multiply\n path: builtin\n global-config:\n input-parameters: ["cpu-energy-kwh", "grid/carbon-intensity"]\n output-parameter: "carbon-operational"\n "sci":\n path: "builtin"\n method: Sci\n global-config:\n functional-unit-time: 1 sec\n functional-unit: requests # factor to convert per time to per f.unit\n "sum-carbon":\n path: "builtin"\n method: Sum\n global-config:\n input-parameters:\n - carbon-operational\n - carbon-embodied\n output-parameter: carbon\n "time-sync":\n method: TimeSync\n path: "builtin"\n global-config:\n start-time: "2023-12-12T00:00:00.000Z"\n end-time: "2023-12-12T00:01:00.000Z"\n interval: 5\n allow-padding: true\ntree:\n children:\n child-1:\n pipeline:\n - interpolate\n - cpu-factor-to-wattage\n - wattage-times-duration\n - wattage-to-energy-kwh\n - calculate-vcpu-ratio\n - correct-cpu-energy-for-vcpu-ratio\n - sci-embodied\n - operational-carbon\n - sum-carbon\n - time-sync\n # - sci\n config:\n group-by:\n group:\n - cloud/region\n - cloud/instance-type\n defaults:\n cpu/thermal-design-power: 100\n grid/carbon-intensity: 800\n device/emissions-embodied: 1533.120 # gCO2eq\n time-reserved: 3600 # 1hr in seconds\n device/expected-lifespan: 94608000 # 3 years in seconds\n vcpus-total: 8\n vcpus-allocated: 1\n inputs:\n - timestamp: "2023-12-12T00:00:00.000Z"\n cloud/instance-type: A1\n cloud/region: uk-west\n duration: 1\n cpu/utilization: 10\n requests: 10\n - timestamp: "2023-12-12T00:00:01.000Z"\n duration: 5\n cpu/utilization: 20\n cloud/instance-type: A1\n cloud/region: uk-west\n requests: 5\n - timestamp: "2023-12-12T00:00:06.000Z"\n duration: 7\n cpu/utilization: 15\n cloud/instance-type: A1\n cloud/region: uk-west\n requests: 15\n - timestamp: "2023-12-12T00:00:13.000Z"\n duration: 30\n cloud/instance-type: A1\n cloud/region: uk-west\n cpu/utilization: 15\n requests: 30\n\n')),(0,a.kt)("h3",{id:"complex-applications"},"Complex applications"),(0,a.kt)("p",null,"The manifest examples provided so far have only had a single component. However, Impact Framework can handle any number of nested ",(0,a.kt)("inlineCode",{parentName:"p"},"children"),"."),(0,a.kt)("p",null,"In this way, you can combine complex plugin pipelines and application architectures to calculate the energy and carbon outputs of complicated systems."),(0,a.kt)("h2",{id:"choosing-which-plugins-to-run"},"Choosing which plugins to run"),(0,a.kt)("p",null,"The plugins are designed to be composable, but they each have specific input requirements that must be met in order for the plugins to run correctly. For example, the ",(0,a.kt)("inlineCode",{parentName:"p"},"teads-curve")," plugin requires ",(0,a.kt)("inlineCode",{parentName:"p"},"cpu/thermal-design-power")," to be available in the manifest. If it is not there, the plugin cannot use it to calculate ",(0,a.kt)("inlineCode",{parentName:"p"},"cpu/energy"),"."),(0,a.kt)("p",null,"It is also possible to leapfrog some plugins if you have access to high-level data. For example, perhaps you already know the energy being used by your CPU. In this case, there is no need to run ",(0,a.kt)("inlineCode",{parentName:"p"},"teads-curve"),", you can simply provide ",(0,a.kt)("inlineCode",{parentName:"p"},"cpu/energy")," as an ",(0,a.kt)("inlineCode",{parentName:"p"},"input")," and omit ",(0,a.kt)("inlineCode",{parentName:"p"},"teads-curve")," from the plugin pipeline."),(0,a.kt)("p",null,"We have deliberately made the plugins modular and composable so that you can be creative in developing new plugins to replace those provided as part of IF."),(0,a.kt)("h2",{id:"adding-real-life-inputs"},"Adding real-life inputs"),(0,a.kt)("p",null,"The examples above already include inputs for the components. However, you may want to input real-life data into the manifest file."),(0,a.kt)("p",null,"There is no one-size-fits-all solution for getting data into the manifest file. This is because there are so many possible sources for your input data, all of which have their own particular requirements related to authorization, API request syntax and return types. Therefore, the approach taken by IF is to have specific plugins for specific services."),(0,a.kt)("p",null,"The recommended method for integrating data is to use the plugin system of the Impact Framework. You can either use an existing specific importer plugin or write your own."),(0,a.kt)("p",null,"There are already some community plugins available, including plugins for fetching data from Kubernetes, GCP, and third-party data aggregators like Datadog."),(0,a.kt)("p",null,"If there is no fitting plugin available yet, we encourage you to write and add one for your specific use case. See ",(0,a.kt)("a",{parentName:"p",href:"./developers/"},"developer documentation")," for more information on how to build a plugin. There is a ",(0,a.kt)("a",{parentName:"p",href:"https://github.com/Green-Software-Foundation/if-unofficial-plugins/blob/main/src/lib/azure-importer/README.md"},"Azure-Importer")," you can as a prototype and starting point for your own development.\nIf you already have external scripts you might have a look at the ",(0,a.kt)("a",{parentName:"p",href:"https://github.com/Green-Software-Foundation/if-plugins/blob/main/src/lib/shell/README.md"},"shell plugin")," to integrate them with the Impact Framework."),(0,a.kt)("p",null,"If you just need data for testing purposes, you can use the ",(0,a.kt)("a",{parentName:"p",href:"https://github.com/Green-Software-Foundation/if-plugins/blob/main/src/lib/mock-observations/README.md"},"mock-observation")," plugin."),(0,a.kt)("h2",{id:"running-a-manifest"},"Running a manifest"),(0,a.kt)("p",null,"You run a manifest by providing its path to our command line tool and a path to save the results file to. You can run a manifest named ",(0,a.kt)("inlineCode",{parentName:"p"},"my-manifest.yml")," using the following command:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-sh"},"if-run --manifest my-manifest.yml\n")))}c.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/assets/js/935f2afb.59ef3ce8.js b/assets/js/935f2afb.59ef3ce8.js new file mode 100644 index 00000000..16af3a4c --- /dev/null +++ b/assets/js/935f2afb.59ef3ce8.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkgreen_software_training=self.webpackChunkgreen_software_training||[]).push([[53],{1109:e=>{e.exports=JSON.parse('{"pluginId":"default","version":"current","label":"Next","banner":null,"badge":false,"className":"docs-version-current","isLast":true,"docsSidebars":{"tutorialSidebar":[{"type":"link","label":"Introduction","href":"/intro","docId":"intro"},{"type":"category","label":"Major Concepts","collapsible":true,"collapsed":true,"items":[{"type":"link","label":"Design philosophy","href":"/major-concepts/design-philosophy","docId":"major-concepts/design-philosophy"},{"type":"link","label":"Aggregation","href":"/major-concepts/aggregation","docId":"major-concepts/aggregation"},{"type":"link","label":"Pipelines","href":"/major-concepts/pipelines","docId":"major-concepts/pipelines"},{"type":"link","label":"Exhaust scripts","href":"/major-concepts/exhaust-script","docId":"major-concepts/exhaust-script"},{"type":"link","label":"Group-by","href":"/major-concepts/groupby","docId":"major-concepts/groupby"},{"type":"link","label":"Impact Engine (CLI)","href":"/major-concepts/if","docId":"major-concepts/if"},{"type":"link","label":"Manifest File","href":"/major-concepts/manifest-file","docId":"major-concepts/manifest-file"},{"type":"link","label":"Parameters","href":"/major-concepts/parameters","docId":"major-concepts/parameters"},{"type":"link","label":"Plugins","href":"/major-concepts/plugins","docId":"major-concepts/plugins"},{"type":"link","label":"Time","href":"/major-concepts/time","docId":"major-concepts/time"}],"href":"/major-concepts/"},{"type":"category","label":"Users","collapsible":true,"collapsed":true,"items":[{"type":"link","label":"Quick start","href":"/users/quick-start","docId":"users/quick-start"},{"type":"link","label":"How to install Impact Framework","href":"/users/how-to-install-if","docId":"users/how-to-install-if"},{"type":"link","label":"How to load plugins","href":"/users/how-to-import-plugins","docId":"users/how-to-import-plugins"},{"type":"link","label":"How to write a manifest file","href":"/users/how-to-write-manifests","docId":"users/how-to-write-manifests"},{"type":"link","label":"Exporting CSV file with `if-csv`","href":"/users/how-to-export-csv-file-with-if-csv","docId":"users/how-to-export-csv-file-with-if-csv"},{"type":"link","label":"How to compare files with `if-diff`","href":"/users/how-to-compare-files-with-if-diff","docId":"users/how-to-compare-files-with-if-diff"},{"type":"link","label":"Verifying IF outputs with `if-check`","href":"/users/how-to-verify-files-with-if-check","docId":"users/how-to-verify-files-with-if-check"}],"href":"/users/"},{"type":"category","label":"Developers","collapsible":true,"collapsed":true,"items":[{"type":"link","label":"How to build plugins","href":"/developers/how-to-build-plugins","docId":"developers/how-to-build-plugins"},{"type":"link","label":"How to create an exhaust script","href":"/developers/how-to-create-exhaust-script","docId":"developers/how-to-create-exhaust-script"},{"type":"link","label":"How to make plugins production ready","href":"/developers/how-to-refine-plugins","docId":"developers/how-to-refine-plugins"},{"type":"link","label":"How to submit plugins","href":"/developers/how-to-submit-plugins","docId":"developers/how-to-submit-plugins"},{"type":"link","label":"How to write unit tests","href":"/developers/how-to-write-unit-tests","docId":"developers/how-to-write-unit-tests"}],"href":"/developers/"},{"type":"category","label":"Pipelines","collapsible":true,"collapsed":true,"items":[{"type":"link","label":"Teads CPU pipeline","href":"/pipelines/teads","docId":"pipelines/teads"},{"type":"link","label":"Instance metadata pipeline","href":"/pipelines/instance-metadata","docId":"pipelines/instance-metadata"},{"type":"link","label":"Software Carbon Intensity (SCI)","href":"/pipelines/sci","docId":"pipelines/sci"}],"href":"/pipelines/"},{"type":"category","label":"Reference","collapsible":true,"collapsed":true,"items":[{"type":"link","label":"Command line tool","href":"/reference/cli","docId":"reference/cli"},{"type":"link","label":"Errors","href":"/reference/errors","docId":"reference/errors"},{"type":"link","label":"Plugins","href":"/reference/plugins","docId":"reference/plugins"}],"href":"/reference/"},{"type":"link","label":"FAQs","href":"/FAQ","docId":"FAQ"}]},"docs":{"developers/how-to-build-plugins":{"id":"developers/how-to-build-plugins","title":"How to build plugins","description":"The IF is designed to be as composable as possible. This means you can develop your own plugins and use them in a pipeline.","sidebar":"tutorialSidebar"},"developers/how-to-create-exhaust-script":{"id":"developers/how-to-create-exhaust-script","title":"How to create an exhaust script","description":"The If framework outputs data in yaml format. Any other output formats require a separate script that takes the yaml output data and processes it. We provide if-csv for outputting data in csv format bundled with IF. For any other format, you need to write an exhaust script.","sidebar":"tutorialSidebar"},"developers/how-to-refine-plugins":{"id":"developers/how-to-refine-plugins","title":"How to make plugins production ready","description":"Our How to build plugins guide covered the basics for how to construct an Impact Framework plugin. This guide will help you to refine your plugin to make it production-ready. These are best practice guidelines - if you intend to contribute your plugin to one of our repositories, following these guidelines will help your PR to get merged. Even if you are not aiming to have a plugin merged into one of our repositories, consistency with our norms is useful for debugging and maintaining and for making your plugin as useful as possible for other Impact Framework developers.","sidebar":"tutorialSidebar"},"developers/how-to-submit-plugins":{"id":"developers/how-to-submit-plugins","title":"How to submit plugins","description":"Once you have built a plugin and made it production ready you probably want to share it with the world!","sidebar":"tutorialSidebar"},"developers/how-to-write-unit-tests":{"id":"developers/how-to-write-unit-tests","title":"How to write unit tests","description":"Impact Framework unit tests follow a standard format. We use the jest testing library. You can run all our existing tests by opening the project directory and running npm test. This page explains how you can add new unit tests for your plugins (or add some for our plugins if you notice a gap).","sidebar":"tutorialSidebar"},"developers/index":{"id":"developers/index","title":"Developers","description":"This section contains information for Impact Framework developers. You are a developer if you want to change or update the Impact Framework by adding new features, fixing bugs or building new plugins.","sidebar":"tutorialSidebar"},"FAQ":{"id":"FAQ","title":"FAQs","description":"How can I contribute to Impact framework?","sidebar":"tutorialSidebar"},"intro":{"id":"intro","title":"Introduction","description":"Impact Framework","sidebar":"tutorialSidebar"},"major-concepts/aggregation":{"id":"major-concepts/aggregation","title":"Aggregation","description":"Aggregation is the process of summarizing a set of metrics.","sidebar":"tutorialSidebar"},"major-concepts/design-philosophy":{"id":"major-concepts/design-philosophy","title":"Design philosophy","description":"Transparency","sidebar":"tutorialSidebar"},"major-concepts/exhaust-script":{"id":"major-concepts/exhaust-script","title":"Exhaust scripts","description":"Exhaust scripts are scripts that can run independently of IF itself that take an executed manifest file (one with outputs) as an input, parse the yaml data and reformat it into some other representation. We provide if-csv bundled with IF, but if you want other data formats, you\'ll have to create an exhaust script yourself.","sidebar":"tutorialSidebar"},"major-concepts/groupby":{"id":"major-concepts/groupby","title":"Group-by","description":"Groupby is an IF plugin that reorganizes a tree according to keys provided by the user. This allows users to regroup their observations according to various properties of their application. For example, the following manifest file contains a flat array of observations. This is how you might expect data to arrive from an importer plugin, maybe one that hits a metrics API for a cloud service.","sidebar":"tutorialSidebar"},"major-concepts/if":{"id":"major-concepts/if","title":"Impact Engine (CLI)","description":"Introduction","sidebar":"tutorialSidebar"},"major-concepts/index":{"id":"major-concepts/index","title":"Major Concepts","description":"Here you will find explanations for the fundamental Impact Framework concepts. This includes:","sidebar":"tutorialSidebar"},"major-concepts/manifest-file":{"id":"major-concepts/manifest-file","title":"Manifest File","description":"Manifest files are fundamental to Impact Framework and they serve multiple important purposes, including:","sidebar":"tutorialSidebar"},"major-concepts/parameters":{"id":"major-concepts/parameters","title":"Parameters","description":"Parameters are handled carefully in IF because of the risk of errors propagating through plugin pipelines. We have a set of canonical parameters that are stored in a file called params.ts in the IF repository.","sidebar":"tutorialSidebar"},"major-concepts/pipelines":{"id":"major-concepts/pipelines","title":"Pipelines","description":"Pipelines are chains of plugins that operate in sequence over the input data in your manifest file.","sidebar":"tutorialSidebar"},"major-concepts/plugins":{"id":"major-concepts/plugins","title":"Plugins","description":"Plugins are self-contained units of code that do one thing. They can be loaded into IF and chained together in a pipeline so that simple individual plugins can form a complicated procedure for computing a manifest file.","sidebar":"tutorialSidebar"},"major-concepts/time":{"id":"major-concepts/time","title":"Time","description":"Every observation in an array of inputs represents a snapshot with a known start time and a known duration. For example, the following observation shows that the CPU utilization for a resource was 20% for the 10 second period starting at 1500 on the 22nd January 2024:","sidebar":"tutorialSidebar"},"pipelines/index":{"id":"pipelines/index","title":"Pipelines","description":"This section contains walkthrough guides for some common pipelines you may want to use in your manifest files.","sidebar":"tutorialSidebar"},"pipelines/instance-metadata":{"id":"pipelines/instance-metadata","title":"Instance metadata pipeline","description":"The instance metadata pipeline simply looks up a metadata for a given virtual machine instance name using the csv-lookup plugin from the IF standard library. However, the target dataset can return multiple processor names for a given VM instance where there are multiple possibilitiers. This means we need to create a pipeline that includes the regex plugin so parse out just one of the possible values.","sidebar":"tutorialSidebar"},"pipelines/sci":{"id":"pipelines/sci","title":"Software Carbon Intensity (SCI)","description":"The software carbon intensity (SCI) score is perhaps the most important value that can be generated using Impact Framework.","sidebar":"tutorialSidebar"},"pipelines/teads":{"id":"pipelines/teads","title":"Teads CPU pipeline","description":"The Teads CPU power curve CPU utilization (as a percentage) against a scaling factor that can be applied to the CPUs thermal design power to estimate the power drawn by the CPU in Watts.","sidebar":"tutorialSidebar"},"reference/cli":{"id":"reference/cli","title":"Command line tool","description":"A core feature of the Impact Framework is the if-run command line tool (CLI). This is how you trigger Impact Framework to execute a certain manifest file.","sidebar":"tutorialSidebar"},"reference/errors":{"id":"reference/errors","title":"Errors","description":"IF defines a finite set of error classes. All error messages emitted by IF are attached to one of these classes.","sidebar":"tutorialSidebar"},"reference/index":{"id":"reference/index","title":"Reference","description":"In this section you will find reference documentation for the core data structures and features used in the Impact Framework.","sidebar":"tutorialSidebar"},"reference/plugins":{"id":"reference/plugins","title":"Plugins","description":"Impact Framework works by executing pipelines of plugins over input data. Those plugins are re-useable units of code that can be thought of as Lego bricks - simple blocks of code that can be assembled into complex workflows.","sidebar":"tutorialSidebar"},"users/how-to-compare-files-with-if-diff":{"id":"users/how-to-compare-files-with-if-diff","title":"How to compare files with `if-diff`","description":"if-diff is a command line tool that allows you to compare two if-run output files. They either match according to if-diff\'s matching rules, or they don\'t. If they match, then if-diff returns a simple success response. If the differ, then if-diff returns a report of the differences it finds.","sidebar":"tutorialSidebar"},"users/how-to-export-csv-file-with-if-csv":{"id":"users/how-to-export-csv-file-with-if-csv","title":"Exporting CSV file with `if-csv`","description":"IF includes a command line tool called if-csv which is designed to export CSV files based on a specified manifest file and metric.","sidebar":"tutorialSidebar"},"users/how-to-import-plugins":{"id":"users/how-to-import-plugins","title":"How to load plugins","description":"Plugins are developed separately to the Impact Framework core. However, the IF core developers maintain a standard library of plugins come bundled with IF. These are known as builtins.","sidebar":"tutorialSidebar"},"users/how-to-install-if":{"id":"users/how-to-install-if","title":"How to install Impact Framework","description":"You can install Impact Framework either globally or locally. For most users, we recommend installing our official releases globally using npm. You can do this using the following command:","sidebar":"tutorialSidebar"},"users/how-to-verify-files-with-if-check":{"id":"users/how-to-verify-files-with-if-check","title":"Verifying IF outputs with `if-check`","description":"IF includes a command line tool called if-check that can be used to verify the results in a manifest file.","sidebar":"tutorialSidebar"},"users/how-to-write-manifests":{"id":"users/how-to-write-manifests","title":"How to write a manifest file","description":"The Impact Framework receives all its configuration and input data in the form of a manifest file known as an manifest. To use the framework, you will need to write a manifest file and pass its path to the command line tool. This guide will help you to understand how to construct one of these files and use it to measure the energy and carbon usage of your app.","sidebar":"tutorialSidebar"},"users/index":{"id":"users/index","title":"Users","description":"This section contains information for Impact Framework users. You are a user if you want to apply the Impact Framework to your own use-case, such as using it to measure the environmental impact of your own apps running on some cloud platform.","sidebar":"tutorialSidebar"},"users/quick-start":{"id":"users/quick-start","title":"Quick start","description":"This page will provide the basic instructions for getting up and running with Impact Framework.","sidebar":"tutorialSidebar"}}}')}}]); \ No newline at end of file diff --git a/assets/js/935f2afb.74bbde3d.js b/assets/js/935f2afb.74bbde3d.js deleted file mode 100644 index e8853545..00000000 --- a/assets/js/935f2afb.74bbde3d.js +++ /dev/null @@ -1 +0,0 @@ -"use strict";(self.webpackChunkgreen_software_training=self.webpackChunkgreen_software_training||[]).push([[53],{1109:e=>{e.exports=JSON.parse('{"pluginId":"default","version":"current","label":"Next","banner":null,"badge":false,"className":"docs-version-current","isLast":true,"docsSidebars":{"tutorialSidebar":[{"type":"link","label":"Introduction","href":"/intro","docId":"intro"},{"type":"category","label":"Major Concepts","collapsible":true,"collapsed":true,"items":[{"type":"link","label":"Design philosophy","href":"/major-concepts/design-philosophy","docId":"major-concepts/design-philosophy"},{"type":"link","label":"Aggregation","href":"/major-concepts/aggregation","docId":"major-concepts/aggregation"},{"type":"link","label":"Pipelines","href":"/major-concepts/pipelines","docId":"major-concepts/pipelines"},{"type":"link","label":"Exhaust scripts","href":"/major-concepts/exhaust-script","docId":"major-concepts/exhaust-script"},{"type":"link","label":"Group-by","href":"/major-concepts/groupby","docId":"major-concepts/groupby"},{"type":"link","label":"Impact Engine (CLI)","href":"/major-concepts/if","docId":"major-concepts/if"},{"type":"link","label":"Manifest File","href":"/major-concepts/manifest-file","docId":"major-concepts/manifest-file"},{"type":"link","label":"Parameters","href":"/major-concepts/parameters","docId":"major-concepts/parameters"},{"type":"link","label":"Plugins","href":"/major-concepts/plugins","docId":"major-concepts/plugins"},{"type":"link","label":"Time","href":"/major-concepts/time","docId":"major-concepts/time"}],"href":"/major-concepts/"},{"type":"category","label":"Users","collapsible":true,"collapsed":true,"items":[{"type":"link","label":"Quick start","href":"/users/quick-start","docId":"users/quick-start"},{"type":"link","label":"How to install Impact Framework","href":"/users/how-to-install-if","docId":"users/how-to-install-if"},{"type":"link","label":"How to load plugins","href":"/users/how-to-import-plugins","docId":"users/how-to-import-plugins"},{"type":"link","label":"How to write a manifest file","href":"/users/how-to-write-manifests","docId":"users/how-to-write-manifests"},{"type":"link","label":"how-to-export-csv-file-with-if-csv","href":"/users/how-to-export-csv-file-with-if-csv","docId":"users/how-to-export-csv-file-with-if-csv"},{"type":"link","label":"How to compare files with `if-diff`","href":"/users/how-to-compare-files-with-if-diff","docId":"users/how-to-compare-files-with-if-diff"},{"type":"link","label":"Verifying IF outputs with `if-check`","href":"/users/how-to-verify-files-with-if-check","docId":"users/how-to-verify-files-with-if-check"}],"href":"/users/"},{"type":"category","label":"Developers","collapsible":true,"collapsed":true,"items":[{"type":"link","label":"How to build plugins","href":"/developers/how-to-build-plugins","docId":"developers/how-to-build-plugins"},{"type":"link","label":"How to create an exhaust script","href":"/developers/how-to-create-exhaust-script","docId":"developers/how-to-create-exhaust-script"},{"type":"link","label":"How to make plugins production ready","href":"/developers/how-to-refine-plugins","docId":"developers/how-to-refine-plugins"},{"type":"link","label":"How to submit plugins","href":"/developers/how-to-submit-plugins","docId":"developers/how-to-submit-plugins"},{"type":"link","label":"How to write unit tests","href":"/developers/how-to-write-unit-tests","docId":"developers/how-to-write-unit-tests"}],"href":"/developers/"},{"type":"category","label":"Pipelines","collapsible":true,"collapsed":true,"items":[{"type":"link","label":"Teads CPU pipeline","href":"/pipelines/teads","docId":"pipelines/teads"},{"type":"link","label":"Instance metadata pipeline","href":"/pipelines/instance-metadata","docId":"pipelines/instance-metadata"},{"type":"link","label":"Software Carbon Intensity (SCI)","href":"/pipelines/sci","docId":"pipelines/sci"}],"href":"/pipelines/"},{"type":"category","label":"Reference","collapsible":true,"collapsed":true,"items":[{"type":"link","label":"Command line tool","href":"/reference/cli","docId":"reference/cli"},{"type":"link","label":"Errors","href":"/reference/errors","docId":"reference/errors"},{"type":"link","label":"Plugins","href":"/reference/plugins","docId":"reference/plugins"}],"href":"/reference/"},{"type":"link","label":"FAQs","href":"/FAQ","docId":"FAQ"}]},"docs":{"developers/how-to-build-plugins":{"id":"developers/how-to-build-plugins","title":"How to build plugins","description":"The IF is designed to be as composable as possible. This means you can develop your own plugins and use them in a pipeline.","sidebar":"tutorialSidebar"},"developers/how-to-create-exhaust-script":{"id":"developers/how-to-create-exhaust-script","title":"How to create an exhaust script","description":"The If framework outputs data in yaml format. Any other output formats require a separate script that takes the yaml output data and processes it. We provide if-csv for outputting data in csv format bundled with IF. For any other format, you need to write an exhaust script.","sidebar":"tutorialSidebar"},"developers/how-to-refine-plugins":{"id":"developers/how-to-refine-plugins","title":"How to make plugins production ready","description":"Our How to build plugins guide covered the basics for how to construct an Impact Framework plugin. This guide will help you to refine your plugin to make it production-ready. These are best practice guidelines - if you intend to contribute your plugin to one of our repositories, following these guidelines will help your PR to get merged. Even if you are not aiming to have a plugin merged into one of our repositories, consistency with our norms is useful for debugging and maintaining and for making your plugin as useful as possible for other Impact Framework developers.","sidebar":"tutorialSidebar"},"developers/how-to-submit-plugins":{"id":"developers/how-to-submit-plugins","title":"How to submit plugins","description":"Once you have built a plugin and made it production ready you probably want to share it with the world!","sidebar":"tutorialSidebar"},"developers/how-to-write-unit-tests":{"id":"developers/how-to-write-unit-tests","title":"How to write unit tests","description":"Impact Framework unit tests follow a standard format. We use the jest testing library. You can run all our existing tests by opening the project directory and running npm test. This page explains how you can add new unit tests for your plugins (or add some for our plugins if you notice a gap).","sidebar":"tutorialSidebar"},"developers/index":{"id":"developers/index","title":"Developers","description":"This section contains information for Impact Framework developers. You are a developer if you want to change or update the Impact Framework by adding new features, fixing bugs or building new plugins.","sidebar":"tutorialSidebar"},"FAQ":{"id":"FAQ","title":"FAQs","description":"How can I contribute to Impact framework?","sidebar":"tutorialSidebar"},"intro":{"id":"intro","title":"Introduction","description":"Impact Framework","sidebar":"tutorialSidebar"},"major-concepts/aggregation":{"id":"major-concepts/aggregation","title":"Aggregation","description":"Aggregation is the process of summarizing a set of metrics.","sidebar":"tutorialSidebar"},"major-concepts/design-philosophy":{"id":"major-concepts/design-philosophy","title":"Design philosophy","description":"Transparency","sidebar":"tutorialSidebar"},"major-concepts/exhaust-script":{"id":"major-concepts/exhaust-script","title":"Exhaust scripts","description":"Exhaust scripts are scripts that can run independently of IF itself that take an executed manifest file (one with outputs) as an input, parse the yaml data and reformat it into some other representation. We provide if-csv bundled with IF, but if you want other data formats, you\'ll have to create an exhaust script yourself.","sidebar":"tutorialSidebar"},"major-concepts/groupby":{"id":"major-concepts/groupby","title":"Group-by","description":"Groupby is an IF plugin that reorganizes a tree according to keys provided by the user. This allows users to regroup their observations according to various properties of their application. For example, the following manifest file contains a flat array of observations. This is how you might expect data to arrive from an importer plugin, maybe one that hits a metrics API for a cloud service.","sidebar":"tutorialSidebar"},"major-concepts/if":{"id":"major-concepts/if","title":"Impact Engine (CLI)","description":"Introduction","sidebar":"tutorialSidebar"},"major-concepts/index":{"id":"major-concepts/index","title":"Major Concepts","description":"Here you will find explanations for the fundamental Impact Framework concepts. This includes:","sidebar":"tutorialSidebar"},"major-concepts/manifest-file":{"id":"major-concepts/manifest-file","title":"Manifest File","description":"Manifest files are fundamental to Impact Framework and they serve multiple important purposes, including:","sidebar":"tutorialSidebar"},"major-concepts/parameters":{"id":"major-concepts/parameters","title":"Parameters","description":"Parameters are handled carefully in IF because of the risk of errors propagating through plugin pipelines. We have a set of canonical parameters that are stored in a file called params.ts in the IF repository.","sidebar":"tutorialSidebar"},"major-concepts/pipelines":{"id":"major-concepts/pipelines","title":"Pipelines","description":"Pipelines are chains of plugins that operate in sequence over the input data in your manifest file.","sidebar":"tutorialSidebar"},"major-concepts/plugins":{"id":"major-concepts/plugins","title":"Plugins","description":"Plugins are self-contained units of code that do one thing. They can be loaded into IF and chained together in a pipeline so that simple individual plugins can form a complicated procedure for computing a manifest file.","sidebar":"tutorialSidebar"},"major-concepts/time":{"id":"major-concepts/time","title":"Time","description":"Every observation in an array of inputs represents a snapshot with a known start time and a known duration. For example, the following observation shows that the CPU utilization for a resource was 20% for the 10 second period starting at 1500 on the 22nd January 2024:","sidebar":"tutorialSidebar"},"pipelines/index":{"id":"pipelines/index","title":"Pipelines","description":"This section contains walkthrough guides for some common pipelines you may want to use in your manifest files.","sidebar":"tutorialSidebar"},"pipelines/instance-metadata":{"id":"pipelines/instance-metadata","title":"Instance metadata pipeline","description":"The instance metadata pipeline simply looks up a metadata for a given virtual machine instance name using the csv-lookup plugin from the IF standard library. However, the target dataset can return multiple processor names for a given VM instance where there are multiple possibilitiers. This means we need to create a pipeline that includes the regex plugin so parse out just one of the possible values.","sidebar":"tutorialSidebar"},"pipelines/sci":{"id":"pipelines/sci","title":"Software Carbon Intensity (SCI)","description":"The software carbon intensity (SCI) score is perhaps the most important value that can be generated using Impact Framework.","sidebar":"tutorialSidebar"},"pipelines/teads":{"id":"pipelines/teads","title":"Teads CPU pipeline","description":"The Teads CPU power curve CPU utilization (as a percentage) against a scaling factor that can be applied to the CPUs thermal design power to estimate the power drawn by the CPU in Watts.","sidebar":"tutorialSidebar"},"reference/cli":{"id":"reference/cli","title":"Command line tool","description":"A core feature of the Impact Framework is the if-run command line tool (CLI). This is how you trigger Impact Framework to execute a certain manifest file.","sidebar":"tutorialSidebar"},"reference/errors":{"id":"reference/errors","title":"Errors","description":"IF defines a finite set of error classes. All error messages emitted by IF are attached to one of these classes.","sidebar":"tutorialSidebar"},"reference/index":{"id":"reference/index","title":"Reference","description":"In this section you will find reference documentation for the core data structures and features used in the Impact Framework.","sidebar":"tutorialSidebar"},"reference/plugins":{"id":"reference/plugins","title":"Plugins","description":"Impact Framework works by executing pipelines of plugins over input data. Those plugins are re-useable units of code that can be thought of as Lego bricks - simple blocks of code that can be assembled into complex workflows.","sidebar":"tutorialSidebar"},"users/how-to-compare-files-with-if-diff":{"id":"users/how-to-compare-files-with-if-diff","title":"How to compare files with `if-diff`","description":"if-diff is a command line tool that allows you to compare two if-run output files. They either match according to if-diff\'s matching rules, or they don\'t. If they match, then if-diff returns a simple success response. If the differ, then if-diff returns a report of the differences it finds.","sidebar":"tutorialSidebar"},"users/how-to-export-csv-file-with-if-csv":{"id":"users/how-to-export-csv-file-with-if-csv","title":"how-to-export-csv-file-with-if-csv","description":"Exporting CSV file with if-csv","sidebar":"tutorialSidebar"},"users/how-to-import-plugins":{"id":"users/how-to-import-plugins","title":"How to load plugins","description":"Plugins are developed separately to the Impact Framework core. However, the IF core developers maintain a standard library of plugins come bundled with IF. These are known as builtins.","sidebar":"tutorialSidebar"},"users/how-to-install-if":{"id":"users/how-to-install-if","title":"How to install Impact Framework","description":"You can install Impact Framework either globally or locally. For most users, we recommend installing our official releases globally using npm. You can do this using the following command:","sidebar":"tutorialSidebar"},"users/how-to-verify-files-with-if-check":{"id":"users/how-to-verify-files-with-if-check","title":"Verifying IF outputs with `if-check`","description":"IF includes a command line tool called if-check that can be used to verify the results in a manifest file.","sidebar":"tutorialSidebar"},"users/how-to-write-manifests":{"id":"users/how-to-write-manifests","title":"How to write a manifest file","description":"The Impact Framework receives all its configuration and input data in the form of a manifest file known as an manifest. To use the framework, you will need to write a manifest file and pass its path to the command line tool. This guide will help you to understand how to construct one of these files and use it to measure the energy and carbon usage of your app.","sidebar":"tutorialSidebar"},"users/index":{"id":"users/index","title":"Users","description":"This section contains information for Impact Framework users. You are a user if you want to apply the Impact Framework to your own use-case, such as using it to measure the environmental impact of your own apps running on some cloud platform.","sidebar":"tutorialSidebar"},"users/quick-start":{"id":"users/quick-start","title":"Quick start","description":"This page will provide the basic instructions for getting up and running with Impact Framework.","sidebar":"tutorialSidebar"}}}')}}]); \ No newline at end of file diff --git a/assets/js/runtime~main.2bcaa095.js b/assets/js/runtime~main.2bcaa095.js deleted file mode 100644 index ae7e9ad3..00000000 --- a/assets/js/runtime~main.2bcaa095.js +++ /dev/null @@ -1 +0,0 @@ -(()=>{"use strict";var e,a,t,r,f,c={},o={};function d(e){var a=o[e];if(void 0!==a)return a.exports;var t=o[e]={id:e,loaded:!1,exports:{}};return c[e].call(t.exports,t,t.exports,d),t.loaded=!0,t.exports}d.m=c,d.c=o,e=[],d.O=(a,t,r,f)=>{if(!t){var c=1/0;for(i=0;i=f)&&Object.keys(d.O).every((e=>d.O[e](t[n])))?t.splice(n--,1):(o=!1,f0&&e[i-1][2]>f;i--)e[i]=e[i-1];e[i]=[t,r,f]},d.n=e=>{var a=e&&e.__esModule?()=>e.default:()=>e;return d.d(a,{a:a}),a},t=Object.getPrototypeOf?e=>Object.getPrototypeOf(e):e=>e.__proto__,d.t=function(e,r){if(1&r&&(e=this(e)),8&r)return e;if("object"==typeof e&&e){if(4&r&&e.__esModule)return e;if(16&r&&"function"==typeof e.then)return e}var f=Object.create(null);d.r(f);var c={};a=a||[null,t({}),t([]),t(t)];for(var o=2&r&&e;"object"==typeof o&&!~a.indexOf(o);o=t(o))Object.getOwnPropertyNames(o).forEach((a=>c[a]=()=>e[a]));return c.default=()=>e,d.d(f,c),f},d.d=(e,a)=>{for(var t in a)d.o(a,t)&&!d.o(e,t)&&Object.defineProperty(e,t,{enumerable:!0,get:a[t]})},d.f={},d.e=e=>Promise.all(Object.keys(d.f).reduce(((a,t)=>(d.f[t](e,a),a)),[])),d.u=e=>"assets/js/"+({53:"935f2afb",54:"667a5e8e",95:"95ad2285",99:"449bc0d9",109:"0a19367a",112:"cca8bc57",117:"29966d47",162:"486bc80e",168:"fd565be6",169:"89d906ef",229:"75e434b4",237:"1df93b7f",278:"3dce4fd2",288:"ad895e75",352:"f744e480",426:"244c9605",448:"8e29a039",468:"5c5a410f",485:"37c09b06",491:"b5ada7f5",514:"1be78505",527:"859a76f9",533:"bdfbfcca",642:"fbcc0412",644:"68ffc9f1",671:"0e384e19",672:"a4954f21",681:"425ff8ea",712:"ab3e713c",713:"93f138be",751:"1a3c9b31",760:"069226b7",796:"0569acb0",806:"28f080da",822:"63925da8",890:"1ead5b6d",918:"17896441",939:"0734eb39",940:"d57fc049",954:"8c7895a0",960:"91c76d4c"}[e]||e)+"."+{53:"74bbde3d",54:"0a66180c",95:"e3482d9c",99:"96810bc5",109:"7747066b",112:"8d17d27a",117:"d1761346",162:"e37292f9",168:"1a85a6a4",169:"79c3efec",229:"99169d05",237:"b7b5a3d3",248:"d6c76d13",278:"c0aed4f8",288:"c7013a94",352:"7d6a9191",426:"d6c3fa12",448:"0882b3db",468:"c6f0c19c",485:"40fe1089",491:"5f91c40d",514:"a282f924",527:"aa11797a",533:"d4823cdb",642:"4e280288",644:"3aac1eaa",671:"b41b0840",672:"97c940d4",681:"8c7173b7",712:"23be38d6",713:"febc7216",751:"a0e912eb",760:"19270c29",796:"fd39d925",806:"983696d1",822:"0131eb0f",890:"9a69b942",918:"dd24f0b7",939:"7fd12fdb",940:"0dbe304a",954:"e3cb11fb",960:"04d974b1"}[e]+".js",d.miniCssF=e=>{},d.g=function(){if("object"==typeof globalThis)return globalThis;try{return this||new Function("return this")()}catch(e){if("object"==typeof window)return window}}(),d.o=(e,a)=>Object.prototype.hasOwnProperty.call(e,a),r={},f="green-software-training:",d.l=(e,a,t,c)=>{if(r[e])r[e].push(a);else{var o,n;if(void 0!==t)for(var b=document.getElementsByTagName("script"),i=0;i{o.onerror=o.onload=null,clearTimeout(s);var f=r[e];if(delete r[e],o.parentNode&&o.parentNode.removeChild(o),f&&f.forEach((e=>e(t))),a)return a(t)},s=setTimeout(l.bind(null,void 0,{type:"timeout",target:o}),12e4);o.onerror=l.bind(null,o.onerror),o.onload=l.bind(null,o.onload),n&&document.head.appendChild(o)}},d.r=e=>{"undefined"!=typeof Symbol&&Symbol.toStringTag&&Object.defineProperty(e,Symbol.toStringTag,{value:"Module"}),Object.defineProperty(e,"__esModule",{value:!0})},d.p="/",d.gca=function(e){return e={17896441:"918","935f2afb":"53","667a5e8e":"54","95ad2285":"95","449bc0d9":"99","0a19367a":"109",cca8bc57:"112","29966d47":"117","486bc80e":"162",fd565be6:"168","89d906ef":"169","75e434b4":"229","1df93b7f":"237","3dce4fd2":"278",ad895e75:"288",f744e480:"352","244c9605":"426","8e29a039":"448","5c5a410f":"468","37c09b06":"485",b5ada7f5:"491","1be78505":"514","859a76f9":"527",bdfbfcca:"533",fbcc0412:"642","68ffc9f1":"644","0e384e19":"671",a4954f21:"672","425ff8ea":"681",ab3e713c:"712","93f138be":"713","1a3c9b31":"751","069226b7":"760","0569acb0":"796","28f080da":"806","63925da8":"822","1ead5b6d":"890","0734eb39":"939",d57fc049:"940","8c7895a0":"954","91c76d4c":"960"}[e]||e,d.p+d.u(e)},(()=>{var e={303:0,532:0};d.f.j=(a,t)=>{var r=d.o(e,a)?e[a]:void 0;if(0!==r)if(r)t.push(r[2]);else if(/^(303|532)$/.test(a))e[a]=0;else{var f=new Promise(((t,f)=>r=e[a]=[t,f]));t.push(r[2]=f);var c=d.p+d.u(a),o=new Error;d.l(c,(t=>{if(d.o(e,a)&&(0!==(r=e[a])&&(e[a]=void 0),r)){var f=t&&("load"===t.type?"missing":t.type),c=t&&t.target&&t.target.src;o.message="Loading chunk "+a+" failed.\n("+f+": "+c+")",o.name="ChunkLoadError",o.type=f,o.request=c,r[1](o)}}),"chunk-"+a,a)}},d.O.j=a=>0===e[a];var a=(a,t)=>{var r,f,c=t[0],o=t[1],n=t[2],b=0;if(c.some((a=>0!==e[a]))){for(r in o)d.o(o,r)&&(d.m[r]=o[r]);if(n)var i=n(d)}for(a&&a(t);b{"use strict";var e,a,t,r,f,c={},d={};function o(e){var a=d[e];if(void 0!==a)return a.exports;var t=d[e]={id:e,loaded:!1,exports:{}};return c[e].call(t.exports,t,t.exports,o),t.loaded=!0,t.exports}o.m=c,o.c=d,e=[],o.O=(a,t,r,f)=>{if(!t){var c=1/0;for(i=0;i=f)&&Object.keys(o.O).every((e=>o.O[e](t[n])))?t.splice(n--,1):(d=!1,f0&&e[i-1][2]>f;i--)e[i]=e[i-1];e[i]=[t,r,f]},o.n=e=>{var a=e&&e.__esModule?()=>e.default:()=>e;return o.d(a,{a:a}),a},t=Object.getPrototypeOf?e=>Object.getPrototypeOf(e):e=>e.__proto__,o.t=function(e,r){if(1&r&&(e=this(e)),8&r)return e;if("object"==typeof e&&e){if(4&r&&e.__esModule)return e;if(16&r&&"function"==typeof e.then)return e}var f=Object.create(null);o.r(f);var c={};a=a||[null,t({}),t([]),t(t)];for(var d=2&r&&e;"object"==typeof d&&!~a.indexOf(d);d=t(d))Object.getOwnPropertyNames(d).forEach((a=>c[a]=()=>e[a]));return c.default=()=>e,o.d(f,c),f},o.d=(e,a)=>{for(var t in a)o.o(a,t)&&!o.o(e,t)&&Object.defineProperty(e,t,{enumerable:!0,get:a[t]})},o.f={},o.e=e=>Promise.all(Object.keys(o.f).reduce(((a,t)=>(o.f[t](e,a),a)),[])),o.u=e=>"assets/js/"+({53:"935f2afb",54:"667a5e8e",95:"95ad2285",99:"449bc0d9",109:"0a19367a",112:"cca8bc57",117:"29966d47",162:"486bc80e",168:"fd565be6",169:"89d906ef",229:"75e434b4",237:"1df93b7f",278:"3dce4fd2",288:"ad895e75",352:"f744e480",426:"244c9605",448:"8e29a039",468:"5c5a410f",485:"37c09b06",491:"b5ada7f5",514:"1be78505",527:"859a76f9",533:"bdfbfcca",642:"fbcc0412",644:"68ffc9f1",671:"0e384e19",672:"a4954f21",681:"425ff8ea",712:"ab3e713c",713:"93f138be",751:"1a3c9b31",760:"069226b7",796:"0569acb0",806:"28f080da",822:"63925da8",890:"1ead5b6d",918:"17896441",939:"0734eb39",940:"d57fc049",954:"8c7895a0",960:"91c76d4c"}[e]||e)+"."+{53:"59ef3ce8",54:"0a66180c",95:"e3482d9c",99:"96810bc5",109:"7747066b",112:"8d17d27a",117:"d1761346",162:"e37292f9",168:"1a85a6a4",169:"79c3efec",229:"99169d05",237:"b7b5a3d3",248:"d6c76d13",278:"169f590e",288:"c7013a94",352:"7d6a9191",426:"d6c3fa12",448:"0882b3db",468:"c6f0c19c",485:"40fe1089",491:"5f91c40d",514:"a282f924",527:"aa11797a",533:"d4823cdb",642:"4e280288",644:"d6e6b40c",671:"b41b0840",672:"97c940d4",681:"8c7173b7",712:"23be38d6",713:"febc7216",751:"a0e912eb",760:"19270c29",796:"fd39d925",806:"983696d1",822:"0131eb0f",890:"9a69b942",918:"dd24f0b7",939:"d090e628",940:"0dbe304a",954:"e3cb11fb",960:"04d974b1"}[e]+".js",o.miniCssF=e=>{},o.g=function(){if("object"==typeof globalThis)return globalThis;try{return this||new Function("return this")()}catch(e){if("object"==typeof window)return window}}(),o.o=(e,a)=>Object.prototype.hasOwnProperty.call(e,a),r={},f="green-software-training:",o.l=(e,a,t,c)=>{if(r[e])r[e].push(a);else{var d,n;if(void 0!==t)for(var b=document.getElementsByTagName("script"),i=0;i{d.onerror=d.onload=null,clearTimeout(s);var f=r[e];if(delete r[e],d.parentNode&&d.parentNode.removeChild(d),f&&f.forEach((e=>e(t))),a)return a(t)},s=setTimeout(l.bind(null,void 0,{type:"timeout",target:d}),12e4);d.onerror=l.bind(null,d.onerror),d.onload=l.bind(null,d.onload),n&&document.head.appendChild(d)}},o.r=e=>{"undefined"!=typeof Symbol&&Symbol.toStringTag&&Object.defineProperty(e,Symbol.toStringTag,{value:"Module"}),Object.defineProperty(e,"__esModule",{value:!0})},o.p="/",o.gca=function(e){return e={17896441:"918","935f2afb":"53","667a5e8e":"54","95ad2285":"95","449bc0d9":"99","0a19367a":"109",cca8bc57:"112","29966d47":"117","486bc80e":"162",fd565be6:"168","89d906ef":"169","75e434b4":"229","1df93b7f":"237","3dce4fd2":"278",ad895e75:"288",f744e480:"352","244c9605":"426","8e29a039":"448","5c5a410f":"468","37c09b06":"485",b5ada7f5:"491","1be78505":"514","859a76f9":"527",bdfbfcca:"533",fbcc0412:"642","68ffc9f1":"644","0e384e19":"671",a4954f21:"672","425ff8ea":"681",ab3e713c:"712","93f138be":"713","1a3c9b31":"751","069226b7":"760","0569acb0":"796","28f080da":"806","63925da8":"822","1ead5b6d":"890","0734eb39":"939",d57fc049:"940","8c7895a0":"954","91c76d4c":"960"}[e]||e,o.p+o.u(e)},(()=>{var e={303:0,532:0};o.f.j=(a,t)=>{var r=o.o(e,a)?e[a]:void 0;if(0!==r)if(r)t.push(r[2]);else if(/^(303|532)$/.test(a))e[a]=0;else{var f=new Promise(((t,f)=>r=e[a]=[t,f]));t.push(r[2]=f);var c=o.p+o.u(a),d=new Error;o.l(c,(t=>{if(o.o(e,a)&&(0!==(r=e[a])&&(e[a]=void 0),r)){var f=t&&("load"===t.type?"missing":t.type),c=t&&t.target&&t.target.src;d.message="Loading chunk "+a+" failed.\n("+f+": "+c+")",d.name="ChunkLoadError",d.type=f,d.request=c,r[1](d)}}),"chunk-"+a,a)}},o.O.j=a=>0===e[a];var a=(a,t)=>{var r,f,c=t[0],d=t[1],n=t[2],b=0;if(c.some((a=>0!==e[a]))){for(r in d)o.o(d,r)&&(o.m[r]=d[r]);if(n)var i=n(o)}for(a&&a(t);b How to build plugins | Impact Framework - + @@ -16,7 +16,7 @@
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

How to build plugins

The IF is designed to be as composable as possible. This means you can develop your own plugins and use them in a pipeline. To help developers write Typescript plugins to integrate easily into IF, we provide the PluginInterface interface. Here's an overview of the stages you need to follow to integrate your plugin:

  • create a Typescript file that implements the PluginInterface
  • install the plugin
  • initialize and invoke the plugin in your manifest file

Step 1: Use our template repository​

Instead of building up your plugin repository and all the configuration from scratch, you can use our plugin template repository. To use the template, visit the Github repository and click the Use this template button. You will have the option to create a new repository under your own account. Then, you can clone that repository to your local machine.

use our template repository

Inside that repository, all you have to do is run npm install typescript in the template folder, rename the project in package.json and write your plugin code inside index.ts. All the configuration and setup is taken care of for you.

Step 2: Writing your plugin code​

Now your project is setup, you can focus on your plugin logic. The entry point for your plugin is index.ts. In this guide it is assumed that all your plugin logic is in index.ts but depending on the copmplexity of your plugin you might want to split the code across multiple files. index.ts should always be your entry point, though.

The following sections describe the rules your plugin code should conform to. We also have an appendix that deep dives a real plugin.

The plugin interface​

The PluginInterface is structured as follows:

export type PluginInterface = {
  execute: (
    inputs: PluginParams[],
    config?: Record<string, any>
  ) => PluginParams[];
  metadata: {
    kind: string;
  };
  [key: string]: any;
};

The interface requires an execute function where your plugin logic is implemented. It should also return metadata. This can include any relevant metadata you want to include, with a minimum requirement being kind: execute.

Global config​

Global config is passed as an argument to the plugin. In your plugin code you can handle it as follows:

// Here's the function definition - notice that global config is passed in here!
export const Plugin = (globalConfig: YourConfig): PluginInterface => {

// in here you have access to globalConfig[your-params]

}

The parameters available to you in globalConfig depends upon the parameters you pass in the manifest file. For example, the Sum plugin has access to input-parameters and output-parameter in its global config, and it is defined in the Initialize block in the manifest file as follows:

initialize:
  plugins:
    sum:
      method: Sum
      path: 'builtin'
      global-config:
        input-parameters: ['cpu/energy', 'network/energy']
        output-parameter: 'energy'

Methods​

execute​

execute() is where the main calculation logic of the plugin is implemented. It always takes inputs (an array of PluginParams) as an argument and returns an updated set of inputs.

Params​

ParamTypePurpose
inputsPluginParams[]Array of data provided in the inputs field of a component in a manifest file

Returns​

Return valueTypePurpose
outputsPromise<PluginParams[]>Promise resolving to an array of updated PluginParams[]

What are PluginParams?​

What are PluginParams?​

PluginParams are a fundamental data type in the Impact Framework. The type is defined as follows:

export type PluginParams = {
  [key: string]: any;
};

The PluginParams type therefore defines an array of key-value pairs.

IF needs to know about all the parameters used in each pipeline. The default behaviour is that it grabs parameters from a local file, params.ts. This file defines the standard set of parameter names, their units, a descriptiona nd the method used to aggregate them across time or across a tree.

If your new plugin uses new parameters that are not included in params.ts, you can simply add them to your manifest file in a section named params. For example:

name: params-demo
description: null
tags:
params: 
  - name: new-param-1
    description: dummy
    aggregation: sum
    unit: MT
  - name: new-param-2
    description: dummy
    aggregation: sum
    unit: s

This will append the new parameter informatrion to the object loaded from params.ts and you can use your plugin as normal. In effect, you have append-only access to params.ts via your manifest file without ever having to change any IF source code.

However, if you are an advanced user and you want to use something other than out recommended standard set of parameters, you can provide a replacement params.ts file on the command line. This file should be a json or js/ts file with the ame structure as our params.ts. You can rename the file. You then pass the path to the file to the override-params command.

if-run --manifest <path-to-manifest> --override-params <path-to-your-params-file>

Step 3: Install your plugin​

Now your plugin code is written, you can install it to make it available to IF.

npm run build

Then use npm link to create a package that can be installed into IF:

npm link

Step 4: Load your plugin into IF​

Now your plugin is ready to run in IF. First install your plugin by navigating to the if project folder and running:

npm link new-plugin

replacing new-plugin with your plugin name as defined in the plugin's package.json. If you are not sure, the name can be checked by running npm ls -g --depth=0 --link=true.

Your plugin is now ready to be run in IF. All that remains is to add your plugin to your manifest file. This means adding it to the initialize block and adding it to the component pipelines where you want your plugin to be executed. For example, an initilize block might look as follows:

initialize:
  plugins:
    new-plugin:
      method: YourFunctionName
      path: 'new-plugin'
      global-config:
        something: true

Run your manifest uisng

np run if-run -- --manifest <path-to-manifest>

If you have to link more than one local plugin, for example to test your plugin in a pipeline, you can do so with

npm link new-plugin --save

This will create an entry like "new-plugin": "file:path/to/your/plugin" in the package.json which links to your local plugin. This way, multiple plugins can be linked at once. Of course, these changes should not be committed, but they can be helpful for local testing.

Step 5: Publishing your plugin​

Now you have run your plugin locally and you are happy with how it works, you can make it public by publishing it to a public Github repository. Now all you have to do to use it in a manifest file is npm install it and pass the path to the Github repository in the plugin initialize block.

For example, for a plugin saved in github.com/my-repo/new-plugin you can do the following:

npm install https://github.com/my-repo/new-plugin

Then, in your manifest file, provide the path in the plugin instantiation. You also need to specify which function the plugin instantiates. Let's say you are using the Sum plugin from the example above:

name: plugin-demo
description: loads plugin
tags: null
initialize:
  plugins:
    - name: new-plugin
      kind: plugin
      method: FunctionName
      path: https://github.com/my-repo/new-plugin
tree:
  children:
    child:
      config:
      inputs:

Now, when you run the manifest file, it will load the plugin automatically.

You can run this using the globally installed IF as follows:

if-run --manifest <path-to-my-manifest>

Summary of steps​

  • Copy our template repository and update package.json
  • Add your plugin code to index.ts
  • Build and link the plugin using npm run build && npm link
  • Load your plugin into if using npm link
  • Initialize your plugin and add it to a pipeline in your manifest file.
  • Publish your plugin to Github

You should also create unit tests for your plugin to demonstrate correct execution and handling of corner cases.

Next steps​

You can read our more advanced guide on how to refine your plugins.

Appendix: Walk-through of the Sum plugin​

To demonstrate how to build a plugin that conforms to the pluginInterface, let's examine the sum plugin.

The sum plugin implements the following logic:

  • sum whatever is provided in the input-parameters field from globalConfig.
  • append the result to each element in the output array with the name provided as output-parameter in globalConfig.

Let's look at how you would implement this from scratch:

The plugin must be a function conforming to PluginInterface. You can call the function Sum, and inside the body you can add the signature for the execute method:

export const Sum = (globalConfig: SumConfig): PluginInterface => {
  const errorBuilder = buildErrorMessage(Sum.name);
  const metadata = {
    kind: 'execute',
  };

  /**
   * Calculate the sum of each input.
   */
  const execute = async (inputs: PluginParams[]): Promise<PluginParams[]> => {

  };

  return {
    metadata,
    execute,
  };

}

Your plugin now has the basic structure required for IF integration. Your next task is to add code to the body of execute to enable the actual plugin logic to be implemented.

The execute function should grab the input-parameters (the values to sum) from globalConfig. it should then iterate over the inputs array, get the values for each of the input-parameters and append them to the inputs array, using the name from the output-parameter value in globalConfig. Here's what this can look like, with the actual calculation pushed to a separate function, calculateSum. 

  /**
   * Calculate the sum of each input.
   */
  const execute = async (inputs: PluginParams[]): Promise<PluginParams[]> => {
    const inputParameters = globalConfig['input-parameters'];
    const outputParameter = globalConfig['output-parameter'];

    return inputs.map(input => {
      return {
        ...input,
        [outputParameter]: calculateSum(input, inputParameters),
      };
    });

  return {
    metadata,
    execute,
  };
}

Now we just need to define what happens in calculateSum - this can be a simple reduce:

  /**
   * Calculates the sum of the energy components.
   */
  const calculateSum = (input: PluginParams, inputParameters: string[]) =>
    inputParameters.reduce(
      (accumulator, metricToSum) => accumulator + input[metricToSum],
      0
    );

Note that this example did not include any validation or error handling - you will likely want to add some for a real plugin.

Finally, if your plugin used any fields in inputs or created new outputs that have not been used in the Impact Framework before, then you should add them to params.ts.

params.ts can be found in the path src/config.

Each entry in params.ts looks as follows:

carbon:
  description: an amount of carbon emitted into the atmosphere
  unit: gCO2e
  aggregation: sum

This information allows IF to programmatically make decisions about how to handle values in features such as aggregation, time normalization and visualizations, and also acts as a global reference document for understanding IF data. The example above is for carbon.

You should add your new data, give a name, define a unit and short description. The aggregation field determines how the value is treated when some manipulation has to be done to spread the value over time or aggregate it.

For absolute metrics like carbon, the right value is sum because you would want to add carbon emissions from each timestep when you aggregate over time.

For proportional metrics, the right value is avg. For example, you would want to calculate the average cpu/utilization - it would not make sense to sum it when aggregating over multiple timesteps.

Finally, values that should always be presented identically regardless of any aggregation, such as names or global constants, should be given the aggregation-method value none.

Managing errors​

If framework provides it's own set of error classes which will make user's live much more easier! If Core plugin has a set of error classes which can be used for having full integration with the IF framework. More details about each error class can be found at Errors Reference

Now you are ready to run your plugin using the if-run CLI tool!

- + \ No newline at end of file diff --git a/developers/how-to-create-exhaust-script/index.html b/developers/how-to-create-exhaust-script/index.html index fc40992b..f0a7e712 100644 --- a/developers/how-to-create-exhaust-script/index.html +++ b/developers/how-to-create-exhaust-script/index.html @@ -8,14 +8,14 @@ How to create an exhaust script | Impact Framework - +
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

How to create an exhaust script

The If framework outputs data in yaml format. Any other output formats require a separate script that takes the yaml output data and processes it. We provide if-csv for outputting data in csv format bundled with IF. For any other format, you need to write an exhaust script. This guide will help you create your own exhaust script.

In this example, we'll create a script that executes the manifest and outputs the data in json format.

const IfJson = async () => {
  const { manifest, output } = await parseIfCsvArgs();

  if (manifest) {
    const { rawManifest } = await load(manifest);
    const { children } = rawManifest.tree;

    if (!(children?.child || children?.['child-0']).outputs) {
      throw new ManifestValidationError(FAILURE_MESSAGE_OUTPUTS);
    }

    // Add logic to export the executed manifest to `json` format.
  }

  process.exit(0);
};

IfJson().catch(handleError);

To add this script to your package.json, include the following entry in the scripts section:

"scripts": {
  "if-json": "npx ts-node if-json.ts"
}

This setup ensures that your script will execute the manifest and output the data in JSON format.

- + \ No newline at end of file diff --git a/developers/how-to-refine-plugins/index.html b/developers/how-to-refine-plugins/index.html index 163192ba..e2672ec1 100644 --- a/developers/how-to-refine-plugins/index.html +++ b/developers/how-to-refine-plugins/index.html @@ -8,7 +8,7 @@ How to make plugins production ready | Impact Framework - + @@ -16,7 +16,7 @@
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

How to make plugins production ready

Our How to build plugins guide covered the basics for how to construct an Impact Framework plugin. This guide will help you to refine your plugin to make it production-ready. These are best practice guidelines - if you intend to contribute your plugin to one of our repositories, following these guidelines will help your PR to get merged. Even if you are not aiming to have a plugin merged into one of our repositories, consistency with our norms is useful for debugging and maintaining and for making your plugin as useful as possible for other Impact Framework developers.

1. Naming conventions​

We prefer not to use abbreviations of contractions in parameter names. Using fully descriptive names makes the code more readable, which in turn helps reviewers and anyone else aiming to understand how the plugin works. It also helps to avoid ambiguity and naming collisions within and across plugins. Your name should describe what an element does as precisely as practically possible.

For example, we prefer cpu/energy to e-cpu and we prefer functionalUnit to funcUnit, fUnit, or any other abbreviation.

In Typescript code we use lower Camel case (likeThis) for variable and function names and Pascal/Upper Camel case for class, type, enum, and interface names (LikeThis).

For example:

  • sci is the name for the SCI value normalized per second.
  • energy is the name for the array of energy metrics available to be summed in the sci-e plugin

In yaml files, we prefer to use kebab-case (like-this) for field names. For example:

  • network/energy is the field name for the energy consumed by networking for an application
  • functional-unit is the unit in which to express an SCI value.

Global constants can be given capitalized names, such as TIME_UNITS_IN_SECONDS.

2. Plugin code​

Imports​

We prefer the following ordering of imports in your plugin code:

  1. Node built-in modules (e.g. import fs from 'fs';)
  2. External modules (e.g. import {z} from 'zod';)
  3. Internal modules (e.g. import config from 'src/config';)
  4. Interfaces (e.g. import type {PluginInterface} from 'interfaces')
  5. Types (e.g. import {PluginParams} from '../../types/common';)

Comments​

Each logical unit in the code should be preceded by an appropriate explanatory comment. Sometimes it is useful to include short comments inside a function that clarifies the purpose of a particular statement. Here's an example from our codebase:

  /**
   * Calculates the energy consumption for a single input.
   */
  const calculateEnergy = (input: PluginParams) => {
    const {
      'memory/capacity': totalMemory,
      'memory/utilization': memoryUtil,
      'energy-per-gb': energyPerGB,
    } = input;

    // GB * kWh/GB == kWh
    return totalMemory * (memoryUtil / 100) * energyPerGB;
  };

Error handling​

We use custom errors across our codebase to make it as easy as possible to understand the root cause of a problem. You can use our error handlers by importing if-core as a dependency of your plugin. This provides you with our error handling code and predefined list of error classes that you can invoke. This gives you tight integration with IF, because the framework can recognize those error classes and automatically incorporate them into the framework's error handling routines.

Just import ERRORS from if-core and use the error classes that are appropriate for your use-case.

e.g.

import {ERRORS} from '@grnsft/if-core/util';

const {MissingInputDataError} = ERRORS;

... 

throw new MissingInputDataError("my-plugin is missing my-parameter from inputs[0]");

Validation​

We recommend using validation techniques to ensure the integrity of input data. Validate input parameters against expected types, ranges, or constraints to prevent runtime errors and ensure data consistency.

We use zod to validate data. Here's an example from our codebase:

/**
 * Checks for required fields in input.
 */
const validateInput = (input: PluginParams) => {
  const schema = z
    .object({
      'cpu/name': z.string(),
    })
    .refine(allDefined, {
      message: '`cpu/name` should be present.',
    });

  return validate<z.infer<typeof schema>>(schema, input);
}

Code Modularity​

Break down complex functionality into smaller, manageable methods with well-defined responsibilities. Encapsulate related functionality into private methods to promote code reusability and maintainability.

3. Unit tests​

Your plugin should have unit tests with 100% coverage. We use jest to handle unit testing. We strive to have one describe per function. Each possible outcome from each function is separated using it with a precise and descriptive message.

Here's an example that covers plugin initialization and the happy path for the execute() function.

import {Sum} from '../../../../lib';

import {ERRORS} from '@grnsft/if-core/util/';

const {InputValidationError} = ERRORS;

describe('lib/sum: ', () => {
  describe('Sum: ', () => {
    const globalConfig = {
      'input-parameters': ['cpu/energy', 'network/energy', 'memory/energy'],
      'output-parameter': 'energy',
    };
    const sum = Sum(globalConfig);

    describe('init: ', () => {
      it('successfully initalized.', () => {
        expect(sum).toHaveProperty('metadata');
        expect(sum).toHaveProperty('execute');
      });
    });

    describe('execute(): ', () => {
      it('successfully applies Sum strategy to given input.', async () => {
        expect.assertions(1);

        const expectedResult = [
          {
            duration: 3600,
            'cpu/energy': 1,
            'network/energy': 1,
            'memory/energy': 1,
            energy: 3,
            timestamp: '2021-01-01T00:00:00Z',
          },
        ];

        const result = await sum.execute([
          {
            duration: 3600,
            'cpu/energy': 1,
            'network/energy': 1,
            'memory/energy': 1,
            timestamp: '2021-01-01T00:00:00Z',
          },
        ]);

        expect(result).toStrictEqual(expectedResult);
      });
    }
  })
})

We have a dedicated page explaining in more detail how to write great unit tests for Impact Framework plugins.

4. Linting​

We use ESLint to format our code. We use a very simple configuration file (eslintrc.json), as follows:

{
  "extends": "./node_modules/gts/",
  "rules": {
    "@typescript-eslint/no-explicit-any": ["off"]
  }
}

For our repositories we use Github CI to enforce the linting rules for any pull requests.

Summary​

On this page, we have outlined best practices for refining your plugins so that they conform to our expected norms. This will help you write clean, efficient, and understandable code!

- + \ No newline at end of file diff --git a/developers/how-to-submit-plugins/index.html b/developers/how-to-submit-plugins/index.html index 2974741c..641a7ac2 100644 --- a/developers/how-to-submit-plugins/index.html +++ b/developers/how-to-submit-plugins/index.html @@ -8,13 +8,13 @@ How to submit plugins | Impact Framework - +
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

How to submit plugins

Once you have built a plugin and made it production ready you probably want to share it with the world!

We provide an IF Explorer website where you can list your plugins for the world to see.

By submitting your plugins to the Explorer you make them discoverable to other IF users. Users can browse or search the listed plugins and install them in their own pipelines. This is a great way to get exposure for your work and help grow the IF ecosystem.

Submitting a plugin​

When you are happy with your plugin code, you can simply head to our submission portal and fill in the form. As long as you provide the requested information and your README is sufficiently detailed, your plugin will get listed on the site.

It's as easy as filling in the form - we handle the rest!

Acceptance criteria​

You should fill in alll the fields on the submission form in order to list your plugin (it is ok to leave the npm field blank if you did not yet publish your plugin to npm).

However, we do want to see some specific information provided in your README.

Your plugin documentation should include the following information:

  • A list of the required and optional parameters and return values, with types and any valid ranges/properties clearly defined (REQUIRED).
  • A list of required and optional configuration (REQUIRED)
  • A description of any environment setup such as credentials in environment variables, etc. (REQUIRED)
  • Installation instructions (REQUIRED)
  • A written description of the plugin behaviour (REQUIRED)
  • A demo manifest that executes the plugin correctly without errors (REQUIRED)
  • A link to your unit tests
  • A list of errors that your plugin can raise, the behaviours that cause them and potential remedies.
  • A reference list of any publications or other material supporting the approach you have taken in your plugin.

The more comprehensive the documentation, the more likely users are to download and use your plugin.

Disclaimer​

PLEASE BE ADVISED THAT IF YOU RUN CODE INCLUDED IN THESE PLUGINS OR USE INFORMATION GENERATED BY THESE PLUGINS, INCLUDING MAKING ANY DECISIONS BASED UPON THE INFORMATION, YOU DO SO AT YOUR OWN RISK. GREEN SOFTWARE FOUNDATION MAKES NO REPRESENTATION, GUARANTEE OR WARRANTY, EXPRESS OR IMPLIED, STATUTORY OR OTHERWISE, INCLUDING MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE NOR ANY WARRANTIES ARISING FROM COURSE OF PERFORMANCE, COURSE OF DEALING OR USAGE IN TRADE.

We are only doing minimal QA on plugins. This amounts to checking that the necessary information is provided in a submission form and README. This means we are not checking that plugins execute correctly, provide accurate results or that they are free from security vulnerabilities. You must do your own research and do your own due diligence for any use of plugins listed on the explorer.

- + \ No newline at end of file diff --git a/developers/how-to-write-unit-tests/index.html b/developers/how-to-write-unit-tests/index.html index eb46b4f1..d241a3b6 100644 --- a/developers/how-to-write-unit-tests/index.html +++ b/developers/how-to-write-unit-tests/index.html @@ -8,14 +8,14 @@ How to write unit tests | Impact Framework - +
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

How to write unit tests

Impact Framework unit tests follow a standard format. We use the jest testing library. You can run all our existing tests by opening the project directory and running npm test. This page explains how you can add new unit tests for your plugins (or add some for our plugins if you notice a gap).

Test files​

Both the IF and the project repositories include a __test__ directory. Inside, you will find subdirectory unit/lib containing directories for each plugin. Your plugin repository should also follow this structure. Inside the plugin directory you can add index.test.ts. This is where you write your unit tests. For example, here's the directory tree for our teads-curve test file:


if-unofficial-plugins
 |
 |- src
    |
    |-__tests__
        |
        |-unit
          |
          |-lib
             |
             teads-curve
                 |
                 |- index.test.ts

Setting up your test file​

You will need to import your plugin so that it can be instantiated and tested. You will also need some elements from jest/globals: For example, these are the imports for our Sum plugin.

import {Sum} from '../../../../lib';
import {ERRORS} from '../../../../util/errors';
const {InputValidationError} = ERRORS;

You may require other imports for your specific set of tests.

Describe​

Each method should have its own dedicated describe block.

Your unit tests should have at least two describe blocks, one to test the plugin initialization and one for execute.

describe("init", ()=> {})
describe("execute", ()=> {})

For example, here is a describe block checking that the Sum plugin initializes correctly:

describe('lib/sum: ', () => {
  describe('Sum: ', () => {
    const globalConfig = {
      'input-parameters': ['cpu/energy', 'network/energy', 'memory/energy'],
      'output-parameter': 'energy',
    };
    const sum = Sum(globalConfig);

    describe('init: ', () => {
      it('successfully initalized.', () => {
        expect(sum).toHaveProperty('metadata');
        expect(sum).toHaveProperty('execute');
      });
    });
  })
})

It​

Within each describe block, each effect to be tested should have a dedicated it block.

Here's an example of a new describe block for the execute() method on the Sum plugin. The describe block indicates that we are testing effects of the execute() method. it is specific to a single outcome - in this case there are two it blocks that test that the plugin returns a specific result in the happy path and throws an exception if the user has provided invalid config data, specifically that the user-provided cpu/energy parameter is missing:

    describe('execute(): ', () => {
      it('successfully applies Sum strategy to given input.', async () => {
        expect.assertions(1);

        const expectedResult = [
          {
            duration: 3600,
            'cpu/energy': 1,
            'network/energy': 1,
            'memory/energy': 1,
            energy: 3,
            timestamp: '2021-01-01T00:00:00Z',
          },
        ];

        const result = await sum.execute([
          {
            duration: 3600,
            'cpu/energy': 1,
            'network/energy': 1,
            'memory/energy': 1,
            timestamp: '2021-01-01T00:00:00Z',
          },
        ]);

        expect(result).toStrictEqual(expectedResult);
      });

      it('throws an error on missing params in input.', async () => {
        const expectedMessage =
          'Sum: cpu/energy is missing from the input array.';

        expect.assertions(1);

        try {
          await sum.execute([
            {
              duration: 3600,
              timestamp: '2021-01-01T00:00:00Z',
            },
          ]);
        } catch (error) {
          expect(error).toStrictEqual(
            new InputValidationError(expectedMessage)
          );
        }
      });
    })

Errors​

We prefer to use expect to check the errors returned from a test. We do this by writing expect in a catch block. Here's an example from our sci plugin tests:

it('throws an exception on missing functional unit data.', async () => {
  const inputs = [
    {
      timestamp: '2021-01-01T00:00:00Z',
      'operational-carbon': 0.002,
      'embodied-carbon': 0.0005,
      'functional-unit': 'requests',
      duration: 1,
    },
  ];
  expect.assertions(1);

  try {
    await sciModel.execute(inputs);
  } catch (error) {
    expect(error).toBeInstanceOf(InputValidationError);
  }
});

It is also necessary to include expect.assertions(n) for testing asynchronous code, where n is the number of assertiosn that should be tested before the test completes.

Mocks​

Please try to avoid mocking data if possible. However, if it is necessary to mock (e.g. if your plugin relies on a third party credentialed API) then please make your mock data as realistic as possible (no foo, bar, baz style mock data, please).

We do have mock backends in several of our tests, and we also have a mock data generator plugin that can create realistic dummy data to your specific requirements.

Coverage​

Please use jest --coverage to see a coverage report for your plugin - your unit tests should yield 100% coverage. The snippet below shows what to expect from the coverage report:

-------------------------------|---------|----------|---------|---------|-------------------
| File                        | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s |
| --------------------------- | ------- | -------- | ------- | ------- | ----------------- |
| All files                   | 100     | 100      | 100     | 100     |
| lib                         | 100     | 100      | 100     | 100     |
| index.ts                    | 100     | 100      | 100     | 100     |
| lib/cloud-metadata          | 100     | 100      | 100     | 100     |
| index.ts                    | 100     | 100      | 100     | 100     |
| lib/e-mem                   | 100     | 100      | 100     | 100     |
| index.ts                    | 100     | 100      | 100     | 100     |
| lib/e-net                   | 100     | 100      | 100     | 100     |
| index.ts                    | 100     | 100      | 100     | 100     |
- + \ No newline at end of file diff --git a/developers/index.html b/developers/index.html index 4b8bf61f..e1473f2f 100644 --- a/developers/index.html +++ b/developers/index.html @@ -8,13 +8,13 @@ Developers | Impact Framework - +
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

Developers

This section contains information for Impact Framework developers. You are a developer if you want to change or update the Impact Framework by adding new features, fixing bugs or building new plugins.

The developer documentation includes:

If you are looking for guidance for how to use IF to measure the environmental impact of your apps, you should go to our user documentation instead.

- + \ No newline at end of file diff --git a/index.html b/index.html index 5c087b4c..1014c982 100644 --- a/index.html +++ b/index.html @@ -8,13 +8,13 @@ Welcome to Impact Framework | Impact Framework - +
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

Impact Framework

Impact Framework is a way to compute and report the environmental impacts of software applications accurately.

Read The DocsExplore IF Plugins

Transform Observations into Impacts

Take easily observable metrics like CPU utilization, page views and installs and convert them into environmental impacts such as carbon emissions, water usage, energy consumption, and air quality.

Buildable Plugin Ecosystem

Engage an existing library of plugins or create new plugins within Impact Framework based on your assumptions and observations.

Explore What-If Scenarios

Uncover the environmental impact of your software changes in real-time by examining how your software’s environmental performance changes if your application moves to the cloud or experiences shifts in its runtime.

Decentralize Data

Record your observations, chosen plugins, configurations, and computed environmental impacts in a manifest file. Open the door for others to understand, verify, and challenge the entire process.

Democratize Measurement

Empower others to rerun your manifest file, validate your findings, or question your assumptions. They can tweak configurations, select different plugins, and run the analysis themselves.

- + \ No newline at end of file diff --git a/intro/index.html b/intro/index.html index 2b428a75..d9cdbc4e 100644 --- a/intro/index.html +++ b/intro/index.html @@ -8,13 +8,13 @@ Introduction | Impact Framework - +
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

Introduction


Impact Framework​

Impact Framework (IF) aims to make the environmental impacts of software easier to calculate and share.

IF allows you to calculate the environmental impacts, such as carbon, of your software applications without writing any code. All you have to do is write a simple manifest file and IF handles the rest.

The project is entirely open source and composability is a core design principle - we want you to be able to create your own plugins and plug them in to our framework, or pick from a broad universe of open source plugins created by others.

Motivation​

If you can't measure, you can't improve. Software has many negative environmental impacts which we need to optimize, carbon, water, and energy, to name just a few.

Unfortunately, measuring software impact metrics like carbon, water, and energy is complex and nuanced.

Modern applications are composed of many smaller pieces of software (components) running on different environments, for example, private cloud, public cloud, bare-metal, virtualized, containerized, mobile, laptops, desktops, embedded, and IoT. Many components that make up a typical software application are run on something other than resources you own or control, which makes including the impact of managed services in your measurement especially hard.

The impacts of software components also vary over time, so as well as understanding which components contribute most to the overall impacts, there is also a question of when they contribute the most.

Only through a granular analysis of the impacts of your software system can investments in reducing its impact be prioritized and verified. Measurement is the first and most crucial step in greening a software system, and the first step in that process with the Impact Framework is to create a tree.

Background​

This project has evolved over the two years of the GSF's existence.

During the development of the SCI, we acknowledged that the biggest blocker to adoption was data regarding the emissions of software components on different platforms and runtimes.

We then launched the sci-data project to help create the data sets required to calculate an SCI score.

After some investigation, the original sci-data team quickly realized that there were several existing data sources, and many more were in development, free open source or private commercial. The future challenge wouldn't be to source them, it would be knowing which data set to use for which use case, how data sets differed in their methodology and interface and when to use one over the other, the pros/cons, and trade-offs.

The project evolved into the sci-guide to document existing data sets, providing guidance for when to use one over another and how to use it to create your own software measurement reports.

Finally, we had enough information, and SCI case studies started to be written. This was a milestone moment.

But now we are in the next evolution, to have software measurement be a mainstream activity. For this to be an industry with thousands of professionals working to decarbonize software, for businesses to grow and thrive in a commercial software measurement ecosystem, we need to formalize software measurement into a discipline with standards and tooling. The SCI Specification is the standard, and the Impact Framework is the tooling.

Project Structure​

The IF source code can be found in the IF Github repository. The code there covers the framework, which includes all the infrastructure for reading and writing input and output yamls, invoking plugins, running the command line tool and associated helper functions. However, it does not include the actual plugins themselves. Part of the IF design philosophy is that plugins should aim to do one thing as simply as possible, so that the IF is as composable and configurable as possible. Therefore, to use IF, you have to either create your own plugins or find some prebuilt ones and install them yourself. This also implies that you take responsibility for the plugins you choose to install.

We do provide a standard library of plugins built and maintained by the IF core team. These come bundled with IF. Their source code and README documentation and can be found in if/src/builtins.

There are also a wide range of community-owned plugins that we make discoverable on our Explorer website

Finally, the source code for this documentation website is available at the if-docs Github repository.

The lefthand sidebar contains links to all the information you need to understand Impact Framework.

You can explore the key ideas underpinning Impact Framework in the Major Concepts section.

Users can read our guides explaining how to use IF, including installation, using the CLI and loading plugins.

We also have developer documentation to help you get started building with IF.

You will find documentation for the individual built-in plugin implementations in plugins.

- + \ No newline at end of file diff --git a/major-concepts/aggregation/index.html b/major-concepts/aggregation/index.html index fd8ebcb0..1b4de66a 100644 --- a/major-concepts/aggregation/index.html +++ b/major-concepts/aggregation/index.html @@ -8,13 +8,13 @@ Aggregation | Impact Framework - +
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

Aggregation

Aggregation is the process of summarizing a set of metrics.

Two types of aggregation can be executed in IF. The first takes a time series and condenses it down into a single number representing an entire observation period, for example, if your time series contains three timesteps, the value of the metric being aggregated is 1 in each timestep and the aggregate is being computed as a sum, the aggregated value for the whole observation period is 3. We refer to this as "time series aggregation" or "horizontal aggregation".

The second type of aggregation happens across components in a tree. Where time series exist for multiple child nodes under a parent, their time series are aggregated together into one summary time series that is pushed to the parent node. For example, in the following tree, each child has a time series with three timesteps. At each timestep, the metric value is 1 for both children:

         parent
        /      \
       /        \
child-1          child-2
data: [1,1,1]    data: [1,1,1]

Assuming the aggregation method is sum, the parent would receive an aggregated time series [2,2,2], representing the aggregated values from both children. We refer to this as "tree aggregation" or "vertical aggregation".

Configuration​

Aggregate is a built-in feature of the IF. This means you do not need to initialize it along with the plugins you are using in your pipeline. All you need to do is to add a small piece of config to your manifest file.

The aggregate config looks as follows:

aggregation:
  metrics:
    - "carbon"
    - "energy"
  type: "both"

There are two fields: metrics and type.

metrics is an array of metrics that you want to aggregate. You can provide any value here, but they must match a key that exists in your output data (i.e. if you tell IF to aggregate carbon but carbon is not in your outputs you will receive an error message and aggregation will fail). You can provide any number of metrics. In the example above, the aggregation feature will operate on the carbon and energy values.

type determines which kind of aggregation you want to perform. The choices are horizontal (time-series aggregation only), vertical (tree aggregation only) or both (both kinds of aggregation will be performed). In the example above, both types of aggregation will be performed over the two selected metrics.

Aggregation methods​

Aggregation can happen in several ways. In the example above, both of the selected metrics are absolute values measured in gCO2eq (carbon) and kWh (energy). To aggregate, it makes sense to add the values in each timestep for time-series aggregation and element-wise across components for tree aggregation. However, summing the values is not always the right way to aggregate.

For example, some values are constants that apply to every timestep - these should simply be copied into the aggregated metric and not summed. For example, if you are using a hard-coded value for grid carbon intensity that applies globally, then you simply want to persist that value into the aggregate - adding them together would provide a misleading result.

Similarly, some values are proportions or percentages. In these cases, the right way to aggregate is usually to take an average rather than summing over a set of values.

The decisions about how to aggregate are made on a case-by-case basis for each individual parameter. We track the aggregation method for each parameter in our params.ts file, which contains the canonical set of parameters, their units and aggregation methods. You can append to the parameters in this file by providing params data in your manifest file, or you can override our recommended set of parameters entirely by providing a new file to the --override-params command in the CLI. In either case, you need to provide an aggregation method with your parameters so that IF can look up the right way to aggregate the values.

Aggregation outputs​

The aggregation process adds new output data to your manifest file. The two types of aggregation add different outputs. The horizontal (time-series) aggregation adds a new field called aggregated to each node whose time series has been aggregated. In the aggregated block, you will find the aggregated value for each of the aggregation metrics defined in the aggregation config.

The vertical aggregation adds a new array of output observations. These are simply named outputs and they always contain a timestamp and duration along with the aggregated metrics for each timestep.

The example below shows the result of running both kinds of aggregation for a single component:

   "outputs": [
      {
        "carbon": 0.04846481793320214,
        "energy": 0.00030285447154471535,
        "timestamp": "2023-12-12T00:00:00.000Z"
      },
      {
        "carbon": 0.037777724630840566,
        "energy": 0.00023606013840495548,
        "timestamp": "2023-12-12T00:00:05.000Z"
      },
      {
        "carbon": 0.03630388278027921,
        "energy": 0.000226848626838947,
        "timestamp": "2023-12-12T00:00:10.000Z"
      },
      {
        "carbon": 0.0360935970659935,
        "energy": 0.0002255343411246613,
        "timestamp": "2023-12-12T00:00:15.000Z"
      },
      {
        "carbon": 0.0360935970659935,
        "energy": 0.0002255343411246613,
        "timestamp": "2023-12-12T00:00:20.000Z"
      },
      {
        "carbon": 0.0360935970659935,
        "energy": 0.0002255343411246613,
        "timestamp": "2023-12-12T00:00:25.000Z"
      },
      {
        "carbon": 0.0360935970659935,
        "energy": 0.0002255343411246613,
        "timestamp": "2023-12-12T00:00:30.000Z"
      },
      {
        "carbon": 0.0360935970659935,
        "energy": 0.0002255343411246613,
        "timestamp": "2023-12-12T00:00:35.000Z"
      },
    ],
    "aggregated": {
      "carbon": 0.3246705689138855,
      "energy": 0.0020287555470867216
    }
- + \ No newline at end of file diff --git a/major-concepts/design-philosophy/index.html b/major-concepts/design-philosophy/index.html index 8f6ad628..a9033b90 100644 --- a/major-concepts/design-philosophy/index.html +++ b/major-concepts/design-philosophy/index.html @@ -8,13 +8,13 @@ Design philosophy | Impact Framework - +
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

Design philosophy

Transparency​

The manifest file is the lifeblood of Impact Framework. It defines all the context for an environmental impact calculation, defining the architecture of an application, the observation period, the pipeline of calculations and transformations to execute, and the environmental impacts to track. This document can then be executed to generate impact values. This gives unparalleled transparency to an environmental audit, all in a standard, easy to read format that anyone can read or re-execute.

Verifiability​

An Impact Framework manifest file is powerful because anyone can re-execute it and verify an organization's impact calculation. You can even experiment by swapping out different plugins. The critical concept is that everything you need to calculate an impact is provided in the manifest file and anyone can re-run a calculation with the manifest file and the lightweight Impact Framework command line tool.

Flexibility

We aim to bake the minimum of constraints into the Imapct Framework, balancing the helpers and standards that make plugins interoperable and consistent against freedom of expression and creativity. The real power of Impact Framework comes from the community. This includes the community of experts contributing to the design decisions and standards baked into the protocol and the community of plugin developers experimenting at the margins, and the organizations using Impact Framework to measure, report and mitigate their environmental imapct.

Impact Framework can be a tool for transparent, verifiable environmental impact audits, but it can also be a platform for experimentation. Your manifest file is a foundation for forecasting into the future or exploring where you can tweak your stack to most effectively minimize your impact. Impact Framework can be a tool for research, hypothesis testing, R&D and business decision making as well as environemntal reporting. To realize this vision, we know we have to make Impact Framework as flexible as possible, imposing the absolute minimum of constraints in the underlying protocol, while also providing the necessary functionality and safeguards our users require.

Modularity​

Impact Framework is the minimal set of features that enable a manifest file to be processed according to some agreed principles. We provide a tool for processing manifest files and a set of standards and norms. This allows builders to create plugins that do some specific task, such as grabbing data from a particular cloud provider, or applying some calculation over some particular data.

Anyone can build a plugin and share them with the world, meaning Impact Framework development can be bottom-up and community driven. It also means that if you are not satisfied with how some calculation was done, you can easily fork it and replace it with your own.

What we provide is a minimal set of rules and guardrails for plugin builders to conform to to ensure compatibility with Impact Framework.

Neutrality​

Impact Framework aims to support maximally decentralized plugin development. We want anyone to be able to build plugins and use them to calculate their environmental impacts. We do not want to gatekeep what people can measure and monitor - we want to encourage people to build freely and experiment on Impact Framework rails!

At the same time, we want to provide the helpers and guardrails that make impact calculations as friction free as possible. This means we focus on providing the minimal protocol required to support community plugin development and make it as safe as possible from unit errors and other footguns.

We want to see the universe of Impact Framework plugins grow organically and permissionlessly in ways we can't even imagine today!

To this end, what we are really building is a protocol. Impact Framework is just a Typescript implementation of the protocol. The protocol itself is a set of fundamental principles that define how a manifest file should be processed, such that any implementation in any language will yield the same result from a given manifest file.

The Impact Protocol is the result of countless discussions, experiments, conversations with industry partners, academics, researchers and developers, and represents a community consensus for how certain actions should be executed, for example, how should a series of observations be aggregated, what standard units should be used, how should a manifest file be structured, etc.

This means we can be neutral about what can be built with IF while also providing a set of canonical processes and standards.

Where to go next​

This page has outlined the design philosophies that guide Impact Framework development.

Explore the other pages in this section to see how these principles have been applied to specific Impact Framework features, or head to our user documentation to get started running Impact Framework for yourself.

- + \ No newline at end of file diff --git a/major-concepts/exhaust-script/index.html b/major-concepts/exhaust-script/index.html index fab603e8..0ec02bab 100644 --- a/major-concepts/exhaust-script/index.html +++ b/major-concepts/exhaust-script/index.html @@ -8,13 +8,13 @@ Exhaust scripts | Impact Framework - +
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

Exhaust scripts

Exhaust scripts are scripts that can run independently of IF itself that take an executed manifest file (one with outputs) as an input, parse the yaml data and reformat it into some other representation. We provide if-csv bundled with IF, but if you want other data formats, you'll have to create an exhaust script yourself.

if-run​

if-run isn't really an exhaust script, because it also grabs input data, regroups data, computes the pipeline and aggregates. However, we're mentioning it here because it does have some built-in exhaust functionality. Specifically, if-run outputs yaml data. if-run can only output yaml data. This yaml data can be dumped to the console or saved to a yaml file.

How to Use if-run​

To use if-run, you need to provide a manifest file in yaml or yml foramt. The output will be in yaml format if you specify the output file path.

Here's a simple manifest file example. This manifest sums two components, cpu/energy and network/energy and assigns the result to energy in the outputs array.

name: sum
description: successful path
tags:
initialize:
  plugins:
    sum:
      method: Sum
      path: 'builtin'
      global-config:
        input-parameters: ['cpu/energy', 'network/energy']
        output-parameter: 'energy'
tree:
  children:
    child:
      pipeline:
        - sum
      config:
        sum:
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001

To execute this manifest with if-run, use the following command:

if-run -m sum.yaml -o output-sum

You will get the executed manifest in the output-sum.yaml file.

if-csv​

The if-csv script allows users to pass in yaml and yml files created using if-run and save the output in csv format. Yopu have to define the parameters you want to export from the yaml file, e.g. energy or carbon.

For the above example, you can get the following result:

Path,2023-08-06T00:00
tree.children.child.energy,0.002

by running:

if-csv -m sum.yaml -p energy -o output-sum

This command specifies the manifest file (sum.yaml), the parameter to export (energy), and the output file path (output-sum).

- + \ No newline at end of file diff --git a/major-concepts/groupby/index.html b/major-concepts/groupby/index.html index 20563977..61ae0589 100644 --- a/major-concepts/groupby/index.html +++ b/major-concepts/groupby/index.html @@ -8,14 +8,14 @@ Group-by | Impact Framework - +
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

Group-by

Groupby is an IF plugin that reorganizes a tree according to keys provided by the user. This allows users to regroup their observations according to various properties of their application. For example, the following manifest file contains a flat array of observations. This is how you might expect data to arrive from an importer plugin, maybe one that hits a metrics API for a cloud service.

name: if-demo
description: demo pipeline
graph:
  children:
    my-app:
      pipeline:     
        - group-by
        - teads-curve
      config:
        group-by:
          - cloud-region
          - instance-type
      inputs:
        - timestamp: 2023-07-06T00:00
          duration: 300 
          instance-type: A1 
          region: uk-west
          cpu-util: 99
        - timestamp: 2023-07-06T05:00 
          duration: 300 
          instance-type: A1 
          region: uk-west
          cpu-util: 23    
        - timestamp: 2023-07-06T10:00
          duration: 300
          instance-type: A1 
          region: uk-west
          cpu-util: 12
        - timestamp: 2023-07-06T00:00 # note this time restarts at the start timstamp
          duration: 300 
          instance-type: B1
          region: uk-west
          cpu-util: 11
        - timestamp: 2023-07-06T05:00 
          duration: 300 
          instance-type: B1
          region: uk-west
          cpu-util: 67
        - timestamp: 2023-07-06T10:00
          duration: 300 
          instance-type: B1
          region: uk-west
          cpu-util: 1

However, each observation contains an instance-type field that varies between observations. There are two instance types being represented in this array of observations. This means there are duplicate entries for the same timestamp in this array. This is the problem that group-by solves. You provide instance-type as a key to the group-by plugin and it extracts the data belonging to the different instances and separates them into independent arrays. The above example would be restructured so that instance types A1 and B1 have their own data, as follows:

graph:
  children:
    my-app:
      pipeline:
        # - group-by
        - teads-curve
      config:
        group-by:
          groups:
            - cloud-region
            - instance-type
      children:
        A1:
            inputs:
            - timestamp: 2023-07-06T00:00
                duration: 300
                instance-type: A1
                region: uk-west
                cpu-util: 99
            - timestamp: 2023-07-06T05:00
                duration: 300
                instance-type: A1
                region: uk-west
                cpu-util: 23
            - timestamp: 2023-07-06T10:00
                duration: 300
                instance-type: A1
                region: uk-west
                cpu-util: 12
        B1:
            inputs:
            - timestamp: 2023-07-06T00:00
                duration: 300
                instance-type: B1
                region: uk-east
                cpu-util: 11
            - timestamp: 2023-07-06T05:00
                duration: 300
                instance-type: B1
                region: uk-east
                cpu-util: 67
            - timestamp: 2023-07-06T10:00
                duration: 300
                instance-type: B1
                region: uk-east
                cpu-util: 1

Using group-by​

To use group-by, you have to initialize it as a plugin and invoke it in a pipeline.

The initialization looks as follows:

initialize:
plugins:
group-by: 
    path: 'builtin'
    method: GroupBy

You then have to provide config defining which keys to group by in each component. This is done at the component level (i.e. not global config). For example:

tree:
  children:
    my-app:
      pipeline:     
        - group-by
      config:
        group-by:
          group:
            - region
            - instance-type

In the example above, the plugin would regroup the input data for the specific component by region and by instance-type.

Assuming the values A1 and B1 are found for instance-type and the values uk-east and uk-west are found for region, the result of group-by would look similar to the following:

tree:
  children:
    my-app:
      pipeline:
        - group-by
      config:
        group-by:
          groups:
            - region
            - instance-type
      children:
        uk-west:
          children:
            A1:
              inputs:
                - timestamp: 2023-07-06T00:00
                  duration: 300
                  instance-type: A1
                  region: uk-west
                  cpu-util: 99
                - timestamp: 2023-07-06T05:00
                  duration: 300
                  instance-type: A1
                  region: uk-west
                  cpu-util: 23
                - timestamp: 2023-07-06T10:00
                  duration: 300
                  instance-type: A1
                  region: uk-west
                  cpu-util: 12
        uk-east:
          children:
            B1:
              inputs:
                - timestamp: 2023-07-06T00:00
                  duration: 300
                  instance-type: B1
                  region: uk-east
                  cpu-util: 11
                - timestamp: 2023-07-06T05:00
                  duration: 300
                  instance-type: B1
                  region: uk-east
                  cpu-util: 67
                - timestamp: 2023-07-06T10:00
                  duration: 300
                  instance-type: B1
                  region: uk-east
                  cpu-util: 1

This reorganized data can then be used to feed the rest of a computation pipeline.

- + \ No newline at end of file diff --git a/major-concepts/if/index.html b/major-concepts/if/index.html index ac4f6caf..5e5c0ce4 100644 --- a/major-concepts/if/index.html +++ b/major-concepts/if/index.html @@ -8,14 +8,14 @@ Impact Engine (CLI) | Impact Framework - +
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

Impact Engine (CLI)

Introduction​

if-run is a command line tool that computes Manifest files. It is the portal allowing users to interact with the Impact Framework.

The available options and their shortcuts are:

  • --manifest or -m: path to an input manifest file
  • --output or -o (optional): path to the output file where the results as saved
  • --no-output or -n (optional): suppress the output to console
  • --override-params or -p (optional): if you are an advanced user and you want to override our standard set of parameters and their definitions, you can provide the path to an alternative file as an argument to this command.
  • --help or -h: prints out help instruction
  • --debug: enables IF execution logs

The only required command is --manifest. Without a valid path to a manifest file, if-run has nothing to execute.

To use if-run, you must first write a manifest file. Then, you can simply pass the path to the manifest file to if-run on the command line.

if-run --manifest /my-manifest.yml
## or using aliases
if-run -m /my-manifest.yml

You can also pass a path where you would like to save the output file to. For example:

if-run --manifest ./my-manifest.yml --output ./my-results.yml
## or using aliases
if-run -m ./my-manifest.yml -o ./my-results.yml

If you omit the --output command, your results will only be displayed in the console.

For more information on the if-run commands see the CLI reference documentation.

- + \ No newline at end of file diff --git a/major-concepts/index.html b/major-concepts/index.html index a86372cb..4f395283 100644 --- a/major-concepts/index.html +++ b/major-concepts/index.html @@ -8,13 +8,13 @@ Major Concepts | Impact Framework - +
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.
- + \ No newline at end of file diff --git a/major-concepts/manifest-file/index.html b/major-concepts/manifest-file/index.html index 901993a7..5233cd55 100644 --- a/major-concepts/manifest-file/index.html +++ b/major-concepts/manifest-file/index.html @@ -8,14 +8,14 @@ Manifest File | Impact Framework - +
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

Manifest File

Manifest files are fundamental to Impact Framework and they serve multiple important purposes, including:

  • They contain all the necessary configurations for Impact Framework
  • They define your application architecture
  • They hold your input data
  • They are shareable, portable and human-readable
  • They can be used as verifiable audits form your application

The manifest is a yaml file with a particular structure. It can be thought of as an executable audit because the file itself can be shared with others and re-executed to verify your environmental impact calculations.

It is a formal report detailing not just the end impact but all the assumptions, inputs, and plugins used in calculating the impact.

This is possible because all the configuration and data required to run Impact Framework is contained in the manifest file.

Anyone can download Impact Framework and execute a manifest file to verify the results.

Structure of a manifest file​

Overview​

Manifest files can be simple or very intricate, depending on the plugin pipeline you want to use and the complexity of your application. However, all manifest files conform to a basic structure that looks as follows:

name:
description:
tags:
initialize:
  plugins:
    <PLUGIN-NAME-HERE>:
      method:
      path:
tree:
  children:
    child:
      pipeline:
      config:
      defaults:
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600

Everything above the tree is collectively referred to as the context. The tree contains the input data and is structured according to the architecture of the application being examined, with individual components being nodes in the tree. Individual components can be grouped under parent nodes.

Context​

Metadata​

The global metadata includes the name, description, and tags that can be used to describe the nature of the manifest file. For example, you might name the file Carbon Jan 2024 or similar. A short description might briefly outline the scope of the manifest file, e.g. company x's carbon emissions due to web serves from Jab 24 - July 24. Tags can be used to group manifest files (we do not explicitly use this field for anything currently).

Initialize​

The initialize section is where you define which plugins will be used in your manifest file and provide the global configuration for them. Below is sample for initialization:

initialize:
  plugins:
    <PLUGIN-NAME-HERE>:
      method: <METHOD-NAME-HERE>

Where required values are:

  • method: the name of the function exported by the plugin.
  • path: the path to the plugin code. For example, for a plugin from our standard library, this value would be builtin

There is also an optional global-config field that can be used to set global configuration that is common to a plugin wherever it is invoked across the entire manifest file.

Impact Framework uses the initialize section to instantiate each plugin. A plugin cannot be invoked elsewhere in the manifest file unless it is included in this section.

Execution (auto-generated)​

This section is auto generated by IF at runtime. You don't have to include this section in your manifest. The execution node contains all the necessary information to rebuild the environment, which can support debugging or verifying output files.

execution:
  status: success
  command: if-run --manifest examples/basic.yml
  environment:
    if-version: v0.3.2
    os: ubuntu
    os-version: 22.04.6
    node-version: v21.4.0
    date-time: 2023-12-12T00:00:00.000Z (UTC)
    dependencies:
      - '@babel/core@7.22.10'
      - ...
  error: 'InputValidationError:   "duration" parameter is required. Error code: invalid_type'. ## appears when execution failed
  • status: execution state: success (indicating that IF successfully executed this manifest) or fail (indicating that IF encountered a problem that halted execution).
  • command: exact command which was used to run the framework to execute this manifest (it may include full path to tools in place of aliases such as run)
  • environment: information about the environment the manifest was executed in, including the local operating system, Node.js version, time, and dependencies.
  • error: this field only appears if execution failed. The error message returned by IF is captured here.

Tree​

The tree section of a manifest file defines the topology of all the components being measured. The shape of the tree defines the grouping of components. It describes the architecture of the application being studied and contains all the usage observations for each component. The tree has individual components such as leaves, intermediate nodes representing groupings, and the top level is the root.

For example, a web application could be organized as follows:

tree:
  children:
    front-end:
      children:
        build-pipeline:
          children:
            vercel:
            github-pages:
    backend-database:
      children:
        server1:
        server2:
        server3:
    front-end:
    networking:

This example has a relatively straightforward structure with a maximum of 3 levels of nesting. You can continue to nest components to any depth.

Each component has some configuration, some input data, and a plugin pipeline.

  • pipeline: a list of plugins that should be executed for a specific component
  • config: contains configuration for each plugin that applies just inside this specific component.
  • defaults: fallback values that IF defaults to if they are not present in an input observation.
  • inputs: an array of observation data, with each observation containing usage data for a given timestep.

If a component does not include its own pipeline, config, or inputs values, they are inherited from the closest parent.

Here's an example of a moderately complex tree:

tree:
  children:
    child-0:
      pipeline:
        - sci-e
      children:
        child-0-1:
          pipeline:
            - sci-e
          config: null
          defaults: null
          inputs:
            - timestamp: 2023-07-06T00:00
              duration: 10
              cpu-util: 50
              energy-network: 0.000811
          outputs:
            - timestamp: 2023-07-06T00:00
              duration: 10
              cpu-util: 50
              energy-network: 0.000811
              energy: 0.000811
        child-0-2:
          children:
            child-0-2-1:
              pipeline:
                - sci-e
              config: null
              defaults: null
              inputs:
                - timestamp: 2023-07-06T00:00
                  duration: 10
                  cpu-util: 50
                  energy-network: 0.000811
              outputs:
                - timestamp: 2023-07-06T00:00
                  duration: 10
                  cpu-util: 50
                  energy-network: 0.000811
                  energy: 0.000811

Node config​

Node level configuration is intended to include any configuration data that is constant from timestep to timestep, but varies across components in the tree. Values that are constant both within and across components should go in global config instead.

The plugin must be written such that it expects these values to exist in node level config and explicitly reads them in. The values expected in node level config should be defined in the plugin README.

Defaults​

Defaults are fallback values that are only used if a given value is missing in the inputs array. For example, if you have a value that could feasibly be missing in a given timestep, perhaps because your plugin relies on a third party API that can fail, you can provide a value in defaults that can be used as a fallback value.

The values in defaults are applied to every timestep where the given value is missing. This means that as well as acting as a fallback defaults can be used as a convenience tool for efficiently adding a constant value to every timestep in your inputs array.

Inputs​

Every component includes an inputs field that gets read into plugins as an array. inputs are divided into observations, each having a timestamp and a duration. Every observation refers to an element in inputs representing some snapshot in time.

Each plugin takes the inputs array and applies some calculation or transformation to each observation in the array.

Observations can include any type of data, including human judgment, assumptions, other plugins, APIs, survey data or telemetry.

The separation of timestamps in the inputs array determines the temporal granularity of your impact calculations. The more frequent your observations, the more accurate your impact assessment.

Computing a manifest file​

Impact Framework computes manifest files. For each component in the tree, the inputs array is passed to each plugin in the pipeline in sequence.

Each plugin enriches the inputs array in some specific way, typically by adding a new key-value pair to each observation in the array. For example, the teads-curve plugin takes in CPU utilization expressed as a percentage as an input and appends cpu/energy expressed in kWh. cpu/energy is then available to be passed as an input to, for example, the sci-e plugin.

This implies a sequence of plugins where the inputs for some plugins must either be present in the original manifest file or be outputs of the preceding plugins in the pipeline.

There are also plugins and built-in features that can synchronize time series of observations across an entire tree and aggregate data across time or across components.

Outputs​

When Impact Framework computes a manifest file, it appends new data to the manifest file and the final result is an enriched manifest that includes all the configuration and contextual data, the input data, and the results of executing each plugin. This means the output file is completely auditable - the manifest file can be recovered simply by deleting the outputs section of the output file.

Here's an example output file:

name: sum
description: successful path
tags: null
initialize:
  plugins:
    sum:
      path: builtin
      method: Sum
      global-config:
        input-parameters:
          - cpu/energy
          - network/energy
        output-parameter: energy
execution:
  command: >-
    /home/user/.npm/_npx/1bf7c3c15bf47d04/node_modules/.bin/ts-node
    /home/user/Code/if/src/index.ts -m
    /home/user/Code/if/manifests/plugins/sum/success.yml
  environment:
    if-version: 0.3.3-beta.0
    os: linux
    os-version: 5.15.0-105-generic
    node-version: 21.4.0
    date-time: 2024-05-31T09:18:48.895Z (UTC)
    dependencies:
      - '@babel/core@7.22.10'
      - '@babel/preset-typescript@7.23.3'
      - '@commitlint/cli@18.6.0'
      - '@commitlint/config-conventional@18.6.0'
      - '@jest/globals@29.7.0'
      - '@types/jest@29.5.8'
      - '@types/js-yaml@4.0.9'
      - '@types/luxon@3.4.2'
      - '@types/node@20.9.0'
      - csv-stringify@6.4.6
      - fixpack@4.0.0
      - gts@5.2.0
      - husky@8.0.3
      - jest@29.7.0
      - js-yaml@4.1.0
      - lint-staged@15.2.2
      - luxon@3.4.4
      - release-it@16.3.0
      - rimraf@5.0.5
      - ts-command-line-args@2.5.1
      - ts-jest@29.1.1
      - typescript-cubic-spline@1.0.1
      - typescript@5.2.2
      - winston@3.11.0
      - zod@3.22.4
  status: success
tree:
  children:
    child:
      pipeline:
        - sum
      config:
        sum: null
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001
      outputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001
          energy: 0.002
- + \ No newline at end of file diff --git a/major-concepts/parameters/index.html b/major-concepts/parameters/index.html index 3264b43e..2fb7a751 100644 --- a/major-concepts/parameters/index.html +++ b/major-concepts/parameters/index.html @@ -8,13 +8,13 @@ Parameters | Impact Framework - +
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

Parameters

Parameters are handled carefully in IF because of the risk of errors propagating through plugin pipelines. We have a set of canonical parameters that are stored in a file called params.ts in the IF repository.

Each entry in params.ts has the following structure:

name: {
description: 
unit: 
aggregation:
}

The name is the accepted name for a particular parameter. For example, IF has a parameter, carbon that represents carbon in gCO2eq. Anywhere carbon exists as a parameter anywhere across the IF and its plugins, the name carbon is reserved for carbon with this specific unit.

The description is a short sentence describing the parameter.

The unit is the unit the parameter is expected to be expressed in. Prescribing this here is important because it allows everyone to know what units to expect from a plugin that returns this parameter, protecting against unit errors in pipelines.

The aggregation parameter can accept sum, avg or none. This is the method IF should use to aggregate values over time or across a tree. Some values should be summed to aggregate them over time, such as carbon or energy - it makes sense to add these up as they are absolute values that can be added up over time or across components. However, it does not make sense to aggregate metrics expressed as percentages or proportions - these cases are better aggregated by averaging their values. Some other values should not be manipulated when they are aggregated and instead they are treated as constants that should simply be copied into the aggregate, such as names.

Adding parameters​

The canonical set of parameters protects users against introducing unit errors into plugin pipelines by asserting the units and aggregation methods associated with a specific named parameter. However, this also means that if you create a new plugin that deals with a parameter not currently available in params.ts, IF with throw an error. To handle this, you can add your plugin's new parameters to a block of config in the manifest file, and they will be added to the canonical set of parameters at runtime. This effectively gives you read-only access to params.ts via the manifest file.

Here's an example of the config block that could be added to a manifest to enable a plugin that uses the dummy parameter:

params:
  - name: 
    description:
    unit:
    aggregaton:

Later, you could raise a pull request to IF to add your parameters to the canonical set in params.ts, where it would be discussed and agreed among the community. Eventually, we would like to move the governance of params.ts to a separate repository with independent custodians and a dedicated discussion forum.

Overriding parameters​

If you want to, you can also reject our canonical set of parameters and use your own instead. We do not recommend this, but we do make it possible via a CLI command, override-params. In this case, you can provide the path to a new file, structured the same way as params.ts that will be used instead of ours. For example:

if-run --manifest <manifest> --override-params my-params.ts

The file must parse as JSON or a Javascript object.

- + \ No newline at end of file diff --git a/major-concepts/pipelines/index.html b/major-concepts/pipelines/index.html index 69dc8343..48561e03 100644 --- a/major-concepts/pipelines/index.html +++ b/major-concepts/pipelines/index.html @@ -8,13 +8,13 @@ Pipelines | Impact Framework - +
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

Pipelines

Pipelines are chains of plugins that operate in sequence over the input data in your manifest file.

Your input data is fed through your pipeline of plugins, with each plugin adding a key-value pair to the inputs array or updating the value of an existing entry.

Each plugin does some specific operation. The idea is that individual plugins are simple - they do one specific thing only - but they act like Impact Legos, building up into complex logic operating on your manifest file.

Prebuilt pipelines​

We have designed our "standard library" of builtins to cover many generic operations such as file i/o, arithetic, queries etc so that in any cases you can build up complex pipelines without having to install any third party dependencies. One of the downsides of this is that logic that could be abstracted away into plugin code has to be implemented inside your manifest. We think this is great for transparency, auditability and reproducability, but it does come with a moderate learning curve. For this reason, we have provided prebuilt pipelines for several of our common operations, such as implementing the Teads curve for estimating cpu energy consumption from CPU utilization, looking up metadata for given cloud instance types anc calculating software carbon intensity (SCI) scores.

We recommend looking at the manifests in the manifests/examples folder that comes bundled with IF.

We also have a set of pipeline walkthroughs on this website, including:

  • Teads curve: calculate CPU energy from CPU utilization and the thermal design power of your processor
  • Cloud-instance metadata: lookup information about your cloud instance froma CSV file
  • SCI: calculate a software carbon intensiy score

These pipelines can be modified or chained together with other pipelines to make larger pipelines. Just as each plugin is a building block, pipelines themselves can be building blocks too.

- + \ No newline at end of file diff --git a/major-concepts/plugins/index.html b/major-concepts/plugins/index.html index 228d724c..2cb3941f 100644 --- a/major-concepts/plugins/index.html +++ b/major-concepts/plugins/index.html @@ -8,13 +8,13 @@ Plugins | Impact Framework - +
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

Plugins

Plugins are self-contained units of code that do one thing. They can be loaded into IF and chained together in a pipeline so that simple individual plugins can form a complicated procedure for computing a manifest file.

What are plugins?​

A plugin is a small, reuseable unit of code that can either observe some usage metric, grab some data from a file or API, or execute some transformation or calculation over some existing input data, for example, some plugins convert an input of CPU utilization into an output of energy.

The set of available plugins is growing; however, no single plugin can cover all impacts, scenarios, environments, contexts, and use cases. To calculate the end-to-end impact of a software application, you need to stitch together many different plugins. Plugins differ in fundamental ways in the inputs inputs they accept, their interface, their calculation methodology, their outputs, their granularity, and their coverage.

We expect the choice of which plugin to use for which software component to come down to an expert decision by a green software professional.

Why do we need to standardize the interface to plugins?​

Currently, suppose you want to consume a plugin in your measurement application. In that case, you must craft custom code to interact with a custom interface since every plugin has its unique interface. Swapping one plugin for another requires code changes, and comparing plugins or validating their accuracy/precision is challenging.

If every plugin exposes the same interface, then those plugins can easily be plugged into different applications, swapped in and out, upgraded, and compared.

Our thesis is simple: if we want a large, vibrant ecosystem of people and tooling around measurement, we need a standard, common interface for all plugins.

Ecosystems grow and thrive through standards.

You can explore more in the plugins reference docs or our plugin building tutorial.

Finding community plugins​

Anyone can build IF plugins. To make them discoverable, we host the IF Explorer website. There you can search for plugins that you need for your specific use-case.

You can also add your own plugins to the explorer. Simply fill in the submission form.

- + \ No newline at end of file diff --git a/major-concepts/time/index.html b/major-concepts/time/index.html index e78de139..23ce6459 100644 --- a/major-concepts/time/index.html +++ b/major-concepts/time/index.html @@ -8,13 +8,13 @@ Time | Impact Framework - +
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

Time

Every observation in an array of inputs represents a snapshot with a known start time and a known duration. For example, the following observation shows that the CPU utilization for a resource was 20% for the 10 second period starting at 1500 on the 22nd January 2024:

inputs:
    - timestamp: 2024-01-15T00:00:00.000Z
      duration: 10
      cpu-util: 20

The total time covered by an inputs array is determined by the timestamp of the first observation in the array and the timestamp and duration in the last observation in the array. Since every observation needs both a timestamp and a duration, an inputs array is always a time series.

Synchronizing time series'

The time series for each component is defined by its inputs array. However, a manifest file can contain many separate components, each with their own time series. There is no guarantee that an individual time series is continuous, or that all the components in a manifest file have the same start time, end time and resolution. This makes it difficult to aggregate, visualize or do like-for-like comparisons between components.

To solve this problem, we provide a built-in time-sync feature that synchronizes the time series' across all the components in a tree. The time-sync feature takes a global start time, end time and interval, then forces every individual time series to conform to this global configuration.

  • This works by first upsampling each time series to a common base resolution (typically 1s).
  • Any gaps in the time series are filled in with "zero objects", which have an identical structure to the real observations but with usage metrics set to zero (we assume that when there is no data, there is no usage).
  • Next, we check to see whether the first timestamp in each time series is before, after or identical to the global start time.
  • If a component's time series starts after the global start time, we pad the start of the time series with "zero objects" so that the start times are identical.
  • If the component's time series starts *before* the global start time, we trim the time series down, discarding observations from before the global start time. The same trimming logic is applied to the end times.
  • After synchronizing the start and end times and padding any discontinuities, we have a set of continuous time series' of identical length.
  • Next, we batch observations together into time bins whose size is define by the global interval value. This means that the resolution of the final time series' are identical and equal to interval.

This process yields synchronized time series for all components across a tree, enabling easy visualization and intercomparison. This synchronization is also a prerequisite for our aggregation function.

Toggling off time sync​

Some applications will not want to pad with zero values, and may be strict about continuous time series' being provided in the raw manifest file. In these cases, simply toggle the padding off in the manifest file.

- + \ No newline at end of file diff --git a/pipelines/index.html b/pipelines/index.html index 8aaa1d75..5ba5e278 100644 --- a/pipelines/index.html +++ b/pipelines/index.html @@ -8,13 +8,13 @@ Pipelines | Impact Framework - +
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

Pipelines

This section contains walkthrough guides for some common pipelines you may want to use in your manifest files.

These pipelines only use plugins from our standard library that comes bundled with IF and do not require any third party plugins to be installed. They all work out of the box with IF.

These walkthroughs will help you to understand how complex operations can be implemented using simple generic features and provide templates that you can use, modify or chain together in your own workflows.

The pipeline guides include:

You can find the manifests themselves in your copy of IF in manifests/examples or find them on our Github.

- + \ No newline at end of file diff --git a/pipelines/instance-metadata/index.html b/pipelines/instance-metadata/index.html index 560d49fc..d8efd6f5 100644 --- a/pipelines/instance-metadata/index.html +++ b/pipelines/instance-metadata/index.html @@ -8,13 +8,13 @@ Instance metadata pipeline | Impact Framework - +
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

Instance metadata pipeline

The instance metadata pipeline simply looks up a metadata for a given virtual machine instance name using the csv-lookup plugin from the IF standard library. However, the target dataset can return multiple processor names for a given VM instance where there are multiple possibilitiers. This means we need to create a pipeline that includes the regex plugin so parse out just one of the possible values.

For this demo we'll just extract the first value if there are m,ultiple available for the processor-name.

Start by creating a manifest and adding the following boilerplate code:

name: metadata-demo
description:
tags:
initialize:
  plugins:
tree:
  children:
    child:
      pipeline:
      inputs:

Step 1: grab metadata using csv-lookup​

There is a cloud instance metadata file in the if-data Github repository. You can use the csv-lookup plugin to grab data from that file. You do not need to have a local copy of the file, you can simply provide the URL of the remote file.

You can create an instance of CSVLookup and name it cloud-instance-metadata and add it to the initialize block in your manifest file.

The lookup query is configured in global-config. You provide the parameters you want to use as selectors, and the selector value is a field from your inputs array. You also provide the target columns you want to return data from (we'll use a wildcard and grab everything).

You want to retrieve all available data where instance-class is equal to Standard_A1_v2. So you need to make sure that Standard_A1_v2 is available in your inputs array - we'll put it there with the key cloud/instance-type.

Add the following data to your inputs array:

- timestamp: 2023-08-06T00:00
    duration: 3600
    cpu/energy: 0.001
    cloud/instance-type: Standard_A1_v2

Now, add the CSVLookup instance to your initialize block. Configure your query so that you select your row based on the value in the instance-class column. The value should be cloud/instance-type. You want data from all the other rows, so output can be a wildcard "*".

name: csv-demo
description:
tags:
initialize:
  plugins:
    cloud-instance-metadata:
      method: CSVLookup
      path: "builtin"
      global-config:
        filepath: https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/cloud-metdata-azure-instances.csv
        query:
          instance-class: "cloud/instance-type"
        output: "*"

The CSV lookup can return multiple values for the processor name, because the same instance can use different processors in different circumstances. Multiple values are returned as a single string, separated using commas. Therefore, you can easily parse out the first individual value by selecting the entire string up to the first comma. This is a simple regex task.

Create an instance of your regex plugin, and select all characters up to the first comma, by adding the following to your initialize block:

extract-processor-name:
    method: Regex
    path: "builtin"
    global-config:
    parameter: cpu-model-name
    match: /^([^,])+/g
    output: cpu/name

That's it!

Run the manifest​

Here's the complete manifest:

name: instance-metadata
description:
tags:
initialize:
  plugins:
    cloud-instance-metadata:
      method: CSVLookup
      path: "builtin"
      global-config:
        filepath: https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/cloud-metdata-azure-instances.csv
        query:
          instance-class: "cloud/instance-type"
        output: "*"
    extract-processor-name:
      method: Regex
      path: "builtin"
      global-config:
        parameter: cpu-model-name
        match: /^([^,])+/g
        output: cpu/name
tree:
  children:
    child:
      pipeline:
        - cloud-instance-metadata
        - extract-processor-name
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          cloud/provider: gcp
          cloud/region: asia-east
          cloud/instance-type: Standard_A1_v2

Now you can run this manifest using:

if-run -m instance-metadata.yml -o output.yml

Your new output.yml file will contain the following:

name: csv-demo
description: null
tags: null
initialize:
  plugins:
    cloud-instance-metadata:
      path: builtin
      method: CSVLookup
      global-config:
        filepath: >-
          https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/cloud-metdata-azure-instances.csv
        query:
          instance-class: cloud/instance-type
        output: '*'
    extract-processor-name:
      path: builtin
      method: Regex
      global-config:
        parameter: cpu-model-name
        match: /^([^,])+/g
        output: cpu/name
execution:
  command: >-
    /home/user/.npm/_npx/1bf7c3c15bf47d04/node_modules/.bin/ts-node
    /home/user/Code/if/src/index.ts -m manifests/examples/instance-metadata.yml
  environment:
    if-version: 0.3.3-beta.0
    os: linux
    os-version: 5.15.0-107-generic
    node-version: 21.4.0
    date-time: 2024-06-06T15:21:50.108Z (UTC)
    dependencies:
      - '@babel/core@7.22.10'
      - '@babel/preset-typescript@7.23.3'
      - '@commitlint/cli@18.6.0'
      - '@commitlint/config-conventional@18.6.0'
      - '@grnsft/if-unofficial-plugins@v0.3.1'
      - '@jest/globals@29.7.0'
      - '@types/jest@29.5.8'
      - '@types/js-yaml@4.0.9'
      - '@types/luxon@3.4.2'
      - '@types/node@20.9.0'
      - axios-mock-adapter@1.22.0
      - axios@1.7.2
      - csv-parse@5.5.6
      - csv-stringify@6.4.6
      - fixpack@4.0.0
      - gts@5.2.0
      - husky@8.0.3
      - jest@29.7.0
      - js-yaml@4.1.0
      - lint-staged@15.2.2
      - luxon@3.4.4
      - release-it@16.3.0
      - rimraf@5.0.5
      - ts-command-line-args@2.5.1
      - ts-jest@29.1.1
      - typescript-cubic-spline@1.0.1
      - typescript@5.2.2
      - winston@3.11.0
      - zod@3.22.4
  status: success
tree:
  children:
    child:
      pipeline:
        - cloud-instance-metadata
        - extract-processor-name
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          cloud/provider: gcp
          cloud/region: asia-east
          cloud/instance-type: Standard_A1_v2
      outputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          cloud/provider: gcp
          cloud/region: asia-east
          cloud/instance-type: Standard_A1_v2
          cpu-cores-available: 52
          cpu-cores-utilized: 1
          cpu-manufacturer: Intel
          cpu-model-name: >-
            Intel® Xeon® Platinum 8272CL,Intel® Xeon® 8171M 2.1 GHz,Intel® Xeon®
            E5-2673 v4 2.3 GHz,Intel® Xeon® E5-2673 v3 2.4 GHz
          cpu-tdp: 205
          gpu-count: nan
          gpu-model-name: nan
          gpu-tdp: nan
          memory-available: 2
          cpu/name: Intel® Xeon® Platinum 8272CL
- + \ No newline at end of file diff --git a/pipelines/sci/index.html b/pipelines/sci/index.html index 3bb81908..006be4a8 100644 --- a/pipelines/sci/index.html +++ b/pipelines/sci/index.html @@ -8,13 +8,13 @@ Software Carbon Intensity (SCI) | Impact Framework - +
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

Software Carbon Intensity (SCI)

The software carbon intensity (SCI) score is perhaps the most important value that can be generated using Impact Framework.

SCI is an ISO-recognized standard for reporting the carbon costs of running software. This tutorial demonstrates how to organize a pipeline of Impact framework plugins to calculate SCI scores from some simple observations of the CPU utilization of a software application running in the cloud.

Prerequisites​

This tutorial builds on top of the Teads curve pipeline tutorial. That tutorial demonstrates how to organize a pipeline that converts CPOU utilization observations into CPU energy. This tutorial uses the same pipeline but goes several steps further, including converting the CPU energy estimates into carbon, adding the embodied carbon associated with the hardware being used and calculating the SCI score.

Step 1: Pasting in the Teads pipeline​

You can start by copying the manifest you created in the Teads curve tutorial into a new file called sci.yml. You can use that manifest as a template and build off of it here. Here's a reminder of what the file should look like:

name: carbon-intensity plugin demo
description: 
tags:
initialize:
  plugins:
    interpolate:
      method: Interpolation
      path: 'builtin'
      global-config:
        method: linear
        x: [0, 10, 50, 100]
        y: [0.12, 0.32, 0.75, 1.02]
        input-parameter: 'cpu/utilization'
        output-parameter: 'cpu-factor'
    cpu-factor-to-wattage:
      method: Multiply
      path: builtin
      global-config:
        input-parameters: ["cpu-factor", "cpu/thermal-design-power"]
        output-parameter: "cpu-wattage"
    wattage-times-duration:
      method: Multiply
      path: builtin
      global-config:
        input-parameters: ["cpu-wattage", "duration"]
        output-parameter: "cpu-wattage-times-duration"
    wattage-to-energy-kwh:
      method: Divide
      path: "builtin"
      global-config:
        numerator: cpu-wattage-times-duration
        denominator: 3600000
        output: cpu-energy-raw
    calculate-vcpu-ratio:
      method: Divide
      path: "builtin"
      global-config:
        numerator: vcpus-total
        denominator: vcpus-allocated
        output: vcpu-ratio
    correct-cpu-energy-for-vcpu-ratio:
      method: Divide
      path: "builtin"
      global-config:
        numerator: cpu-energy-raw
        denominator: vcpu-ratio
        output: cpu-energy-kwh
tree:
  children:
    child:
      pipeline:
        - interpolate
        - cpu-factor-to-wattage
        - wattage-times-duration
        - wattage-to-energy-kwh
        - calculate-vcpu-ratio
        - correct-cpu-energy-for-vcpu-ratio
      defaults:
        cpu/thermal-design-power: 100
        vcpus-total: 8
        vcpus-allocated: 2
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 360
          cpu/utilization: 1
          carbon: 30
        - timestamp: 2023-09-06T00:00
          duration: 360
          carbon: 30
          cpu/utilization: 10
        - timestamp: 2023-10-06T00:00
          duration: 360
          carbon: 30
          cpu/utilization: 50
        - timestamp: 2023-10-06T00:00
          duration: 360
          carbon: 30
          cpu/utilization: 100

Step 2: Adding a network/energy component​

Your Teads curve manifest only accounted for CPU energy, but this SCI manifest will also consider the energy consumed during data ingress and egress, collectively known as "network energy". This can be calculated from data ingress and egress observations, but in this example, we will shortcut it by adding the network/energy value, measured in kWh, directly to the input data. You can do this by adding network/energy and a value to each element in the inputs array. This is just an example, so you can create dummy values. In a real example, these data would come from observations of a real system.

The SCI score will take into account all the energy used by the application, which in this case includes CPU energy and network energy. Therefore, you need to add a plugin that sums these components of energy together and adds the result to the inputs array. The Sum plugin exists for precisely this purpose. There are going to be multiple instances of the Sum plugin in your SCI pipeline, so you should choose a descriptive name for it so that you can easily invoke it in the right position in your pipeline.

Add the following to your initialize: plugins: block:

    sum-energy-components:
      path: "builtin"
      method: Sum
      global-config:
        input-parameters:
          - cpu/energy
          - network/energy
        output-parameter: energy

This will create an instance of Sum called sum-energy-components, and it will sum cpu/energy and network/energy and append the result to inputs as energy.

Step 3: Account for embodied carbon​

Embodied carbon is the carbon emitted during the production and disposal of the hardware used to run an application. The total embodied carbon for a unit of hardware is scaled down by the proportion of its expected lifespan used up by an application. This is all handled by another IF builtin called SciEmbodied. The result is embodied-carbon in units of gCO2eq. You can simply create an instance of it and add it to your pipeline. It requires no global configuration.

embodied-carbon:
  path: "builtin"
  method: SciEmbodied

embodied-carbon does expect some specific values to be available in the inputs array. These include:

device/emissions-embodied: # the embodied emissions for the entire component
time-reserved: # time the component is used by the application
device/expected-lifespan: # lifespan of the component in seconds
resources-reserved: # proportion of the total component being allocated
resources-total: # size of the component

Most of these values can be found in manufacturer documentation for specific processors and other hardware. In the present case, you can again provide some default values for a hypothetical system. You can assume the resource is a processor being used in a cloud virtual machine. In this case, the resources-total can be the total number of VCPUs for the processor and the resources-allocated can be the number of VCPUs actually being used by your application. Remembering back to the Teads curve example, you already have that information available to you in the form of the vcpus-total and vcpus-allocated fields, which you can pass by name as values to resources-total and resources-reserved`.

Add the following to your defaults section:

device/emissions-embodied: 1533.120 # gCO2eq
time-reserved: 3600 # 1hr in seconds
device/expected-lifespan: 94608000 # 3 years in seconds
resources-reserved: vcpus-allocated
resources-total: vcpus-total

Step 4: Calculate operational carbon​

So far, you have calculated the embodied carbon for your application, but your usage values are still expressed as units of energy. To calculate the carbon emissions that result from that energy consumption, you need to multiply your total energy by the carbon intensity of the electricity you consume. This value is known as the operational-carbon. In a real example, the grid carbon intensity could be a time-varying value that also depends on your physical location. However, here you will hardcode it for simplicity.

grid/carbon-intensity

Now create an instance of Multiply that will calculate the product of energy and grid/carbon-intensity:

    "operational-carbon":
      method: Multiply
      path: builtin
      global-config:
        input-parameters: ["energy", "grid/carbon-intensity"]
        output-parameter: "carbon-operational"

Step 5: Sum carbon components​

At this stage you have two separate sources of carbon emissions treated separately in your inputs: embodied-carbon anf operational-carbon. To account for the total carbon emissions for your application, you need to add these two sources together.

Add the following instance of the Sum plugin to your initialize: plugins: block. It will sum carbon-operational and carbon-embodied and append the result to inputs as carbon.

sum-carbon:
    path: "builtin"
    method: Sum
    global-config:
    input-parameters:
        - carbon-operational
        - carbon-embodied
    output-parameter: carbon

Step 6: Calculate SCI​

Now you have calculated the total carbon emissions due to your application, you can move to the final step which is calculating your SCI score. This is simply the total carbon expressed in terms of some functional unit. Functional units can be measured values such as requests, visits, users or whatever else you want to express your carbon emissions in terms of. In this case, you will simply express your SCI in terms of the entire application, in which case you can just set the functional unit to 1.

Add an instance of the SCI plugin to your initialize: plugins: block as follows:

"sci":
    path: "builtin"
    method: Sci
    global-config:
    functional-unit: "component"

SCI will look in each element in the inputs array for the component key. To ensure it is there, we can add it to defaults as follows:

component: 1

Note that in a real system, you probably don't want to use defaults to define your functional unit unless you are sure it is constant over time. More likely, you'll have observations of some system metric in eachg timestep to use as a functional unit.

Step 7: Create the pipeline​

Now you have initialized all the plugins you will need to compute the SCI score, add them in sequence to your execution pipeline, as follows:

pipeline:
  - interpolate
  - cpu-factor-to-wattage
  - wattage-times-duration
  - wattage-to-energy-kwh
  - calculate-vcpu-ratio
  - correct-cpu-energy-for-vcpu-ratio
  - sum-energy-components
  - embodied-carbon
  - operational-carbon
  - sum-carbon
  - sci

Congratulations, now you have completed your manifest and can calculate your SCI score!

Step 8: Run your manifest​

Assuming your manifest is saved as sci/yml you can run it using the following command:

if-run -m sci.yml

You will see the following data displayed in the console:

name: sci
description: >-
  a full pipeline seeded with some hardcoded input data and yielding an SCI
  score
tags: null
initialize:
  plugins:
    interpolate:
      path: builtin
      method: Interpolation
      global-config:
        method: linear
        x:
          - 0
          - 10
          - 50
          - 100
        'y':
          - 0.12
          - 0.32
          - 0.75
          - 1.02
        input-parameter: cpu/utilization
        output-parameter: cpu-factor
    cpu-factor-to-wattage:
      path: builtin
      method: Multiply
      global-config:
        input-parameters:
          - cpu-factor
          - cpu/thermal-design-power
        output-parameter: cpu-wattage
    wattage-times-duration:
      path: builtin
      method: Multiply
      global-config:
        input-parameters:
          - cpu-wattage
          - duration
        output-parameter: cpu-wattage-times-duration
    wattage-to-energy-kwh:
      path: builtin
      method: Divide
      global-config:
        numerator: cpu-wattage-times-duration
        denominator: 3600000
        output: cpu-energy-raw
    calculate-vcpu-ratio:
      path: builtin
      method: Divide
      global-config:
        numerator: vcpus-total
        denominator: vcpus-allocated
        output: vcpu-ratio
    correct-cpu-energy-for-vcpu-ratio:
      path: builtin
      method: Divide
      global-config:
        numerator: cpu-energy-raw
        denominator: vcpu-ratio
        output: cpu/energy
    sum-energy-components:
      path: builtin
      method: Sum
      global-config:
        input-parameters:
          - cpu/energy
          - network/energy
        output-parameter: energy
    embodied-carbon:
      path: builtin
      method: SciEmbodied
    operational-carbon:
      path: builtin
      method: Multiply
      global-config:
        input-parameters:
          - energy
          - grid/carbon-intensity
        output-parameter: carbon-operational
    sum-carbon:
      path: builtin
      method: Sum
      global-config:
        input-parameters:
          - carbon-operational
          - carbon-embodied
        output-parameter: carbon
    sci:
      path: builtin
      method: Sci
      global-config:
        functional-unit: component
    time-sync:
      path: builtin
      method: TimeSync
      global-config:
        start-time: '2023-12-12T00:00:00.000Z'
        end-time: '2023-12-12T00:01:00.000Z'
        interval: 5
        allow-padding: true
execution:
  command: >-
    /home/user/.npm/_npx/1bf7c3c15bf47d04/node_modules/.bin/ts-node
    /home/user/Code/if/src/index.ts -m manifests/examples/sci.yml -o
    manifests/outputs/test
  environment:
    if-version: 0.3.3-beta.0
    os: linux
    os-version: 5.15.0-107-generic
    node-version: 21.4.0
    date-time: 2024-06-10T09:14:42.383Z (UTC)
    dependencies:
      - '@babel/core@7.22.10'
      - '@babel/preset-typescript@7.23.3'
      - '@commitlint/cli@18.6.0'
      - '@commitlint/config-conventional@18.6.0'
      - '@jest/globals@29.7.0'
      - '@types/jest@29.5.8'
      - '@types/js-yaml@4.0.9'
      - '@types/luxon@3.4.2'
      - '@types/node@20.9.0'
      - axios-mock-adapter@1.22.0
      - axios@1.7.2
      - csv-parse@5.5.6
      - csv-stringify@6.4.6
      - fixpack@4.0.0
      - gts@5.2.0
      - husky@8.0.3
      - jest@29.7.0
      - js-yaml@4.1.0
      - lint-staged@15.2.2
      - luxon@3.4.4
      - release-it@16.3.0
      - rimraf@5.0.5
      - ts-command-line-args@2.5.1
      - ts-jest@29.1.1
      - typescript-cubic-spline@1.0.1
      - typescript@5.2.2
      - winston@3.11.0
      - zod@3.22.4
  status: success
tree:
  children:
    child-1:
      pipeline:
        - interpolate
        - cpu-factor-to-wattage
        - wattage-times-duration
        - wattage-to-energy-kwh
        - calculate-vcpu-ratio
        - correct-cpu-energy-for-vcpu-ratio
        - sum-energy-components
        - embodied-carbon
        - operational-carbon
        - sum-carbon
        - sci
      config: null
      defaults:
        cpu/thermal-design-power: 100
        vcpus-total: 8
        vcpus-allocated: 2
        grid/carbon-intensity: 800
        device/emissions-embodied: 1533.12
        time-reserved: 3600
        device/expected-lifespan: 94608000
        resources-reserved: vcpus-allocated
        resources-total: vcpus-total
        component: 1
      inputs:
        - timestamp: '2023-12-12T00:00:00.000Z'
          cloud/instance-type: A1
          cloud/region: uk-west
          duration: 1
          cpu/utilization: 50
          network/energy: 0.000001
        - timestamp: '2023-12-12T00:00:01.000Z'
          duration: 5
          cpu/utilization: 20
          cloud/instance-type: A1
          cloud/region: uk-west
          network/energy: 0.000001
        - timestamp: '2023-12-12T00:00:06.000Z'
          duration: 7
          cpu/utilization: 15
          cloud/instance-type: A1
          cloud/region: uk-west
          network/energy: 0.000001
        - timestamp: '2023-12-12T00:00:13.000Z'
          duration: 30
          cloud/instance-type: A1
          cloud/region: uk-west
          cpu/utilization: 15
          network/energy: 0.000001
      outputs:
        - timestamp: '2023-12-12T00:00:00.000Z'
          cloud/instance-type: A1
          cloud/region: uk-west
          duration: 1
          cpu/utilization: 50
          network/energy: 0.000001
          cpu/thermal-design-power: 100
          vcpus-total: 8
          vcpus-allocated: 2
          grid/carbon-intensity: 800
          device/emissions-embodied: 1533.12
          time-reserved: 3600
          device/expected-lifespan: 94608000
          resources-reserved: vcpus-allocated
          resources-total: vcpus-total
          component: 1
          cpu-factor: 0.75
          cpu-wattage: 75
          cpu-wattage-times-duration: 75
          cpu-energy-raw: 0.000020833333333333333
          vcpu-ratio: 4
          cpu/energy: 0.000005208333333333333
          energy: 0.000006208333333333333
          carbon-embodied: 0.000004051243023845763
          carbon-operational: 0.004966666666666666
          carbon: 0.004970717909690512
          sci: 0.004970717909690512
        - timestamp: '2023-12-12T00:00:01.000Z'
          duration: 5
          cpu/utilization: 20
          cloud/instance-type: A1
          cloud/region: uk-west
          network/energy: 0.000001
          cpu/thermal-design-power: 100
          vcpus-total: 8
          vcpus-allocated: 2
          grid/carbon-intensity: 800
          device/emissions-embodied: 1533.12
          time-reserved: 3600
          device/expected-lifespan: 94608000
          resources-reserved: vcpus-allocated
          resources-total: vcpus-total
          component: 1
          cpu-factor: 0.4275
          cpu-wattage: 42.75
          cpu-wattage-times-duration: 213.75
          cpu-energy-raw: 0.000059375
          vcpu-ratio: 4
          cpu/energy: 0.00001484375
          energy: 0.00001584375
          carbon-embodied: 0.000020256215119228814
          carbon-operational: 0.012674999999999999
          carbon: 0.012695256215119228
          sci: 0.012695256215119228
        - timestamp: '2023-12-12T00:00:06.000Z'
          duration: 7
          cpu/utilization: 15
          cloud/instance-type: A1
          cloud/region: uk-west
          network/energy: 0.000001
          cpu/thermal-design-power: 100
          vcpus-total: 8
          vcpus-allocated: 2
          grid/carbon-intensity: 800
          device/emissions-embodied: 1533.12
          time-reserved: 3600
          device/expected-lifespan: 94608000
          resources-reserved: vcpus-allocated
          resources-total: vcpus-total
          component: 1
          cpu-factor: 0.37375
          cpu-wattage: 37.375
          cpu-wattage-times-duration: 261.625
          cpu-energy-raw: 0.00007267361111111111
          vcpu-ratio: 4
          cpu/energy: 0.000018168402777777778
          energy: 0.000019168402777777778
          carbon-embodied: 0.00002835870116692034
          carbon-operational: 0.015334722222222222
          carbon: 0.015363080923389142
          sci: 0.015363080923389142
        - timestamp: '2023-12-12T00:00:13.000Z'
          duration: 30
          cloud/instance-type: A1
          cloud/region: uk-west
          cpu/utilization: 15
          network/energy: 0.000001
          cpu/thermal-design-power: 100
          vcpus-total: 8
          vcpus-allocated: 2
          grid/carbon-intensity: 800
          device/emissions-embodied: 1533.12
          time-reserved: 3600
          device/expected-lifespan: 94608000
          resources-reserved: vcpus-allocated
          resources-total: vcpus-total
          component: 1
          cpu-factor: 0.37375
          cpu-wattage: 37.375
          cpu-wattage-times-duration: 1121.25
          cpu-energy-raw: 0.00031145833333333335
          vcpu-ratio: 4
          cpu/energy: 0.00007786458333333334
          energy: 0.00007886458333333333
          carbon-embodied: 0.0001215372907153729
          carbon-operational: 0.06309166666666667
          carbon: 0.06321320395738204
          sci: 0.06321320395738204

What next?​

Now you have a basic SCI pipeline, you can use it as a base for more advanced IF runs. Try experimenting with:

  • aggregating the individual SCI scores over time using our aggregate feature
  • using mock observations to generate dummy input data
  • using a third-party plugin to grab real grid carbon intensity data
  • grabbing real metadata for your processor using csv-lookup
- + \ No newline at end of file diff --git a/pipelines/teads/index.html b/pipelines/teads/index.html index a9b9137a..c146d4eb 100644 --- a/pipelines/teads/index.html +++ b/pipelines/teads/index.html @@ -8,7 +8,7 @@ Teads CPU pipeline | Impact Framework - + @@ -20,7 +20,7 @@ 4) Determine the power drawn by your CPU by multiplying your scaling factor by the CPU's thermal design power 5) Perform a unit conversion to convert power in Watts to energy in kwH 6) Scale the energy estimated for the entire chip to the portion of the chip that is actually in use.

These steps can be executed in IF using just three plugins:

  • Interpolate
  • Multiply
  • Divide

We'll go through each step in the energy estimate and examine how to implement it in a manifest file using IF's standard library of builtins.

Impact Framework implementation​

First, create a manifest file and add this following boilerplate code:

name: carbon-intensity plugin demo
description: 
tags:
initialize:
  plugins:
tree:
  children:
    child:
      pipeline:
      defaults:
      inputs:

If this structure looks unfamiliar to you, you can go back to our manifests page.

Step 1: measure CPU utilization​

The first step was to measure your CPU utilization. In real use cases you would typoically do this using an importer plugin that grabs data from a monitor API or similar. However, for this example we will just manually create some dummy data. Add some timestamps, durations and cpu/utilization data to your inputs array, as follows:

name: teads demo
description: 
tags:
initialize:
  plugins:
tree:
  children:
    child:
      pipeline:
      defaults:
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 360
          cpu/utilization: 1
          carbon: 30
        - timestamp: 2023-09-06T00:00
          duration: 360
          carbon: 30
          cpu/utilization: 10
        - timestamp: 2023-10-06T00:00
          duration: 360
          carbon: 30
          cpu/utilization: 50
        - timestamp: 2023-10-06T00:00
          duration: 360
          carbon: 30
          cpu/utilization: 100

Step 2: Determine the thermal design power of your processor​

Typically determinign the TDP of your processor would be done using a CSV lookup. We have a pipeline example for tdp-finder in these docs - combining this pipeline with the tdp-finder pipeline would eb a great follow on exercise after you have finished this tutorial. Foir now, we will just hartd code some TDP data into your manifest so we can focus on the CPU utilization to energy calculations. Add thermal-design-power to defaults - this is a shortcut to providing it in every timestep in your inputs array.

default:
  thermal-design-power: 100

Step 3: Interpolate the Teads curve​

The Teads curve has CPU utilization ont he x axis and a scaling factor on the y axis. There are only four points on the published curve. Your task is to get the scaling factor for your specific CPU utilization values by interpolating between the known points. Luckily, we have a builtin for that purpose!

Add the Interpolation plugin to your list of plugins in the initialize block. 

initialize:
  plugins:
    interpolate:
      method Interpolation
      path: builtin

The details about the interpolation you want to do and the values to return are configured in the global-config whoch is also added int he initialize block. Specifically, you have to provide the known points of the curve you want to interpolate, the input-parameter (which is the x value whose correspondiong y value you want to find out, i.e. your CPU utilization value) and the output-parameter (the name you want to give to your retrieved y value).

You want to interpolate the Teads curve, so you can provide the x and y values obtained from the articles linked in the introduction section above:

x: [0, 10, 50, 100]
y: [0.12, 0.32, 0.75, 1.02]

Your input-parameter is your cpu/utilization and we'll name give the output-parameter the name cpu-factor.

Your compelted initialize block for interpolate should look as follows:

    interpolate:
      method: Interpolation
      path: 'builtin'
      global-config:
        method: linear
        x: [0, 10, 50, 100]
        y: [0.12, 0.32, 0.75, 1.02]
        input-parameter: 'cpu/utilization'
        output-parameter: 'cpu-factor'

Step 4: Convert CPU factor to power​

The interpoaltion only gave use the scaling factor; we need to apply that scaling factor to the processor's TDP to get the power drawn by the CPU at your specific CPU utilization.

To do this, we can use the Multiply plugin in the IF standard library. We'll give the instance of Multiply the name cpu-factor-to-wattage and int he global-config we'll define cpu-factor and thermal-design-power as the two elements in our inputs array that we want to multiply together. Then we'll name the result cpu-wattage:

cpu-factor-to-wattage:
    method: Multiply
    path: builtin
    global-config:
    input-parameters: ["cpu-factor", "thermal-design-power"]
    output-parameter: "cpu-wattage"

Add this to your initialize block.

Step 5: Convert wattage to energy​

Next we have to perform some unit conversions. Wattage is a measure of power (energy over time). To convert to energy, we can first multiply by the number of seconds our observation covers (duration) to yield energy in joules. Then, convert to kWh by applying a scaling factor that takes seconds to hours and watts to kilowatts.

You can do this in two steps: the first uses another instance of Multiply an the second uses Divide:

To do the initial multiplication of the CPU wattage and the observation duration, add the following config to your initialize block:

wattage-times-duration:
    method: Multiply
    path: builtin
    global-config:
    input-parameters: ["cpu-wattage", "duration"]
    output-parameter: "cpu-wattage-times-duration"

next, use the Divide plugin to do the unit conversion:

wattage-to-energy-kwh:
    method: Divide
    path: "builtin"
    global-config:
    numerator: cpu-wattage-times-duration
    denominator: 3600000
    output: cpu-energy-raw

Step 6: Scale the energy by the allocated CPUs​

The cpu-energy-raw value you just configured is for the entire chip. But your application probably doesn't use the entire chip. Chances are you have some number of VCPUs allocated to you that is less than the total available. So you can scale your energy estimate by the ratio of VCPUs allocated to VCPUS available.

Let's assume you know the number of VCPUs allocated and available in advance and that they are the same in every timestep. In this case, you can just add the values to defaults so they become available in every timestep, just as you did with thermal-design-power.

defaults:
  thermal-design-power: 100
  vcpus-total: 8
  vcpus-allocated: 2

You need one instance of Divide to calculate the vcpu-ratio and another to apply that vcpu-ratio to your cpu-energy-raw value and yield your final result: cpu-energy-kwh. Add the following to your initialize block to achieve those steps:

calculate-vcpu-ratio:
    method: Divide
    path: "builtin"
    global-config:
    numerator: vcpus-total
    denominator: vcpus-allocated
    output: vcpu-ratio
correct-cpu-energy-for-vcpu-ratio:
    method: Divide
    path: "builtin"
    global-config:
    numerator: cpu-energy-raw
    denominator: vcpu-ratio
    output: cpu-energy-kwh

Step 7: Define your pipeline​

Now you have configured all your plugins, covering all the stages of the calculation, you can simple define them in order in the pipeline section of your manifest, as follows:

tree:
  children:
    child:
      pipeline:
        - interpolate
        - cpu-factor-to-wattage
        - wattage-times-duration
        - wattage-to-energy-kwh
        - calculate-vcpu-ratio
        - correct-cpu-energy-for-vcpu-ratio

You also need to add some input data that your pipeline can operate over.

You can see the full manifest in the IF repository.

That's it! Your manifest is ready to run!

Running the manifest​

Having saved your manifest as teads-curve.yaml you can run it using IF:

if-run -m teads-curve.yml -o teads-output.yml

This will yield the following output file:

name: teads curve demo
description: null
tags: null
initialize:
  plugins:
    interpolate:
      path: builtin
      method: Interpolation
      global-config:
        method: linear
        x:
          - 0
          - 10
          - 50
          - 100
        'y':
          - 0.12
          - 0.32
          - 0.75
          - 1.02
        input-parameter: cpu/utilization
        output-parameter: cpu-factor
    cpu-factor-to-wattage:
      path: builtin
      method: Multiply
      global-config:
        input-parameters:
          - cpu-factor
          - thermal-design-power
        output-parameter: cpu-wattage
    wattage-times-duration:
      path: builtin
      method: Multiply
      global-config:
        input-parameters:
          - cpu-wattage
          - duration
        output-parameter: cpu-wattage-times-duration
    wattage-to-energy-kwh:
      path: builtin
      method: Divide
      global-config:
        numerator: cpu-wattage-times-duration
        denominator: 3600000
        output: cpu-energy-raw
    calculate-vcpu-ratio:
      path: builtin
      method: Divide
      global-config:
        numerator: vcpus-total
        denominator: vcpus-allocated
        output: vcpu-ratio
    correct-cpu-energy-for-vcpu-ratio:
      path: builtin
      method: Divide
      global-config:
        numerator: cpu-energy-raw
        denominator: vcpu-ratio
        output: cpu-energy-kwh
execution:
  command: >-
    /home/user/.npm/_npx/1bf7c3c15bf47d04/node_modules/.bin/ts-node
    /home/user/if/src/index.ts -m manifests/examples/teads-curve.yml
  environment:
    if-version: 0.3.3-beta.0
    os: linux
    os-version: 5.15.0-107-generic
    node-version: 21.4.0
    date-time: 2024-06-06T14:33:25.188Z (UTC)
    dependencies:
      - '@babel/core@7.22.10'
      - '@babel/preset-typescript@7.23.3'
      - '@commitlint/cli@18.6.0'
      - '@commitlint/config-conventional@18.6.0'
      - '@grnsft/if-unofficial-plugins@v0.3.1'
      - '@jest/globals@29.7.0'
      - '@types/jest@29.5.8'
      - '@types/js-yaml@4.0.9'
      - '@types/luxon@3.4.2'
      - '@types/node@20.9.0'
      - axios-mock-adapter@1.22.0
      - axios@1.7.2
      - csv-parse@5.5.6
      - csv-stringify@6.4.6
      - fixpack@4.0.0
      - gts@5.2.0
      - husky@8.0.3
      - jest@29.7.0
      - js-yaml@4.1.0
      - lint-staged@15.2.2
      - luxon@3.4.4
      - release-it@16.3.0
      - rimraf@5.0.5
      - ts-command-line-args@2.5.1
      - ts-jest@29.1.1
      - typescript-cubic-spline@1.0.1
      - typescript@5.2.2
      - winston@3.11.0
      - zod@3.22.4
  status: success
tree:
  children:
    child:
      pipeline:
        - interpolate
        - cpu-factor-to-wattage
        - wattage-times-duration
        - wattage-to-energy-kwh
        - calculate-vcpu-ratio
        - correct-cpu-energy-for-vcpu-ratio
      defaults:
        thermal-design-power: 100
        vcpus-total: 8
        vcpus-allocated: 2
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 360
          cpu/utilization: 1
          carbon: 30
        - timestamp: 2023-09-06T00:00
          duration: 360
          carbon: 30
          cpu/utilization: 10
        - timestamp: 2023-10-06T00:00
          duration: 360
          carbon: 30
          cpu/utilization: 50
        - timestamp: 2023-10-06T00:00
          duration: 360
          carbon: 30
          cpu/utilization: 100
      outputs:
        - timestamp: 2023-08-06T00:00
          duration: 360
          cpu/utilization: 1
          carbon: 30
          thermal-design-power: 100
          vcpus-total: 8
          vcpus-allocated: 2
          cpu-factor: 0.13999999999999999
          cpu-wattage: 13.999999999999998
          cpu-wattage-times-duration: 5039.999999999999
          cpu-energy-raw: 0.0013999999999999998
          vcpu-ratio: 4
          cpu-energy-kwh: 0.00034999999999999994
        - timestamp: 2023-09-06T00:00
          duration: 360
          carbon: 30
          cpu/utilization: 10
          thermal-design-power: 100
          vcpus-total: 8
          vcpus-allocated: 2
          cpu-factor: 0.32
          cpu-wattage: 32
          cpu-wattage-times-duration: 11520
          cpu-energy-raw: 0.0032
          vcpu-ratio: 4
          cpu-energy-kwh: 0.0008
        - timestamp: 2023-10-06T00:00
          duration: 360
          carbon: 30
          cpu/utilization: 50
          thermal-design-power: 100
          vcpus-total: 8
          vcpus-allocated: 2
          cpu-factor: 0.75
          cpu-wattage: 75
          cpu-wattage-times-duration: 27000
          cpu-energy-raw: 0.0075
          vcpu-ratio: 4
          cpu-energy-kwh: 0.001875
        - timestamp: 2023-10-06T00:00
          duration: 360
          carbon: 30
          cpu/utilization: 100
          thermal-design-power: 100
          vcpus-total: 8
          vcpus-allocated: 2
          cpu-factor: 1.02
          cpu-wattage: 102
          cpu-wattage-times-duration: 36720
          cpu-energy-raw: 0.0102
          vcpu-ratio: 4
          cpu-energy-kwh: 0.00255
- + \ No newline at end of file diff --git a/reference/cli/index.html b/reference/cli/index.html index 9ac7afe2..7822f500 100644 --- a/reference/cli/index.html +++ b/reference/cli/index.html @@ -8,13 +8,13 @@ Command line tool | Impact Framework - +
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

Command line tool

A core feature of the Impact Framework is the if-run command line tool (CLI). This is how you trigger Impact Framework to execute a certain manifest file.

We also provide several other command line tools that work in concert with if-run to enable flows such as comparing, re-executing and verifying IF output files.

This page includes reference documentation for the CLI tools, including the various commands and flags each tool exposes.

We also provide tutorial-style user documentation for these tools in the Users section.

if-run​

If you have globally installed our if npm package, you can invoke the CLI using the if-run command directly in your terminal. The if-run command is an alias to npx ts-node src/index.ts, which executes the Impact Framework's src/index.ts script and acts as the entry point for Impact Framework.

if-run <args>

--manifest , -m​

The --manifest flag is the only required flag and tells if-run where to find the manifest file that you want to execute. This command expects to receive the path where your manifest file is saved, as shown in the following example:

if-run --manifest examples/manifests/my-manifest.yml

--output , -o​

The --output flag is optional and is used for defining a path to save your output data.

Here is an example of --output being used to define a path:

if-run --manifest examples/manifests/my-manifest.yml --output examples/outputs/my-outdata
## or using aliases
if-run -m examples/manifests/my-manifest.yml -o examples/outputs/my-outdata

--override-params , -p​

The override-params command is used when you want to discard our recommended set of parameters and associated units and aggregation methods and instead provide your own. We do not recommend this, and if you use this feature you take full responsibility for any errors you introduce downstream, including unit or aggregation errors. This is why we hide the ability to override the parameters behind a CLI command - it is an advanced feature that you should only use if you really know what you are doing.

You pass the path to your new parameter file as an argument. The file is expected to conform to the same structure as our src/config/params.ts file.

For example:

if-run --manifest <your manifest> --override-params <path-to-your-params-file>
## or using aliases
if-run -m <your manifest> -p <path-to-your-params-file>

--help , -h​

The --help command provides information about all available commands in order to help you easily find the command you need.

Example:

if-run --help
## or using alias
if-run -h

--debug​

You can provide the --debug flag to ie in order to display execution logs to the console. These logs show messages for each operation IF and its plugins are executing. For example, your debug logs will look similar to the following:

INFO: 2024-06-12T08:48:02.918Z: Starting IF
DEBUG: 2024-06-12T08:48:02.919Z: Loading manifest
DEBUG: 2024-06-12T08:48:02.924Z: Capturing runtime environment data
DEBUG: 2024-06-12T08:48:03.978Z: Validating manifest
DEBUG: 2024-06-12T08:48:03.980Z: Syncing parameters
DEBUG: 2024-06-12T08:48:03.980Z: Initializing plugins
DEBUG: 2024-06-12T08:48:03.981Z: Initializing Sum
DEBUG: 2024-06-12T08:48:03.981Z: Loading Sum from builtin
DEBUG: 2024-06-12T08:48:04.859Z: Initializing Coefficient
DEBUG: 2024-06-12T08:48:04.859Z: Loading Coefficient from builtin
DEBUG: 2024-06-12T08:48:04.860Z: Initializing Multiply
DEBUG: 2024-06-12T08:48:04.860Z: Loading Multiply from builtin
DEBUG: 2024-06-12T08:48:04.860Z: Computing pipeline for `sum`
DEBUG: 2024-06-12T08:48:04.861Z: Computing pipeline for `coefficient`
DEBUG: 2024-06-12T08:48:04.861Z: Computing pipeline for `multiply`
DEBUG: 2024-06-12T08:48:04.862Z: Aggregating outputs
DEBUG: 2024-06-12T08:48:04.862Z: Preparing output data

You can use the --debug flag to help debug failing IF runs. You will see exactly where in the execution pipeline an error arose. If the error arose from a plugin, this will be clear from the execution logs, for example:

INFO: 2024-06-12T08:53:21.376Z: Starting IF
DEBUG: 2024-06-12T08:53:21.376Z: Loading manifest
DEBUG: 2024-06-12T08:53:21.381Z: Capturing runtime environment data
DEBUG: 2024-06-12T08:53:22.367Z: Validating manifest
DEBUG: 2024-06-12T08:53:22.369Z: Syncing parameters
DEBUG: 2024-06-12T08:53:22.369Z: Initializing plugins
DEBUG: 2024-06-12T08:53:22.369Z: Initializing Sum
DEBUG: 2024-06-12T08:53:22.370Z: Loading Sum from builtin
DEBUG: 2024-06-12T08:53:23.165Z: Initializing Coefficient
DEBUG: 2024-06-12T08:53:23.165Z: Loading Coefficient from builtin
DEBUG: 2024-06-12T08:53:23.165Z: Initializing Multiply
DEBUG: 2024-06-12T08:53:23.165Z: Loading Multiply from builtin
DEBUG: 2024-06-12T08:53:23.165Z: Computing pipeline for `sum`
[2024-06-12 09:53:23.166 AM] error:     cpu/energy is missing from the input array.

if-diff​

The if-diff command line tool allows you to determine whether two manifest or output files are the same, and if not, how they differ.

if-diff needs two files to compare - a source and a target. The source file is considered to be the "true" file that another file, the target, is compared against. Note that for most purposes, it doesn't matter which file is assigned as source or target - the important thing is that if-diff receives two files. Both files should be yaml files. They are expected to be IF output files, meaning they contain all the required fields of a manifest plus the IF-generated output and execution blocks.

if-diff is run as follows:

if-diff --source file-1.yml --target file2.yml

You can also pipe the outputs from if-run directly into if-diff. This means you only provide one file to if-diff and the other comes from a new if-run run configured to send its output data to the console. This is an important feature because it allows you to receive an output file and verify that it was computed correctly and not tampered with post-execution. For example, if someone provides you with an output file, you can strip out the outputs section and re-run it with if-run, piping the outputs straight to if-diff to compare against the original you received.

If the original was correctly and honestly reported, if-diff will return a success response.

e.g.

if-run -m my-manifest | if-diff --target my-output-file.yml

if-diff matching rules​

if-diff looks for differences between the source and target. However, if-diff applies its own IF-specific matching rules, ensuring that the outputs are functionally identical even if they are not precisely identical. For example, if-diff allows the order of nodes in a tree to vary between files as long as identically named components contain identical data.

Difference identifiedReport or ignore?Note
trees contain different number of nodesreportworks the same regardless whether source or target has more nodes
nodes in tree have different namesreportThere should be no named nodes existing in one file that aren't also in the other
nodes in tree contain non-identical fields and/or valuesreportthe data inside each tree component should contain identical keys/values
keys and values in context field are non- identicalreportthe same fields should exist in the context section and their values should be identical
status and error fields in execution blockreportOnly these two fields in execution are considered
order of nodes in tree are differentignoreif data is identical, position of node in tree is ignored
order of fields in contextignoreif data is identical, position of field in context is ignored
content of execution block EXCEPT status and errorignoreenvironment information is ignored

if-diff outputs​

If if-diff finds no in-scope differences between the source and target then it returns a success message and exit code 0:

FILES MATCH and exit code 0.

If if-diff detects an in-scope difference between the files, it halts execution, returns exit code 1 and reports the difference to the command line.

The report includes the yaml path to the differing element in the tree, the value in the source and the value in the target, using the following schema:

Files do not match!
<yaml path to non-matching element>
source: <value in source file>
target: <value in target file>

If the difference relates to a missing node in the tree for source or target then <value in x file> should be either exists or missing and the yaml path should point to the highest level element that is missing (e.g. if an entire child component is missing, provide the path to the child component).

e.g. different values detected for a given key in an input array:

Files do not match!
tree.children.vm1[4].cpu/utilization
source: 45
target:  43

e.g. different components in tree in source and target:

Files do not match!
tree.children.child1
source: missing
target:  exists

if-env​

if-env is a command line tool that helps you to create local development environments where you can run manifests.

There are two use cases for this:

  1. setting up a new development environment for plugin building
  2. replicating a runtime environment for a given manifest, so you can re-execute it

commands​

  • --manifest or -m: the path to a manifest whose dependencies you want to install
  • --install or -i: instructs if-env to automatically install the dependencies in the local package.json
  • --cwd or -c: forces if-env to create or update the package.json in the current working directory. This is already default behaviour when no arguments are passed to if-env, but when a manifest is passed to -m, if-env defaults to saving a package.json in the same folder as the manifest. using -cwd overrides that behaviour and uses the current working directory as the package.json target path.

Setting up new development environments using if-env​

If you are creating a new manifest from scratch and want to bootstrap your way in, you can use if-env with no arguments to generate a template manifest and package.json in your current working directory. Then, all you need to do is tweak the templates for your specific use case.

For example:

mkdir my-manifest && cd my-manifest
if-env

After running these commands, you will see the following files in my-manifest:

ls my-manifest

> package.json manifest.yaml

Now, you can use these files as templates for your manifest development.

Replicating runtime environments using if-env​

If you are given an IF output file and you want to re-run it, you can use if-env to install that output file's dependencies so that all the plugins in its execution pipeline can be executed.

For example, if you are given a file, output-file.yml, you can save the file to if and run

cd if
if-env -m output-file.yml

if-env will compare the installed dependencies in the package.json it sees in if with the dependencies listed in output-file.yaml. Any dependencies that are in output-file.yaml and not in if/package.json will be added to if-package.json. Then, you can run:

npm i

and you are ready to re-execute output-file.yaml in your local environment. We also provide the --install flag to instruct if-env to automatically run npm i after merging the dependencies, so you could craft a single command to install all the relevant dependencies and then run the manifest, as follows:

if-env -m output-file.yml -i && if-run -m output-file.yml

if-check​

if-check is a manifest verification tool that is equivalent to running if-env and if-diff on a given manifest file. The manifest file must have outputs and an execution section for if-check to run.

The intended use case is to verify that a manifest's outputs are correct and honest. Say someone handed you a manifest as evidence of their environmental impact. You could choose to trust them, or you could run if-check to verify that their calculations are correct. Under the hood, IF is creating a development environment using the dependencies listed in the given file's execution section and then executing the file locally, then comparing the newly generated results to those in the given file.

To check a file:

if-check -m <path-to-file>

If the if-check is successful you will receive the following response:

if-check: successfully verified <filename>

If if-check was not able to verify the file because there were differences in the given and re-executed files, then you will receive the following response which includes the details of how the files differ, as per if-diff.

if-check: could not verify <filename>. The re-executed file does not match the original.

Running IF over multiple manifests with --d​

Alice could also run if-check over any number of manifests in a single command, using the --directory or -d subcommand. For a folder containing multiple manifests, pass the folder path:

if-check -d /my-folder-of-manifests

Each manifest will be run through if-check in sequence.

if-csv​

if-csv is a command line tool that helps to save data to CSV file.

commands​

  • --manifest or -m: (optional) the path to an executed manifest
  • --output or -o: (optional) the path to save your output data in csv format
  • --params or -p: (required) the metric to export the data

There are three use cases for this:

  1. Exporting CSV with the --output flag: When the --output flag is provided, if-csv exports the data to a CSV file at the specified path. This is useful for saving data for later use or sharing with others.
if-csv -m ./my-manifest.yml -p carbon -o ./my-outdata
  1. Printing CSV to the console without the --output flag: If the --output flag is omitted, if-csv will print the CSV data directly to the console. This is useful for quick checks.
if-csv -m ./my-manifest.yml -p carbon
  1. Piping output from if-run to if-csv. By piping the output from if-run, you can chain commands to execute a manifest and then immediately export the data to a CSV file.
if-run -m ./my-manifest.yml | if-csv -p carbon -o ./my-outdata
- + \ No newline at end of file diff --git a/reference/errors/index.html b/reference/errors/index.html index 998e9a09..e4a62cde 100644 --- a/reference/errors/index.html +++ b/reference/errors/index.html @@ -8,7 +8,7 @@ Errors | Impact Framework - + @@ -17,7 +17,7 @@ The message itself indicates that the problematic element is initialize and the problem is that it is missing.

The remedy for this issue is to add an initialize block into the manifest.

InvalidGroupingError​

Errors of the InvalidGroupingError are only emitted by the group-by plugin. There is only one associated message; it is emitted when the requested groups do not exist in the tree.

messagecauseremedy
Invalid group ${type}.you are requested groupby to regroup the tree based on fields that do not existCheck the spelling of the values passed to groupby and ensure the values exist in the tree

WriteFileError​

Errors of the WriteFileError class are caused by problems writing output data to files. Typically, this can occur when the user does not have sufficient permissions to write to a given file.

Messages​

messagecauseremedy
Failed to write CSV to ${outputPath}: ${error}There was a problem writing data to filecheck that you have provided a valid output path and that you have valid permissions to write to that location

CliSourceFileError​

Errors of the CliSourceFileError class are caused by problems with source manifest.

Messages​

messagecauseremedy
Manifest is missing.Source manifest is not providedcheck that you have provided a path to source manifest
Given source file is not in yaml format.Source file is provided, but format is not a yaml formatcheck that you have provided valid yaml manifest

CliTargetFileError​

Errors of the CliTargetFileError class are caused by problems with target manifest.

messagecauseremedy
Given target file is not in yaml format.Target file is provided, but format is not a yaml formatcheck that you have provided valid yaml manifest

PluginInitializationError​

Errors of the PluginInitializationError arise when a plugin is invoked in a pipeline without having been initialized in the initialize block of the manifest being executed.

messagecauseremedy
Not initalized plugin: ${name}. Check if ${name} is in 'manifest.initalize.plugins'.a plugin invoked in a pipeline is not initializedensure all plugins that exist in pipelines across your manifest have been included in the manifest's initialize block
Provided module ${path} is invalid or not found. ${error ?? ''}a plugin invoked in a pipeline is not initializedensure all plugins that exist in pipelines across your manifest have been included in the manifest's initialize block

InvalidAggregationMethodError​

Errors of the InvalidAggregationMethodError class are caused by problems in the configuration of the aggregation feature.

messagecauseremedy
Aggregation is not possible for given ${metric} since method is 'none'.You are trying to aggregate a metric whose method is set to nonethe aggregation method is defined either in if/src/config/params.ts or you defined it in a params block in your manifest. Either update the aggregation method, or choose a different metric to aggregate.

MissingAggregationParamError​

Errors of the MissingAggregationParamError class are caused by problems in the configuration of the aggregation feature. Typically, the aggregation method may be undefined or you have tried to aggregate a metric that IF cannot find in the input data.

Messages​

messagecauseremedy
Aggregation metric ${metric} is not found in inputs[${index}].You are trying to aggegate a metric that doesn't exist in the input dataCheck that your chosen metric is spelled correctly and that it exists in the input data by the time the aggregate feature executes.

MissingPluginMethodError​

Errors of the MissingPluginMethodError class are caused by missing information in manifest's initalize.plugins section.

messagecauseremedy
Initalization param 'method' is missing.The required method field is missing from the initialize block for a given plugin.Ensure the method field is added to the initialize block for each plugin. The value should be the name of the function exported by the plugin.

MissingPluginPathError​

Errors of the MissingPluginPathError class are caused by missing information in manifest's initalize.plugins section.

messagecauseremedy
Initalization param 'path' is missing.The required path field is missing from the initialize block for a given pluginEnsure the path field is added to the initialize block for each plugin. The value should be the path to the directory in if/node_modules for your plugin.

InvalidExhaustPluginError​

Errors of the InvalidExhaustPluginError class are caused by using unsupported exhaust plugin.

messagecauseremedy
Invalid exhaust plugin: ${pluginName}.Unsupported or misspelled plugin was used as output methodEnsure the pluginName corresponds to supported plugins.

Plugin Errors​

Plugins can emit their own custom error messages, but we still prefer those messages to be attached to one of a finite set of predefined error classes. Those classes are listed in this section.

GlobalConfigError​

Errors of the GlobalConfigError are used when part of the config data provided to a plugin is invalid or missing.

For example the Divide plugin throws a GlobalConfigError when it receives a denominator equal to zero.

The message should name the config element that was invalid and describe the reason why. For example:

GlobalConfigError: "denominator" parameter is number must be greater than 0. Error code: too_small.

MissingInputDataError​

Errors of the MissingInputDataError class arise because your plugin is not receiving the data it expects in input data, global config or node-level config. The specific messages depend on the plugin. It is expected that the messages emitted by each plugin are listed in their own documentation.

The example below is a message emitted by the interpolation plugin when the method given in global config is not one of the expected enum variants:

MissingInputDataError: "interpolation" parameter is invalid enum value. expected 'spline' | 'linear', received 'dummy'. Error code: invalid_enum_value.

ProcessExecutionError​

Errors of the ProcessExecutionError class arise because shell plugin have faced problems while executing the script you have provided.

RegexMismatchError​

Errors of the RegexMismatchError class arise because regex plugin have faced problems while parsing given string with specified regex.

messagecauseremedy
${input} does not match the ${match} regex expressionGiven string doesn't contain anything matching given regexEnsure that input contains string which can be matched by your regex.

FetchingFileError​

Errors of the FetchingFileError class arise because csv-lookup plugin have faced problems fetching given url.

messagecauseremedy
Failed fetching the file: ${filepath}.Fetching the file with given URL failedEnsure that file's url is accessible

ReadFileError​

Errors of the ReadFileError class arise because csv-lookup plugin have faced problems reading given file path. The error should include the file path and the system error that was encountered when IF attempted to read data from the file:

messagecauseremedy
Failed reading the file: ${filepath}.Reading the file with given path failedEnsure that file's path is correct

MissingCSVColumnError​

Errors of the MissingCSVColumnError class arise because csv-lookup plugin can't access given csv file column.

messagecauseremedy
There is no column with the name: ${columnName}.CSV file doens't contain such columnEnsure that specified query is correct and contains existing column name.

QueryDataNotFoundError​

Errors of the QueryDataNotFoundError class arise because csv-lookup plugin can't find query related data in given CSV file.

messagecauseremedy
One or more of the given query parameters are not found in the target CSV file column headers.CSV file doens't contain data with given criteria.Ensure that specified query and input values have intersection with CSV file's data.

InvalidDateInInputError​

Errors of the InvalidDateInInputError class arise because time-sync plugin can't parse date from inputs.

messagecauseremedy
Unexpected date datatype: ${typeof date}: ${date}Unsupported type for date.Ensure that dates in inputs are correct timestamps.

InvalidPaddingError​

Errors of the InvalidPaddingError class arise when there is misconfiguration of time-sync plugin.

messagecauseremedy
Avoiding padding at ${start or end}Error on padding is enabled and config is missing padding configuration.Make sure padding is correctly configured.

InvalidInputError​

Errors of the InvalidInputError class arise when there is input timestamps incompatibility while using time-sync plugin.

messagecauseremedy
Observation timestamps overlap, please check inputs.Input timestamps have overlap.Make sure that input timestamps are continuous.

ExhaustOutputArgError​

Errors of the ExhaustOutputArgError class arise when there is output path issues while exporting file or exporting criteria misconfiguration.

messagecauseremedy
Output path is required, please make sure output is configured properly.Missed output path.Make sure that output path is present in your cli command.

CSVParseError​

Errors of the CSVParseError occur due to a problem reading CSV file. Typically, this can occur when provided file is not a CSV.

Capturing errors in manifests​

When you run a manifest, IF generates output data and either displays it in the console or saves it to a file. If IF or one of the plugins being executed throws an exception, IF can still return an output file, except instead of adding outputs, it captures the error message that caused IF to fail in the manifest's execution section. Inside the execution section, you will find two fields: status and error. The status field is either success or fail, and the error field contains the error message.

For example, the following is an output file generated by running a manifest whose input data omitted the required duration field:

name: basic-error-demo
description:
tags:
initialize:
  plugins:
    teads-curve: 
      path: '@grnsft/if-unofficial-plugins'
      method: TeadsCurve
      global-config:
        interpolation: spline
execution:
  status: fail
  error: 'InputValidationError: "duration" parameter is required. Error code: invalid_type'.
tree:
  children:
    child-0:
      defaults:
        cpu/thermal-design-power: 100
      pipeline:
        - teads-curve
      inputs:
        - timestamp: 2023-07-06T00:00
          cpu/utilization: 20

No configuration is necessary - this is the default behaviour for IF if the output is configured to save to yaml and the manifest has an error causing IF to fail.

- + \ No newline at end of file diff --git a/reference/index.html b/reference/index.html index 908c92e9..87c3f9b0 100644 --- a/reference/index.html +++ b/reference/index.html @@ -8,13 +8,13 @@ Reference | Impact Framework - +
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

Reference

In this section you will find reference documentation for the core data structures and features used in the Impact Framework.

This includes:

These are developer focused reference docs. If you are not a developer and looking for usage guides, please head over to the Using IF section.

- + \ No newline at end of file diff --git a/reference/plugins/index.html b/reference/plugins/index.html index 99153bb4..e97f2420 100644 --- a/reference/plugins/index.html +++ b/reference/plugins/index.html @@ -8,13 +8,13 @@ Plugins | Impact Framework - +
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

Plugins

Impact Framework works by executing pipelines of plugins over input data. Those plugins are re-useable units of code that can be thought of as Lego bricks - simple blocks of code that can be assembled into complex workflows.

IF comes bundled with a standard library of builtins that allow you to do basic and/or generic operations over your data. These include simple arithmetic, regrouping data, calculating SCI scores, and running processes in a spawned shell.

Most IF plugins are created and maintained by the community. Anyone can create a plugin and share it so that other iF users can install it and use it in their pipelines.

We provide a website where you can search for plugins:

IF Explorer

You can also add your own plugins to the Explorer using this form.

built-in​

IF builtins all come bundled with IF. Below you will find a list of each builtin along with a brief description of its purpose and a link to its README documentation.

  • Time Sync: Takes a heterogeneous set of time series data that might be offset, discontinuous or irregularly spaces and returns time series conforming to a user defined time grid. E.g. a user can define that all sets of observations should start at some global start time, end at some global end time and have a specific temporal resolution.

  • Groupby: Allows a user to regroup their output data according to given keys.

  • SCI-embodied - Calculates the embodied carbon for a component.

  • SCI: Calculates the software carbon intensity.

  • Shell - A plugin that enables external code in any language to be run in a child process

  • Sum: a generic arithmetic plugin that allows you to sum any set of input parameters.

  • Multiply: a generic arithmetic plugin that allows you to multiply any set of input parameters.

  • Coefficient: a generic arithmetic plugin that allows you to multiply any input value by a coefficient.

  • Mock Observations: A plugin for mocking observations (inputs) for testing and demo purposes.

  • Subtract: a generic plugin for subtracting one value from another

  • Divide: A generic plugin for doing arithmetic division of two values.

  • Regex: A generic plugin to match part of one string and extract it into another.

  • Exponent: A generic plugin for raising a value to a power

  • Interpolation: A generic plugin for interpolating between known points.

  • Copy Param: A generic plugin for copying a parameter to a new element in the input array, optionally deleting the original. Useful as a way to rename parameters.

  • CSV lookup: A generic plugin for querying data from CSV files.

- + \ No newline at end of file diff --git a/users/how-to-compare-files-with-if-diff/index.html b/users/how-to-compare-files-with-if-diff/index.html index b90d51ca..7f6b6317 100644 --- a/users/how-to-compare-files-with-if-diff/index.html +++ b/users/how-to-compare-files-with-if-diff/index.html @@ -8,13 +8,13 @@ How to compare files with `if-diff` | Impact Framework - +
-
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

How to compare files with if-diff

if-diff is a command line tool that allows you to compare two if-run output files. They either match according to if-diff's matching rules, or they don't. If they match, then if-diff returns a simple success response. If the differ, then if-diff returns a report of the differences it finds.

Why is this useful?​

if-diff can be used to verify that a given output file was correctly executed and that it was not tampered with after it was computed. Imagine you received an output file from someone, reporting their carbon expenditure. It is better to verify than trust this person's report, so you simply delete the outputs block from the file (creating a manifest), run it through if-run and compare the output to the original file you received. All being well, the two files are identical. if not, you can see exactly where the differences are.

if-diff can also be used for debugging your own files. Maybe you have some large manifest files in development and have accidentally introduced some changes that you are now struggling to identify, but are leading to different if-run outcomes. if-diff can quickly scan your two files and tell you where the differences are so you can get back in sync.

Example: output verification​

Let's say someone provides you with this output file, given-output-file.yml:

name: sum
description: successful path
tags:
initialize:
  plugins:
    sum:
      method: Sum
      path: 'builtin'
      global-config:
        input-parameters: ['cpu/energy', 'network/energy']
        output-parameter: 'energy'
tree:
  children:
    child:
      pipeline:
        - sum
      config:
        sum:
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001
      outputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001
          energy: 0.0005

This manifest simply sums two components, cpu/energy and network/energy and assigns the result to energy in the outputs array. You receive this file and feel like something's not quite right. So you delete the outputs block to create test-manifest.yml:

name: sum
description: successful path
tags:
initialize:
  plugins:
    sum:
      method: Sum
      path: "builtin"
      global-config:
        input-parameters: ['cpu/energy', 'network/energy']
        output-parameter: 'energy'
tree:
  children:
    child:
      pipeline:
        - sum
      config:
        sum:
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001

Now you want to run the manifest through if-run and compare the result to the given output file. You can do this by piping the result of if-run directly into if-diff as follows:

if-run -m test-manifest.yml | if-diff --target given-output-file.yml

The result is:

Files do not match!
tree.children.child[0].energy
source: 0.002
target:  0.0005

Uh oh. It seems there has been some mistake or tampering with the outputs in given-output-file.yml. The right result of summing cpu/energy and network/energy is 0.002, but they reported 0.0005. You can now query that result with the sender and ask them to fix it.

Obviously, this is an arbitrary, simplified example, but if-diff enables you to do this kind of output verification on very complex manifests where errors are harder to spot by eye and to do it programmatically over large numbers of files.

Example: debugging​

Imagine you developed a manifest that was giving you a consistent result, but now when you run it your result is different and you are not sure why. Maybe one of your colleagues changed something and forgot to tell you, maybe you accidentally inserted or removed something while you were working.

You could revert to an archived version, but you moved all the components around into a structure you prefer! if-diff has you covered. It will step through the files identifying all the functional differences between the files to help you identify the problematic one(s).

You have original-manifest.yml and new-manifest.yml. Pass them to if-diff as follows:

if-diff --source original-manifest.yml --target new-manifest.yml

if-diff will report each difference it finds. You can fix the difference and run if-diff again until you get a success response - since if-diff ignores positional differences and only considers differences in context and tree keys and values, your two manifests will run identically even though you persist your tree reorganization.

- +
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

How to compare files with if-diff

if-diff is a command line tool that allows you to compare two if-run output files. They either match according to if-diff's matching rules, or they don't. If they match, then if-diff returns a simple success response. If the differ, then if-diff returns a report of the differences it finds.

Why is this useful?​

if-diff can be used to verify that a given output file was correctly executed and that it was not tampered with after it was computed. Imagine you received an output file from someone, reporting their carbon expenditure. It is better to verify than trust this person's report, so you simply delete the outputs block from the file (creating a manifest), run it through if-run and compare the output to the original file you received. All being well, the two files are identical. if not, you can see exactly where the differences are.

if-diff can also be used for debugging your own files. Maybe you have some large manifest files in development and have accidentally introduced some changes that you are now struggling to identify, but are leading to different if-run outcomes. if-diff can quickly scan your two files and tell you where the differences are so you can get back in sync.

Example: output verification​

Let's say someone provides you with this output file, given-output-file.yml:

name: sum
description: successful path
tags:
initialize:
  plugins:
    sum:
      method: Sum
      path: 'builtin'
      global-config:
        input-parameters: ['cpu/energy', 'network/energy']
        output-parameter: 'energy'
tree:
  children:
    child:
      pipeline:
        - sum
      config:
        sum:
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001
      outputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001
          energy: 0.0005

This manifest simply sums two components, cpu/energy and network/energy and assigns the result to energy in the outputs array. You receive this file and feel like something's not quite right. So you delete the outputs block to create test-manifest.yml:

name: sum
description: successful path
tags:
initialize:
  plugins:
    sum:
      method: Sum
      path: "builtin"
      global-config:
        input-parameters: ['cpu/energy', 'network/energy']
        output-parameter: 'energy'
tree:
  children:
    child:
      pipeline:
        - sum
      config:
        sum:
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001

Now you want to run the manifest through if-run and compare the result to the given output file. You can do this by piping the result of if-run directly into if-diff as follows:

if-run -m test-manifest.yml | if-diff --target given-output-file.yml

The result is:

Files do not match!
tree.children.child[0].energy
source: 0.002
target:  0.0005

Uh oh. It seems there has been some mistake or tampering with the outputs in given-output-file.yml. The right result of summing cpu/energy and network/energy is 0.002, but they reported 0.0005. You can now query that result with the sender and ask them to fix it.

Obviously, this is an arbitrary, simplified example, but if-diff enables you to do this kind of output verification on very complex manifests where errors are harder to spot by eye and to do it programmatically over large numbers of files.

Example: debugging​

Imagine you developed a manifest that was giving you a consistent result, but now when you run it your result is different and you are not sure why. Maybe one of your colleagues changed something and forgot to tell you, maybe you accidentally inserted or removed something while you were working.

You could revert to an archived version, but you moved all the components around into a structure you prefer! if-diff has you covered. It will step through the files identifying all the functional differences between the files to help you identify the problematic one(s).

You have original-manifest.yml and new-manifest.yml. Pass them to if-diff as follows:

if-diff --source original-manifest.yml --target new-manifest.yml

if-diff will report each difference it finds. You can fix the difference and run if-diff again until you get a success response - since if-diff ignores positional differences and only considers differences in context and tree keys and values, your two manifests will run identically even though you persist your tree reorganization.

+ \ No newline at end of file diff --git a/users/how-to-export-csv-file-with-if-csv/index.html b/users/how-to-export-csv-file-with-if-csv/index.html index 9e1f06ae..078a8216 100644 --- a/users/how-to-export-csv-file-with-if-csv/index.html +++ b/users/how-to-export-csv-file-with-if-csv/index.html @@ -7,14 +7,14 @@ -how-to-export-csv-file-with-if-csv | Impact Framework - +Exporting CSV file with `if-csv` | Impact Framework +
-
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

how-to-export-csv-file-with-if-csv

Exporting CSV file with if-csv​

IF includes a command line tool called if-csv which is designed to export CSV files based on a specified manifest file and metric.

Example:​

Let's execute this manifest file. This manifest simply sums two components, cpu/energy and network/energy and assigns the result to energy in the outputs array.

name: sum
description: successful path
tags: null
initialize:
  plugins:
    sum:
      path: builtin
      method: Sum
      global-config:
        input-parameters:
          - cpu/energy
          - network/energy
        output-parameter: energy
  outputs:
    - yaml
execution:
  command: >-
    /Users/manushak/.npm/_npx/1bf7c3c15bf47d04/node_modules/.bin/ts-node
    /Users/manushak/Documents/Projects/Green-Software/if/src/if-run/index.ts -m
    ./manifests/test.yaml -o ./manifests/re-test
  environment:
    if-version: 0.5.0
    os: macOS
    os-version: 13.6.7
    node-version: 18.20.0
    date-time: 2024-07-09T16:00:58.218Z (UTC)
    dependencies:
      - '@babel/core@7.22.10'
      - '@babel/preset-typescript@7.23.3'
      - '@commitlint/cli@18.6.0'
      - '@commitlint/config-conventional@18.6.0'
      - '@grnsft/if-core@0.0.10'
      - '@grnsft/if-plugins@v0.3.2 extraneous -> file:../../../if-models'
      - >-
        @grnsft/if-unofficial-plugins@v0.3.0 extraneous ->
        file:../../../if-unofficial-models
      - '@jest/globals@29.7.0'
      - '@types/jest@29.5.8'
      - '@types/js-yaml@4.0.9'
      - '@types/luxon@3.4.2'
      - '@types/node@20.9.0'
      - axios-mock-adapter@1.22.0
      - axios@1.7.2
      - cross-env@7.0.3
      - csv-parse@5.5.6
      - csv-stringify@6.4.6
      - fixpack@4.0.0
      - gts@5.2.0
      - husky@8.0.3
      - jest@29.7.0
      - js-yaml@4.1.0
      - lint-staged@15.2.2
      - luxon@3.4.4
      - release-it@16.3.0
      - rimraf@5.0.5
      - ts-command-line-args@2.5.1
      - ts-jest@29.1.1
      - typescript-cubic-spline@1.0.1
      - typescript@5.2.2
      - winston@3.11.0
      - zod@3.22.4
  status: success
tree:
  children:
    child:
      pipeline:
        - sum
      config:
        sum: null
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001
      outputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001
          energy: 0.002

To generate a CSV file in the provided path, run the following command:

if-csv -m ./sum.yaml -p energy -o ./output-sum

To print data in the console, you need to run:

if-csv -m ./sum.yaml -p energy

The output will be:

Path,2023-08-06T00:00
tree.children.child.energy,0.002

Alternatively, you can pipe the result from if-run.

if-run -m ./sum.yaml | if-csv -p energy
- +
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

Exporting CSV file with if-csv

IF includes a command line tool called if-csv which is designed to export CSV files based on a specified manifest file and metric.

Example:​

Let's execute this manifest file. This manifest simply sums two components, cpu/energy and network/energy and assigns the result to energy in the outputs array.

name: sum
description: successful path
tags: null
initialize:
  plugins:
    sum:
      path: builtin
      method: Sum
      global-config:
        input-parameters:
          - cpu/energy
          - network/energy
        output-parameter: energy
  outputs:
    - yaml
execution:
  command: >-
    /Users/manushak/.npm/_npx/1bf7c3c15bf47d04/node_modules/.bin/ts-node
    /Users/manushak/Documents/Projects/Green-Software/if/src/if-run/index.ts -m
    ./manifests/test.yaml -o ./manifests/re-test
  environment:
    if-version: 0.5.0
    os: macOS
    os-version: 13.6.7
    node-version: 18.20.0
    date-time: 2024-07-09T16:00:58.218Z (UTC)
    dependencies:
      - '@babel/core@7.22.10'
      - '@babel/preset-typescript@7.23.3'
      - '@commitlint/cli@18.6.0'
      - '@commitlint/config-conventional@18.6.0'
      - '@grnsft/if-core@0.0.10'
      - '@grnsft/if-plugins@v0.3.2 extraneous -> file:../../../if-models'
      - >-
        @grnsft/if-unofficial-plugins@v0.3.0 extraneous ->
        file:../../../if-unofficial-models
      - '@jest/globals@29.7.0'
      - '@types/jest@29.5.8'
      - '@types/js-yaml@4.0.9'
      - '@types/luxon@3.4.2'
      - '@types/node@20.9.0'
      - axios-mock-adapter@1.22.0
      - axios@1.7.2
      - cross-env@7.0.3
      - csv-parse@5.5.6
      - csv-stringify@6.4.6
      - fixpack@4.0.0
      - gts@5.2.0
      - husky@8.0.3
      - jest@29.7.0
      - js-yaml@4.1.0
      - lint-staged@15.2.2
      - luxon@3.4.4
      - release-it@16.3.0
      - rimraf@5.0.5
      - ts-command-line-args@2.5.1
      - ts-jest@29.1.1
      - typescript-cubic-spline@1.0.1
      - typescript@5.2.2
      - winston@3.11.0
      - zod@3.22.4
  status: success
tree:
  children:
    child:
      pipeline:
        - sum
      config:
        sum: null
      inputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001
      outputs:
        - timestamp: 2023-08-06T00:00
          duration: 3600
          cpu/energy: 0.001
          network/energy: 0.001
          energy: 0.002

To generate a CSV file in the provided path, run the following command:

if-csv -m ./sum.yaml -p energy -o ./output-sum

To print data in the console, you need to run:

if-csv -m ./sum.yaml -p energy

The output will be:

Path,2023-08-06T00:00
tree.children.child.energy,0.002

Alternatively, you can pipe the result from if-run.

if-run -m ./sum.yaml | if-csv -p energy
+ \ No newline at end of file diff --git a/users/how-to-import-plugins/index.html b/users/how-to-import-plugins/index.html index 627dc865..71e329b3 100644 --- a/users/how-to-import-plugins/index.html +++ b/users/how-to-import-plugins/index.html @@ -8,13 +8,13 @@ How to load plugins | Impact Framework - +
-
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

How to load plugins

Plugins are developed separately to the Impact Framework core. However, the IF core developers maintain a standard library of plugins come bundled with IF. These are known as builtins.

Builtins have to be initialized in a manifest file using the path builtin. Then they can be invoked in pipelines.

name: if-demo
description: demo pipeline
tags:
initialize:
  plugins:
    "sum":
      path: "builtin"
      method: Sum
      global-config:
        input-parameters:
          - cpu/energy
          - network/energy
        output-parameter: energy-sum

Other plugins are hosted externally to the IF. Anyone can build a plugin and provide it as an npm package or a public code repository (such as Github) and share it using our Explorer.

These external plugins are loaded into IF by installing locally and initializing in a manifest.

First, install the plugin by providing the path to the repository to npm install as follows:

npm install https://github.com/some-account/some-repo

Then, in the manifest's initialize section, you'll need to provide the following fields:

  • YOUR-PLUGIN-HERE: a name to reference this specific instance of the plugin. The same name has to be used to refer to this plugin instance everywhere across the manifest
  • method: the function name exported by your plugin, e.g. AzureImporter
  • path: the path to the plugin

And, if your plugin requires it, add its global-config too.

name: plugin-demo
description: loads plugin
tags: null
initialize:
  plugins:
    <YOUR-PLUGIN-HERE:
      method: OutputPlugin
      path: https://github.com/my-repo/my-plugin

Anyone can develop plugins. As long as you conform to our plugin specification, you can load your plugin into the Impact Framework and run it as part of a pipeline. We provide a guide to building plugins and a template to help you structure them correctly.

- +
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

How to load plugins

Plugins are developed separately to the Impact Framework core. However, the IF core developers maintain a standard library of plugins come bundled with IF. These are known as builtins.

Builtins have to be initialized in a manifest file using the path builtin. Then they can be invoked in pipelines.

name: if-demo
description: demo pipeline
tags:
initialize:
  plugins:
    "sum":
      path: "builtin"
      method: Sum
      global-config:
        input-parameters:
          - cpu/energy
          - network/energy
        output-parameter: energy-sum

Other plugins are hosted externally to the IF. Anyone can build a plugin and provide it as an npm package or a public code repository (such as Github) and share it using our Explorer.

These external plugins are loaded into IF by installing locally and initializing in a manifest.

First, install the plugin by providing the path to the repository to npm install as follows:

npm install https://github.com/some-account/some-repo

Then, in the manifest's initialize section, you'll need to provide the following fields:

  • YOUR-PLUGIN-HERE: a name to reference this specific instance of the plugin. The same name has to be used to refer to this plugin instance everywhere across the manifest
  • method: the function name exported by your plugin, e.g. AzureImporter
  • path: the path to the plugin

And, if your plugin requires it, add its global-config too.

name: plugin-demo
description: loads plugin
tags: null
initialize:
  plugins:
    <YOUR-PLUGIN-HERE:
      method: OutputPlugin
      path: https://github.com/my-repo/my-plugin

Anyone can develop plugins. As long as you conform to our plugin specification, you can load your plugin into the Impact Framework and run it as part of a pipeline. We provide a guide to building plugins and a template to help you structure them correctly.

+ \ No newline at end of file diff --git a/users/how-to-install-if/index.html b/users/how-to-install-if/index.html index 1ede2cf0..5b53ff2d 100644 --- a/users/how-to-install-if/index.html +++ b/users/how-to-install-if/index.html @@ -8,13 +8,13 @@ How to install Impact Framework | Impact Framework - +
-
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

How to install Impact Framework

You can install Impact Framework either globally or locally. For most users, we recommend installing our official releases globally using npm. You can do this using the following command:

npm install -g @grnsft/if

Then, run the package using the if-run command:

if-run --manifest <path to manifest file>

Installing locally​

You can also clone the Impact Framework repositories and install them locally, useful for developers who want to make changes or build new plugins. Use the following command for local installation:

git clone https://github.com/Green-Software-Foundation/if && cd if
npm install

Then, use the following command to run Impact Framework:

npm run if-run -- --manifest <path to your manifest file>

Next, install local plugin repositories using npm link. You can do this by entering the plugin folder and running the following command:

npm link

This creates a global package with the same name as your project root directory which you can then load by passing the path in your manifest file.

Read our detailed guide to installing plugins.

- +
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

How to install Impact Framework

You can install Impact Framework either globally or locally. For most users, we recommend installing our official releases globally using npm. You can do this using the following command:

npm install -g @grnsft/if

Then, run the package using the if-run command:

if-run --manifest <path to manifest file>

Installing locally​

You can also clone the Impact Framework repositories and install them locally, useful for developers who want to make changes or build new plugins. Use the following command for local installation:

git clone https://github.com/Green-Software-Foundation/if && cd if
npm install

Then, use the following command to run Impact Framework:

npm run if-run -- --manifest <path to your manifest file>

Next, install local plugin repositories using npm link. You can do this by entering the plugin folder and running the following command:

npm link

This creates a global package with the same name as your project root directory which you can then load by passing the path in your manifest file.

Read our detailed guide to installing plugins.

+ \ No newline at end of file diff --git a/users/how-to-verify-files-with-if-check/index.html b/users/how-to-verify-files-with-if-check/index.html index 8bf591b6..c921d602 100644 --- a/users/how-to-verify-files-with-if-check/index.html +++ b/users/how-to-verify-files-with-if-check/index.html @@ -8,13 +8,13 @@ Verifying IF outputs with `if-check` | Impact Framework - +
-
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

Verifying IF outputs with if-check

IF includes a command line tool called if-check that can be used to verify the results in a manifest file.

Imagine that someone provides you with a computed manifest file that they claim demonstrates the environmental impact of their software application.

You could trust them and take their results at face value, but there are probably cases where you think someone might have made a mistake or might be massaging their data in a dishonest way. Maybe you just need to demonstrate due diligence in the quality of the data you receive and use. In these cases, you can make use of if-check.

if-check is a single command that takes a given manifest file, sets up an environment where it can be executed, executes it, and compares the newly generated results to those in the original file. This means you can independently verify the results in the file using your own local copy of IF.

Under the hood, if-check is wrapping calls to if-env and if-diff, so if you need granular control over the information flow for some reason, you could achieve the same result using separate calls to those commands.

Example​

Alice is a manifest verifier. She receives a manifest from Bob. Bob is a good actor and has provided a valid file. Alice is a good cypherpunk who verifies everything.

Here is Bob's manifest:

name: bobs-manifest
description:
tags: null
initialize:
  plugins:
    sci:
      path: builtin
      method: Sci
      global-config:
        functional-unit: requests
execution:
  command: >-
    /home/bob/.npm/_npx/1bf7c3c15bf47d04/node_modules/.bin/ts-node
    /home/bob/Code/if/src/index.ts -m manifests/plugins/sci/success.yml -s
  environment:
    if-version: 0.4.0
    os: linux
    os-version: 5.15.0-113-generic
    node-version: 21.4.0
    date-time: 2024-06-27T14:10:29.830Z (UTC)
    dependencies:
      - '@babel/core@7.22.10'
      - '@babel/preset-typescript@7.23.3'
      - '@commitlint/cli@18.6.0'
      - '@commitlint/config-conventional@18.6.0'
      - '@grnsft/if-core@0.0.9'
      - '@jest/globals@29.7.0'
      - '@types/jest@29.5.8'
      - '@types/js-yaml@4.0.9'
      - '@types/luxon@3.4.2'
      - '@types/node@20.9.0'
      - axios-mock-adapter@1.22.0
      - axios@1.7.2
      - cross-env@7.0.3
      - csv-parse@5.5.6
      - csv-stringify@6.4.6
      - fixpack@4.0.0
      - gts@5.2.0
      - husky@8.0.3
      - jest@29.7.0
      - js-yaml@4.1.0
      - lint-staged@15.2.2
      - luxon@3.4.4
      - release-it@16.3.0
      - rimraf@5.0.5
      - ts-command-line-args@2.5.1
      - ts-jest@29.1.1
      - typescript-cubic-spline@1.0.1
      - typescript@5.2.2
      - winston@3.11.0
      - zod@3.22.4
  status: success
tree:
  children:
    child-1:
      pipeline:
        - sci
      config: null
      inputs:
        - timestamp: 2023-07-06T00:00
          duration: 3600
          energy: 5
          carbon-operational: 5
          carbon-embodied: 0.02
          carbon: 5.02
          requests: 100
      outputs:
        - timestamp: 2023-07-06T00:00
          duration: 3600
          energy: 5
          carbon-operational: 5
          carbon-embodied: 0.02
          carbon: 5.02
          requests: 100
          sci: 0.050199999999999995
    child-2:
      pipeline:
        - sci
      config: null
      inputs:
        - timestamp: 2023-07-06T00:00
          duration: 3600
          energy: 8
          carbon-operational: 8
          carbon-embodied: 0.02
          carbon: 8.02
          requests: 100
      outputs:
        - timestamp: 2023-07-06T00:00
          duration: 3600
          energy: 8
          carbon-operational: 8
          carbon-embodied: 0.02
          carbon: 8.02
          requests: 100
          sci: 0.0802

Alice runs :

if-check -m bobs-manifest.yml

And receives the response:

if-check: successfully verified bobs-manifest

Charlie also has a copy of Bob's manifest. He wants to trick Alice into thinking his SCI score is lower, so he overwrites the values in the manifest file, making them lower. Charlie's manifest looks like this:

# start
name: charlies-manifest
description:
tags: null
initialize:
  plugins:
    sci:
      path: builtin
      method: Sci
      global-config:
        functional-unit: requests
execution:
  command: >-
    /home/charlie/.npm/_npx/1bf7c3c15bf47d04/node_modules/.bin/ts-node
    /home/charlie/Code/if/src/index.ts -m manifests/plugins/sci/success.yml -s
  environment:
    if-version: 0.4.0
    os: linux
    os-version: 5.15.0-113-generic
    node-version: 21.4.0
    date-time: 2024-06-27T14:10:29.830Z (UTC)
    dependencies:
      - '@babel/core@7.22.10'
      - '@babel/preset-typescript@7.23.3'
      - '@commitlint/cli@18.6.0'
      - '@commitlint/config-conventional@18.6.0'
      - '@grnsft/if-core@0.0.9'
      - '@jest/globals@29.7.0'
      - '@types/jest@29.5.8'
      - '@types/js-yaml@4.0.9'
      - '@types/luxon@3.4.2'
      - '@types/node@20.9.0'
      - axios-mock-adapter@1.22.0
      - axios@1.7.2
      - cross-env@7.0.3
      - csv-parse@5.5.6
      - csv-stringify@6.4.6
      - fixpack@4.0.0
      - gts@5.2.0
      - husky@8.0.3
      - jest@29.7.0
      - js-yaml@4.1.0
      - lint-staged@15.2.2
      - luxon@3.4.4
      - release-it@16.3.0
      - rimraf@5.0.5
      - ts-command-line-args@2.5.1
      - ts-jest@29.1.1
      - typescript-cubic-spline@1.0.1
      - typescript@5.2.2
      - winston@3.11.0
      - zod@3.22.4
  status: success
tree:
  children:
    child-1:
      pipeline:
        - sci
      config: null
      inputs:
        - timestamp: 2023-07-06T00:00
          duration: 3600
          energy: 5
          carbon-operational: 5
          carbon-embodied: 0.02
          carbon: 5.02
          requests: 100
      outputs:
        - timestamp: 2023-07-06T00:00
          duration: 3600
          energy: 5
          carbon-operational: 5
          carbon-embodied: 0.02
          carbon: 5.02
          requests: 100
          sci: 0.020199999999999995
    child-2:
      pipeline:
        - sci
      config: null
      inputs:
        - timestamp: 2023-07-06T00:00
          duration: 3600
          energy: 8
          carbon-operational: 8
          carbon-embodied: 0.02
          carbon: 8.02
          requests: 100
      outputs:
        - timestamp: 2023-07-06T00:00
          duration: 3600
          energy: 8
          carbon-operational: 8
          carbon-embodied: 0.02
          carbon: 8.02
          requests: 100
          sci: 0.0102

Now, when Alice runs if-check -m charlies-manifest, she receives:

if-check could not verify charlies-manifest. The re-executed file does not match the original.

Files do not match!
tree.children.child-1.outputs.0.sci
source: 0.050199999999999995
target: 0.020199999999999995

Not only can Alice see that the files do not match, she can see which values Charlie manipulated.

Running IF over multiple manifests​

Alice could also run if-check over any number of manifests in a single command, using the -d subcommand. For a folder containing n manifests, pass the folder path:

if-check -d /my-folder-of-manifests

Each manifest will be run through if-check in sequence.

if-check limitations​

if-check can verify that a manifest is correctly calculated. However, if someone really wanted to use a fraudulent manifest, they could provide fraudulent input data not output data. There's little we can really do about this - if someone provides fake input data it is out of IF's remit. This means that although the examples above are good for demonstrating how if-check works, it's more likely to be used to check for bugs and configuration errors than it is to be used to detect fraud.

- +
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

Verifying IF outputs with if-check

IF includes a command line tool called if-check that can be used to verify the results in a manifest file.

Imagine that someone provides you with a computed manifest file that they claim demonstrates the environmental impact of their software application.

You could trust them and take their results at face value, but there are probably cases where you think someone might have made a mistake or might be massaging their data in a dishonest way. Maybe you just need to demonstrate due diligence in the quality of the data you receive and use. In these cases, you can make use of if-check.

if-check is a single command that takes a given manifest file, sets up an environment where it can be executed, executes it, and compares the newly generated results to those in the original file. This means you can independently verify the results in the file using your own local copy of IF.

Under the hood, if-check is wrapping calls to if-env and if-diff, so if you need granular control over the information flow for some reason, you could achieve the same result using separate calls to those commands.

Example​

Alice is a manifest verifier. She receives a manifest from Bob. Bob is a good actor and has provided a valid file. Alice is a good cypherpunk who verifies everything.

Here is Bob's manifest:

name: bobs-manifest
description:
tags: null
initialize:
  plugins:
    sci:
      path: builtin
      method: Sci
      global-config:
        functional-unit: requests
execution:
  command: >-
    /home/bob/.npm/_npx/1bf7c3c15bf47d04/node_modules/.bin/ts-node
    /home/bob/Code/if/src/index.ts -m manifests/plugins/sci/success.yml -s
  environment:
    if-version: 0.4.0
    os: linux
    os-version: 5.15.0-113-generic
    node-version: 21.4.0
    date-time: 2024-06-27T14:10:29.830Z (UTC)
    dependencies:
      - '@babel/core@7.22.10'
      - '@babel/preset-typescript@7.23.3'
      - '@commitlint/cli@18.6.0'
      - '@commitlint/config-conventional@18.6.0'
      - '@grnsft/if-core@0.0.9'
      - '@jest/globals@29.7.0'
      - '@types/jest@29.5.8'
      - '@types/js-yaml@4.0.9'
      - '@types/luxon@3.4.2'
      - '@types/node@20.9.0'
      - axios-mock-adapter@1.22.0
      - axios@1.7.2
      - cross-env@7.0.3
      - csv-parse@5.5.6
      - csv-stringify@6.4.6
      - fixpack@4.0.0
      - gts@5.2.0
      - husky@8.0.3
      - jest@29.7.0
      - js-yaml@4.1.0
      - lint-staged@15.2.2
      - luxon@3.4.4
      - release-it@16.3.0
      - rimraf@5.0.5
      - ts-command-line-args@2.5.1
      - ts-jest@29.1.1
      - typescript-cubic-spline@1.0.1
      - typescript@5.2.2
      - winston@3.11.0
      - zod@3.22.4
  status: success
tree:
  children:
    child-1:
      pipeline:
        - sci
      config: null
      inputs:
        - timestamp: 2023-07-06T00:00
          duration: 3600
          energy: 5
          carbon-operational: 5
          carbon-embodied: 0.02
          carbon: 5.02
          requests: 100
      outputs:
        - timestamp: 2023-07-06T00:00
          duration: 3600
          energy: 5
          carbon-operational: 5
          carbon-embodied: 0.02
          carbon: 5.02
          requests: 100
          sci: 0.050199999999999995
    child-2:
      pipeline:
        - sci
      config: null
      inputs:
        - timestamp: 2023-07-06T00:00
          duration: 3600
          energy: 8
          carbon-operational: 8
          carbon-embodied: 0.02
          carbon: 8.02
          requests: 100
      outputs:
        - timestamp: 2023-07-06T00:00
          duration: 3600
          energy: 8
          carbon-operational: 8
          carbon-embodied: 0.02
          carbon: 8.02
          requests: 100
          sci: 0.0802

Alice runs :

if-check -m bobs-manifest.yml

And receives the response:

if-check: successfully verified bobs-manifest

Charlie also has a copy of Bob's manifest. He wants to trick Alice into thinking his SCI score is lower, so he overwrites the values in the manifest file, making them lower. Charlie's manifest looks like this:

# start
name: charlies-manifest
description:
tags: null
initialize:
  plugins:
    sci:
      path: builtin
      method: Sci
      global-config:
        functional-unit: requests
execution:
  command: >-
    /home/charlie/.npm/_npx/1bf7c3c15bf47d04/node_modules/.bin/ts-node
    /home/charlie/Code/if/src/index.ts -m manifests/plugins/sci/success.yml -s
  environment:
    if-version: 0.4.0
    os: linux
    os-version: 5.15.0-113-generic
    node-version: 21.4.0
    date-time: 2024-06-27T14:10:29.830Z (UTC)
    dependencies:
      - '@babel/core@7.22.10'
      - '@babel/preset-typescript@7.23.3'
      - '@commitlint/cli@18.6.0'
      - '@commitlint/config-conventional@18.6.0'
      - '@grnsft/if-core@0.0.9'
      - '@jest/globals@29.7.0'
      - '@types/jest@29.5.8'
      - '@types/js-yaml@4.0.9'
      - '@types/luxon@3.4.2'
      - '@types/node@20.9.0'
      - axios-mock-adapter@1.22.0
      - axios@1.7.2
      - cross-env@7.0.3
      - csv-parse@5.5.6
      - csv-stringify@6.4.6
      - fixpack@4.0.0
      - gts@5.2.0
      - husky@8.0.3
      - jest@29.7.0
      - js-yaml@4.1.0
      - lint-staged@15.2.2
      - luxon@3.4.4
      - release-it@16.3.0
      - rimraf@5.0.5
      - ts-command-line-args@2.5.1
      - ts-jest@29.1.1
      - typescript-cubic-spline@1.0.1
      - typescript@5.2.2
      - winston@3.11.0
      - zod@3.22.4
  status: success
tree:
  children:
    child-1:
      pipeline:
        - sci
      config: null
      inputs:
        - timestamp: 2023-07-06T00:00
          duration: 3600
          energy: 5
          carbon-operational: 5
          carbon-embodied: 0.02
          carbon: 5.02
          requests: 100
      outputs:
        - timestamp: 2023-07-06T00:00
          duration: 3600
          energy: 5
          carbon-operational: 5
          carbon-embodied: 0.02
          carbon: 5.02
          requests: 100
          sci: 0.020199999999999995
    child-2:
      pipeline:
        - sci
      config: null
      inputs:
        - timestamp: 2023-07-06T00:00
          duration: 3600
          energy: 8
          carbon-operational: 8
          carbon-embodied: 0.02
          carbon: 8.02
          requests: 100
      outputs:
        - timestamp: 2023-07-06T00:00
          duration: 3600
          energy: 8
          carbon-operational: 8
          carbon-embodied: 0.02
          carbon: 8.02
          requests: 100
          sci: 0.0102

Now, when Alice runs if-check -m charlies-manifest, she receives:

if-check could not verify charlies-manifest. The re-executed file does not match the original.

Files do not match!
tree.children.child-1.outputs.0.sci
source: 0.050199999999999995
target: 0.020199999999999995

Not only can Alice see that the files do not match, she can see which values Charlie manipulated.

Running IF over multiple manifests​

Alice could also run if-check over any number of manifests in a single command, using the -d subcommand. For a folder containing n manifests, pass the folder path:

if-check -d /my-folder-of-manifests

Each manifest will be run through if-check in sequence.

if-check limitations​

if-check can verify that a manifest is correctly calculated. However, if someone really wanted to use a fraudulent manifest, they could provide fraudulent input data not output data. There's little we can really do about this - if someone provides fake input data it is out of IF's remit. This means that although the examples above are good for demonstrating how if-check works, it's more likely to be used to check for bugs and configuration errors than it is to be used to detect fraud.

+ \ No newline at end of file diff --git a/users/how-to-write-manifests/index.html b/users/how-to-write-manifests/index.html index f0a186cd..8b5cfbf4 100644 --- a/users/how-to-write-manifests/index.html +++ b/users/how-to-write-manifests/index.html @@ -8,14 +8,14 @@ How to write a manifest file | Impact Framework - +
-
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

How to write a manifest file

The Impact Framework receives all its configuration and input data in the form of a manifest file known as an manifest. To use the framework, you will need to write a manifest file and pass its path to the command line tool. This guide will help you to understand how to construct one of these files and use it to measure the energy and carbon usage of your app.

Structure of a manifest​

The basic structure of a manifest is as follows: 

name: 
description: 
tags: 
initialize:
  plugins:
    <PLUGIN-NAME-HERE>: 
      method:
      path: 
tree:
  children:
    child:
      pipeline:
        - 
      config:
      defaults:
      inputs:

Project metadata​

The file starts with some metadata about the project. There are no strict specifications for what to put in these fields, they are for you to keep track of your manifest files and to help other users to understand your use case.

name:
description:
tags:

Initialize​

The initialize fields are where you specify each individual plugin that will be initialized in your pipeline. The plugins can be initialized in any order, but can only be invoked elsewhere in the manifest if they have been initialized first here. In each case, you will need to provide the name, path and method (and global-config if your plugin requires it):

initialize:
  plugins:
    sci-m:
      path: ''
      method:
  • The name is the name you want this plugin instance to be recognized as by Impact Framework.
  • The path defines where IF should look for the installed plugin. For example, for our standard library of plugins you would specify builtin, for other installed plugins you use the name of the directory they are installed into in node_modules.
  • For the method field, you should provide the name of the function exported by your plugin. For example, for the sum plugin, the correct value is Sum.

Tree​

The tree fields are where you define the various components of your application. Each component is defined as children, where each child's output is summed to give the overall impact. Each child can have its own plugin pipeline and its own configuration, but when none is provided, it is inherited from the tree-level configuration.

In the following example, there is only one component but the plugin pipeline contains two plugins; teads-curve and sci-m. Neither requires any config data, but certain information is required in inputs.

tree:
  children:
    child:
      pipeline:
        - teads-curve
        - sci-m
      config:
      defaults:
      inputs:
        - timestamp: '2023-11-02T10:35:31.820Z'
          duration: 3600
          total-embodied-emissions: 1533.12
          time-reserved: 1
          expected-lifespan: 3
          resources-reserved: 1
          total-resources: 8

Inputs​

The most granular level of the manifest file are the inputs. This is where you can add specific data for each child. Inputs must always include a timestamp and a duration.

inputs:
  - timestamp: 2023-07-06T00:00
    duration: 3600
    cpu-util: 45

You now have a simple manifest file that will use the plugin config and input data to run the teads-curve and sci-m plugins. The output data will be appended to the manifest under a new outputs field and saved as an output file.

More complex manifests​

Complex pipelines​

Whilst the manifest file we looked at above works perfectly well, it will only return the most basic output data. Most users will want to calculate an SCI score, which implies a number of additional steps:

  • operational-carbon and embodied-carbon must appear as inputs.
  • This means that sci will need to be preceded by sci-m and sci-o in the plugin pipeline.
  • In most cases, sci-o will have to be preceded by sci-e to ensure energy is available to be piped to sci-o.
  • The inputs to sci-e will most likely be coming from a plugin such as teads-curve or boavizta.
  • The sci plugin also requires functional-unit information so it can convert the estimated carbon into a useful unit.
  • You may also wish to grab your input data by querying a metrics API on a virtual machine.

The example below gives you the full pipeline implemented in a manifest. There are also several other executable example manifests in if/manifests/examples that you can run for yourself.

name: pipeline-with-aggregate
description: a full pipeline with the aggregate feature enabled
tags:
aggregation:
  metrics:
    - "carbon"
  type: "both"
initialize:
  plugins:
    "interpolate":
      method: Interpolation
      path: 'builtin'
      global-config:
        method: linear
        x: [0, 10, 50, 100]
        y: [0.12, 0.32, 0.75, 1.02]
        input-parameter: 'cpu/utilization'
        output-parameter: 'cpu-factor'
    "cpu-factor-to-wattage":
      method: Multiply
      path: builtin
      global-config:
        input-parameters: ["cpu-factor", "cpu/thermal-design-power"]
        output-parameter: "cpu-wattage"
    "wattage-times-duration":
      method: Multiply
      path: builtin
      global-config:
        input-parameters: ["cpu-wattage", "duration"]
        output-parameter: "cpu-wattage-times-duration"
    "wattage-to-energy-kwh":
      method: Divide
      path: "builtin"
      global-config:
        numerator: cpu-wattage-times-duration
        denominator: 3600000
        output: cpu-energy-raw
    "calculate-vcpu-ratio":
      method: Divide
      path: "builtin"
      global-config:
        numerator: vcpus-total
        denominator: vcpus-allocated
        output: vcpu-ratio
    "correct-cpu-energy-for-vcpu-ratio":
      method: Divide
      path: "builtin"
      global-config:
        numerator: cpu-energy-raw
        denominator: vcpu-ratio
        output: cpu-energy-kwh
    "sci-embodied":
      path: "builtin"
      method: SciEmbodied
    "operational-carbon":
      method: Multiply
      path: builtin
      global-config:
        input-parameters: ["cpu-energy-kwh", "grid/carbon-intensity"]
        output-parameter: "carbon-operational"
    "sci":
      path: "builtin"
      method: Sci
      global-config:
        functional-unit-time: 1 sec
        functional-unit: requests # factor to convert per time to per f.unit
    "sum-carbon":
      path: "builtin"
      method: Sum
      global-config:
        input-parameters:
          - carbon-operational
          - carbon-embodied
        output-parameter: carbon
    "time-sync":
      method: TimeSync
      path: "builtin"
      global-config:
        start-time: "2023-12-12T00:00:00.000Z"
        end-time: "2023-12-12T00:01:00.000Z"
        interval: 5
        allow-padding: true
tree:
  children:
    child-1:
      pipeline:
        - interpolate
        - cpu-factor-to-wattage
        - wattage-times-duration
        - wattage-to-energy-kwh
        - calculate-vcpu-ratio
        - correct-cpu-energy-for-vcpu-ratio
        - sci-embodied
        - operational-carbon
        - sum-carbon
        - time-sync
        # - sci
      config:
        group-by:
          group:
            - cloud/region
            - cloud/instance-type
      defaults:
        cpu/thermal-design-power: 100
        grid/carbon-intensity: 800
        device/emissions-embodied: 1533.120 # gCO2eq
        time-reserved: 3600 # 1hr in seconds
        device/expected-lifespan: 94608000 # 3 years in seconds
        vcpus-total: 8
        vcpus-allocated: 1
      inputs:
        - timestamp: "2023-12-12T00:00:00.000Z"
          cloud/instance-type: A1
          cloud/region: uk-west
          duration: 1
          cpu/utilization: 10
          requests: 10
        - timestamp: "2023-12-12T00:00:01.000Z"
          duration: 5
          cpu/utilization: 20
          cloud/instance-type: A1
          cloud/region: uk-west
          requests: 5
        - timestamp: "2023-12-12T00:00:06.000Z"
          duration: 7
          cpu/utilization: 15
          cloud/instance-type: A1
          cloud/region: uk-west
          requests: 15
        - timestamp: "2023-12-12T00:00:13.000Z"
          duration: 30
          cloud/instance-type: A1
          cloud/region: uk-west
          cpu/utilization: 15
          requests: 30

Complex applications​

The manifest examples provided so far have only had a single component. However, Impact Framework can handle any number of nested children.

In this way, you can combine complex plugin pipelines and application architectures to calculate the energy and carbon outputs of complicated systems.

Choosing which plugins to run​

The plugins are designed to be composable, but they each have specific input requirements that must be met in order for the plugins to run correctly. For example, the teads-curve plugin requires cpu/thermal-design-power to be available in the manifest. If it is not there, the plugin cannot use it to calculate cpu/energy.

It is also possible to leapfrog some plugins if you have access to high-level data. For example, perhaps you already know the energy being used by your CPU. In this case, there is no need to run teads-curve, you can simply provide cpu/energy as an input and omit teads-curve from the plugin pipeline.

We have deliberately made the plugins modular and composable so that you can be creative in developing new plugins to replace those provided as part of IF.

Adding real-life inputs​

The examples above already include inputs for the components. However, you may want to input real-life data into the manifest file.

There is no one-size-fits-all solution for getting data into the manifest file. This is because there are so many possible sources for your input data, all of which have their own particular requirements related to authorization, API request syntax and return types. Therefore, the approach taken by IF is to have specific plugins for specific services.

The recommended method for integrating data is to use the plugin system of the Impact Framework. You can either use an existing specific importer plugin or write your own.

There are already some community plugins available, including plugins for fetching data from Kubernetes, GCP, and third-party data aggregators like Datadog.

If there is no fitting plugin available yet, we encourage you to write and add one for your specific use case. See developer documentation for more information on how to build a plugin. There is a Azure-Importer you can as a prototype and starting point for your own development. -If you already have external scripts you might have a look at the shell plugin to integrate them with the Impact Framework.

If you just need data for testing purposes, you can use the mock-observation plugin.

Running a manifest​

You run a manifest by providing its path to our command line tool and a path to save the results file to. You can run a manifest named my-manifest.yml using the following command:

if-run --manifest my-manifest.yml
- +
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

How to write a manifest file

The Impact Framework receives all its configuration and input data in the form of a manifest file known as an manifest. To use the framework, you will need to write a manifest file and pass its path to the command line tool. This guide will help you to understand how to construct one of these files and use it to measure the energy and carbon usage of your app.

Structure of a manifest​

The basic structure of a manifest is as follows: 

name: 
description: 
tags: 
initialize:
  plugins:
    <PLUGIN-NAME-HERE>: 
      method:
      path: 
tree:
  children:
    child:
      pipeline:
        - 
      config:
      defaults:
      inputs:

Project metadata​

The file starts with some metadata about the project. There are no strict specifications for what to put in these fields, they are for you to keep track of your manifest files and to help other users to understand your use case.

name:
description:
tags:

Initialize​

The initialize fields are where you specify each individual plugin that will be initialized in your pipeline. The plugins can be initialized in any order, but can only be invoked elsewhere in the manifest if they have been initialized first here. In each case, you will need to provide the name, path and method (and global-config if your plugin requires it):

initialize:
  plugins:
    sci-m:
      path: ''
      method:
  • The name is the name you want this plugin instance to be recognized as by Impact Framework.
  • The path defines where IF should look for the installed plugin. For example, for our standard library of plugins you would specify builtin, for other installed plugins you use the name of the directory they are installed into in node_modules.
  • For the method field, you should provide the name of the function exported by your plugin. For example, for the sum plugin, the correct value is Sum.

Tree​

The tree fields are where you define the various components of your application. Each component is defined as children, where each child's output is summed to give the overall impact. Each child can have its own plugin pipeline and its own configuration, but when none is provided, it is inherited from the tree-level configuration.

In the following example, there is only one component but the plugin pipeline contains two plugins; teads-curve and sci-m. Neither requires any config data, but certain information is required in inputs.

tree:
  children:
    child:
      pipeline:
        - teads-curve
        - sci-m
      config:
      defaults:
      inputs:
        - timestamp: '2023-11-02T10:35:31.820Z'
          duration: 3600
          total-embodied-emissions: 1533.12
          time-reserved: 1
          expected-lifespan: 3
          resources-reserved: 1
          total-resources: 8

Inputs​

The most granular level of the manifest file are the inputs. This is where you can add specific data for each child. Inputs must always include a timestamp and a duration.

inputs:
  - timestamp: 2023-07-06T00:00
    duration: 3600
    cpu-util: 45

You now have a simple manifest file that will use the plugin config and input data to run the teads-curve and sci-m plugins. The output data will be appended to the manifest under a new outputs field and saved as an output file.

More complex manifests​

Complex pipelines​

Whilst the manifest file we looked at above works perfectly well, it will only return the most basic output data. Most users will want to calculate an SCI score, which implies a number of additional steps:

  • operational-carbon and embodied-carbon must appear as inputs.
  • This means that sci will need to be preceded by sci-m and sci-o in the plugin pipeline.
  • In most cases, sci-o will have to be preceded by sci-e to ensure energy is available to be piped to sci-o.
  • The inputs to sci-e will most likely be coming from a plugin such as teads-curve or boavizta.
  • The sci plugin also requires functional-unit information so it can convert the estimated carbon into a useful unit.
  • You may also wish to grab your input data by querying a metrics API on a virtual machine.

The example below gives you the full pipeline implemented in a manifest. There are also several other executable example manifests in if/manifests/examples that you can run for yourself.

name: pipeline-with-aggregate
description: a full pipeline with the aggregate feature enabled
tags:
aggregation:
  metrics:
    - "carbon"
  type: "both"
initialize:
  plugins:
    "interpolate":
      method: Interpolation
      path: 'builtin'
      global-config:
        method: linear
        x: [0, 10, 50, 100]
        y: [0.12, 0.32, 0.75, 1.02]
        input-parameter: 'cpu/utilization'
        output-parameter: 'cpu-factor'
    "cpu-factor-to-wattage":
      method: Multiply
      path: builtin
      global-config:
        input-parameters: ["cpu-factor", "cpu/thermal-design-power"]
        output-parameter: "cpu-wattage"
    "wattage-times-duration":
      method: Multiply
      path: builtin
      global-config:
        input-parameters: ["cpu-wattage", "duration"]
        output-parameter: "cpu-wattage-times-duration"
    "wattage-to-energy-kwh":
      method: Divide
      path: "builtin"
      global-config:
        numerator: cpu-wattage-times-duration
        denominator: 3600000
        output: cpu-energy-raw
    "calculate-vcpu-ratio":
      method: Divide
      path: "builtin"
      global-config:
        numerator: vcpus-total
        denominator: vcpus-allocated
        output: vcpu-ratio
    "correct-cpu-energy-for-vcpu-ratio":
      method: Divide
      path: "builtin"
      global-config:
        numerator: cpu-energy-raw
        denominator: vcpu-ratio
        output: cpu-energy-kwh
    "sci-embodied":
      path: "builtin"
      method: SciEmbodied
    "operational-carbon":
      method: Multiply
      path: builtin
      global-config:
        input-parameters: ["cpu-energy-kwh", "grid/carbon-intensity"]
        output-parameter: "carbon-operational"
    "sci":
      path: "builtin"
      method: Sci
      global-config:
        functional-unit-time: 1 sec
        functional-unit: requests # factor to convert per time to per f.unit
    "sum-carbon":
      path: "builtin"
      method: Sum
      global-config:
        input-parameters:
          - carbon-operational
          - carbon-embodied
        output-parameter: carbon
    "time-sync":
      method: TimeSync
      path: "builtin"
      global-config:
        start-time: "2023-12-12T00:00:00.000Z"
        end-time: "2023-12-12T00:01:00.000Z"
        interval: 5
        allow-padding: true
tree:
  children:
    child-1:
      pipeline:
        - interpolate
        - cpu-factor-to-wattage
        - wattage-times-duration
        - wattage-to-energy-kwh
        - calculate-vcpu-ratio
        - correct-cpu-energy-for-vcpu-ratio
        - sci-embodied
        - operational-carbon
        - sum-carbon
        - time-sync
        # - sci
      config:
        group-by:
          group:
            - cloud/region
            - cloud/instance-type
      defaults:
        cpu/thermal-design-power: 100
        grid/carbon-intensity: 800
        device/emissions-embodied: 1533.120 # gCO2eq
        time-reserved: 3600 # 1hr in seconds
        device/expected-lifespan: 94608000 # 3 years in seconds
        vcpus-total: 8
        vcpus-allocated: 1
      inputs:
        - timestamp: "2023-12-12T00:00:00.000Z"
          cloud/instance-type: A1
          cloud/region: uk-west
          duration: 1
          cpu/utilization: 10
          requests: 10
        - timestamp: "2023-12-12T00:00:01.000Z"
          duration: 5
          cpu/utilization: 20
          cloud/instance-type: A1
          cloud/region: uk-west
          requests: 5
        - timestamp: "2023-12-12T00:00:06.000Z"
          duration: 7
          cpu/utilization: 15
          cloud/instance-type: A1
          cloud/region: uk-west
          requests: 15
        - timestamp: "2023-12-12T00:00:13.000Z"
          duration: 30
          cloud/instance-type: A1
          cloud/region: uk-west
          cpu/utilization: 15
          requests: 30

Complex applications​

The manifest examples provided so far have only had a single component. However, Impact Framework can handle any number of nested children.

In this way, you can combine complex plugin pipelines and application architectures to calculate the energy and carbon outputs of complicated systems.

Choosing which plugins to run​

The plugins are designed to be composable, but they each have specific input requirements that must be met in order for the plugins to run correctly. For example, the teads-curve plugin requires cpu/thermal-design-power to be available in the manifest. If it is not there, the plugin cannot use it to calculate cpu/energy.

It is also possible to leapfrog some plugins if you have access to high-level data. For example, perhaps you already know the energy being used by your CPU. In this case, there is no need to run teads-curve, you can simply provide cpu/energy as an input and omit teads-curve from the plugin pipeline.

We have deliberately made the plugins modular and composable so that you can be creative in developing new plugins to replace those provided as part of IF.

Adding real-life inputs​

The examples above already include inputs for the components. However, you may want to input real-life data into the manifest file.

There is no one-size-fits-all solution for getting data into the manifest file. This is because there are so many possible sources for your input data, all of which have their own particular requirements related to authorization, API request syntax and return types. Therefore, the approach taken by IF is to have specific plugins for specific services.

The recommended method for integrating data is to use the plugin system of the Impact Framework. You can either use an existing specific importer plugin or write your own.

There are already some community plugins available, including plugins for fetching data from Kubernetes, GCP, and third-party data aggregators like Datadog.

If there is no fitting plugin available yet, we encourage you to write and add one for your specific use case. See developer documentation for more information on how to build a plugin. There is a Azure-Importer you can as a prototype and starting point for your own development. +If you already have external scripts you might have a look at the shell plugin to integrate them with the Impact Framework.

If you just need data for testing purposes, you can use the mock-observation plugin.

Running a manifest​

You run a manifest by providing its path to our command line tool and a path to save the results file to. You can run a manifest named my-manifest.yml using the following command:

if-run --manifest my-manifest.yml
+ \ No newline at end of file diff --git a/users/index.html b/users/index.html index f475d918..855b34af 100644 --- a/users/index.html +++ b/users/index.html @@ -8,13 +8,13 @@ Users | Impact Framework - +
-
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

Users

This section contains information for Impact Framework users. You are a user if you want to apply the Impact Framework to your own use-case, such as using it to measure the environmental impact of your own apps running on some cloud platform.

The user documentation includes:

If you are looking for guidance for how to change or update the Impact Framework by adding new features, fixing bugs or building new plugins, you should go to our developers documentation instead.

- +
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

Users

This section contains information for Impact Framework users. You are a user if you want to apply the Impact Framework to your own use-case, such as using it to measure the environmental impact of your own apps running on some cloud platform.

The user documentation includes:

If you are looking for guidance for how to change or update the Impact Framework by adding new features, fixing bugs or building new plugins, you should go to our developers documentation instead.

+ \ No newline at end of file diff --git a/users/quick-start/index.html b/users/quick-start/index.html index 80a1d9b1..766ed680 100644 --- a/users/quick-start/index.html +++ b/users/quick-start/index.html @@ -8,13 +8,13 @@ Quick start | Impact Framework - +
-
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

Quick start

This page will provide the basic instructions for getting up and running with Impact Framework.

1: Install Impact Framework​

Install the Impact Framework globally using npm.

npm install -g @grnsft/if

Read our detailed guide to installing IF.

2: Install some plugins​

Install some of the plugins you want to include in your pipeline. The following commands will install both the official and unofficial IF model packages.

npm install -g @grnsft/if-plugins
npm install -g @grnsft/if-unofficial-plugins

Read our detailed guide to loading plugins.

3: Create a manifest file​

A manifest file contains all the configuration and input data required to measure your application's energy and carbon impacts and should have a .yml extension.

Open the file, add your data and save the file. The simple example below runs a single snapshot observation through a single plugin.

name: basic-demo
description:
tags:
initialize:
  plugins:
    teads-curve: 
      path: '@grnsft/if-unofficial-plugins'
      method: TeadsCurve
      global-config:
        interpolation: spline
tree:
  children:
    child-0:
      defaults:
        cpu/thermal-design-power: 100
      pipeline:
        - teads-curve
      inputs:
        - timestamp: 2023-07-06T00:00
          duration: 1
          cpu/utilization: 20
        - timestamp: 2023-07-06T00:01
          duration: 1
          cpu/utilization: 80
        - timestamp: 2023-07-06T00:02
          duration: 1
          cpu/utilization: 20

Read our detailed guide to writing manifest files.

4: Compute your manifest file​

Run the pipeline by passing the path to your manifest file to the if-run command line tool:

if-run --manifest <path-to-your-manifest>

🎉Congratulations 🎉! You have just used the Impact Framework to compute the energy consumed by an application!

Next steps​

Now you know how to use the if-run you can start building more complex pipelines of plugins and more complicated manifest files. Your overall aim is to create a manifest file that accurately represents a real software application, and a plugin pipeline that yields an environmental metric that's important to you (e.g. carbon).

Experiment by adding more plugins to the pipeline, for example add sci-o to convert energy into operational-carbon. Your output data will be displayed in your console.

You can also configure if to save your output data to another yaml file. To do this, add the --output flag and the path to the output file where the results are saved.

The command is then as follows:

if-run --manifest <path-to-your-impl> --output <save-path>

Explore our user documentation for walkthrough guides to common Impact Framework tasks:

- +
This project is an incubation project being run inside the Green Software Foundation; as such, we DON’T recommend using it in any critical use case. Incubation projects are experimental, offer no support guarantee, have minimal governance and process, and may be retired at any moment. This project may one day Graduate, in which case this disclaimer will be removed.

Quick start

This page will provide the basic instructions for getting up and running with Impact Framework.

1: Install Impact Framework​

Install the Impact Framework globally using npm.

npm install -g @grnsft/if

Read our detailed guide to installing IF.

2: Install some plugins​

Install some of the plugins you want to include in your pipeline. The following commands will install both the official and unofficial IF model packages.

npm install -g @grnsft/if-plugins
npm install -g @grnsft/if-unofficial-plugins

Read our detailed guide to loading plugins.

3: Create a manifest file​

A manifest file contains all the configuration and input data required to measure your application's energy and carbon impacts and should have a .yml extension.

Open the file, add your data and save the file. The simple example below runs a single snapshot observation through a single plugin.

name: basic-demo
description:
tags:
initialize:
  plugins:
    teads-curve: 
      path: '@grnsft/if-unofficial-plugins'
      method: TeadsCurve
      global-config:
        interpolation: spline
tree:
  children:
    child-0:
      defaults:
        cpu/thermal-design-power: 100
      pipeline:
        - teads-curve
      inputs:
        - timestamp: 2023-07-06T00:00
          duration: 1
          cpu/utilization: 20
        - timestamp: 2023-07-06T00:01
          duration: 1
          cpu/utilization: 80
        - timestamp: 2023-07-06T00:02
          duration: 1
          cpu/utilization: 20

Read our detailed guide to writing manifest files.

4: Compute your manifest file​

Run the pipeline by passing the path to your manifest file to the if-run command line tool:

if-run --manifest <path-to-your-manifest>

🎉Congratulations 🎉! You have just used the Impact Framework to compute the energy consumed by an application!

Next steps​

Now you know how to use the if-run you can start building more complex pipelines of plugins and more complicated manifest files. Your overall aim is to create a manifest file that accurately represents a real software application, and a plugin pipeline that yields an environmental metric that's important to you (e.g. carbon).

Experiment by adding more plugins to the pipeline, for example add sci-o to convert energy into operational-carbon. Your output data will be displayed in your console.

You can also configure if to save your output data to another yaml file. To do this, add the --output flag and the path to the output file where the results are saved.

The command is then as follows:

if-run --manifest <path-to-your-impl> --output <save-path>

Explore our user documentation for walkthrough guides to common Impact Framework tasks:

+ \ No newline at end of file