-
Notifications
You must be signed in to change notification settings - Fork 1
/
plugins.html
executable file
·379 lines (265 loc) · 24.1 KB
/
plugins.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Plugins — Rumal 0.0.1 alpha documentation</title>
<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
<link rel="top" title="Rumal 0.0.1 alpha documentation" href="index.html"/>
<link rel="next" title="Development" href="development.html"/>
<link rel="prev" title="Advanced Search" href="advancedsearch.html"/>
<script src="_static/js/modernizr.min.js"></script>
</head>
<body class="wy-body-for-nav" role="document">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search">
<a href="index.html" class="icon icon-home"> Rumal
</a>
<div class="version">
0.0.1 alpha
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">
<input type="text" name="q" placeholder="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div>
<div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="intro.html">Introduction</a></li>
<li class="toctree-l1"><a class="reference internal" href="architecture.html">Architecture</a></li>
<li class="toctree-l1"><a class="reference internal" href="install.html">Installation</a></li>
<li class="toctree-l1"><a class="reference internal" href="docker.html">Docker</a></li>
<li class="toctree-l1"><a class="reference internal" href="quickstart.html">Quickstart</a></li>
<li class="toctree-l1"><a class="reference internal" href="advancedusage.html">Advanced Usage</a></li>
<li class="toctree-l1"><a class="reference internal" href="social.html">Rumal Social</a></li>
<li class="toctree-l1"><a class="reference internal" href="advancedsearch.html">Advanced Search</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="#">Plugins</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#what-and-why">What and Why?</a></li>
<li class="toctree-l2"><a class="reference internal" href="#adding-removing-plugins">Adding Removing Plugins</a></li>
<li class="toctree-l2"><a class="reference internal" href="#how-plugins-work">How Plugins work?</a></li>
<li class="toctree-l2"><a class="reference internal" href="#writing-your-own-plugins">Writing your own Plugins</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="development.html">Development</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="index.html">Rumal</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="breadcrumbs navigation">
<ul class="wy-breadcrumbs">
<li><a href="index.html">Docs</a> »</li>
<li>Plugins</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<div class="section" id="plugins">
<h1>Plugins<a class="headerlink" href="#plugins" title="Permalink to this headline">¶</a></h1>
<div class="section" id="what-and-why">
<h2>What and Why?<a class="headerlink" href="#what-and-why" title="Permalink to this headline">¶</a></h2>
<p>One of Rumal’s major strength is its ability to augment the information generated by Thug. This enrichment is performed on the fron-end
server by making use of Plugins. The additional information is often obtained by sending the current one (say domain name) to fetch some
extra information (IP Address). Rumal’s GUI has a separate panel for displaying this augmented information.</p>
<p>As of now Rumal comes bundled with a WhoisPlugin and GeoPlugin by default. The WhoisPlugin fetches the WhoisInfo while the GeoPlugin tries
to match the IP Address obtained on back-end to locations on the Earth’s map. Work is in progress to expand the set of default plugins to add
more useful information by default.</p>
</div>
<div class="section" id="adding-removing-plugins">
<h2>Adding Removing Plugins<a class="headerlink" href="#adding-removing-plugins" title="Permalink to this headline">¶</a></h2>
<p>Adding and Removing plugins is a straight forward procedure. To add a plugin you just need to drop its .py file into the plugins directory of
the frontend (/rumal/interface/plugins/). This section on advanced usage has details regarding how to configure the plugins.</p>
<div class="admonition warning">
<p class="first admonition-title">Warning</p>
<ul class="last simple">
<li>Removing the default plugins can make Rumal unstable. They must not be modified.</li>
<li>All plugins should have a config file even if they are blank. The name of the config files should not be changed.</li>
</ul>
</div>
</div>
<div class="section" id="how-plugins-work">
<h2>How Plugins work?<a class="headerlink" href="#how-plugins-work" title="Permalink to this headline">¶</a></h2>
<p>Plugins come in the workflow once the scan details are retrieved from the backend. The bson object stored for the
scan is then modified to add details. The bson has a key named “Plugins” under which all the generated plugin info
is stored. For example a the default WhoisPlugin saves its info under. “flat_tree_node” => “some_node” =>”WhoisPlugin” key.</p>
<p>The plugin system works in the form of a pipeline and it is possible to add plugins such that they are dependent on each other. Rumal automatically finds out the correct order in which plugins need to be run given that dependencies are correctly specified in the .py file of the plugins.</p>
<p>The UI has a plugins panel which displays information by parsing the same file and automatically displays the json of the info stored in the particular plugin’s key.</p>
</div>
<div class="section" id="writing-your-own-plugins">
<h2>Writing your own Plugins<a class="headerlink" href="#writing-your-own-plugins" title="Permalink to this headline">¶</a></h2>
<p>Rumal Plugins are derived from the abstract class PluginBase found in rumal/interface/plug.py file. PluginBase
class has been impemented using Python’s abc module. To get a better understanding of the implementation we recommend the excellent tutorial at <a class="reference external" href="https://pymotw.com/2/abc/">https://pymotw.com/2/abc/</a></p>
<p>The methods with @abc.abstractproperty and @abc.abstractmethod need to be overridden while implementing the Plugin class. Below is a walk-through of how the GeoPlugin was implemented.</p>
<p>Relevant code from PluginBase. Implementation of some methods has been omitted.</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">PluginBase</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="n">__metaclass__</span> <span class="o">=</span> <span class="n">abc</span><span class="o">.</span><span class="n">ABCMeta</span>
<span class="nd">@abc</span><span class="o">.</span><span class="n">abstractproperty</span>
<span class="k">def</span> <span class="nf">dependencies</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""List of dependencies - put class names of required plugins here.</span>
<span class="sd"> Return blank list if none."""</span>
<span class="k">return</span> <span class="p">[]</span>
<span class="nd">@abc</span><span class="o">.</span><span class="n">abstractproperty</span>
<span class="k">def</span> <span class="nf">module_dependencies</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""List of non-standard lib module dependencies</span>
<span class="sd"> - put module names as keys and versions as</span>
<span class="sd"> values.Return blank list if none."""</span>
<span class="k">return</span> <span class="p">{}</span>
<span class="k">def</span> <span class="nf">input_run</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">data</span><span class="p">):</span>
<span class="sd">"""Adds data to object and calls self.get_config and self.run"""</span>
<span class="k">def</span> <span class="nf">get_config</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""Gets config file data and stores it under self.config of object instance"""</span>
<span class="k">def</span> <span class="nf">save_data</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""Add to plugins list and return modified data."""</span>
<span class="k">def</span> <span class="nf">check_dependencies</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""Check if all the dependencies are met."""</span>
<span class="nd">@abc</span><span class="o">.</span><span class="n">abstractmethod</span>
<span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""Run and make changes to data"""</span>
<span class="c1"># 1. Call check for dependencies</span>
<span class="bp">self</span><span class="o">.</span><span class="n">check_dependencies</span><span class="p">()</span>
<span class="c1"># 2. Append all changes to x.data["flat_tree"]["url_link/node"]["plugin_name"]</span>
<span class="c1"># 3. Call save data</span>
<span class="bp">self</span><span class="o">.</span><span class="n">save_data</span><span class="p">()</span>
</pre></div>
</div>
<p>Now for implementing GeoPlugin the class must be named properly. This is important so that the
enrichment daemon is able to sucessfully detect plugins. All plugin classes are required to be named as
XyzPlugin. This is the name of the key under “Plugins” where the info is saved. This is also important
for the plugin info to be displayed in the UI.</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">GeoPlugin</span><span class="p">(</span><span class="n">PluginBase</span><span class="p">):</span>
<span class="nd">@property</span>
<span class="k">def</span> <span class="nf">dependencies</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""List of dependencies - put class names of required plugins here.</span>
<span class="sd"> Return blank list if none."""</span>
<span class="k">return</span> <span class="p">[]</span>
<span class="nd">@property</span>
<span class="k">def</span> <span class="nf">module_dependencies</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""List of non-standard lib module dependencies</span>
<span class="sd"> - put module names as keys and versions as</span>
<span class="sd"> values.Return blank list if none."""</span>
<span class="k">return</span>
<span class="p">{</span>
<span class="s2">"geoip2"</span><span class="p">:</span> <span class="s2">"2.2.0"</span><span class="p">,</span>
<span class="s2">"ipaddr"</span><span class="p">:</span> <span class="s2">"2.1.11"</span><span class="p">,</span>
<span class="s2">"maxminddb"</span><span class="p">:</span> <span class="s2">"1.2.0"</span><span class="p">,</span>
<span class="s2">"requests"</span><span class="p">:</span> <span class="s2">"2.7.0"</span><span class="p">,</span>
<span class="p">}</span>
<span class="k">def</span> <span class="nf">config_plugin</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">""" Use data in self.config_dict to configure required settings</span>
<span class="sd"> also create reader object as creation is expensive.</span>
<span class="sd"> """</span>
<span class="bp">self</span><span class="o">.</span><span class="n">readers</span> <span class="o">=</span> <span class="p">{}</span>
<span class="c1"># Now check which dbs are enabled.</span>
<span class="bp">self</span><span class="o">.</span><span class="n">enabled_dbs</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">db_path_dict</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">config_dict</span><span class="p">[</span><span class="s2">"db_path"</span><span class="p">]</span>
<span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">value</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">config_dict</span><span class="p">[</span><span class="s2">"dbs"</span><span class="p">]</span><span class="o">.</span><span class="n">iteritems</span><span class="p">():</span>
<span class="k">if</span> <span class="n">value</span> <span class="o">==</span> <span class="s2">"True"</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">enabled_dbs</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
<span class="n">db_path</span> <span class="o">=</span> <span class="n">db_path_dict</span><span class="p">[</span><span class="n">name</span><span class="p">]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">readers</span><span class="p">[</span><span class="n">name</span><span class="p">]</span> <span class="o">=</span> <span class="n">geoip2</span><span class="o">.</span><span class="n">database</span><span class="o">.</span><span class="n">Reader</span><span class="p">(</span><span class="n">db_path</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">pretty_response</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span><span class="n">response</span><span class="p">,</span><span class="n">db_type</span><span class="p">):</span>
<span class="c1"># Not relevant to this example.</span>
<span class="k">def</span> <span class="nf">get_geo</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">ip</span><span class="p">,</span> <span class="n">db_type</span><span class="p">):</span>
<span class="c1"># Not relevant to this example.</span>
<span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""Run and make changes to data"""</span>
<span class="bp">self</span><span class="o">.</span><span class="n">check_dependencies</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">config_plugin</span><span class="p">()</span>
<span class="c1"># 2. Append all changes to x.data["flat_tree"]["url_link/node"]["plugin_name"]</span>
<span class="k">for</span> <span class="n">db_type</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">enabled_dbs</span><span class="p">:</span> <span class="c1"># will be set in config method</span>
<span class="n">ip_geo_map</span> <span class="o">=</span> <span class="p">{}</span> <span class="c1"># keys are IPs and values contain respective geo data.</span>
<span class="c1"># Reset ip_geo_map for each DB run.</span>
<span class="k">for</span> <span class="n">node</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">data</span><span class="p">[</span><span class="s2">"flat_tree"</span><span class="p">]:</span>
<span class="n">node_ip</span> <span class="o">=</span> <span class="n">node</span><span class="p">[</span><span class="s2">"ip"</span><span class="p">]</span>
<span class="k">if</span> <span class="n">node_ip</span> <span class="o">==</span> <span class="bp">None</span><span class="p">:</span>
<span class="n">node</span><span class="p">[</span><span class="s2">"GeoPlugin"</span><span class="p">][</span><span class="n">db_type</span><span class="p">]</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">continue</span>
<span class="k">if</span> <span class="s2">"GeoPlugin"</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">node</span><span class="o">.</span><span class="n">keys</span><span class="p">():</span>
<span class="n">node</span><span class="p">[</span><span class="s2">"GeoPlugin"</span><span class="p">]</span> <span class="o">=</span> <span class="p">{}</span> <span class="c1"># To avoid key error as further data is according to db.</span>
<span class="k">if</span> <span class="n">node_ip</span> <span class="ow">in</span> <span class="n">ip_geo_map</span><span class="o">.</span><span class="n">keys</span><span class="p">():</span>
<span class="n">node</span><span class="p">[</span><span class="s2">"GeoPlugin"</span><span class="p">][</span><span class="n">db_type</span><span class="p">]</span> <span class="o">=</span> <span class="n">ip_geo_map</span><span class="p">[</span><span class="n">node_ip</span><span class="p">]</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">node</span><span class="p">[</span><span class="s2">"GeoPlugin"</span><span class="p">][</span><span class="n">db_type</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_geo</span><span class="p">(</span><span class="n">node_ip</span><span class="p">,</span> <span class="n">db_type</span><span class="p">)</span>
<span class="n">ip_geo_map</span><span class="p">[</span><span class="n">node_ip</span><span class="p">]</span> <span class="o">=</span> <span class="n">node</span><span class="p">[</span><span class="s2">"GeoPlugin"</span><span class="p">][</span><span class="n">db_type</span><span class="p">]</span>
<span class="c1"># 3. Call save data</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">save_data</span><span class="p">()</span>
</pre></div>
</div>
<p>To override the methods dependencies and properties @property decorator is used. dependencies method
returns a list of strings consisting of class names of other plugins that must be run before this plugin.
The module_dependecies method lists the required python packages for the plugin to function. These must be
manually installed by the user.</p>
<p>Next, the config_plugin makes use of the plugin confiugration parameters that have already been loaded in
self.config_dict by the get_config method in the PluginBase class. The config_dict contains the config details
in key,value form from the config file.</p>
<p>The most important part of the plugin is the run method. A fixed procedure must be followed for writing this method. self.check_dependencies() and self.config_plugin() must be called in the beginning and result from self.save_data() should be returned towards the end. Between these the required processing should be performed and the changes must be saved to self.data object which is passed to the next plugin. New information must always be saved to self.data[“flat_tree_node”][particular_node][“XyzPlugin”]</p>
<p>The configuration file should be named by the name XyzPlugin-config in the plugins directory. The example config file is shown below:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="p">[</span><span class="n">db_path</span><span class="p">]</span>
<span class="n">city</span> <span class="o">=</span> <span class="o">/</span><span class="n">absolute</span><span class="o">/</span><span class="n">path</span><span class="o">/</span><span class="n">to</span><span class="o">/</span><span class="n">your</span><span class="o">/</span><span class="n">db</span><span class="o">/</span><span class="n">file</span>
<span class="n">anonymous_ip</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">connection_type</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">isp</span> <span class="o">=</span> <span class="kc">None</span>
<span class="p">[</span><span class="n">dbs</span><span class="p">]</span>
<span class="n">city</span> <span class="o">=</span> <span class="kc">True</span>
<span class="n">anonymous_ip</span> <span class="o">=</span> <span class="kc">False</span>
<span class="n">connection_type</span> <span class="o">=</span> <span class="kc">False</span>
<span class="n">isp</span> <span class="o">=</span> <span class="kc">False</span>
</pre></div>
</div>
<p>self.config_dict stores all info from the file in the form of a nested dictonary.</p>
</div>
</div>
</div>
</div>
<footer>
<div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
<a href="development.html" class="btn btn-neutral float-right" title="Development" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a>
<a href="advancedsearch.html" class="btn btn-neutral" title="Advanced Search" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a>
</div>
<hr/>
<div role="contentinfo">
<p>
© Copyright 2014-2016 Rumal Developers.
</p>
</div>
Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT:'./',
VERSION:'0.0.1 alpha',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<script type="text/javascript" src="_static/js/theme.js"></script>
<script type="text/javascript">
jQuery(function () {
SphinxRtdTheme.StickyNav.enable();
});
</script>
</body>
</html>