Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ELK] Change raw mappings to support search fields #665

Draft
wants to merge 38 commits into
base: main
Choose a base branch
from
Draft

[ELK] Change raw mappings to support search fields #665

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
aab4af9
[elastic_items] Change filter-raw parameter
valeriocos Jul 21, 2019
5c989fe
[raw-jira] Change mapping
valeriocos Jul 21, 2019
3ce862f
[raw-askbot] Change mapping
valeriocos Jul 26, 2019
3ae3458
[raw-bugzilla] Change mapping
valeriocos Jul 26, 2019
256e4cc
[raw-bugzillarest] Change mapping
valeriocos Jul 26, 2019
6c86f0e
[raw-confluence] Change mapping
valeriocos Jul 26, 2019
fdce2e2
[raw-discourse] Change mapping
valeriocos Jul 26, 2019
65f1c17
[raw-dockerhub] Change mapping
valeriocos Jul 26, 2019
f16c193
[raw-gerrit] Change mapping
valeriocos Jul 26, 2019
809f46a
[raw-git] Change mapping
valeriocos Jul 26, 2019
b7bd993
[raw-github] Change mapping
valeriocos Jul 26, 2019
6895013
[raw-gitlab] Change mapping
valeriocos Jul 26, 2019
da494c0
[raw-googlehits] Change mapping
valeriocos Jul 27, 2019
e5877cd
[raw-hyperkitty] Change mapping
valeriocos Jul 27, 2019
b331084
[raw-jenkins] Change mapping
valeriocos Jul 27, 2019
4adfc39
[raw-launchpad] Change mapping
valeriocos Jul 27, 2019
769a1c7
[raw-mattermost] Change mapping
valeriocos Jul 27, 2019
31f1a08
[raw-mbox] Change mapping
valeriocos Jul 27, 2019
4f23dc5
[raw-mediawiki] Change mapping
valeriocos Jul 27, 2019
738ee4d
[raw-meetup] Change mapping
valeriocos Jul 27, 2019
39d5063
[raw-nntp] Change mapping
valeriocos Jul 29, 2019
dbd7341
[raw-redmine] Change mapping
valeriocos Jul 29, 2019
77e685a
[raw-phabricator] Change mapping
valeriocos Jul 29, 2019
65a0357
[raw-rss] Change mapping
valeriocos Jul 29, 2019
6db8454
[raw-slack] Change mapping
valeriocos Jul 29, 2019
9d587fb
[raw-stackexchange] Change mapping
valeriocos Jul 29, 2019
9066023
[raw-supybot] Change mapping
valeriocos Jul 29, 2019
dff989f
[raw-telegram] Change mapping
valeriocos Jul 29, 2019
cacb20e
[raw-twitter] Change mapping
valeriocos Jul 29, 2019
25bcd9c
[raw-remo] Change mapping
valeriocos Mar 31, 2020
1cd233a
[raw-finosmeetings] Change mapping
valeriocos Mar 31, 2020
0609637
[raw-functest] Change mapping
valeriocos Mar 31, 2020
f54381c
[raw-crates] Change mapping
valeriocos Mar 31, 2020
93843e9
[raw-graal] Change mapping
valeriocos Mar 31, 2020
7e55380
[raw-kitsune] Change mapping
valeriocos Mar 31, 2020
f68c273
[raw-mozillaclub] Change mapping
valeriocos Mar 31, 2020
8493671
[raw-puppetforge] Change mapping
valeriocos Mar 31, 2020
04f2b38
[tests] Update test_all_properties in test_elastic
valeriocos Mar 31, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion grimoire_elk/elastic_items.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@
HEADER_JSON = {"Content-Type": "application/json"}
MAX_BULK_UPDATE_SIZE = 1000

FILTER_DATA_ATTR = 'data.'
FILTER_DATA_ATTR = 'search_fields.'
FILTER_SEPARATOR = r",\s*%s" % FILTER_DATA_ATTR
PROJECTS_JSON_LABELS_PATTERN = r".*(--labels=\[(.*)\]).*"

Expand Down
29 changes: 6 additions & 23 deletions grimoire_elk/raw/askbot.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,34 +29,17 @@ def get_elastic_mappings(es_major):
"""Get Elasticsearch mapping.

:param es_major: major version of Elasticsearch, as string
:returns: dictionary with a key, 'items', with the mapping
:returns: dictionary with a key, 'items', with the mapping
"""

mapping = '''
{
"dynamic":true,
"properties": {
"data": {
"properties": {
"answers": {
"dynamic":false,
"properties": {}
},
"author": {
"dynamic":false,
"properties": {}
},
"comments": {
"dynamic":false,
"properties": {}
},
"summary": {
"type": "text",
"index": true
}
}
}
"properties": {
"data": {
"dynamic":false,
"properties": {}
}
}
}
'''

Expand Down
30 changes: 6 additions & 24 deletions grimoire_elk/raw/bugzilla.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,35 +32,17 @@ def get_elastic_mappings(es_major):
"""Get Elasticsearch mapping.

:param es_major: major version of Elasticsearch, as string
:returns: dictionary with a key, 'items', with the mapping
:returns: dictionary with a key, 'items', with the mapping
"""

mapping = '''
{
"dynamic":true,
"properties": {
"data": {
"properties": {
"long_desc": {
"dynamic": false,
"properties": {}
},
"short_desc": {
"dynamic": false,
"properties": {
"__text__": {
"type": "text",
"index": true
}
}
},
"activity": {
"dynamic": false,
"properties": {}
}
}
}
"properties": {
"data": {
"dynamic":false,
"properties": {}
}
}
}
'''

Expand Down
74 changes: 6 additions & 68 deletions grimoire_elk/raw/bugzillarest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,79 +32,17 @@ def get_elastic_mappings(es_major):
"""Get Elasticsearch mapping.

:param es_major: major version of Elasticsearch, as string
:returns: dictionary with a key, 'items', with the mapping
:returns: dictionary with a key, 'items', with the mapping
"""

mapping = '''
{
"dynamic":true,
"properties": {
"data": {
"properties": {
"comments": {
"dynamic":false,
"properties": {
"raw_text": {
"type": "text",
"index": true
},
"text": {
"type": "text",
"index": true
}
}
},
"attachments": {
"properties": {
"description" : {
"type": "text",
"index": true
},
"summary" : {
"type": "text",
"index": true
}
}
},
"clones": {
"dynamic":false,
"properties": {}
},
"description": {
"type": "text",
"index": true
},
"history": {
"properties": {
"changes": {
"properties": {
"removed": {
"type": "text",
"index": true
},
"added": {
"type": "text",
"index": true
},
"comment": {
"properties": {
"thetext": {
"type": "text",
"index": true
}
}
}
}
}
}
},
"summary": {
"type": "text",
"index": true
}
}
}
"properties": {
"data": {
"dynamic":false,
"properties": {}
}
}
}
'''

Expand Down
32 changes: 6 additions & 26 deletions grimoire_elk/raw/confluence.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,38 +29,18 @@ class Mapping(BaseMapping):
def get_elastic_mappings(es_major):
"""Get Elasticsearch mapping.

Non dynamic discovery of type for:
* data.extensions

:param es_major: major version of Elasticsearch, as string
:returns: dictionary with a key, 'items', with the mapping
:returns: dictionary with a key, 'items', with the mapping
"""

mapping = '''
{
"dynamic":true,
"properties": {
"data": {
"properties": {
"extensions": {
"dynamic":false,
"properties": {}
},
"ancestors": {
"properties": {
"extensions": {
"dynamic":false,
"properties": {}
}
}
},
"body": {
"dynamic":false,
"properties": {}
}
}
}
"properties": {
"data": {
"dynamic":false,
"properties": {}
}
}
}
'''

Expand Down
14 changes: 5 additions & 9 deletions grimoire_elk/raw/crates.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,16 +36,12 @@ def get_elastic_mappings(es_major):
mapping = '''
{
"dynamic":true,
"properties": {
"data": {
"properties": {
"versions_data": {
"dynamic":false,
"properties": {}
}
}
}
"properties": {
"data": {
"dynamic":false,
"properties": {}
}
}
}
'''

Expand Down
55 changes: 6 additions & 49 deletions grimoire_elk/raw/discourse.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,60 +30,17 @@ def get_elastic_mappings(es_major):
"""Get Elasticsearch mapping.

:param es_major: major version of Elasticsearch, as string
:returns: dictionary with a key, 'items', with the mapping
:returns: dictionary with a key, 'items', with the mapping
"""

mapping = '''
{
"dynamic":true,
"properties": {
"data": {
"properties": {
"details": {
"properties": {
"suggested_topics": {
"dynamic": false,
"properties": {
"slug": {
"type": "text",
"index": true
},
"title": {
"type": "text",
"index": true
}
}
}
}
},
"fancy_title": {
"type": "text",
"index": true
},
"slug": {
"type": "text",
"index": true
},
"title": {
"type": "text",
"index": true
},
"post_stream": {
"dynamic": false,
"properties": {
"posts": {
"properties": {
"cooked": {
"type": "text",
"index": true
}
}
}
}
}
}
}
"properties": {
"data": {
"dynamic":false,
"properties": {}
}
}
}
'''

Expand Down
21 changes: 6 additions & 15 deletions grimoire_elk/raw/dockerhub.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,26 +30,17 @@ def get_elastic_mappings(es_major):
"""Get Elasticsearch mapping.

:param es_major: major version of Elasticsearch, as string
:returns: dictionary with a key, 'items', with the mapping
:returns: dictionary with a key, 'items', with the mapping
"""

mapping = '''
{
"dynamic":true,
"properties": {
"data": {
"properties": {
"description": {
"type": "text",
"index": true
},
"full_description": {
"type": "text",
"index": true
}
}
}
"properties": {
"data": {
"dynamic":false,
"properties": {}
}
}
}
'''

Expand Down
7 changes: 2 additions & 5 deletions grimoire_elk/raw/finosmeetings.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,11 +38,8 @@ def get_elastic_mappings(es_major):
"dynamic":true,
"properties": {
"data": {
"properties": {
"date": {
"type": "keyword"
}
}
"dynamic":false,
"properties": {}
}
}
}
Expand Down
18 changes: 5 additions & 13 deletions grimoire_elk/raw/functest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,20 +42,12 @@ def get_elastic_mappings(es_major):
mapping = '''
{
"dynamic":true,
"properties": {
"data": {
"properties": {
"details": {
"dynamic":false,
"properties": {}
},
"trust_indicator": {
"dynamic":false,
"properties": {}
}
}
}
"properties": {
"data": {
"dynamic":false,
"properties": {}
}
}
}
'''

Expand Down
Loading