Merge pull request #242 from v3io/development

Development --> Master (v0.5.6)

README.md

@@ -6,7 +6,186 @@
 
 V3IO Frames is a high-speed server and client library for accessing time-series (TSDB), NoSQL, and streaming data in the [Iguazio Continuous Data Platform](https://www.iguazio.com).
 
-## Components
+## Documentation
+
+Frames currently supports 3 backends and basic CRUD functionality for each.
+
+Supported Backends:
+1. TSDB
+2. KV
+3. Stream
+4. CSV - for testing purposes
+
+
+All of frames operations are executed via the `client` object. To create a client object simply provide the Iguazio web-api endpoint and optional credentials.
+```python
+import v3io_frames as v3f
+client = v3f.Client('web-api:8081', user='user1', password='pass')
+```
+Note: When running from within the managed jupyter notebook on the iguazio platform there is no need to add credentials as this is handled by the platform.  
+Next, for every operation we need to provide a `backend`, and a `table` parameters and optionally other function specific arguments.
+
+### Create
+Creates a new table for the desired backend. Not all backends require a table to be created prior to ingestion. For example KV table will be created while ingesting new data, on the other hand since TSDB tables have mandatory fields we need to create a table before ingesting new data.  
+```python
+client.create(backend=<backend>, table=<table>, attrs=<backend_specefic_attributes>)
+```
+
+#### backend specific parameters
+##### TSDB
+* rate 
+* aggregates (optional)
+* aggregation-granularity (optional)
+
+For detailed info on these parameters please visit [TSDB](https://github.com/v3io/v3io-tsdb#v3io-tsdb) docs.  
+Example:
+```python
+client.create('tsdb', '/mytable', attrs={'rate': '1/m'})
+```
+
+##### Stream
+* shards=1 (optional)
+* retention_hours=24 (optional)
+
+For detailed info on these parameters please visit [Stream](https://www.iguazio.com/docs/concepts/latest-release/streams) docs.  
+Example:
+```python
+client.create('stream', '/mystream', attrs={'shards': '6'})
+```
+
+### Write
+Writes a Dataframe into one of the supported backends.  
+Common write parameters:
+* dfs - list of Dataframes to write
+* index_cols=None (optional) - specify specific index columns, by default Dataframe's index columns will be used.
+* labels=None (optional)
+* max_in_message=0 (optional)
+* partition_keys=None (optional)
+
+Example:
+```python
+data = [['tom', 10], ['nick', 15], ['juli', 14]] 
+df = pd.DataFrame(data, columns = ['name', 'age'])
+df.set_index('name')
+client.write(backend='kv', table='mytable', dfs=df)
+```
+
+#### backend specific parameters
+##### KV
+* expression=' ' (optional) - for detailed information on update expressions see [docs](https://www.iguazio.com/docs/reference/latest-release/expressions/update-expression/)
+* condition=' ' (optional) - for detailed information on condition expressions see [docs](https://www.iguazio.com/docs/reference/latest-release/expressions/condition-expression/)
+
+Example:
+```python
+data = [['tom', 10, 'TLV'], ['nick', 15, 'Berlin'], ['juli', 14, 'NY']] 
+df = pd.DataFrame(data, columns = ['name', 'age', 'city'])
+df.set_index('name')
+v3c.write(backend='kv', table='mytable', dfs=tsdf, expression='city="NY"', condition='age>14')
+```
+
+### Read
+Reads data from a backend.  
+Common read parameters:
+* query: string - Query in SQL format
+* iterator: bool - Return iterator of DataFrames or (if False) just one DataFrame
+* filter: string - Query filter (can't be used with query)
+* group_by: string - Query group by (can't be used with query)
+* columns: []str - List of columns to pass (can't be used with query)
+* limit: int - Maximal number of rows to return
+* row_layout: bool - Weather to use row layout (vs the default column layout)
+* max_in_message: int - Maximal number of rows per message
+* data_format: string - Data format (Not yet supported)
+* marker: string - Query marker (Not yet supported)
+
+
+#### backend specific parameters
+##### TSDB
+* start: string
+* end: string
+* step: string
+* aggragators: string
+* aggregationWindow: string
+
+For detailed info on these parameters please visit [TSDB](https://github.com/v3io/v3io-tsdb#v3io-tsdb) docs.  
+Example:
+```python
+df = client.read(backend='tsdb', query="select avg(cpu) as cpu, avg(diskio), avg(network)from mytable", start='now-1d', end='now', step='2h')
+```
+
+##### KV
+* sharding_keys: []string - list of specific sharding keys to query. For range scan formatted tables only.
+* segments: []int64 (Not yet supported)
+* total_segments: int64 (Not yet supported)
+* sort_key_range_start: string (Not yet supported)
+* sort_key_range_end: string (Not yet supported)
+
+For detailed info on these parameters please visit KV docs.
+
+Example:
+```python
+df = client.read(backend='kv', table='mytable', filter='col1>666')
+```
+
+##### Stream
+* seek: string - excepted values:  time | seq/sequence | latest | earliest.  
+if `seq` seek type is requested, need to provide the desired sequence id via `sequence` parameter.  
+if `time` seek type is requested, need to provide the desired start time via `start` parameter.
+* shard_id: string
+* sequence: int64 (optional)
+
+For detailed info on these parameters please visit [Stream](https://www.iguazio.com/docs/concepts/latest-release/streams) docs.
+
+Example:
+```python
+df = client.read(backend='stream', table='mytable', seek='latest', shard_id='5')
+```
+
+### Delete
+Deletes a table of a specific backend.
+
+Example:
+```python
+df = client.delete(backend='<backend>', table='mytable')
+```
+
+#### backend specific parameters
+##### TSDB
+* start: string - delete since start
+* end: string - delete since start
+
+Note: if both `start` and `end` are not specified **all** the TSDB table will be deleted.  
+For detailed info on these parameters please visit [TSDB](https://github.com/v3io/v3io-tsdb#v3io-tsdb) docs.  
+Example:
+```python
+df = client.delete(backend='tsdb', table='mytable', start='now-1d', end='now-5h')
+```
+##### KV
+* filter: string - Filter for selective delete
+
+Example:
+```python
+df = client.delete(backend='kv', table='mytable', filter='age>40')
+```
+
+### Execute
+Provides additional functions that are not covered in the basic CRUD functionality.
+
+##### TSDB
+Currently no `execute` commands are available for the TSDB backend.
+
+##### KV
+* infer, inferschema - inferring and creating a schema file for a given kv table.
+Example: `client.execute(backend='kv', table='mytable', command='infer')`
+* update - perform an update expression for a specific key.
+Example: `client.execute(backend='kv', table='mytable', command='update', args={'key': 'somekey', 'expression': 'col2=30', 'condition': 'col3>15'})`
+
+##### Stream
+* put - putting a new object to a stream.
+Example: `client.execute(backend='stream', table='mystream', command='put', args={'data': 'this a record', 'clientinfo': 'some_info', 'partition': 'partition_key'})`
+
+## Contributing
+
+### Components
 
 - Go server with support for both the gRPC and HTTP protocols
 - Go client

clients/py/v3io_frames/client.py

@@ -49,12 +49,12 @@ def __init__(self, address, session, frame_factory=pd.DataFrame,
     def read(self, backend='', table='', query='', columns=None, filter='',
              group_by='', limit=0, data_format='', row_layout=False,
              max_in_message=0, marker='', iterator=False, **kw):
-        """Run a query in nuclio
+        """Run a query
 
-        Parameters
+        Common Parameters
         ----------
         backend : str
-            Backend name
+            Backend name ('kv', 'tsdb', 'stream')
         table : str
             Table to query (can't be used with query)
         query : str
@@ -123,7 +123,6 @@ def write(self, backend, table, dfs, expression='', condition='',
             Columns to use as indices
         partition_keys : list of str
             Partition keys
-
         Returns:
             Write result
         """

go.mod

@@ -6,10 +6,12 @@ require (
 	github.com/nuclio/logger v0.0.1
 	github.com/nuclio/zap v0.0.2
 	github.com/pkg/errors v0.8.1
-	github.com/v3io/v3io-go v0.0.0-20190526000000-99f1b77b1592ed16f3e65be03a9fde54e408cb8b
+	github.com/v3io/v3io-go v0.0.0-20190606000000-9441f7c028db0b9642b541b07f30e35a1b812c58
 	github.com/v3io/v3io-tsdb v0.9.4
 	github.com/valyala/fasthttp v1.2.0
 	github.com/xwb1989/sqlparser v0.0.0-20180606152119-120387863bf2
 	golang.org/x/net v0.0.0-20181114220301-adae6a3d119a
 	google.golang.org/grpc v1.17.0
 )
+
+replace github.com/xwb1989/sqlparser => github.com/v3io/sqlparser v0.0.0-20190306105200-4d7273501871

go.sum

@@ -72,17 +72,14 @@ github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXf
 github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q=
 github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
 github.com/v3io/frames v0.0.0-20190328123118-1dad1ff610509e7b087d9cd390ed1b452caecf15/go.mod h1:6aKW4Wl4A+gQhXH0JRCVOLgwvcrLyk+fqEpemuie094=
+github.com/v3io/sqlparser v0.0.0-20190306105200-4d7273501871 h1:myF4tU/HdFWU1UzMdf16cHRbownzsyvL7VKIHqkrSvo=
+github.com/v3io/sqlparser v0.0.0-20190306105200-4d7273501871/go.mod h1:QD2Bo64oyTWzeV8RFehXS0hZEDFgOK99/h2a6ErRu6E=
 github.com/v3io/v3io-go v0.0.0-20180415000000-1486c75b0e590a14580f7d9b6cef7a944a231ca7/go.mod h1:MHc+d/Jg/y8lV4B9sgwTvuS3tEE9wS+kqtU0+D0Sr78=
-github.com/v3io/v3io-go v0.0.0-20180415000000-a0c7cc523b6e17283300c46b385096bdf0278aad/go.mod h1:MHc+d/Jg/y8lV4B9sgwTvuS3tEE9wS+kqtU0+D0Sr78=
-github.com/v3io/v3io-go v0.0.0-20190526000000-99f1b77b1592ed16f3e65be03a9fde54e408cb8b h1:tpz3V548Pv/sQ0c7C46x7Jo4SyBdLnJUPOPkURwDKIg=
-github.com/v3io/v3io-go v0.0.0-20190526000000-99f1b77b1592ed16f3e65be03a9fde54e408cb8b/go.mod h1:MHc+d/Jg/y8lV4B9sgwTvuS3tEE9wS+kqtU0+D0Sr78=
+github.com/v3io/v3io-go v0.0.0-20190606000000-9441f7c028db0b9642b541b07f30e35a1b812c58 h1:rrs6oeAEvoK4HeAm7iQbH9inyWwt8HIDLJktbq0GH9Y=
+github.com/v3io/v3io-go v0.0.0-20190606000000-9441f7c028db0b9642b541b07f30e35a1b812c58/go.mod h1:MHc+d/Jg/y8lV4B9sgwTvuS3tEE9wS+kqtU0+D0Sr78=
 github.com/v3io/v3io-go-http v0.0.0-20190221115935-53e2b487c9a2 h1:NJc63wM25iS+ci5z7LVwjWD4QM0QpTQw/fovKzatss0=
 github.com/v3io/v3io-go-http v0.0.0-20190221115935-53e2b487c9a2/go.mod h1:GXYcR9MxgfbE3BJdkXki5EclvtS8Nxu2RQNLA8hMMog=
 github.com/v3io/v3io-tsdb v0.0.0-20190328071546-4e85f3df2d205fc7368d54184bb2ceff949ab4bd/go.mod h1:A+5yKC16QxLf+Fy5v7VvIxSw+jwsKHLhUS7dCYFDLAA=
-github.com/v3io/v3io-tsdb v0.0.0-20190415000000-3748a7a3d0ce082633692719de3628de7925a0de h1:jf0x5vDaFUqQGfxcgpbgJxSqIjMjXRs9VUln0WZXLRM=
-github.com/v3io/v3io-tsdb v0.0.0-20190415000000-3748a7a3d0ce082633692719de3628de7925a0de/go.mod h1:QFbAvfxDpokOk9nseiZbcsiMq1v9Aor4BEV+PEQxKhg=
-github.com/v3io/v3io-tsdb v0.9.2 h1:9taOa/SrAA/cWMgTLO8M12FNqgH4SyhrBzKzdCRp/Z0=
-github.com/v3io/v3io-tsdb v0.9.2/go.mod h1:cPLq5KvhxzdvRaVzy8HTDxousSFUaSRBRdhtd131JgU=
 github.com/v3io/v3io-tsdb v0.9.3 h1:GHnQ1deelC5riYNuQsmQhN7Bzj/SWRv02WE3mGOobQA=
 github.com/v3io/v3io-tsdb v0.9.3/go.mod h1:cPLq5KvhxzdvRaVzy8HTDxousSFUaSRBRdhtd131JgU=
 github.com/v3io/v3io-tsdb v0.9.4 h1:s1gI+WVOHj4GoY6dt/yfvUldBLcQPWwPhP1wiOxIz2M=
@@ -94,8 +91,6 @@ github.com/valyala/fasthttp v1.0.0/go.mod h1:4vX61m6KN+xDduDNwXrhIAVZaZaZiQ1luJk
 github.com/valyala/fasthttp v1.2.0 h1:dzZJf2IuMiclVjdw0kkT+f9u4YdrapbNyGAN47E/qnk=
 github.com/valyala/fasthttp v1.2.0/go.mod h1:4vX61m6KN+xDduDNwXrhIAVZaZaZiQ1luJk8LWSxF3s=
 github.com/valyala/tcplisten v0.0.0-20161114210144-ceec8f93295a/go.mod h1:v3UYOV9WzVtRmSR+PDvWpU/qWl4Wa5LApYYX4ZtKbio=
-github.com/xwb1989/sqlparser v0.0.0-20180606152119-120387863bf2 h1:zzrxE1FKn5ryBNl9eKOeqQ58Y/Qpo3Q9QNxKHX5uzzQ=
-github.com/xwb1989/sqlparser v0.0.0-20180606152119-120387863bf2/go.mod h1:hzfGeIUDq/j97IG+FhNqkowIyEcD88LrW6fyU3K3WqY=
 go.uber.org/atomic v1.3.2 h1:2Oa65PReHzfn29GpvgsYwloV9AVFHPDk8tYxt2c2tr4=
 go.uber.org/atomic v1.3.2/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
 go.uber.org/multierr v1.1.0 h1:HoEmRHQPVSqub6w2z2d2EOVs2fjyFRGyofhKuyDq0QI=