Skip to content

Commit

Permalink
update version
Browse files Browse the repository at this point in the history
  • Loading branch information
@xnuinside
xnuinside committed Sep 18, 2021
1 parent c5e2fe6 commit 1ff8e1b
Show file tree
Hide file tree
Showing 3 changed files with 41 additions and 133 deletions.
67 changes: 11 additions & 56 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ Build with ply (lex & yacc in python). A lot of samples in 'tests/.

### Is it Stable?

Yes, library already has about 5000+ usage per day - https://pypistats.org/packages/simple-ddl-parser.
Yes, library already has about 7000+ downloads per day - https://pypistats.org/packages/simple-ddl-parser.

As maintainer I guarantee that any backward incompatible changes will not be done in patch or minor version. Only additionals & new features.
As maintainer, I guarantee that any backward incompatible changes will not be done in patch or minor version. Only additionals & new features.

However, in process of adding support for new statements & features I see that output can be structured more optimal way and I hope to release version `1.0.*` with more struct output result. But, it will not be soon, first of all, I want to add support for so much statements as I can. So I don't think make sense to expect version 1.0.* before, for example, version `0.26.0` :)

Expand Down Expand Up @@ -363,6 +363,15 @@ Big thanks for the involving & contribution with test cases with DDL samples & o


## Changelog
**v0.19.7**
Fixes:
1. Add support for more special symbols to strings - https://github.com/xnuinside/simple-ddl-parser/issues/68

Features:
1. Added support for HQL statements:
STORED AS INPUTFORMAT, OUTPUTFORMAT - https://github.com/xnuinside/simple-ddl-parser/issues/69
SKEWED BY

**v0.19.6**
Fixes:

Expand Down Expand Up @@ -444,57 +453,3 @@ Will be fixed in next releases.
3. Half of changelogs moved to ARCHIVE_CHANGELOG.txt
4. Added base support for CREATE DOMAIN statement
5. Added base support for CREATE SCHEMA [IF NOT EXISTS] ... [AUTHORIZATION] statement, added new type keyword 'schemas'


**v0.15.0**
1. Garbage like '‘’' quotes are ignored now and changed to normal.
2. Added support for HQL: LINES TERMINATED BY, COMMENT (for table), ROW FORMAT SERDE, WITH SERDEPROPERTIES (
"input.regex" = "..some regex..")
3. Fixed issue when primary key with default option was not parsed correct - https://github.com/xnuinside/simple-ddl-parser/issues/40
4. Fixed issue when expression in default value was not parsed correct - https://github.com/xnuinside/simple-ddl-parser/issues/39
5. Added support for comments in Columns (except one case when COMMENT goes after DEFAULT word, in this case does not parse correct now - will be fixed in next releases)


**v0.14.0**
1. Added support for CONSTRAINT ... PRIMARY KEY ...
2. Added support for ENCRYPT [+ NO SALT, SALT, USING] statements for Oracle dialect. All default values taken from this doc https://docs.oracle.com/en/database/oracle/oracle-database/21/asoag/encrypting-columns-tables2.html
Now if you use output_mode='oracle' in column will be showed new property 'encrypt'.
If no ENCRYPT statement will be in table defenition - then value will be 'None', but if ENCRYPT exists when in encrypt property you will find this information:

{'encrypt' : {
'salt': True,
'encryption_algorithm': 'AES192',
'integrity_algorithm': 'SHA-1'
}}

3. Added support for oracle STORAGE statement, 'oracle' output_mode now has key 'storage' in table data defenition.
4. Added support for TABLESPACE statement after columns defenition

**v0.12.1**
1. () after DEFAULT now does not cause an issue
2. ' and " does not lost now in DEFAULT values

**v0.12.0**
1. Added support for MSSQL: types with 2 words like 'int IDENTITY',
FOREIGN KEY REFERENCES statement, supported 'max' as type size, CONSTRAINT ... UNIQUE statement in table defenition,
CONSTRAINT ... CHECK, CONSTRAINT ... FOREIGN KEY
2. Added output_mode types: 'mysql', 'mssql' for SQL Server, 'oracle'. If chosed one of the above -
added key 'constraints' in table defenition by default. 'constraints' contain dict with keys 'uniques', 'checks', 'references'
it this is a COSTRAINT .. CHECK 'checks' key will be still in data output, but it will be duplicated to 'constraints': {'checks': ...}
3. Added support for ALTER ADD ... UNIQUE
4. Added support for CREATE CLUSTERED INDEX, if output_mode = 'mssql' then index will have additional arg 'clustered'.
5. Added support for DESC & NULLS in CREATE INDEX statements. Detailed information places in key 'detailed_columns' in 'indexes', example: '
'index': [{'clustered': False,
'columns': ['extra_funds'],
'detailed_columns': [{'name': 'extra_funds',
'nulls': 'LAST',
'order': 'ASC'}],
6. Added support for statement ALTER TABLE ... ADD CONSTRAINT ... DEFAULT ... FOR ... ;

**v0.11.0**
1. Now table can has name 'table'
2. Added base support for statement CREATE TYPE: AS ENUM, AS OBJECT, INTERNALLENGTH, INPUT, OUTPUT (not all properties & types supported yet.)
3. Added argument 'group_by_type' in 'run' method that will group output by type of parsed entities like:
'tables': [all_pasrsed_tables], 'sequences': [all_pasrsed_sequences], 'types': [all_pasrsed_types], 'domains': [all_pasrsed_domains]
4. Type in column defenition also can be "schema"."YourCustomType"
5. " now are not dissapeared if you use them in DDL.
105 changes: 29 additions & 76 deletions docs/README.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ Build with ply (lex & yacc in python). A lot of samples in 'tests/.
Is it Stable?
^^^^^^^^^^^^^

Yes, library already has about 5000+ usage per day - https://pypistats.org/packages/simple-ddl-parser.
Yes, library already has about 7000+ downloads per day - https://pypistats.org/packages/simple-ddl-parser.

As maintainer I guarantee that any backward incompatible changes will not be done in patch or minor version. Only additionals & new features.

Expand All @@ -34,7 +34,12 @@ However, in process of adding support for new statements & features I see that o
How does it work?
^^^^^^^^^^^^^^^^^

Parser tested on different DDLs for PostgreSQL & Hive. But idea to support as much as possible DDL dialects (AWS Redshift, Oracle, Hive, MsSQL, etc.). You can check dialects sections after ``Supported Statements`` section to get more information that statements from dialects already supported by parser.
Parser tested on different DDLs mostly for PostgreSQL & Hive. But idea to support as much as possible DDL dialects (AWS
Redshift, Oracle, Hive, MsSQL, etc.). You can check dialects sections after ``Supported Statements`` section to get more information that statements from dialects already supported by parser.

Feel free to open Issue with DDL sample
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

**If you need some statement, that not supported by parser yet**\ : please provide DDL example & information about that is it SQL dialect or DB.

Types that are used in your DB does not matter, so parser must also work successfuly to any DDL for SQL DB. Parser is NOT case sensitive, it did not expect that all queries will be in upper case or lower case. So you can write statements like this:
Expand Down Expand Up @@ -332,11 +337,12 @@ HQL Dialect statements
* PARTITIONED BY statement
* ROW FORMAT, ROW FORMAT SERDE
* WITH SERDEPROPERTIES ("input.regex" = "..some regex..")
* STORED AS
* STORED AS (AVRO, PARQUET, etc), STORED AS INPUTFORMAT, OUTPUTFORMAT
* COMMENT
* LOCATION
* FIELDS TERMINATED BY, LINES TERMINATED BY, COLLECTION ITEMS TERMINATED BY, MAP KEYS TERMINATED BY
* TBLPROPERTIES ('parquet.compression'='SNAPPY' & etc.)
* SKEWED BY

MSSQL / MySQL/ Oracle
^^^^^^^^^^^^^^^^^^^^^
Expand Down Expand Up @@ -386,8 +392,6 @@ TODO in next Releases (if you don't see feature that you need - open the issue)
#. Add support for ALTER TABLE ... ADD COLUMN
#. Add more support for CREATE type IS TABLE (example: CREATE OR REPLACE TYPE budget_tbl_typ IS TABLE OF NUMBER(8,2);
#. Add support (ignore correctly) ALTER TABLE ... DROP CONSTRAINT ..., ALTER TABLE ... DROP INDEX ...
#. Add support for COMMENT ON statement
#. Add support for SKEWED BY for HQL

non-feature todo
----------------
Expand All @@ -396,17 +400,32 @@ non-feature todo
#. Provide API to get result as Python Object
#. Add online demo (UI) to parse ddl

Historical context
^^^^^^^^^^^^^^^^^^
Thanks for involving & contributions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Big thanks for the involving & contribution with test cases with DDL samples & opening issues goes to:

This library is an extracted parser code from https://github.com/xnuinside/fakeme (Library for fake relation data generation, that I used in several work projects, but did not have time to make from it normal open source library)

For one of the work projects I needed to convert SQL ddl to Python ORM models in auto way and I tried to use https://github.com/andialbrecht/sqlparse but it works not well enough with ddl for my case (for example, if in ddl used lower case - nothing works, primary keys inside ddl are mapped as column name not reserved word and etc.).
So I remembered about Parser in Fakeme and just extracted it & improved.
* https://github.com/kukigai ,
* https://github.com/Awalkman90 ,
* https://github.com/geob3d

Changelog
---------

**v0.19.7**
Fixes:


#. Add support for more special symbols to strings - https://github.com/xnuinside/simple-ddl-parser/issues/68

Features:


#. Added support for HQL statements:
STORED AS INPUTFORMAT, OUTPUTFORMAT - https://github.com/xnuinside/simple-ddl-parser/issues/69
SKEWED BY

**v0.19.6**
Fixes:

Expand Down Expand Up @@ -508,69 +527,3 @@ create temp table tempevent(like event);
#. Half of changelogs moved to ARCHIVE_CHANGELOG.txt
#. Added base support for CREATE DOMAIN statement
#. Added base support for CREATE SCHEMA [IF NOT EXISTS] ... [AUTHORIZATION] statement, added new type keyword 'schemas'

**v0.15.0**


#. Garbage like '‘’' quotes are ignored now and changed to normal.
#. Added support for HQL: LINES TERMINATED BY, COMMENT (for table), ROW FORMAT SERDE, WITH SERDEPROPERTIES (
"input.regex" = "..some regex..")
#. Fixed issue when primary key with default option was not parsed correct - https://github.com/xnuinside/simple-ddl-parser/issues/40
#. Fixed issue when expression in default value was not parsed correct - https://github.com/xnuinside/simple-ddl-parser/issues/39
#. Added support for comments in Columns (except one case when COMMENT goes after DEFAULT word, in this case does not parse correct now - will be fixed in next releases)

**v0.14.0**


#. Added support for CONSTRAINT ... PRIMARY KEY ...
#. Added support for ENCRYPT [+ NO SALT, SALT, USING] statements for Oracle dialect. All default values taken from this doc https://docs.oracle.com/en/database/oracle/oracle-database/21/asoag/encrypting-columns-tables2.html
Now if you use output_mode='oracle' in column will be showed new property 'encrypt'.
If no ENCRYPT statement will be in table defenition - then value will be 'None', but if ENCRYPT exists when in encrypt property you will find this information:

{'encrypt' : {
'salt': True,
'encryption_algorithm': 'AES192',
'integrity_algorithm': 'SHA-1'
}}


#. Added support for oracle STORAGE statement, 'oracle' output_mode now has key 'storage' in table data defenition.
#. Added support for TABLESPACE statement after columns defenition

**v0.12.1**


#. () after DEFAULT now does not cause an issue
#. ' and " does not lost now in DEFAULT values

**v0.12.0**


#. Added support for MSSQL: types with 2 words like 'int IDENTITY',
FOREIGN KEY REFERENCES statement, supported 'max' as type size, CONSTRAINT ... UNIQUE statement in table defenition,
CONSTRAINT ... CHECK, CONSTRAINT ... FOREIGN KEY
#. Added output_mode types: 'mysql', 'mssql' for SQL Server, 'oracle'. If chosed one of the above -
added key 'constraints' in table defenition by default. 'constraints' contain dict with keys 'uniques', 'checks', 'references'
it this is a COSTRAINT .. CHECK 'checks' key will be still in data output, but it will be duplicated to 'constraints': {'checks': ...}
#. Added support for ALTER ADD ... UNIQUE
#. Added support for CREATE CLUSTERED INDEX, if output_mode = 'mssql' then index will have additional arg 'clustered'.
#. Added support for DESC & NULLS in CREATE INDEX statements. Detailed information places in key 'detailed_columns' in 'indexes', example: '
'index': [{'clustered': False,
.. code-block::
'columns': ['extra_funds'],
'detailed_columns': [{'name': 'extra_funds',
'nulls': 'LAST',
'order': 'ASC'}],
#. Added support for statement ALTER TABLE ... ADD CONSTRAINT ... DEFAULT ... FOR ... ;

**v0.11.0**


#. Now table can has name 'table'
#. Added base support for statement CREATE TYPE: AS ENUM, AS OBJECT, INTERNALLENGTH, INPUT, OUTPUT (not all properties & types supported yet.)
#. Added argument 'group_by_type' in 'run' method that will group output by type of parsed entities like:
'tables': [all_pasrsed_tables], 'sequences': [all_pasrsed_sequences], 'types': [all_pasrsed_types], 'domains': [all_pasrsed_domains]
#. Type in column defenition also can be "schema"."YourCustomType"
#. " now are not dissapeared if you use them in DDL.
2 changes: 1 addition & 1 deletion pyproject.toml
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[tool.poetry]
name = "simple-ddl-parser"
version = "0.19.6"
version = "0.19.7"
description = "Simple DDL Parser to parse SQL & dialects like HQL, TSQL, Oracle, AWS Redshift, Snowflake, etc ddl files to json/python dict with full information about columns: types, defaults, primary keys, etc.; sequences, alters, custom types & other entities from ddl."
authors = ["Iuliia Volkova <[email protected]>"]
license = "MIT"
Expand Down

0 comments on commit 1ff8e1b

Please sign in to comment.