Questions and Answers

To help to understand what can be done with mETL and how, l note some questions here, that can be answered when time comes, maybe these things can make their way into the docs sometime, if the features are there, and if they are not yet in the docs:

Questions

Transform vs. Manipulation

What exactly is the difference between "transforms" and "manipulations"? From the name I would assume it's a very similar if not same functionality - or manipulating parts of the data would be a subfunction of the process to transform data from one structure into another, what happens between source and target.

Multidimensional data in source and target

How can I use multiple tables as sources to merge and transform them to a single data structure, especially when using a database in table mode? (otherwise I'd probably join the data in a select statement)

And, (how) can I make multi-dimensional data in the target instead of a flat structure? So, I have tables like this(simplified)

Location
--------
name
city_id
content_id
 
City
-----
name
latitude
longitude
 
Content
-------
text
owner

and want to create a single but multi-dimensional json target data from it:

 
[ "locations":
  { 
    'title': 'data from location name'
    'city': {
      'name': 'some name'
      'url': 'external REST API url'
      'lat': 'latitude from source, but named lat'
      'lon': 'longitude from source, but named lon'
     'description': 'text from content text'
    }
  }
]

Extending mETL with custom code

Can I extend some things with custom code? (I need to load some data from an additional REST service that I need to merge into my data, but very likely cannot load it in advance. And I might have some other complex transforms where I will want to plugin my custom code, and in the config file want just to define that a field should be thrown into that custom code and the resulkt being used.

• yes, you can! ;) there's and example in the documentation pdf - search: "transform: tests.test_source.convertToRomanNumber - seems to require usage of "tarr" somehow. Details need to be found out still, though

Answers

• how to make the json output human readable:
  • use "compact: false" in the target definition