-
Notifications
You must be signed in to change notification settings - Fork 118
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Move2Kube : Detailed Insights into Configuration File Generation #1151
Comments
Due to the dynamic composing nature of transformers, the questions change depending on the transformers that get activated. Going through the relevant transformers code (https://github.com/konveyor/move2kube/tree/main/transformer) might give an idea. Or we might have to end up trying out with say a cluster type to see what gets into the config. Using --qa-skip might help quickly check the questions that get asked (at least on the default flow). @HarikrishnanBalagopal do we have any documents explaining the wild card capabilities of a config file? |
@nitinrana-deloitte Our yaml config file generation is heavily based on https://github.com/mikefarah/yq and json path. Especially the ability to create new yaml documents https://mikefarah.gitbook.io/yq/ The config file generation process is very simple.
Example: {} We ask our 1st question with ID {
"aaa": {
"bbb": {
"ccc": "myanswer1"
}
}
} Now we ask our 2nd question with ID {
"aaa": {
"bbb": {
"ccc": "myanswer1",
"ddd": "myanswer2",
}
}
} The above works for For multi-select questions, the answer is an array so we use a special symbol By default we don't store |
@HarikrishnanBalagopal, thanks for this. Could we also get to know more about the decision tree? Are there only a fixed number of questions which m2k is going to ask, we want to answer these questions for the users beforehand, but with increasing code base we have to deal with an increased number of questions, is there any way where we can handle this? |
There is no limit to the number of questions that can be asked. By default we try to avoid asking the user too many questions to keep things simple. Using custom transformers you can ask as many questions as you need. Example using a custom Starlark transformer |
@nitinrana-deloitte There is a feature in move2kube to switch on/off categories of questions in the flow. Here is the tutorial for this: https://move2kube.konveyor.io/tutorials/qa-categorization |
@nitinrana-deloitte Here is the mapping file, which contains the names of different categories and also the mapping between the different categories and the questions. If we disable a category, say $ move2kube transform -s src --qa-disable cicd Multiple categories can be disabled by passing the $ move2kube transform -s src --qa-disable cicd --qa-disable git |
@Akash-Nayak Thanks for this, I just wanted to clarify that we are implementing move2kube via the move2kube API. |
@nitinrana-deloitte |
@seshapad could we get a document which talks about the wildcard in the config file? |
@nitinrana-deloitte I will try to summarize the semantics here for now. We always try to get the answer from the config file first. We do this by looking at the ID of the question, splitting that ID into a bunch of sub keys separated by If we are unable to find the answer in the config, then we ask the user for the answer. There are 2 wildcards
Lines 74 to 77 in 565394b
Wildcard
|
// starting from 2nd last subkey replace with match all selector * | |
// Example: Given a.b.c.d.e this matches a.b.c.*.e, then a.b.*.d.e, then a.*.c.d.e |
Wildcard []
The []
symbol is used for Multi-Select
type questions. It can match against any key in a map/dict.
Multi-Select
type questions take a list of strings as the answer to the question.
move2kube/types/qaengine/config.go
Lines 135 to 137 in 565394b
// Given 'a.b.[].d' there should be at least one key 'c' such that 'a.b.c.d' exists. | |
// Note that 'c' in 'a.b.c.d' does not have to be a valid option for the question. | |
// We take the mere existence of 'a.b.c.d' to indicate that the user wants to skip the question. |
There are 2 ways such a question can be stored in the config:
move2kube/types/qaengine/config.go
Lines 245 to 256 in 565394b
// multi-select problem has 2 cases | |
key := p.ID | |
idx := strings.LastIndex(key, common.Special) | |
if idx < 0 { | |
// normal case key1 = [val1, val2, val3, ...] | |
set(key, p.Answer, c.yamlMap) | |
set(key, p.Answer, c.writeYamlMap) | |
return nil | |
} | |
// special case | |
baseKey, lastKeySegment := key[:idx-len(common.Delim)], key[idx+len(common.Special)+len(common.Delim):] |
For a Multi-Select question with id a.b.c
, options ["ans1", "ans2", "ans3"]
and answers ["ans1", "ans3"]
we get a config that looks like this
{
"a": {
"b": {
"c": ["ans1", "ans3"]
}
}
}
For a Multi-Select question with id a.[].c
, options ["ans1", "ans2", "ans3"]
and answers ["ans1", "ans3"]
we get a config that looks like this
{
"a": {
"ans1": {
"c": true
},
"ans2": {
"c": false
},
"ans3": {
"c": true
}
}
}
@HarikrishnanBalagopal, thanks. So let's say we have this config file which is only having one service (src), now if the service count is to be increased for future runs, in case of a different code base, so can this wildcard functionality be used for the same? move2kube: |
It just needs to be "*" where you have "src" now. |
Hi @ashokponkumar, using the wildcard *, can we construct the config file in such a way so that we still have the ability to ask the user certain questions for instance port number, while implementing it like this it gets skipped during the transformation. move2kube:
|
Remove the port number from this config move2kube: |
Hi @ashokponkumar. We don't want to include the hostname and url path in our route.yaml which will be generated after trnasformation, but if I'm passing empty string in the config.yaml, it is skipping to create the route.yaml file. This is the config file for reference, doing so skips route.yaml creation.
|
Use a dummy host, and then use a custom transformer like https://move2kube.konveyor.io/tutorials/customizing-the-output/custom-annotations to change the yaml that is generated. |
Hi @ashokponkumar, this is our config.yaml
On placing a wildcard in place of the service name, we only want to answer the port number during transformation, but m2k is asking values for both servicetype and urlpath, is this the expected behaviour while using wildcard? We are having multiple services for this transformation. |
Use a * in the port number section too. Is the url path same for all services? |
No, it will not be same for all the services. |
Hi @ashokponkumar, we are still getting the service type question during our transformation on using the above config.
|
@nitinrana-deloitte if possible try to preserve the indentation while pasting code. YAML especially cares a lot about indentation. You can use the three back ticks ``` to indicate the start of a code block. https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks#fenced-code-blocks You can also upload the config.yaml file here in the comments. That will help us debug. |
Sure @HarikrishnanBalagopal, here's the yaml which we are trying to tweak to get the user to only answer for the port number question, we are dealing with multiple services in this transformation.
|
This is the complete config which gets generated, for now we have only provided sample/dummy values.
|
@nitinrana-deloitte Thanks, the double As mentioned in the comment #1151 (comment) the way the move2kube/types/qaengine/config.go Lines 103 to 104 in 565394b
In the meantime, if you are only using a few ports, then you might try using the minreplicas: "2"
services:
"*":
"8080":
servicetype: Ingress
urlpath: /carbon-emission-ui
"8081":
servicetype: Ingress
urlpath: /carbon-emission-ui
"80":
servicetype: Ingress
urlpath: /carbon-emission-ui
"443":
servicetype: Ingress
urlpath: /carbon-emission-ui
deployment: Deployment
enable: true |
Hi @HarikrishnanBalagopal, thanks for this approach, but we want the end user to provide for the port numbers, so we cannot possibly place it in the config beforehand, what we are trying to achieve is to get only the specific answers from the user, for example port number for all the services that are identified.. Could we figure out a way in the config through which this can be done? |
@nitinrana-deloitte We have added a new feature to override the existing QAMapping file. It has been released in version v0.3.13-rc.0 We have put a sample and instructions here https://github.com/konveyor/move2kube-transformers/tree/main/enable-disable-qa-categories This new feature means you can:
|
@nitinrana-deloitte For your particular use case we have moved the port related questions out of the move2kube/assets/built-in/qa/qamappings.yaml Lines 56 to 60 in 7383e91
So I would suggest using a config like this minreplicas: "2"
services:
"*":
deployment: Deployment
enable: true and using the command below to only enable the categories that you need $ move2kube transform -s source/ --config my-custom-config.yaml --qa-enable ports |
Hi @HarikrishnanBalagopal @ashokponkumar @Akash-Nayak @seshapad We know that Move2Kube generates a basic deployment file upon transformation. We want to add additional fields for specifying CPU requests, memory limits, and other parameters beyond custom annotations. We have reviewed some documents but have only found options to add custom annotations. Is there a way to achieve this? |
The example custom transformer for adding the annotation can be adapted to add cpu requests and other sections of yaml. If you notice the below line, it's only adding values to a dictionary representing the yaml data. In the same way, other fields and values could be added too. |
Disabling certain transformers while planningPlan command with the
Put the following inside the config file
OutputExample plan file generated by Move2Kube apiVersion: move2kube.konveyor.io/v1alpha1
kind: Plan
metadata:
name: myproject
spec:
sourceDir: thingsboard
services:
java:
- transformerName: Gradle
paths:
GradleBuildFile:
- packaging/java/build.gradle
ServiceDirectories:
- packaging/java
ServiceRootDirectory:
- packaging/java
configs:
Gradle:
rootProjectName: java
packagingType: jar
isGradlewPresent: false
childModules:
- name: java
buildScriptPath: build.gradle
js:
- transformerName: Gradle
paths:
GradleBuildFile:
- packaging/js/build.gradle
ServiceDirectories:
- packaging/js
ServiceRootDirectory:
- packaging/js
configs:
Gradle:
rootProjectName: js
packagingType: jar
isGradlewPresent: false
childModules:
- name: js
buildScriptPath: build.gradle
thingsboard:
- transformerName: Maven
paths:
ServiceDirectories:
- netty-mqtt
- common
- rule-engine
- dao
- transport
- ui-ngx
- tools
- application
- msa
- rest-client
- monitoring
ServiceRootDirectory:
- .
pomFiles:
- pom.xml
configs:
Maven:
mavenAppName: thingsboard
packagingType: pom
mavenProfiles:
- default
- download-dependencies
- packaging
isMvnwPresent: false
childModules:
- name: netty-mqtt
pomPath: netty-mqtt/pom.xml
- name: common
pomPath: common/pom.xml
- name: rule-engine
pomPath: rule-engine/pom.xml
- name: dao
pomPath: dao/pom.xml
- name: transport
pomPath: transport/pom.xml
- name: ui-ngx
pomPath: ui-ngx/pom.xml
- name: tools
pomPath: tools/pom.xml
- name: application
pomPath: application/pom.xml
- name: msa
pomPath: msa/pom.xml
- name: rest-client
pomPath: rest-client/pom.xml
- name: monitoring
pomPath: monitoring/pom.xml
- transformerName: DockerfileDetector
paths:
Dockerfile:
- msa/js-executor/docker/Dockerfile
ServiceDirectories:
- msa/js-executor/docker
- transformerName: DockerfileDetector
paths:
Dockerfile:
- msa/monitoring/docker/Dockerfile
ServiceDirectories:
- msa/monitoring/docker
- transformerName: DockerfileDetector
paths:
Dockerfile:
- msa/tb/docker-cassandra/Dockerfile
ServiceDirectories:
- msa/tb/docker-cassandra
- transformerName: DockerfileDetector
paths:
Dockerfile:
- msa/tb/docker-postgres/Dockerfile
ServiceDirectories:
- msa/tb/docker-postgres
- transformerName: DockerfileDetector
paths:
Dockerfile:
- msa/tb-node/docker/Dockerfile
ServiceDirectories:
- msa/tb-node/docker
- transformerName: DockerfileDetector
paths:
Dockerfile:
- msa/transport/coap/docker/Dockerfile
ServiceDirectories:
- msa/transport/coap/docker
- transformerName: DockerfileDetector
paths:
Dockerfile:
- msa/transport/http/docker/Dockerfile
ServiceDirectories:
- msa/transport/http/docker
- transformerName: DockerfileDetector
paths:
Dockerfile:
- msa/transport/lwm2m/docker/Dockerfile
ServiceDirectories:
- msa/transport/lwm2m/docker
- transformerName: DockerfileDetector
paths:
Dockerfile:
- msa/transport/mqtt/docker/Dockerfile
ServiceDirectories:
- msa/transport/mqtt/docker
- transformerName: DockerfileDetector
paths:
Dockerfile:
- msa/transport/snmp/docker/Dockerfile
ServiceDirectories:
- msa/transport/snmp/docker
- transformerName: DockerfileDetector
paths:
Dockerfile:
- msa/vc-executor-docker/docker/Dockerfile
ServiceDirectories:
- msa/vc-executor-docker/docker
- transformerName: DockerfileDetector
paths:
Dockerfile:
- msa/web-ui/docker/Dockerfile
ServiceDirectories:
- msa/web-ui/docker
transformers:
ArgoCD: m2kassets/built-in/transformers/kubernetes/argocd/transformer.yaml
Buildconfig: m2kassets/built-in/transformers/kubernetes/buildconfig/transformer.yaml
CloudFoundry: m2kassets/built-in/transformers/cloudfoundry/transformer.yaml
ClusterSelector: m2kassets/built-in/transformers/kubernetes/clusterselector/transformer.yaml
ComposeGenerator: m2kassets/built-in/transformers/compose/composegenerator/transformer.yaml
ContainerImagesPushScriptGenerator: m2kassets/built-in/transformers/containerimagespushscript/transformer.yaml
DockerfileDetector: m2kassets/built-in/transformers/dockerfile/dockerfiledetector/transformer.yaml
DockerfileImageBuildScript: m2kassets/built-in/transformers/dockerfile/dockerimagebuildscript/transformer.yaml
DockerfileParser: m2kassets/built-in/transformers/dockerfile/dockerfileparser/transformer.yaml
DotNetCore-Dockerfile: m2kassets/built-in/transformers/dockerfilegenerator/dotnetcore/transformer.yaml
EarAnalyser: m2kassets/built-in/transformers/dockerfilegenerator/java/earanalyser/transformer.yaml
EarRouter: m2kassets/built-in/transformers/dockerfilegenerator/java/earrouter/transformer.yaml
Golang-Dockerfile: m2kassets/built-in/transformers/dockerfilegenerator/golang/transformer.yaml
Gradle: m2kassets/built-in/transformers/dockerfilegenerator/java/gradle/transformer.yaml
Jar: m2kassets/built-in/transformers/dockerfilegenerator/java/jar/transformer.yaml
Jboss: m2kassets/built-in/transformers/dockerfilegenerator/java/jboss/transformer.yaml
Knative: m2kassets/built-in/transformers/kubernetes/knative/transformer.yaml
Kubernetes: m2kassets/built-in/transformers/kubernetes/kubernetes/transformer.yaml
KubernetesVersionChanger: m2kassets/built-in/transformers/kubernetes/kubernetesversionchanger/transformer.yaml
Liberty: m2kassets/built-in/transformers/dockerfilegenerator/java/liberty/transformer.yaml
Maven: m2kassets/built-in/transformers/dockerfilegenerator/java/maven/transformer.yaml
Nodejs-Dockerfile: m2kassets/built-in/transformers/dockerfilegenerator/nodejs/transformer.yaml
OperatorTransformer: m2kassets/built-in/transformers/kubernetes/operator/transformer.yaml
PHP-Dockerfile: m2kassets/built-in/transformers/dockerfilegenerator/php/transformer.yaml
Parameterizer: m2kassets/built-in/transformers/kubernetes/parameterizer/transformer.yaml
Python-Dockerfile: m2kassets/built-in/transformers/dockerfilegenerator/python/transformer.yaml
ReadMeGenerator: m2kassets/built-in/transformers/readmegenerator/transformer.yaml
Ruby-Dockerfile: m2kassets/built-in/transformers/dockerfilegenerator/ruby/transformer.yaml
Rust-Dockerfile: m2kassets/built-in/transformers/dockerfilegenerator/rust/transformer.yaml
Tekton: m2kassets/built-in/transformers/kubernetes/tekton/transformer.yaml
Tomcat: m2kassets/built-in/transformers/dockerfilegenerator/java/tomcat/transformer.yaml
WarAnalyser: m2kassets/built-in/transformers/dockerfilegenerator/java/waranalyser/transformer.yaml
WarRouter: m2kassets/built-in/transformers/dockerfilegenerator/java/warrouter/transformer.yaml
WinWebApp-Dockerfile: m2kassets/built-in/transformers/dockerfilegenerator/windows/winweb/transformer.yaml
ZuulAnalyser: m2kassets/built-in/transformers/dockerfilegenerator/java/zuul/transformer.yaml
disabledTransformers:
ComposeAnalyser: m2kassets/built-in/transformers/compose/composeanalyser/transformer.yaml
WinConsoleApp-Dockerfile: m2kassets/built-in/transformers/dockerfilegenerator/windows/winconsole/transformer.yaml
WinSLWebApp-Dockerfile: m2kassets/built-in/transformers/dockerfilegenerator/windows/winsilverlightweb/transformer.yaml |
We can also use this for general discussion https://github.com/konveyor/move2kube/discussions |
Hi @ashokponkumar @HarikrishnanBalagopal @Akash-Nayak Few doubts related to the qa-mapping file, as mentioned in the example https://github.com/konveyor/move2kube-transformers/tree/main/enable-disable-qa-categories we can create a new category of question and then move questions from other categories to this new category and then enable or disable it accordingly. Following this, I disabled the imageregistry category and then created a new category 'image', and tried to include two questions inside it. However, move2kube fails to ask questions around this new category. Here is the qa-mapping file for more clarity.
|
Didn't you want to remove it form the |
Hi Team, We are trying to build the customer service of the enterprise app, we have selected the middleware as Liberty, and this is the Dockerfile generated by move2kube, we have selected the no build stage option and are building the service using mvn clean package.
We are running the container using docker run -p 8080:9080 customers Can you guys check once, as this works when we select the middleware as Tomcat or Jboss. |
Move2Kube dynamically adjusts its questioning process based on different scenarios.
Our goal is to gain a deeper understanding of the configuration file generation process without having to execute the transformation.
Dynamic Questioning based on Cluster Type: When changing the cluster type, such as to OpenShift, Move2Kube modifies subsequent questions, like requesting route creation.
We want to explore how this effects the generation of the config file.
Variation in Questions for Multiple Services: In scenarios where multiple services are detected, Move2Kube poses specific questions tailored to each service. Now, these questions can vary based on the number of services it detects, for a particular service there can be more questions compared to the other detected services.
We are aware of the ability of the config file, but our main goal is to gain a deeper understanding of the configuration file generation process so as to build the config file by taking minimal input from user ( rather than asking users to answer the entire transformation questions) and feed it to pipeline hosting m2k to generate manifest file. ****
The text was updated successfully, but these errors were encountered: