Create rule S7196: Complex logic provided to PySpark withColumn method should be refactored into a separate expression

rules/S7196/metadata.json

@@ -0,0 +1,2 @@
+{
+}

rules/S7196/python/metadata.json

@@ -0,0 +1,26 @@
+{
+  "title": "Complex logic provided to PySpark withColumn method should be refactored into a separate expression",
+  "type": "CODE_SMELL",
+  "status": "ready",
+  "remediation": {
+    "func": "Constant\/Issue",
+    "constantCost": "5min"
+  },
+  "tags": [
+    "data-science",
+    "pyspark"
+  ],
+  "defaultSeverity": "Medium",
+  "ruleSpecification": "RSPEC-7196",
+  "sqKey": "S7196",
+  "scope": "All",
+  "defaultQualityProfiles": ["Sonar way"],
+  "quickfix": "unknown",
+  "code": {
+    "impacts": {
+      "MAINTAINABILITY": "LOW",
+      "RELIABILITY": "MEDIUM",
+    },
+    "attribute": "FOCUSED"
+  }
+}

rules/S7196/python/rule.adoc

@@ -0,0 +1,42 @@
+This rule raises an issue when complex functions or expressions are directly passed to withColumn
+
+== Why is this an issue?
+
+`withColumn` method is commonly used to add or modify columns in a DataFrame. When complex functions or expressions are directly passed to withColumn, it can lead to code that is difficult to read, understand, and maintain. Also, it will become easier to write unit tests for these functions, ensuring that the logic is correct and behaves as expected. This leads to more robust and reliable code. 
+
+== How to fix it
+
+To fix this issue, complex logic within `withColumn` logic should be refactored into separate functions or variables before being passed to `withColumn` to improve code clarity and maintainability,
+
+=== Code examples
+
+==== Noncompliant code example
+
+[source,python,diff-id=1,diff-type=noncompliant]
+----
+from pyspark.sql.functions import *
+df = df.withColumn('Revenue', col('fare_amount').substr(0, 10).cast("float") + col('extra').substr(0, 5).cast("float") + col('tax').substr(0, 3).cast("float"))
+----
+
+==== Compliant solution
+
+[source,python,diff-id=1,diff-type=compliant]
+----
+from pyspark.sql.functions import *
+def convert_to_float(col_str):
+    return col_str.substr(0, 10).cast("float")
+
+def compute_revenue():
+        fare_amount = col('fare_amount').substr(0, 10).cast("float")
+        extra = col('extra').substr(0, 5).cast("float")
+        tax = col('tax').substr(0, 3).cast("float")
+
+        return fare_amount + extra + tax
+
+df = df.withColumn("Revenue", compute_revenue()) # Compliant
+----
+
+== Resources
+=== Documentation
+
+ * PySpark withColumn Documentation - https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.withColumn.html[pyspark.sql.DataFrame.withColumn]