Skip to content
This repository has been archived by the owner on May 27, 2020. It is now read-only.

fix reading properties without changing old configuration #119

Merged
merged 7 commits into from
Apr 27, 2016
Merged

fix reading properties without changing old configuration #119

merged 7 commits into from
Apr 27, 2016

Conversation

wuciawe
Copy link
Contributor

@wuciawe wuciawe commented Apr 21, 2016

it fixes #111, and unlike #112, it doesn't require to change old configuration, it's implemented inspired by the org.apache.spark.sql.execution.datasources.CaseInsensitiveMap in SPARK-SQL.

@pfcoperez
Copy link
Contributor

@wuciawe We'll follow your approach since, as you well pointed out, would keep backward compatibility.
Thanks again for another good contribution.

private val clientOptions = config.properties //config.properties.filterKeys(_.contains(MongodbConfig.ListMongoClientOptions)) // TODO review this Map. Can't filter keys
private val clientOptions = {
val lowerCaseOptions = MongodbConfig.ListMongoClientOptions.map(_.toLowerCase)
config.properties.filter(kv => kv._1.contains(lowerCaseOptions))
Copy link
Contributor

@pfcoperez pfcoperez Apr 27, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logic here is wrong: You are filtering those config properties which does not contain the whole option list.

I'd try something like:

lowerCaseOptions.flatMap(key => config.properties.get(key).map(key -> _)) toMap

Or, cleaner:

private val clientOptions = {
    val lowerCaseOptions = MongodbConfig.ListMongoClientOptions.map(_.toLowerCase).toSet
    config.properties.filterKeys(lowerCaseOptions contains _)
  }

Copy link
Contributor

@pfcoperez pfcoperez Apr 27, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at your commits I see you had already noticed that filterKeys isn't compatible with a map to be distributed to Spark workers as its return instance is not serializable. In any case, you should apply the logic I proposed:

 private val clientOptions = {
   val lowerCaseOptions = MongodbConfig.ListMongoClientOptions.map(_.toLowerCase).toSet
   config.properties.filter { case (k, _) => lowerCaseOptions contains k }
 }

@pfcoperez
Copy link
Contributor

@wuciawe Change as suggested, before committing make sure you pass the project test set, we'll pass them for a second time and if everything is OK we'll merge

@wuciawe
Copy link
Contributor Author

wuciawe commented Apr 27, 2016

@pfcoperez I merged your suggestion and project tests all pass on local machine. Sorry for wrong logic produced in previous pull request, I'm surprised that the wrong code compiles. Is it necessary to squash all those commits into one this time?

@pfcoperez
Copy link
Contributor

@wuciawe No worries. I was surprised too, probably a consequence WrappedString#CanBuildFrom.
You don't need to squash I'll do it using new GitHub merge tool.

Thanks again.

@pfcoperez pfcoperez merged commit f041cf6 into Stratio:branch-0.11 Apr 27, 2016
@wuciawe wuciawe mentioned this pull request Apr 29, 2016
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants