Skip to content

Commit

Permalink
Expose PDAL Mesh API (#27)
Browse files Browse the repository at this point in the history
* Expose PDAL Mesh API

* Make Native Autoclosable

* Add CHANGELOG.md, fix codestyle to avoid syntax deprecated in Dotty

* Update PDAL-Scala reader types

* DSL expressions rename & DSL implicits reorganization

* Update PDAL filters up to 2.0.0
  • Loading branch information
pomadchin authored Mar 3, 2020
1 parent 0add31a commit 7acf3e7
Show file tree
Hide file tree
Showing 45 changed files with 4,438 additions and 361 deletions.
4 changes: 3 additions & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
sudo: false

matrix:
jobs:
include:
- os: linux
jdk: openjdk8
Expand All @@ -20,6 +20,7 @@ matrix:

- os: osx
osx_image: xcode9.3
language: scala
scala:
- 2.13.1
compiler:
Expand All @@ -31,6 +32,7 @@ matrix:

- os: osx
osx_image: xcode9.3
language: scala
scala:
- 2.12.10
compiler:
Expand Down
94 changes: 94 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Changelog
All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

## [2.1.2] - 2020-03-02
### Added
- Expose PDAL Mesh API [#27](https://github.com/PDAL/java/pull/27)
- [CHANGELOG](/CHANGELOG.md)

### Changed
- **Breaking change** // Rename Scala DSL case classes [#28](https://github.com/PDAL/java/issues/28)
- **Breaking change** // Renamed dispose method to close [#27](https://github.com/PDAL/java/pull/27)
- Update Scala DSL up to PDAL 2.0 [#29](https://github.com/PDAL/java/issues/29)

## [2.0.0] - 2020-01-14
### Changed
- PDAL 2.0 compatible release.
- Added Scala 2.13 support.

## [1.9.0] - 2019-05-15
### Changed
- Release process improvements.

## [1.8.6] - 2019-04-26
### Changed
- Release process improvements.

## [1.8.5] - 2019-04-15
### Changed
- PDAL JNI Bindings thread safety [#19](https://github.com/PDAL/java/issues/19)

### Fixed
- PythonFilter gives invalid syntax [#9](https://github.com/PDAL/java/issues/9)
- Option in filter.Python seems to be misspelled [#8](https://github.com/PDAL/java/issues/8)

## [1.8.4] - 2019-04-09
### Changed
- Make Exceptions Handling better [#17](https://github.com/PDAL/java/pull/17)

## [1.8.3] - 2019-04-09
### Fixed
- CSV files read issues [#15](https://github.com/PDAL/java/issues/15)

## [1.8.2] - 2019-03-27
### Fixed
- Fix dimName calls to be better, otherwise CSV reads wont work [#14](https://github.com/PDAL/java/pull/14)

## [1.8.1] - 2019-03-25
### Changed
- Update JTS to make it GeoTrellis compatible [#13](https://github.com/PDAL/java/pull/13)

## [1.7.0-RC4] - 2019-01-14
### Changed
- Scala version update up to 2.12.6.
- Update dependencies and base PDAL version.
- Release process improvements.

### Fixed
- Fix matlab reader ReaderType [#10](https://github.com/PDAL/java/pull/10)
- Fix typo in GdalWrite [#11](https://github.com/PDAL/java/pull/11)

## [1.7.0-RC3] - 2018-04-16
### Changed
- Release process improvements.

## [1.7.0-RC2] - 2018-04-15
### Added
- An [examples](https://github.com/PDAL/java/tree/1.7.0-RC2/examples/pdal-jni) project.

### Changed
- Release process improvements.

## [1.7.0-RC1] - 2018-04-15
### Changed
- Moved from the PDAL repo and established own lifecycle.

[Unreleased]: https://github.com/PDAL/java/compare/2.1.2...HEAD
[2.1.2]: https://github.com/PDAL/java/compare/2.0.0...2.1.2
[2.0.0]: https://github.com/PDAL/java/compare/1.9.0...2.0.0
[1.9.0]: https://github.com/PDAL/java/compare/1.8.6...1.9.0
[1.8.6]: https://github.com/PDAL/java/compare/1.8.5...1.8.6
[1.8.5]: https://github.com/PDAL/java/compare/1.8.4...1.8.5
[1.8.4]: https://github.com/PDAL/java/compare/1.8.3...1.8.4
[1.8.3]: https://github.com/PDAL/java/compare/1.8.2...1.8.3
[1.8.2]: https://github.com/PDAL/java/compare/1.8.1...1.8.2
[1.8.1]: https://github.com/PDAL/java/compare/1.7.0-RC4...1.8.1
[1.7.0-RC4]: https://github.com/PDAL/java/compare/1.7.0-RC3...1.7.0-RC4
[1.7.0-RC3]: https://github.com/PDAL/java/compare/1.7.0-RC2...1.7.0-RC3
[1.7.0-RC2]: https://github.com/PDAL/java/compare/1.7.0-RC1...1.7.0-RC2
[1.7.0-RC1]: https://github.com/PDAL/java/compare/1.7.0-RC1...1.7.0-RC1
27 changes: 13 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,16 +18,16 @@ resolvers ++= Seq(
)

libraryDependencies ++= Seq(
"io.pdal" %% "pdal" % "2.0.0", // core library
"io.pdal" % "pdal-native" % "2.0.0" // jni bindings
"io.pdal" %% "pdal" % "2.1.2", // core library
"io.pdal" % "pdal-native" % "2.1.2" // jni bindings
)
```

It's required to have native JNI binary in `java.library.path`:
If you would like to use your own bindings, it is necessary to set `java.library.path`:

```scala
// Mac OS X example with manual JNI installation
// cp -f native/target/resource_managed/main/native/x86_64-darwin/libpdaljni.2.0.dylib /usr/local/lib/libpdaljni.2.0.dylib
// cp -f native/target/resource_managed/main/native/x86_64-darwin/libpdaljni.2.1.dylib /usr/local/lib/libpdaljni.2.1.dylib
// place built binary into /usr/local/lib, and pass java.library.path to your JVM
javaOptions += "-Djava.library.path=/usr/local/lib"
```
Expand All @@ -37,18 +37,17 @@ Dependency contains bindings for `x86_64-darwin` and `x86_64-linux`, other versi

## PDAL-Scala

Scala API to build pipeline expressions instead of writing a raw JSON.
Scala API allows to build pipeline expressions instead of writing a raw JSON.

```scala
libraryDependencies ++= Seq(
"io.pdal" %% "pdal-scala" % "2.0.0", // scala core library
"io.pdal" % "pdal-native" % "2.0.0" // jni bindings
"io.pdal" %% "pdal-scala" % "2.1.2", // scala core library
"io.pdal" % "pdal-native" % "2.1.2" // jni bindings
)
```

Scala API covers PDAL 1.8.x but is compatible with PDAL >= 1.4.x, to use any custom DSL
that is not covered by the current Scala API you can use `RawExpr` type to build `Pipeline
Expression`.
Scala API covers PDAL 2.0.x, to use any custom DSL that is not covered by the
current Scala API you can use `RawExpr` type to build `Pipeline Expression`.

### Code examples

Expand All @@ -74,11 +73,11 @@ val expected =
""".stripMargin

// The same, but using scala DSL
val pc: PipelineConstructor = LasRead("/path/to/las") ~ CropFilter() ~ LasWrite("/path/to/new/las")
val pc = ReadLas("/path/to/las") ~ FilterCrop() ~ WriteLas("/path/to/new/las")

// The same, but using RawExpr, to support not implemented PDAL Pipeline API features
// RawExpr accepts a circe.Json type, which can be a json object of any desired complexity
val pcWithRawExpr = LasRead("/path/to/las") ~ RawExpr(Map("type" -> "filters.crop").asJson) ~ LasWrite("/path/to/new/las")
val pcWithRawExpr = ReadLas("/path/to/las") ~ RawExpr(Map("type" -> "filters.crop").asJson) ~ WriteLas("/path/to/new/las")
```

### Demo project example
Expand All @@ -87,12 +86,12 @@ JNI bindings basic usage examples can be found [here](./examples).

## How to compile

Development purposes (including binaries):
Development purposes (including binaries) compilation:
1. Install PDAL (using brew / package managers (unix) / build from sources / etc)
2. Build native libs `./sbt native/nativeCompile` (optionally, binaries would be built during tests run)
3. Run `./sbt core/test` to run PDAL tests

Only Java development purposes:
Only Java development purposes compilation:
1. Provide `$LD_LIBRARY_PATH` or `$DYLD_LIBRARY_PATH`
2. If you don't want to provide global variable you can pass `-Djava.library.path=<path>` into sbt:
`./sbt -Djava.library.path=<path>`
Expand Down
7 changes: 4 additions & 3 deletions build.sbt
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
name := "pdal-jni"

lazy val commonSettings = Seq(
version := "2.0.0" + Environment.versionSuffix,
version := "2.1.2" + Environment.versionSuffix,
scalaVersion := "2.13.1",
crossScalaVersions := Seq("2.13.1", "2.12.10", "2.11.12"),
organization := "io.pdal",
Expand Down Expand Up @@ -57,6 +57,8 @@ lazy val root = (project in file("."))

lazy val `core-scala` = project
.settings(commonSettings: _*)
.settings(Dependencies.macroSettings)
.settings(Dependencies.licenseSettings)
.settings(name := "pdal-scala")
.settings(javah / target := (native / nativeCompile / sourceDirectory).value / "include")
.settings(libraryDependencies ++= Seq(
Expand All @@ -67,9 +69,8 @@ lazy val `core-scala` = project
Dependencies.jtsCore,
Dependencies.scalaTest % Test
))
.settings(headerLicense := Some(HeaderLicense.ALv2("2017", "Azavea")))
.settings(licenses := Seq("Apache-2.0" -> url("https://www.apache.org/licenses/LICENSE-2.0.html")))
.dependsOn(core)
.dependsOn(Environment.dependOnNative(native % Runtime): _*)

lazy val core = project
.settings(commonSettings: _*)
Expand Down
9 changes: 9 additions & 0 deletions core-scala/src/main/scala/io/pdal/pipeline/ExprType.scala
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,10 @@

package io.pdal.pipeline

import io.circe.{Decoder, Encoder}
import io.circe.syntax._
import cats.syntax.either._

import scala.util.Try

trait ExprType {
Expand All @@ -26,6 +30,11 @@ trait ExprType {
}

object ExprType {
implicit def exprTypeEncoder[T <: ExprType]: Encoder[T] = Encoder.instance { _.toString.asJson }
implicit def exprTypeDecoder[T <: ExprType]: Decoder[T] = Decoder.decodeString.emap { str =>
Either.catchNonFatal(ExprType.fromName(str).asInstanceOf[T]).leftMap(_ => "ExprType")
}

def fromName(name: String): ExprType =
Try(FilterTypes.fromName(name))
.getOrElse(Try(ReaderTypes.fromName(name))
Expand Down
23 changes: 17 additions & 6 deletions core-scala/src/main/scala/io/pdal/pipeline/FilterTypes.scala
Original file line number Diff line number Diff line change
Expand Up @@ -20,24 +20,30 @@ sealed trait FilterType extends ExprType { val `type` = "filters" }

object FilterTypes {
case object approximatecoplanar extends FilterType
case object assign extends FilterType
case object chipper extends FilterType
case object cluster extends FilterType
case object colorinterp extends FilterType
case object colorization extends FilterType
case object computerange extends FilterType
case object covariancefeatures extends FilterType
case object cpd extends FilterType
case object crop extends FilterType
case object decimation extends FilterType
case object dem extends FilterType
case object delaunay extends FilterType
case object divider extends FilterType
case object eigenvalues extends FilterType
case object estimaterank extends FilterType
case object elm extends FilterType
case object ferry extends FilterType
case object greedyprojection extends FilterType
case object gridprojection extends FilterType
case object groupby extends FilterType
case object hag extends FilterType
case object head extends FilterType
case object hexbin extends FilterType
case object info extends FilterType
case object icp extends FilterType
case object iqr extends FilterType
case object kdistance extends FilterType
Expand All @@ -49,16 +55,21 @@ object FilterTypes {
case object mongus extends FilterType
case object mortonorder extends FilterType
case object movingleastsquares extends FilterType
case object miniball extends FilterType
case object neighborclassifier extends FilterType
case object nndistance extends FilterType
case object normal extends FilterType
case object overlay extends FilterType
case object outlier extends FilterType
case object pclblock extends FilterType
case object planefit extends FilterType
case object pmf extends FilterType
case object poisson extends FilterType
case object python extends FilterType
case object radialdensity extends FilterType
case object range extends FilterType
case object randomize extends FilterType
case object reciprocity extends FilterType
case object reprojection extends FilterType
case object sample extends FilterType
case object smrf extends FilterType
Expand All @@ -72,12 +83,12 @@ object FilterTypes {
case object voxelgrid extends FilterType

lazy val all = List(
approximatecoplanar, chipper, cluster, colorinterp, colorization, computerange, crop,
cpd, decimation, divider, eigenvalues, estimaterank, ferry, greedyprojection, gridprojection, groupby,
hag, head, hexbin, icp, iqr, kdistance, locate, lof, mad, matlab, merge, mongus, mortonorder, movingleastsquares,
normal, outlier, overlay, pclblock, pmf, poisson, python, radialdensity, randomize, range, reprojection,
sample, smrf, sort, splitter, stats, transformation, voxelcenternearestneighbor, voxelcentroidnearestneighbor,
voxelgrid
approximatecoplanar, assign, chipper, cluster, colorinterp, colorization, computerange, covariancefeatures, crop,
cpd, decimation, dem, delaunay, divider, eigenvalues, estimaterank, elm, ferry, greedyprojection, gridprojection, groupby,
hag, head, hexbin, info, icp, iqr, kdistance, locate, lof, mad, matlab, merge, mongus, mortonorder, movingleastsquares,
miniball, neighborclassifier, nndistance, normal, outlier, overlay, pclblock, planefit, pmf, poisson, python, radialdensity,
randomize, reciprocity, range, reprojection, sample, smrf, sort, splitter, stats, transformation, voxelcenternearestneighbor,
voxelcentroidnearestneighbor, voxelgrid
)

def fromName(name: String): FilterType =
Expand Down
7 changes: 3 additions & 4 deletions core-scala/src/main/scala/io/pdal/pipeline/Implicits.scala
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,10 @@ package io.pdal.pipeline
import io.pdal.PointCloud
import org.locationtech.jts.geom.Coordinate

object Implicits extends Implicits
object Implicits extends Implicits with Serializable

trait Implicits extends Serializable {
trait Implicits {
implicit class withPointCloudMethods(pointCloud: PointCloud) {
def getCoordinate(i: Int) =
new Coordinate(pointCloud.getX(i), pointCloud.getY(i), pointCloud.getZ(i))
def getCoordinate(i: Int) = new Coordinate(pointCloud.getX(i), pointCloud.getY(i), pointCloud.getZ(i))
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
/*
* Copyright 2020 Azavea
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package io.pdal.pipeline

import io.pdal.Pipeline

import io.circe.{Decoder, Encoder, Json}
import io.circe.syntax._

case class PipelineConstructor(list: List[PipelineExpr]) {
def ~(e: PipelineExpr): PipelineConstructor =
e match {
case ENil => this
case _ => PipelineConstructor(list :+ e)
}
def ~(e: Option[PipelineExpr]): PipelineConstructor = e.fold(this)(this ~ _)
def map[B](f: PipelineExpr => B): List[B] = list.map(f)
def toPipeline: Pipeline = Pipeline(this.asJson.noSpaces)
}

object PipelineConstructor {
implicit val pipelineConstructorEncoder: Encoder[PipelineConstructor] = Encoder.instance { constructor =>
Json.obj(
"pipeline" -> constructor.list.flatMap {
_.list.flatMap {
case RawExpr(json) => json.asObject
case expr => expr.asJson.asObject
}.map {
_.remove("class_type") // remove type
.filter { case (_, value) => !value.isNull } // cleanup options
}
}.asJson
)
}
implicit val pipelineConstructorDecoder: Decoder[PipelineConstructor] = Decoder.instance {
_.downField("pipeline").as[PipelineConstructor]
}

implicit def pipelineConstructorToJson(expr: PipelineConstructor): Json = expr.asJson
implicit def pipelineConstructorToString(expr: PipelineConstructor): String = expr.asJson.noSpaces
}
Loading

0 comments on commit 7acf3e7

Please sign in to comment.