sparklyr.sedona from GitHub using the
remotes package , run
remotes::install_github(repo = "apache/incubator-sedona", subdir = "R/sparklyr.sedona")
In this blog post, we will provide a quick introduction to
sparklyr.sedona, outlining the motivation behind this
sparklyr extension, and presenting some example
sparklyr.sedona use cases involving Spark spatial RDDs, Spark dataframes, and visualizations.
A suggestion from the mlverse survey results earlier this year mentioned the need for up-to-date R interfaces for Spark-based GIS frameworks. While looking into this suggestion, we learned about Apache Sedona, a geospatial data system powered by Spark that is modern, efficient, and easy to use. We also realized that while our friends from the Spark open-source community had developed a
sparklyr extension for GeoSpark, the predecessor of Apache Sedona, there was no similar extension making more recent Sedona functionalities easily accessible from R yet. We therefore decided to work on
sparklyr.sedona, which aims to bridge the gap between Sedona and R.
The lay of the land
We hope you are ready for a quick tour through some of the RDD-based and Spark-dataframe-based functionalities in
sparklyr.sedona, and also, some bedazzling visualizations derived from geospatial data in Spark.
In Apache Sedona, Spatial Resilient Distributed Datasets(SRDDs) are basic building blocks of distributed spatial data encapsulating “vanilla” RDDs of geometrical objects and indexes. SRDDs support low-level operations such as Coordinate Reference System (CRS) transformations, spatial partitioning, and spatial indexing. For example, with
sparklyr.sedona, SRDD-based operations we can perform include the following:
- Importing some external data source into a SRDD:
library(sparklyr) library(sparklyr.sedona) sedona_git_repo <- normalizePath("~/incubator-sedona") data_dir <- file.path(sedona_git_repo, "core", "src", "test", "resources") sc <- spark_connect(master = "local") pt_rdd <- sedona_read_dsv_to_typed_rdd( sc, location = file.path(data_dir, "arealm.csv"), type = "point" )
- Applying spatial partitioning to all data points:
sedona_apply_spatial_partitioner(pt_rdd, partitioner = "kdbtree")
- Building spatial index on each partition:
sedona_build_index(pt_rdd, type = "quadtree")
- Joining one spatial data set with another using “contain” or “overlap” as the join predicate:
polygon_rdd <- sedona_read_dsv_to_typed_rdd( sc, location = file.path(data_dir, "primaryroads-polygon.csv"), type = "polygon" ) pts_per_region_rdd <- sedona_spatial_join_count_by_key( pt_rdd, polygon_rdd, join_type = "contain", partitioner = "kdbtree" )
It is worth mentioning that
sedona_spatial_join() will perform spatial partitioning and indexing on the inputs using the
index_type only if the inputs are not partitioned or indexed as specified already.
From the examples above, one can see that SRDDs are great for spatial operations requiring fine-grained control, e.g., for ensuring a spatial join query is executed as efficiently as possible with the right types of spatial partitioning and indexing.
Finally, we can try visualizing the join result above, using a choropleth map:
which gives us the following:
Wait, but something seems amiss. To make the visualization above look nicer, we can overlay it with the contour of each polygonal region:
contours <- sedona_render_scatter_plot( polygon_rdd, resolution_x = 1000, resolution_y = 600, output_location = tempfile("scatter-plot-"), boundary = c(-126.790180, -64.630926, 24.863836, 50.000), base_color = c(255, 0, 0), browse = FALSE ) sedona_render_choropleth_map( pts_per_region_rdd, resolution_x = 1000, resolution_y = 600, output_location = tempfile("choropleth-map-"), boundary = c(-126.790180, -64.630926, 24.863836, 50.000), base_color = c(63, 127, 255), overlay = contours )
which gives us the following:
With some low-level spatial operations taken care of using the SRDD API and the right spatial partitioning and indexing data structures, we can then import the results from SRDDs to Spark dataframes. When working with spatial objects within Spark dataframes, we can write high-level, declarative queries on these objects using
dplyr verbs in conjunction with Sedona spatial UDFs, e.g. , the following query tells us whether each of the
8 nearest polygons to the query point contains that point, and also, the convex hull of each polygon.
tbl <- DBI::dbGetQuery( sc, "SELECT ST_GeomFromText("POINT(-66.3 18)") AS `pt`" ) pt <- tbl$pt[] knn_rdd <- sedona_knn_query( polygon_rdd, x = pt, k = 8, index_type = "rtree" ) knn_sdf <- knn_rdd %>% sdf_register() %>% dplyr::mutate( contains_pt = ST_contains(geometry, ST_Point(-66.3, 18)), convex_hull = ST_ConvexHull(geometry) ) knn_sdf %>% print()
# Source: spark<?> [?? x 3] geometry contains_pt convex_hull <list> <lgl> <list> 1 <POLYGON ((-66.335674 17.986328… TRUE <POLYGON ((-66.335674 17.986328,… 2 <POLYGON ((-66.335432 17.986626… TRUE <POLYGON ((-66.335432 17.986626,… 3 <POLYGON ((-66.335432 17.986626… TRUE <POLYGON ((-66.335432 17.986626,… 4 <POLYGON ((-66.335674 17.986328… TRUE <POLYGON ((-66.335674 17.986328,… 5 <POLYGON ((-66.242489 17.988637… FALSE <POLYGON ((-66.242489 17.988637,… 6 <POLYGON ((-66.242489 17.988637… FALSE <POLYGON ((-66.242489 17.988637,… 7 <POLYGON ((-66.24221 17.988799,… FALSE <POLYGON ((-66.24221 17.988799, … 8 <POLYGON ((-66.24221 17.988799,… FALSE <POLYGON ((-66.24221 17.988799, …
The author of this blog post would like to thank Jia Yu, the creator of Apache Sedona, and Lorenz Walthert for their suggestion to contribute
sparklyr.sedona to the upstream incubator-sedona repository. Jia has provided extensive code-review feedback to ensure
sparklyr.sedona complies with coding standards and best practices of the Apache Sedona project, and has also been very helpful in the instrumentation of CI workflows verifying
sparklyr.sedona works as expected with snapshot versions of Sedona libraries from development branches.
The author is also grateful for his colleague Sigrid Keydana for valuable editorial suggestions on this blog post.
That’s all. Thank you for reading!