Skip to content
Snippets Groups Projects

Spark init code

  • Clone with SSH
  • Clone with HTTPS
  • Embed
  • Share
    The snippet can be accessed without any authentication.
    Authored by Frank Sauerburger

    Code snippets needed to start a python-based or scala-based Spark session

    Edited
    dask_on_yarn.py 312 B
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12 
    from dask_yarn import YarnCluster
from dask.distributed import Client


cluster = YarnCluster(environment='venv:///opt/hadoop/pyenv/',
                      worker_vcores=2,
                      worker_memory="4GiB")

# Connect to the cluster
client = Client(cluster)
cluster.adapt(2, 10)
cluster.dashboard_link
    init.py 335 B
    init.scala 804 B
    0% or .
    You are about to add 0 people to the discussion. Proceed with caution.
    Finish editing this message first!
    Please register or to comment