Getting pyspark working

[thomashan]

Hi there.
I couldn't find any documentation for initializing pyspark.

I've initialized the spark sessions with

%%classpath add mvn
org.apache.spark spark-sql_2.11 2.3.1

followed by

%%spark

Everything is all good.

Then when I go to link the spark session in python I don't have an entry point to the spark because spark has been initialized through the jar files and not through python.

Is there a tutorial or documentation which outlines how to use spark in python or through pyspark?

Cheers.

[scottdraves]

Hi Thomas thank you for your question.
We do not yet support pyspark, just scala spark.
Expanding into this area is on our agenda: #7544

[thomashan]

Thanks for the reply.
Should I close the issue or do you want it open for tracking?

[scottdraves]

Please leave this one open.

[marcusianlevine]

Is there any movement on this? We'd really love to see BeakerX's capabilities expanded to include PySpark

[Ohtar10]

I would also love to try BeakerX with PySpark :)

[Ben-Epstein]

Is this improvement still being looked at? Native PySpark support would be really great!

[sapphire008]

I also would love to see BeakerX working with PySpark, a much needed feature.

[ei-grad]

This issue lacks for 👍 responses on the first post. Mine was added second.