在windows上运行spark
在windows上打开sparkG:\Windows\System32>pysparkPython 3.7.4 (tags/v3.7.4:e09359112e, Jul8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)] on win32Type "help", "copyright", "credits" or "license" for more i
·
在windows上打开spark
G:\Windows\System32>pyspark
Python 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
20/05/12 18:19:49 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/05/12 18:19:55 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
20/05/12 18:19:56 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
20/05/12 18:19:57 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 2.1.1
/_/
Using Python version 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019 20:34:20)
SparkSession available as 'spark'.
>>> from pyspark.sql import SparkSession
>>> spark=SparkSession \
... .builder \
... .master("yarn-client") \
... .appName("New SS") \
... .config("spark.executor.instances","10") \
... .config("spark.executor.memory","10g") \
... .getOrCreate()
>>> spark
<pyspark.sql.session.SparkSession object at 0x0000012A9BC5AD88>
>>>
更多推荐
所有评论(0)