- mvn - Pspark-1.6 clean accumulate
- mvn - Pspark-2.1 clean accumulate
Spark is an Apache undertaking publicized as "extremely quick group figuring". It has a flourishing open-source network and is the most dynamic Apache venture right now. Spark gives a quicker and increasingly broad information handling stage. Spark gives you a chance to run projects up to 100x quicker in memory, or 10x quicker on circle, than Hadoop. A year ago, Spark took over Hadoop by finishing the 100 TB Daytona GraySort challenge 3x quicker on one tenth the quantity of machines and it additionally turned into the quickest open source motor for arranging a petabyte.
As of late spark rendition 2.1 was discharged and there is a critical contrast between the 2 forms. Spark 1.6 has DataFrame and SparkContext while 2.1 has Dataset and SparkSession. Presently the inquiry emerges how to compose code with the goal that both the variants of spark are upheld. Luckily expert gives the component of structure your application with various profiles.
This context will get to know how to make your application perfect with various spark renditions. Gives begin by making a vacant expert a chance to extend. You can utilize the expert model quickstart for setting up your undertaking.
Models give an essential format to your venture and expert has a rich gathering of these layouts for every one of your needs. When the venture arrangement is done we have to make 3 modules. Lets name them center, sparkle and spark2 and setting the ancient rarity id of every module to their separate names. For spark modules the antiquity id ought to be spark.
For instance spark2 module would have relic id as spark 2.1.0. Spark module would contain the code for sparkle 1.6 and spark2 would contain the code for flash 2.1.
Begin by making profiles for the 2 spark modules like this in the parent pom:-