Instructions for cross-compiling Ballista to the Raspberry Pi

[andygrove]

Which issue does this PR close?

Closes #247 .

Rationale for this change

Raspberry Pi's are a fun and relatively inexpensive way to experiment with truly distributed computing.

What changes are included in this PR?

Additional content for the user guide.

Are there any user-facing changes?

No.

[codecov-commenter]

Codecov Report

Merging #263 (d9af378) into master (f7bd7b9) will decrease coverage by 1.08%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #263      +/-   ##
==========================================
- Coverage   76.81%   75.72%   -1.09%     
==========================================
  Files         133      143      +10     
  Lines       23295    23840     +545     
==========================================
+ Hits        17895    18054     +159     
- Misses       5400     5786     +386     

Impacted Files	Coverage Δ
datafusion/src/physical_plan/memory.rs	67.27% <0.00%> (-13.17%)	⬇️
datafusion/src/physical_plan/explain.rs	51.35% <0.00%> (-6.23%)	⬇️
datafusion/src/physical_plan/empty.rs	83.82% <0.00%> (-5.73%)	⬇️
datafusion/tests/custom_sources.rs	71.87% <0.00%> (-5.18%)	⬇️
benchmarks/src/bin/tpch.rs	30.82% <0.00%> (-4.54%)	⬇️
datafusion/src/physical_plan/limit.rs	80.83% <0.00%> (-4.26%)	⬇️
datafusion/src/error.rs	51.85% <0.00%> (-3.71%)	⬇️
datafusion/src/physical_plan/coalesce_batches.rs	81.35% <0.00%> (-3.60%)	⬇️
datafusion/src/physical_plan/udaf.rs	71.42% <0.00%> (-3.58%)	⬇️
datafusion/src/physical_plan/expressions/mod.rs	67.85% <0.00%> (-3.58%)	⬇️
... and 81 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f7bd7b9...d9af378. Read the comment docs.

[alamb]

I tried out the instructions for cross compilation and they worked well. I don't have a Raspberry Pi so I can't actually test out what I built, but the instructions seems to work 👍

[andygrove]

For the record, I am now able to run benchmarks against the 100GB data set on my Pi Kubernetes cluster:

apiVersion: v1
kind: Pod
metadata:
  name: tpch
  namespace: default
spec:
  containers:
    - image: andygrove/ballista-arm64
      command: [ "/tpch",
                 "benchmark",
                 "--query=1",
                 "--path=/mnt/tpch/parquet-sf100-partitioned/",
                 "--format=parquet",
                 "--concurrency=24",
                 "--iterations=1",
                 "--debug",
                 "--host=ballista-scheduler",
                 "--port=50050"]
      imagePullPolicy: Always
      name: tpch
      volumeMounts:
          - mountPath: /mnt/tpch/parquet-sf100-partitioned/
            name: data
  restartPolicy: Never
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: data-pv-claim

$ microk8s.kubectl get pods
NAME                   READY   STATUS      RESTARTS   AGE
ballista-scheduler-0   1/1     Running     1          17h
ballista-executor-5    1/1     Running     1          16h
ballista-executor-2    1/1     Running     1          16h
ballista-executor-3    1/1     Running     1          16h
ballista-executor-0    1/1     Running     5          17h
ballista-executor-4    1/1     Running     2          16h
ballista-executor-1    1/1     Running     1          17h
tpch                   0/1     Completed   0          13m

Results

+--------------+--------------+------------+--------------------+--------------------+-------------------+--------------------+--------------------+----------------------+-------------+
| l_returnflag | l_linestatus | sum_qty    | sum_base_price     | sum_disc_price     | sum_charge        | avg_qty            | avg_price          | avg_disc             | count_order |
+--------------+--------------+------------+--------------------+--------------------+-------------------+--------------------+--------------------+----------------------+-------------+
| A            | F            | 3775127758 | 5660776097194.455  | 5377736398183.935  | 5592847429515.929 | 25.499370423275426 | 38236.11698430493  | 0.050002243530928955 | 148047881   |
| N            | F            | 98553062   | 147771098385.98013 | 140384965965.03497 | 145999793032.7757 | 25.501556956882876 | 38237.19938880454  | 0.049985284338054006 | 3864590     |
| N            | O            | 7436302959 | 11150725648169.863 | 10593195276359.283 | 11016932215670.58 | 25.500009433521782 | 38237.227663621445 | 0.0499979183499098   | 291619616   |
| R            | F            | 3775724970 | 5661603032745.348  | 5378513563915.405  | 5593662252666.916 | 25.50006628406532  | 38236.697258453016 | 0.05000130433965413  | 148067261   |
+--------------+--------------+------------+--------------------+--------------------+-------------------+--------------------+--------------------+----------------------+-------------+
Query 1 avg time: 104835.61 ms

[jorgecarleitao]

That is brutal 🤯