# DS Utility Modules

Utility modules for working with Airflow, AWS, Spark, and hopefully more.

## Spark
Every DevHub container has the ability to run a single-node Spark instance.
It is not ideal, but can be handy when trying to run your pipeline on subsets,
or is necessary if you want to save something in Delta format. For these purposes
you can use the `get_spark` (or `getSpark` for backwards compatibility) function.
It will get you a `SparkSession` that works inside DevHub, for our S3 etc.

## Airflow
`write_to_xcom` can be used to pass data between two tasks. 

## AWS
Utility functions to retrieve parameters from SSM's parameter store or secrets from secret manager.

# Change Log

---

## 1.2.4 - 2026-04-01

---

## Changed

- PySpark bump to 3.5.8
- Delta Lake Maven package bump from `delta-core_2.12:2.2.0` to `delta-spark_2.12:3.3.2` for Spark 3.5.x compatibility
- `pyarrow` constrained to `<13.0.0` to stay within Spark 3.5's documented support range
- Fixed the S3A credentials provider config key to use the `spark.hadoop.*` namespace

---

## 1.2.1 - 2024-01-30

---

## Added

- `getGlue` and `get_glue` to create the spark session with Glue support

---

## 1.1.0 - 2023-06-20

---

## Changed

- Scala 2.13 > 2.12
- `get_spark` actually works now

---

## 1.0.1 - 2023-06-19

---

## Added

- `__init__.py` with `get_spark` alias

## Changed

- PySpark bump to 3.3.2
- Delta bump to 2.13
- Removed `hurry.filesize` and `psutils` from deps
