Hahaha. Don't ask how the sausage gets made, so long as the sausage machine is readily maintainable by more than one person or has documentation.

Partially explains how murder came to be https://github.com/lg/murder

> below perf review

That's some cheap bullshit. Fuck marketing-oriented corporate engineering.

It might be rare, but it _can_ happen: https://www.youtube.com/watch?v=ZdIclqha014

It is highly dependent on the snow conditions and the recent weather. Sometimes even just the a couple hours are enough to change the conditions to have a good chance of rollerballs. The climate also has an impact, in my experience more coastal areas have more periods when they form.

And in some cases the rollerballs get too tall for the bonding strength of the snow, so they break into parts that can restart the cycle if the slope is steep enough.

Sounds to me like there's a long blog post waiting to be written.

If you are interested in reading internals of using Parquet with ClickHouse, do read following articles:

- https://clickhouse.com/blog/apache-parquet-clickhouse-local-...

- https://clickhouse.com/blog/apache-parquet-clickhouse-local-...

DuckDB?

https://duckdb.org/2024/03/29/external-aggregation.html

https://duckdb.org/2021/06/25/querying-parquet.html

If your DB is mysql or postgres, then you could read a stream from parquet, transform inline and write out to your DB

https://duckdb.org/2024/01/26/multi-database-support-in-duck...

And an unrelated, but interesting read about the parquet bomb

https://duckdb.org/2024/03/26/42-parquet-a-zip-bomb-for-the-...

I work with pyspark and parquet quite a lot. I never had to deal with parquet outside spark, but this is how I would do this:

- Write a pandas_udf function in pyspark.

- Parition your data into smaller bits so that the pandas_udf does not get too much data at the same time.

Something like:

```

from pyspark.sql import SparkSession

import pyspark.sql.functions as f

@f.pandas_udf(return_type=whatever)

def ingest(doc: pd.Series): # doc is a pandas series now

    # your processing goes here -> write to DB e.t.c

    pd_series_literal = Create a pd.Series that just contains the integer 0 to make spark happy

    return pd_series_literal

spark = SparkSession.builder.getOrCreate()

df = spark.read.parquet("s3 path")

df = df.repartition(1000). # bump up this number if you run into memory issues

df = df.withColumn("foo", ingest(f.col("doc_column"))

```

Now the trick is, you can limit how much data is given to your pandas_udf by repartitioning your data. The more the partitions, the smaller the pd.Series that your pandas_udf gets. There's also the `spark.sql.execution.arrow.maxRecordsPerBatch` config that you can set in spark to limit memory consumption.

^ Probably overkill to bring spark into the equation, but this is one way to do it.

You can use a normal udf (i.e `f.udf()`) instead of a pandas_udf, but apparently that's slower due to java <-> python serialization

I can help! Emailing you now :)

Our company (scratchdata.com, open source) is literally built to solve the problem of schlepping large amounts of data between sources and destinations, so I have worked on this problem a lot personally and happy to nerd out about what works.

Wouldn't partial reads fix the RAM problem? e.g. something like this: https://stackoverflow.com/a/69888274

It might not be fast, but a quick 1-off solution that you let run for a while would probably do that job. There shouldn't be a need to load the whole file into memory.

Have you given DuckDB a try? I'm using it to shuttle some hefty data between postgres and some parquet files on S3 and it's only a couple lines. Haven't noted any memory issues so far

Using polars in Python I've gotten similar to work, using LazyFrame and collect in streaming mode:

``` df = pl.scan_parquet('tmp/'+DUMP_NAME+'_cleaned.parquet')

with open('tmp/'+DUMP_NAME+'_cleaned.jsonl', mode='w', newline='\n', encoding='utf8') as f: for row in df.collect(streaming=True).iter_rows(named=True): row = {k: v for k, v in row.items() if (v is not None and v != [] and v != '')} f.write(json.dumps(row, default=str) + '\n') ```

My suggestion is to load each row group individually, as they generally will be much smaller than your total file size. You can do this via pyarrow.ParquetFile.read_row_group. To truly optimize this for reading from s3 you could use fsspec’s open_parquet_file library which would allow you to only load each row group one at a time.

I had similiar issue, but for aggreagations. Use case was to "compress" large datasets into smaller aggregations for insertion into a costly db. At first we used duckdb but memory became an issue there and we also bumped into a couple of issues with how duckdb handles arrays. We then moved this workload to clickhouse local, which was faster and had more fine tuning options to our liking. in this case was limiting ram usage with i.e. max_bytes_before_external_group_by

I'm puzzled as to why this is a problem that has lasted months. My phone has enough RAM to work with this file in memory. Do not use pyspark, it is unbelievably slow and memory hogging if you hold it even slightly wrong. Spark is for tb-sized data, at minimum.

Have you tried downloading the file from s3 to /tmp, opening it with pandas, iterating through 1000 row chunks, pushing to DB? The default DF to SQL built into pandas doesn't batch the inserts so it will be about 10x slower than necessary, but speeding that up is a quick google->SO away.

Try pyarrow.ParquetFile.iter_batches()

Streams batches of rows

https://arrow.apache.org/docs/python/generated/pyarrow.parqu...

Edit — May need to do some extra work with s3fs too from what I recall with the default pandas s3 reading

Edit 2 — or check out pyarrow.fs.S3FileSystem :facepalm:

Spend a few bucks on an EC2 instance with a few terabytes of RAM for an hour or so. u-3tb1.56xlarge is about $27/hr.

Understand the format of your data.

Look at the parquet file metadata: use whatever tool you want for that. The Python parquet library is useful and supports s3.

How big are your row groups? If it’s one large row group then you will run into this issue.

What’s the number of rows in each row group?

Perhaps look into using dlt from https://dlthub.com, using pyarrow or polars. It handles large datasets well, especially when using generators to process the data in chunks.

You may want to give octosql a try. I was able to read 100GiB parquet files and aggregate them using less than 100MiB RAM and it is pretty fast too

Maybe check out alteryx designer cloud, airbyte, or estuary?

Should be able to do it with Aws Athena?

No-one with the combination of motivation, time and skills needed get it into Debian. Someone wanted to get a Python implementation in, but it looks like they never found the time.

https://bugs.debian.org/838338

These days Debian packaging has become a bit irrelevant, since you can just shove upstream releases into a container and go for it.

Parquet is a file format. Should a file format be in Debian?

  > apt-cache search hdf5 | wc -l
  134
  > apt-cache search netcdf | wc -l
  70
  > apt-cache search parquet | wc -l 
  0

pandas is a python-centric, tabular data handler that works well in clouds (and desktop Debian). Pandas can read parquet data today, among other libs mentioned. The binary dot-so driver style is single-host centric and not the emphasis of these cloudy projects (and their cloudy funders)

https://pandas.pydata.org/docs/reference/api/pandas.read_par...

https://packages.debian.org/buster/python3-pandas

Perhaps more alarm is called for when this python+pandas and parquet does not work on Debian, but that is not the case today.

ps- data access in clouds often uses the S3:// endpoint . Contrast to a POSIX endpoint using _fread()_ or similar.. many parquet-aware clients prefer the cloudy, un-POSIX method to access data and that is another reason it is not a simple package in Debian today.