Readit News logoReadit News
ejwhite commented on Python – Create large ZIP archives without memory inflation   github.com/BuzonIO/zipfly... · Posted by u/pythonscripts2
2bluesc · 5 years ago
I built a streaming zip app using nothing more then the Python stdlib zip implementation and some os primitives.

It runs on a small embedded device that can stream zip archives many times larger then the disk or system ram without any issue.

Example Python Falcon Proof of Concept:

https://gist.github.com/kylemanna/1e22bbf31b7e5ae84bbdfa32c6...

Other then what Python's zipfile buffers in memory, my implementation shouldn't use much more then a os.pipe()'s buffer (typically 64kB?).

ejwhite · 5 years ago
Interesting.

I need to open a very large CSV file in Python, which is around 25GB in .zip format. Any idea how to do this in a streaming way, i.e. stopping after reading the first few thousand rows?

ejwhite commented on I've designed UI Flow Cards because I couldn't find one like it   flowcards.io/... · Posted by u/fatihguner
danmostudco · 5 years ago
For folks wondering "when would I use this?" I have used tools like these early in the design process for broad, net-new features or redesigns in apps where driving alignment early on is crucial to success. In a more buttoned-up Agency process you might see something like:

General Idea > White Board Session > High Level Screen Flows (this) > Individual Wireframes > High-Fidelity UI > Coding.

These are useful tools to step back from individual screens and think of the broader ecosystem of the feature the team is trying to build. If actions on one page affect another page further down a flow, it's easy to reference that in a meeting by having it all laid out in a lower-fidelity, non-distracting way. For example: "if I add a user to this group, where will that user and her derivative information pop up across the experience?" becomes an easier discussion with an artifact like this.

I find when working early on with multi-stakeholder or multi-department initiatives, some high-level UX documentation can be helpful in ensuring when everyone goes back to their desks, each group has the same, general picture in their head.

ejwhite · 5 years ago
Nice explanation, thank you for sharing. Do you have any experience working with mobile apps? I'm wondering if any aspects of the design process you outlined differ when working with mobile apps?
ejwhite commented on I've designed UI Flow Cards because I couldn't find one like it   flowcards.io/... · Posted by u/fatihguner
fatihguner · 5 years ago
Thank you for the motivation!

I don't like these kinds of products sold with a monthly fee, as a designer. That's because I don't think a monthly fee.

What I'm thinking is to create another set, with mobile flows in mind, as an upsell to this product.

Maybe after that, a wireframe kit, as well. Let's see :)

ejwhite · 5 years ago
Well you have a vote right here for mobile flows, I think that would be awesome!

May I ask... do you ever construct information architectures (aka sitemaps) for your clients? If so, what tools would you use for this?

ejwhite commented on New boson appears in nuclear decay, could break standard model if confirmed   arstechnica.com/science/2... · Posted by u/furcyd
reggieband · 6 years ago
Not really directly related to the article, but YouTube recently recommended to me a series of videos from "Strange Loop" which is some kind of tech conference. One of the talks was "Jagged, ragged, awkward arrays" by Jim Pivarski [1]. It is fairly narrow in scope talking about a very particular data modelling problem but it introduced me to the kind of data processing physicists do to make these kinds of discoveries. Just wanted to share in case anyone else finds this kind of thing interesting.

1. https://www.youtube.com/watch?v=2NxWpU7NArk

ejwhite · 6 years ago
Heh, I went to grad school with him. Haven't heard what he's been up to in over a decade.

I was solidly in the "C++" column back then, but have since become a data scientist who now uses numpy/Python for all machine learning. That talk was a very interesting, helped me to understand what they're doing in my old field these days. Thanks for sharing.

u/ejwhite

KarmaCake day2July 24, 2015View Original