For my travel site side hustle (beatthatflight.com.au/), I find and publish cheap flights for Australians. Sounds simple, but even if say, someone gave me the flights each day, there's posting the, sometimes to deals sites, as well as then editing it and posting it on my site, and out to social media, and then to specific mailing lists on mailchimp, depending on the source city of the deal, or type of post.
A LOT of that becomes tedious. Mailchimp can auto post to FB/twitter/instagram etc, but even converting the blog post into mailchimp emails gets tiring, as you need to choose a feature photo, the specific mailing list etc.
I'm a selenium guy, so have built some scripts to minimise some of it, but the rush each time of finding a deal still pains me when I think of the tedium ahead of publishing it.
Ideally, my goal would be to submit [to][from][when] to a script and the whole process would be automatic. I can see that selenium could do it, but my goodness, it'd be slow and potentially flaky all via the web UI for those sites.
I'd love suggestions, examples of automation you've done, and tools used?
Others tell me if my server is getting swamped, forward important emails like renewing a cert, a deploy failed, bug in prod or something else that requires often immediate attention. Chat widgets from sites in production also go directly to Slack so I can reply quickly if needed. (Gmail, Sentry, Freshchat, Ploi etc, integrate them all and filter out what's critical).
Everything generally operates on a 15 min interval or so it's not noisy and I've pretty much tuned it to only receiving specific stuff I want to see. Having a central assistant like hub to go to is very useful and time saving.
* A change in GBP -> NOK exchange rate, to indicate if it's a good time to buy ready for my next trip to Norway.
* Alerts on flight prices to help me purchase flights when they're at their cheapest.
* Daily digest email with content I'm interested in from HN and some web comics.
* A summary of new content coming into my RSS reader.
It works really well for this sort of stuff and I found it fairly easy to get up and running.
Deleted Comment
I wrote this https://github.com/buzzlawless/ynab-live-import to import credit card transactions instantly with no need of giving up my bank credentials.
The whole stack runs on Amazon Web Services. Simple Email Service receives a notification email from the bank that I've made a purchase, saves it to S3, and triggers a particular lambda function tailored to whichever bank the notification came from. The lambda function retrieves the email from S3, parses it for transaction data (account, payee, amount, date), and writes that data to a DynamoDB table. The table has a stream enabled, which triggers another lambda function when the table is updated. The function reads the transaction data from the stream and posts the transaction to YNAB using their API.
I've mentioned this on HN once or twice before and got some positive interest with people even submitting pull requests, which is awesome :) Going to find the time soon to review those and maybe add more features
I am constantly amazed by the people that are willing to do this for various value-added finance sites; people that I would normally consider sane. Unless your bank provides (and you used) a read-only account authentication, if something goes wrong and your money disappears from the bank, the bank is going to tell you to take a hike. I can't imagine taking this kind of risk with my money.
Why don't they have oauth for this?
If your cash spending is sufficiently small that you don't care much to tracking each cash transaction individually, you can just track money as a category instead of an account - just record an outflow transaction in your bank account when you withdraw.
One of our primary CAD suites was Rhino which is very nice and has a great python API. So I wrote a full-fledged 2.5D CAM processor for that. This allowed us to batch process hundreds of these parts with a single click.
Disclaimer: I work with the Playwright team at Microsoft.
https://github.com/microsoft/WinAppDriver
As we build out Playwright, we hope to learn a lot about how to make automation more reliable, ergonomic, and enjoyable for developers. Ideally, we could then apply those principles/learnings to desktop, mobile, etc. automation too. Stay tuned!
> setTimeout-free automation
How ? you still need an upper limit for operations
To make the act of selecting elements more resilient to change, Playwright also supports a collection of “selector engines” that allow you to choose the best strategy for selecting elements (via CSS selector, xpath, text content). Additionally, you can author entirely custom selector engines, and compose them together with the built-in types. This is still somewhat experimental, but we think this can enable another level of robustness to test automation: https://github.com/microsoft/playwright/blob/master/docs/sel....
Finally, in addition to waiting for element visibility, Playwright allows you to wait for specific page events to occur (e.g. network request work made). This enables your code to leverage deterministic “signals”, not timeouts (which are the source of a lot of flakiness!). Overall, we want it to be possible to author automation that is entirely event-driven, and includes zero arbitrary timeouts.
Edit: Missed the Microsoft ref. Not so keen.
The team is working very hard to be open/transparent/available (https://github.com/microsoft/playwright), and we’re very excited to build a better automation stack with the help of the community. I only ask since I’d love to hear any feedback for how we could improve the way we run/position the project. Thanks!
I’d love to hear a bit more about the scenario you have in mind, since that might help inform some improvements we can make. Thanks!
Out of curiosity: are there any specific extensions from puppeteer-extra that you’ve used and/or would be interested to see?
PS: I remember interacting with you from the early days of code-push I think :)
I’m glad to hear the examples worked out of the box! We’ve been iterating on the docs and API quite a bit (based on feedback), so don’t hesitate to let us know if/how they can be improved: https://github.com/microsoft/playwright.
Are you currently using Kantu? If so, I’d love to hear more about your specific use cases, and which capabilities you’d be interested to see. Thanks!
https://github.com/microsoft/playwright
Generally, I've had better luck using undocumented APIs for this kind of stuff. I was a heavy user of Selenium to automate many a task (I work in wholesale construction goods, tons of automations needed everywhere).
But then discovered that using internal APIs (which surprisingly don't change much) is far easier than trying to exception manage changing UIs.
Open up dev tools on your browser and watch the GETS/POSTS as you complete your daily tedium. I use Python (usually just requests and beautifulsoup is enough) to mimic the calls and I have yet to find a use case where I wasn't able to automate something online.
I'm currently developing an SDK for our 18th-century ERP system, which has no API. I've also automated getting shipping container tracking data from a various shipping lines and railroad company websites, many with complicated login processes.
Happy to chat, email in profile.
Were you using an android emulator on desktop or there's some way to easily do this on android?
Man-in-the-middle Android apps to discover (sometimes different API backend).
I did the same with a shipping line which turned out to use a SOAP backend.
I read your post.
Paused.
Opened up debugger on the site I'm currently selenium-ising.
reloaded.
boom. there it is. The request, and beautiful json response.
So...so..so much simpler.
Thank you, my evening is sorted ;)
I had the same experience a couple years ago after spending months selenium-ing our ERP software.
- I mostly automate my bookkeeping with a set of recurring & dependent taskwarrior tasks and scripts as annotations that I run with taskopen[1]. That's creating a bunch of folders, turning some emails in mutt into PDFs, gathering PDFs from emails, fetching bills with selenium, moving files from $DOWNLOADS into the appropriate bookeeping folder, putting a date on some files, turning the whole thing into a zip file, and sending it to the bookkeeper with mailx.
- I automated the email send of my daily summary to my clients with mailx (so I can send it directly from vim)
- I automated turning screen recordings into thumbnails+mp4 link (since GitHub only supports gifs)
- I automated making before/after screen recordings for when I do noticeable performance improvements (page load/animations)
- I automated booting/killing my development servers
- I automated making PRs with `hub pr` (finding the origin/upstream, putting labels, etc.)
- I bound to a key combo switching to the logs of specific development servers
- I turned my client's time tracking (tempo) into a CLI because I got tired of using the UI to say I worked X hours on that ticket and 7.5 - X on the other. Now I only do `tempo log $ticket1 2h $ticket2 3h $supportTicket rest`
[1]: https://github.com/ValiValpas/taskopen
There's the `excel` subcommand you probably don't need and you'll need to export the JIRA_TEMPO_USER and JIRA_TEMPO_TOKEN variables before running it.
- `JIRA_TEMPO_TOKEN` can be found in the Tempo Settings
- `JIRA_TEMPO_USER` can be found by inspecting the requests sent to tempo from JIRA IIRC.
[1]: https://github.com/charlespwd/dotfiles/blob/master/bin/tempo
EDIT: Confirmed, I found my `JIRA_TEMPO_USER` as the only accountID sent as parameter the POST https://app.tempo.io/rest/tempo-timesheets/5/private/days/se... when booting up the app in JIRA.
- switch on when I get home and sun is set
- switch on when I'm home and the sun sets
- switch on in the morning, full brightness, coldest color to help waking up (really helps me a lot and is not as stressful as an alarm clock)
- automatically turn light off when I (my smartphone) leave home (not visible in the WLAN for 10 minutes)
Will look into more automations there. Perhaps something about turning the heat up in the bathroom in the morning when the windows are closed (there are sensors for that).
I ran NodeRed on a Raspberry Pi too for some time, but I found that helped me only little. Mainly did logfile analysis through that but did not look into other use cases for me.
For work I've got the de-facto standard for the printing industry, I'm doing everything possible with that.
I was already looking for an excuse to learn Python and this was perfect.
Instead of recreating the slides I created templates (just the last set of presentations) and update them with info directly queried from the database, using python-pptx and pyodbc.
This isn’t exactly an example of a “tool”, but I think the most important part isn’t really the tool, but to be able to identify what you can automate. How you do it might range from DIY to paying someone to code for you, but the feeling after getting rid of manual processes is great!
https://xkcd.com/1205/