Readit News logoReadit News
Posted by u/podoman 3 years ago
Show HN: We’re open-sourcing our session replay toolgithub.com/highlight/high...
Hey HN! We’re open-sourcing highlight.io (https://github.com/highlight/highlight), a session replay and error monitoring tool. Highlight.io gives you a high-precision video-like replay of what users are doing when an error or exception occurs in your web app, along with a full-fledged error monitoring experience (similar to bugsnag, rollbar, etc..).

The main value prop of highlight.io is that we help you understand the full context surrounding an error and allow you to drill down to the code path that a user invoked (i.e user clicked button X, sent network request Y, and backend code Z was executed). Some of our customers compare this to a “web debugger” of sorts. A picture of what this looks like in our app is here [1].

For some background, when we worked at our previous companies as engineers, we encountered hard-to-reproduce issues spanning across both the frontend and backend. The main issues were (1) if a customer complained about a problem, it was hard to reproduce the issue without asking for a screen-share or jumping on a video call; and (2) when viewing errors caught by tools like BugSnag or Rollbar, understanding the triggered code path required stitching together logs, errors, and trace; all from different sources.

Highlight.io is completely open source and written in Go and Typescript. To build the replay capability, we use an open source project called rrweb [2] and have worked closely with their team to add support for features like canvas recording, shadow dom recording, and more [3]. Beyond that, we use the OpenTelemetry spec for our SDKs [4], which has made it pretty straight forward to support several languages, even with our small 4-person engineering team!

Our product is completely self-serve at app.highlight.io. Installing it is as easy as a npm/yarn import and installing the backend sdk of your choosing. In addition, given the privacy-centric nature of session replay, we also offer the option to self-host [5]. Highlight.io currently makes money off of our hosted offering, and our self-hosted deployment is completely free. We’re also toying with the idea of an “enterprise” self-hosted deployment, similar to gitlab’s billing model, and thoughts from the community on this front would be appreciated!

And as far as what’s next for us: Our customers are asking to render logs and traces on a highlight.io session (and vice versa), and we’re excited to be going deeper into a developer’s debugging stack. The long term goal is to build a platform that connects replay, errors, logs and more so that engineers can “playback” the full state of a web application.

Overall, we’re quite new to the open source scene and would love the HN community to share their feedback on what we’re building. If anyone has opinions on where we’re going, or what they’d like to see in an open source monitoring product, we’re all ears. Check us out at highlight.io and at github.com/highlight/highlight to give us a shot.

[1]: https://www.highlight.io/docs/getting-started/frontend-backe...

[2]: https://github.com/rrweb-io/rrweb

[3]: https://highlight.io/docs/general/product-features/session-r...

[4]: https://opentelemetry.io/docs

[5]: https://www.highlight.io/docs/general/company/open-source/se...

codethief · 3 years ago
Interesting, I hadn't heard of Highlight.io. Thanks for sharing your code!

How does your session replay feature compare to LogRocket (which I've always found quite impressive)? (Side note: I'm missing proper screenshots and/or videos on your website.)

Regarding your pricing: Am I seeing this correctly that, in contrast to LogRocket, you're not charging ridiculous amounts for team seats? This is what drove us (web shop, 30M user sessions / month, ~150 developers) away: For 100k sessions/month and 100 user seats they wanted $30k/year of which $21k were supposed to be just for the user seats. ( ._.)

Now we didn't expect 100 of our developers to use the tool regularly (as in every day) but since LogRocket told us accounts could not be shared or rotated and we expected many devs to access LogRocket at least every now and then, that price was beyond unacceptable.

podoman · 3 years ago
> How does your session replay feature compare to LogRocket

Forgot to reply to this. We've actually had quite a few customers switch off Logrocket because of our support for things like Canvas recording (https://www.highlight.io/docs/general/product-features/sessi...). From a session replay perspective, we have feature parity, i.e. we report everything in the devtools and even report network request/responses. We also support backend error monitoring and pairing those errors with network requests, for example, which is something logrocket doesn't do.

But overall, logrocket is very comparable tool, but as we mature, we're looking to become a more generalized monitoring tool (w/ logging, traces, etc..).

codethief · 3 years ago
Thanks so much for your detailed response! This does sound very interesting!

If I may ask a few more questions, out of pure curiosity: What are, in your experience, the biggest challenges when it comes to frontend error monitoring from the point of view of a company / a developer using it? How can these be addressed? And what are good criteria to evaluate frontend monitoring services by these days?

(Obviously, you'll be biased here but that's alright. :))

EDIT: I should add: Obviously I've got some experience when it comes to frontend monitoring (though not a ton) and could – in theory – partly answer these questions myself. However, sometimes I wonder whether the problems we face in our day-to-day are the "correct" problems. (I.e. is everyone else struggling, too? How do other people solve their problems? How do other people choose their frontend monitoring solution?) Hence my curiosity.

podoman · 3 years ago
Yep, we don't charge for seats at all. Just usage. In fact, we're likely going to be adding a lower tier so that individual developers can use the tool (at about $50/month). With the direction we're going with the product, we see ourselves having several "products" long term (error monitoring, session replay, logging, etc..) so there's a lot of value in making smaller teams happy w/ pricing, even as they grow.

And if you ever give us a try, definitely send feedback our way; we have a discord community up at https://highlight.io/community.

mdaniel · 3 years ago
I wanted to say thank you for picking a business friendly license!

Watch out for the (presumably old) GitHub organization in .gitmodules: https://github.com/highlight/highlight/blob/d818969acddd7c74...

OT1H, GH does a 301 to the new name, but OTOH unknown how long that lasts and is one more thing to go off the rails for fresh clones. It has been my experience that using the https URL makes clones easier for users which may not have set up ssh auth, but it depends on the target audience and whether you have some internal tooling that depends on that submodule coming down over ssh

vadman97 · 3 years ago
That's a great point! Just opened a PR to fix this: https://github.com/highlight/highlight/pull/4362

Switching this to https too since the our rrweb fork is now public (ssh reference was previously necessary for cloning a private fork).

nerdyadventurer · 3 years ago
> I wanted to say thank you for picking a business friendly license!

Is not AGPL business friendly, if not why? Only restriction that I know is you cannot provide it as a service. Databases like ScyllaDB are AGPL and they are doing fine.

seanw265 · 3 years ago
What do you say to potential customers who are concerned with the legality and privacy challenges associated with session replay?

I recognize how tools like this could level up product/marketing-level decision-making. But with litigation as recent as October 2022, I'm wary of incorporating something like this into any software I control.

podoman · 3 years ago
In the past, for customers in healthcare and fintech, we've had to actually pass on signing more restrictive contracts (BAA's, etc..) due to the liability. For such a small company, it was hard to be comfortable with situations like that.

Since then, we've done a couple things that will hopefully open up more opportunities to work with these sorts of companies. First of all, we have a "privacy by default" mode that obfuscates all text by default (see https://www.highlight.io/docs/getting-started/client-sdk/rep...). And secondly, we're hoping that open sourcing the tool will help these companies control the data they're collecting so they don't run any risks, and long-term, these companies pay us a license of sorts.

zeeg · 3 years ago
I think this is a great question, especially with the continued shift towards data residency and customer protection in recent years (GDPR, CCPA).

When we did this at Sentry - which is also built on rrweb (we help fund rrweb btw, everyone else should too!) - we realized privacy was really hard to get right (let alone data security), so we just block all elements that are at risk by default and it turns out the experience is still pretty usable. Ben wrote a bit more about this here: https://blog.sentry.io/2023/02/16/introducing-session-replay...

Not commenting on highlight, but most of the mainstream players in this space take an opt-in protection approach, and often its "you can block this specific HTML element".. which glhf getting right

podoman · 3 years ago
Highlight also has a private by default mode. Totally agree that this is the right way to do it. Also, congrats on the recent launch :)
codethief · 3 years ago
I see[0] your session replay feature also includes network requests. Do you fully capture request & response there and, if so, how do you handle the arising privacy issue? Does the customer have to set up filters by hand?

[0]: https://sentry.io/for/session-replay/

dnsmichi · 3 years ago
GitLab team member here.

Thanks for sharing!

> We’re also toying with the idea of an “enterprise” self-hosted deployment, similar to gitlab’s billing model, and thoughts from the community on this front would be appreciated!

Transparency and an open stewardship of the project can help: https://about.gitlab.com/company/stewardship/ Make it clear what features are paid, and define that for example free features are never moved to the paid tier.

podoman · 3 years ago
That's a good point; we should be clear about what will always be free. Right now, literally everything is free, but what we'd consider putting behind an enterprise tier that are top of mind are: - multi-tenancy - enterprise authentication features (sso, saml, etc..)

Re: the gitlab link you sent, we really like the transparency around not making a free feature paid, and the fact that all tests are OSS.

Overall, we'll work on something like this. Thanks for pointing it out!

rsavage · 3 years ago
Please don’t put basic features behind enterprise licences.

We are a small start up. We use SAML / SSO. We care about security. We care about privacy.

None of this is “enterprise”.

We stopped using competitors to you because of data privacy issues.

We would happily pay for a self host version, where we control all the data recorded.

These type of tools are extremely useful, and it hurts not having it. But they are just way too much of a privacy issue.

Self hosted and paid - is the future of data privacy.

I would love to see you (and others) off these options, it’s really a game changer and makes the switch easy from any product that is not self hosted and collects our customers data.

zeeg · 3 years ago
I would encourage you to consider not even trying to cripple the freely available self-hosted version. Personally I'm not a fan of open core or the GitLab business model, and Sentry has shown you can build a business without removing needed features from the self-hosted version. While folks may not agree Sentry is "open source" these days, our license effectively achieves the same from a customer pov as yours (which is ideal for customers). More people should take the risk IMO.

Ultimately people do not want to run all their own services if they can pay someone else to run it for them.

jonas-w · 3 years ago
Keep https://sso.tax in mind
r_hoods_ghost · 3 years ago
SSO and SAML are not enterprise features. They are table stakes security wise, and intentionally crippling your security model is a good way of driving off customers that take security seriously, no matter their size.
latchkey · 3 years ago
Please don't put SSO/SAML behind a paywall. Security should be a free priority.

Deleted Comment

a_c · 3 years ago
Super interesting project! I'm working on a related/similar project that I'm curious about the technical aspect. Would you mind sharing how are you simulating the reconstructed screen? Do you construct the DOM, together with the CSSOM, on every browser events, and send the result to the backend for replaying?

And do you monkey patch the fetch API to intercept request? Apology for not looking thorough enough on github!

How would you compare yourself with tool like openreplay? Disclaimer, I am not related to them.

An HN comment is not completed without a unsolicited feature request: a demo site, maybe using highlight.io itself, and let user replay their interaction.

Good luck building!

vadman97 · 3 years ago
Hey! Sure thing. Our frontend session replay happens in two parts:

1. The HTML DOM Recording using rrweb (https://github.com/rrweb-io/rrweb) 2. The monkey patching of fetch, xhr, console methods, combined with data from `window.performance`

The rrweb repo has a good in-depth explanation of how the DOM recording works. In short, it captures the full HTML of a page when a user first loads it, then tracks DOM changes via the `MutationObserver` api. From our client, these are all shipped to our backend as serialized events. We process the events per 'user session', storing them temporarily in redis, then compressing a permanent payload once the session no longer is sending new events in the local filesystem or S3.

Monkey patching network and console methods allows us to capture request/response payloads, headers, status code, etc. We then combine that data with the `window.performance` api's notion of network requests to ensure we capture all requests (even ones that happened before our monkey-patch had time to apply), as well as to get precision timing data.

Happy to give more detail about parts of our stack if you're interested!

vadman97 · 3 years ago
Following up regarding the other points I missed. Compared to OpenReplay, we have similar functionality in our session replay, but we've focused a lot of effort on making a cohesive error monitoring (backend and frontend) experience. We closely link sessions with errors (stacktraces and associated metadata) and vice versa to make it easy to get to the root cause of a bug.

To demo your own sessions on highlight, you can easily spin up our next.js example app that has highlight installed: https://github.com/highlight/nextjs-13-sample. All you need to do is make an account on highlight.io and copy your project ID over into the sample app!

a_c · 3 years ago
Thanks for the follow up. Definitely checking up the demo. All the best to your endeavours!
j45 · 3 years ago
> We’re also toying with the idea of an “enterprise” self-hosted deployment, similar to gitlab’s billing model, and thoughts from the community on this front would be appreciated!

First, Thanks for sharing - there might be a few interesting applications in my (non competitive to you) world I wouldn't have considered for the sheer amount of work to begin with.

My experience with B2B on prem has been that maintaining first Class On-Premise support will have an outsized growth vector for you. It's easier to do from the start than to try to shim in. If you really want it to be an unfair advantage and moat, being present as a first class verified citizen in the marketplaces of multiple clouds is good too for those groups who might want it.

For example, being ready to roll (and license for enterprise) int he Microsoft Marketplace is a bit of a superpower, because they already have a good chunk of the Fortune 500 already as customers, so you don't have to go get them and be close to being a one click install where an on-prem m365 environment might allow things from Azure directly or indirectly.

The harder part is making sure enough of the useful and even valuable bits aren't locked away from the spirit of open-source, or critical ones committed moving forward. It's hard but the hope is the magic of creating beginners for your platform remains the chief priority for growth and adoption for your system - people readily available and skilled in it.

One nice way to do this is to perhaps consider some creative licensing around headcount of an organization, sector, or yearly revenues. Appgyver had an interesting approach to this prior to being acquired by SAP and seem to have been able to maintain it. Still, some folks do well with an AGPL type of license to minimize competitors which I respect a groups right to do for their own sustainability.

verghese · 3 years ago
Could you tell me how this compares with other open-source session replay projects such as: 1) OpenReplay (https://openreplay.com/) 2) PostHog (https://posthog.com/)
podoman · 3 years ago
Having used the Posthog product (which is really good btw!), its an "analytics" tool, so their replay product is very good at mapping user funnels to sessions.

On the OpenReplay end, its seems we have feature parity for session replay, but they don't have support for error monitoring and therefore linking errors to corresponding sessions (as far as I can see).

Nonetheless, happy to see other oss tools in this space!

lharries · 3 years ago
PostHog started as a product analytics tool but we now have a fully fledged session recordings product including handling console logging and network requests for debugging. Mobile support is coming shortly too

Disclaimer: I'm Head of Product at PostHog

tough · 3 years ago
I didn't think postHog was a good comparison either,

I knew of rrweb https://www.rrweb.io/

Great to see more open source contendants in the space

mehdios · 3 years ago
Congrats on the open-source move! It's always good to see more players in the OSS space. We do track JS errors at OpenReplay and of course link them to recordings. Errors come with stacktraces, are also fully searchable and can be used as cards to build custom dashboards. In terms of backend errors, we took a different approach at OpenReplay. We pull them from third-party systems (i.e. Sentry, Elastic, CloudWatch, Datadog, etc.) and sync them with the corresponding replays.
a_c · 3 years ago
On debugging side, sentry (recently?) rolled out replay too https://docs.sentry.io/product/session-replay/
bentlegen · 3 years ago
We launched this last week! Right now it's only available on SaaS (sentry.io), but we've been developing it in public in our main server repository (github.com/getsentry/sentry) and plan to have proper instructions for self-hosted users soon.

Deleted Comment

mehdios · 3 years ago
PostHog, Highlight and Sentry are all using rrweb. At OpenReplay, we use a full proprietary solution that we built from the ground up and covers all session replay aspects (tracker, protocol and replayer).
j45 · 3 years ago
> We’re also toying with the idea of an “enterprise” self-hosted deployment, similar to gitlab’s billing model, and thoughts from the community on this front would be appreciated!

First, Thanks for sharing - there might be a few interesting applications in my (non competitive to you) world I wouldn't have considered for the sheer amount of work to begin with.

My experience with B2B on prem has been that maintaining first class on-prem support will have an outsized growth vector for you. The cloud after all is someone else's computer under the control and ownership of someone else, and that won't work for everyone. It's easier to do from the start than to try to shim in. If you really want it to be an unfair advantage and moat, being present as a first class verified citizen in the marketplaces of multiple clouds is good too for those groups who might want it.

For example, being ready to roll (and license for enterprise) int he Microsoft Marketplace is a bit of a superpower, because they already have a good chunk of the Fortune 500 already as customers, so you don't have to go get them and be close to being a one click install where an on-prem m365 environment might allow things from Azure directly or indirectly.

The harder part is making sure enough of the useful and even valuable bits aren't locked away from the spirit of open-source, or critical ones committed moving forward. It's hard but the hope is the magic of creating beginners for your platform remains the chief priority for growth and adoption for your system - people readily available and skilled in it.

One nice way to do this is to perhaps consider some creative licensing around headcount of an organization, sector, or yearly revenues. Appgyver had an interesting approach to this prior to being acquired by SAP and seem to have been able to maintain it.

, some folks do well with an AGPL type of license to minimize competitors which I respect a groups right to do for their own sustainabilty.