Tech @ SpotDraft: Front-end Migration to Monorepo

The front-end team at SpotDraft migrated our conventional Angular workspace to an Nx Monorepo. Read this article to know why we decided to do it and how it was done. Also, learn about the challenges we faced, results obtained, and the plans we have moving forward.

Why Monorepo?

Until January of 2022, customers used SpotDraft only through our main web application. This was built using Angular and we had a conventional Angular workspace with 2 projects (app & component library). Then we saw our integrations roadmap and realized that soon we will be building integrations and add-ons on other platforms, starting with a Word Add-On, followed by a Chrome Extension, and a SalesForce application.

All these (& many more) platforms had one thing in common - their add-on APIs allowed running web-apps in one form or another. This was a great thing, we already had a component library (DraftKit) and if we found a way to reuse parts of the main app, we could give our users a consistent SpotDraft experience wherever they were.

Hence, we were in the lookout for a solution where we could build multiple apps and reuse code between them seamlessly.

Why Nx?

Angular workspaces already allowed us to build different projects (apps & libs). However, these workspaces are made for projects with a library and a reference/demo app. We knew our monorepo would not just contain more Angular apps, but also an Express app, vanilla TS libraries, etc.

In addition to this, the size of our codebase was already a problem with increasing time spent in CI (test & build) which led to slower merge times. Nx comes with a built-in cache that is smart enough to figure out exactly what it needs to lint, test and build, and we knew we could save a lot of time here.

The migration

State of The Repo

When we started migration, our workspace had one library project and one app. We had 330+ modules, over 800 components. And, a team of 8 engineers who were consistently shipping things.

The last point meant that the migration had to happen with minimal disruption of our day-to-day work.

Dry Run

To minimize disruption and avoid surprises, we decided to first perform the migration on a fresh workspace, document issues and steps, and then re-run the script over a weekend on the actual repository.

At this time, we were using Angular 9 (now at 11) and the Nx CLI did not support automatic migration of multi-project workspaces.

We followed the following steps - 

1. Created a tag in the main repository to ensure we have a snapshot of the pre-migration workspace. And, this tag has come in handy a couple of times since. 
2. Created a completely new workspace. 
3. Created a new library project and copied our component library there. 
4. Created a new app project and copied our main app there.
5. Manually copied our custom angular.json configs to their workspace.json version. However, this step was pretty straightforward.

Once we got the test and build to run successfully, it was ready to be merged. At this point, our intent was not to leverage Nx Cloud, but just to get the builds working in the new structure.

Note: Creating a fresh workspace is not always required, however, we knew how big and diverse our monorepo could potentially be. So, we decided to start afresh with the recommended structure to avoid running into issues in the future.

Connecting the Cache

Once the migration was done, we all got busy building and shipping new features and apps.

5 months after adopting Nx, we got the bandwidth to actually connect our repo to the Nx cloud to take advantage of remote caching.

During this time, our monorepo was home to about 40 libs and 4 apps. Since libs and apps are the artifacts that Nx caches, this was also the right time to actually take advantage of remote caching.

Note - Connecting cache when we had 2 projects would not have helped a lot as invariably. But with 40 libs and 4 apps in place, we were able to leverage the remote cache more effectively. 

nx affected

Nx has a very handy command, nx affected. Given a base and head commit, this command figures out which libs and apps have changed, and then is able to test, lint and build only those projects.

Tests

We followed this excellent guide on running affected tests in GitLab CI.

While the guide provided an excellent starting point, we had to make some changes to our before_script implementation as the variables we needed were not always reliably set.

This is what our script ended up looking like -

• If the change is made on the master branch, then we can just compare the latest commit with the commit before that.
• If the change is not made on the master branch, then we compare the latest commit on our branch with the latest commit on the master branch.

Results

In the last 30 days (since August 10th 2022), we have saved 41 hours in running tests in CI.

Incremental Builds

In order to take full advantage of Nx Cache, all libs in your workspace should be buildable. While setting up caching for tests was easy, the build turned out to be a different beast altogether.

In the time between migrating to Nx and connecting the cache, our repo had gained over 40 libs, and many of them moved out from our main app so that they could be used in other apps.

Turns out, when building an app as a whole, the builders are able to take care of a lot of inconsistencies because they treat the entire code as one source. The moment we started building the apps with --with-deps, all kinds of errors started coming up.

This is what we learned - 

1. When building as a single app without --with-deps , circular deps where the lib imports from the apps is not a concern. However, when building the lib individually, it’s a blocker for obvious reasons. 
2. To import styles in your scss files from another lib, it has to be declared in the ng-package.json under styleIncludePaths. Another challenge here was that the error reported was not correct, so this took some time to figure out. 
3. We need to ensure that the import path to a lib under tsconfig.base.json matches that defined in the lib’s package.json. A mismatch here can cause the build to not be recognized by its dependents. 
4. You can have nested folder structures like libs/foo/bar for organization, but your import paths should not be nested, i.e. prefer @org/foo-bar instead of @org/foo/bar. This can also cause issues with how outputs are placed in the dist folder and might require you to manually fix the dest property in your ng-package.json file. 
5. All components, pipes, and directives exported from a module, need to be exported from the public-api/index.ts of the library.

Note: If you forget to enable the buildable flag when creating the library, you can use the@trellisorg/make-buildable package to do it after the fact.

Results

At this time, we are still only building our main (& biggest) app in CI automatically, and here are the numbers for the last 30 days (since August 10, 2022), we have saved around 78 hours of computing time.

These numbers might not look impressive yet, see the Next Steps section on what we plan to do to make this better.

Challenges

While migration has been a huge positive for us, there are some challenges that we have faced along the way.

What’s in a lib?

• This, in my opinion, has been the biggest challenge we have faced - what should be a library, and what’s the exact scope of it?
• We have now made it part of the eng spec process to discuss the creation of any new libraries.

Since we follow Clean Architecture pattern, we usually end up with something like this -

The imports are usually @spotdraft/feature-x-core, @spotdraft/feature-x-api, etc.

Selector Prefix

Angular components, directives and pipes need to have a prefix. When we just had a component library and app, it was sdk-* and app-* respectively. But now we need to think about this prefix every time we create a library.

Test Runner

Nx’s default runner is Jest, but some of our old code still runs on Karma. This is not a pain as such, but something that we would like to standardize.

VS Code Imports

When importing code from another library or the same library, you can set VS Code to import relative or absolute.

For eg, importing from a file from lib-a inside the library, should be a relative import like -

But importing in another library/app, should import like

The second import statement in the first context would break the build.

VSCode has a setting called project-relative that solves this exact problem, however, we cannot use it till we upgrade to TypeScript v4.2. This is something that cannot happen till we upgrade to Angular 12.

Next Steps

• More cache-able libs: Since most of our code still resides in our main app, we still end up with relatively long test and build times. We will continue refactoring our main app to move all features to libs, such that all apps are just shells to compose features together.
• Moving our express server to the monorepo: The FE/TS team manages a supercritical node microservice too. There is an overlap in models and interfaces here, and we would like to move this to the monorepo so we don’t have to copy them manually.
• Moving other mission-critical projects: Once we move the server to the repo, we want to move other projects so that both the server and app rely on monorepo for a more integrated dev experience.
• Automated builds for all apps by using affected more effectively.
• Creating custom generators and schematics to speed up development.
• Upgrading to newer Nx versions to utilize workspace lint rules.

Conclusion

In conclusion, moving to Nx has allowed us to not only ship more apps, but also move faster as our team has grown from 8 to 18 engineers. I really hope this story was helpful to you if you are considering moving to Nx or have made the decision already.

We really wanted to share a real-world account of how we approached this migration, the challenges we faced along the way and our results.

PS: If this sounds exciting, our team is looking for Senior & Principal Engineers to help us take SpotDraft to even more platforms and solve some really interesting problems as we scale our team and our codebase. Feel free to reach out to me on LinkedIn or Twitter or simply apply through our Careers page.