Dynamic Workspaces (alpha) for Windows: A new way to interact with version control

Dynamic Workspaces (alpha) for Windows: A new way to interact with version control

Monday, July 26, 2021 Pablo Santos best practices , plasticfs 0 Comments

Big fanfare and excitement to announce the greatest, biggest addition to the version control space in years!

Here's what you can do with a Dynamic Workspace:

• You can create a workspace pointing to a repository of 102 GB and see it consistently on your computer in under 2 seconds.
• You can mount a huge repository in your laptop where the working tree is 25 TB and navigate it even when you don't have that much disk space available.

How cool is that? We believe it is the most significant capability we've ever added to Plastic. A giant leap. We believe Dynamic Workspaces will transform the way we interact with version control. They are the foundation for what we consider the future of version control.

Don't believe us? Watch this quick demo:

A consistent view of your repo in zero time

You can create a workspace that points to a gigantic repo and virtually navigate it locally as if it was really there. Still, nothing will actually be downloaded unless you really access a file.

This is because a Dynamic Workspace relies on a virtual filesystem. The virtual filesystem has strict control of every file being accessed. This way, while displaying the entire directory tree, only the files required will be hydrated/downloaded onto your machine.

A better way to manage disk space

Some repositories are gigantic, in the range of several GB or even TB in the repository's head. So, downloading a full working copy becomes terribly slow.

This is especially true for monorepos. They are a very extended practice nowadays, which helps simplify projects by simply adding everything inside a single repo, so the team forgets about repo boundaries. The downside is that soon the team will need ways to select what they need to avoid slow download times.

The advantage of Dynamic Workspaces is that it manages the actual disk space for you by grabbing only what you really need.

Example: Your head in the repo is 700k files and 15GB. But to build the "build-level" utility, you'd just need a few hundred files and a few megs. Well, instead of having to configure your cloaked.conf (or some sparse checkout if you are familiar with the Git jargon), you just create a Dynamic Workspace where you see everything. Then you navigate to the correct directory and trigger a build, and only the files you really need will take space in your disk.

On top of that, Dynamic Workspaces add some extras:

• They use a Least Recently Used (LRU) cache system, meaning the actual space used by workspaces can be controlled, and files not recently accessed will be removed from the local cache, freeing up space.
• The same cache is used by multiple workspaces that point to the same repo, so you can have as many copies as you need, but the actual space will be just a single copy.

Let me elaborate on the multiple workspaces. Suppose you are working on your project in workspace_A and want to try a new experiment on a different branch. Of course, you could switch your workspace_A to a different branch, but with Dynamic Workspaces, you can create a new workspace_B pointing to the other branch and work on it in seconds, reusing all the local content already downloaded for workspace_A.

In short: suppose you have a repo where the head of the main branch is 1 GB. Suppose you download it entirely to disk in workspace_A. Now you can create workspace_B, workspace_C, and workspace_D, but instead of using up to 4GB of disk space, the total will still be 1GB because they all reuse the same local cache.

Ultra-fast change detection

A Dynamic Workspace is based on a virtual file system that controls every single disk access. It means Plastic doesn't have to "find" changes because it knows the changes as soon as they are completed.

This means a gigantic performance boost because we can now show changes in almost constant time no matter how many files and directories the workspace has.

Plastic already takes advantage of file system watchers in Linux, Windows, and macOS, but the virtual file system gives us a new tool to precisely track changes.

Ultra-precise change and move detection

Whenever you move a file in your Dynamic Workspace, whether from the command line, Windows Explorer, or any other tool, Plastic is involved in the move because it controls the virtual file system underneath.

This means the detection of files and directories being moved is more precise than it ever was. This opens a new world of possibilities for even better refactoring detection at the directory structure level.

However, as the current generation of editors tend to do all kinds of crazy things to edit files, there are some challenges. This pattern is widespread: do you know what Visual Studio or vim do when you edit and save foo.c? Well, they create a new file foo.c.temp (not the actual name), write to it, then delete foo.c, then rename foo.c.temp into foo.c. An actual "save foo.c" would be much easier for us to deal with, but this is how most editors work. We envision some of this will change as soon as Dynamic Workspaces become widely available.

Machine learning powered download prediction

The Dynamic Workspace downloads files on demand. So when you create a workspace, you see everything, but in reality, not everything is downloaded.

But, when you trigger a build and your build system needs to access a few thousand files, downloading them one by one will be super slow.

So we added a prediction system based on previous use by other teammates. So, plasticfs knows that when you read foo.c, most likely other 100 files will be read after too, so it starts downloading them in parallel, so the one by one fetching doesn't happen, and everything is faster.

And more coming

This is just the beginning. We plan to take advantage of the virtual filesystem to implement extra features like auto-save, so every single change is backed up transparently, and you never lose a change, workspace-wide.

We've already implemented cool things like the ability to browse the history of a file using unique directory entries, as you can see below, but more of this to come!

Watch these videos to learn more:

• Dynamic Workspaces - Extra filesystem features
• Dynamic Workspace - Interact with the filesystem from CLI

How to enable it

If you're more of a visual learner, watch this video to configure Dynamic Workspaces.

Complete these steps:

1. Ensure you have a recent Plastic SCM installation on your Windows machine.
2. Navigate to your Plastic client binaries directory (typically c:\program files\plasticscm5\client).
3. Run plasticfs.exe:
   
4. plasticfs will ask you to install WinFsp, a critical component we use to develop filesystems in userspace.
   
5. It might ask you to add some ignore rules to the Windows Defender for performance reasons. Antiviruses are very intrusive and tend to degrade performance of Dynamic Workspaces tremendously. Of course, you can also do this manually later if you wish.
6. Your plasticfs is started, and you can create dynamic workspaces! You can create it from the command line, checking the cm workspace create command options, but if you want a full GUI experience, you still have to enable a feature flag. Navigate to your AppData\Local\plastic4 directly (mine is c:\Users\pablo\AppData\Local\plastic4) and edit guiclient.conf:
   

There you'll find the following line in the file:

Change EnableCreateDynamicWorkspace from false to true. You'll be able to create dynamic workspaces from the GUI (remember to restart the GUI for the feature flag to take effect).

Extra step: Configure plasticfs to start up automatically when you log in to your Windows. (This is something we'll certainly change in the future, so you won't have to configure this manually). Create a shortcut for it in the "start-up" applications area.

To do that:

1. Type Windows-key + R and something like this will show on your screen:
   
2. Type shell:startup as in the screenshot above.
3. Create a shortcut to your plasticfs (c:\program files\plasticscm5\client\plasticfs.exe) here:
   

And you're good to go!

Unity speed up

While Dynamic Workspaces are general purpose and are not tied in any form with Unity projects, one of our key goals was to help speed up the open time of big Unity projects.

Let me share some numbers (take them with a grain of salt, as I'll explain below):

A Dynamic Workspace (or plasticfs workspace known internally by the dev team) can help speed up Unity when used with Unity Accelerator by a huge percentage. In our example, the open time went down from 40 minutes to just 3.5 minutes! Yes, the network we used was especially slow, and that greatly impacted the result. But the outcome here is that Unity + Dynamic Workspace + Unity Accelerator dramatically reduces the number of files that need to be read to create a build.

Instead of downloading 18GB, the Unity editor just creates the local 4GB required to build, but almost downloading nothing in this case (that's why the "size on disk after Unity open" goes down from 18GB+4GB to just 4GB => 18 GB is the actual size of the working copy).

We expect to continue improving this and making Unity + Dynamic Workspaces a robust solution for big projects.

FAQ

This is on Windows only. Do you plan on supporting Linux and macOS?

Yes, in fact, we already have working prototypes.

This is an alpha release. How ready for production is it?

We have been using plasticfs/Dynamic Workspaces for months internally. However, we expect we might encounter issues with some applications we still haven't tested due to how they interact with the file system and when they use patterns we haven't added support for.

Known Issues

• Remember to stop your plasticfs (tray app icon in your Windows toolbar) before upgrading your Plastic installation. The installer can't stop plasticfs yet.
• The performance of the dynamic workspaces is much slower when the Windows Real-Time Protection is enabled (x2 slower).
  • To work around this, you can add exclusion rules for the plasticfs.exe process and the paths where your dynamic workspaces are located.
• The "Go to Files" feature in VStudio doesn't work with dynamic workspaces. This happens due to an internal issue when the virtual file system is mounted via DefineDosDeviceW and GetFinalPathNameByHandleW in Windows.
• The dynamic workspaces only handle controlled changes. This means that you cannot have a locally changed file. it will always be checked-out.
  • This also means that you cannot locally modify a file that is under exclusive lock rules. This is something we plan to improve in the coming future but, we are not yet there.
• The transformable workspaces rules are not supported for dynamic workspaces. Not that they're in high demand, but just a heads up :)
• The cloaked rules are not supported for dynamic workspaces (and they make much less sense since these workspaces don't download all the server tree content if it's not accessed).
• Dynamic Workspaces only work with Unity projects without Accelerator from version 2021.2a.
• Dynamic workspaces perform much better with Unity projects using Accelerator than standard workspaces. You can take advantage of this using the latest public versions of 2021, 2020, and 2019.

In-depth: plasticfs, a virtual filesystem

Imagine you had a "versioned disk drive". This is what Dynamic Workspaces are. We implemented them through plasticfs: a virtual filesystem that ] interacts closely with the operating system (Windows in this case) to provide added functionality from the ground up.

Every single read or write operation any application performs on a Dynamic Workspace is controlled by plasticfs.

This means when you do notepad foo.c or code foo.cs or open a file with Photoshop, Maya, Unity, anything, Plastic will be in control of every single read and write file.

The possibilities are enormous: you can quickly switch versions, you can explore the history of a file just by interacting with the filesystem, every single change will be in control, even the most complex refactor will be precisely tracked, and many more.

We decided to start on Windows because this is where most of the Plastic SCM and Unity users are, but the goal is to expand plasticfs to macOS and Linux.

Democratizing Dynamic Workspaces

Dynamic Workspaces aren't new. 30+ years ago, the legendary Clearcase had "dynamic views". Today, Google uses its own filesystem to navigate their gigantic monorepo that doesn't even fit a developer's workstation or laptop.

Unity is well-known to have democratized game development. And following the same spirit now that Plastic SCM is part of Unity, we're trying to democratize access to this fantastic concept. You won't need to be a Google-scale company to take advantage of version control virtual filesystems.

The advantages are so huge, the new ways of collaborating and new toolsets so powerful, that we firmly believe that, if we get them right, Dynamic Workspaces will shape how lots of users interact with version control in the coming years.

A dream 10+ years in the making

Back when we started Plastic SCM, we had two dreams. One was developing SemanticMerge, a merge tool that understood code. We did it long ago. The second dream was dynamic workspaces: a filesystem to interact with version control.

In 2009,we had already put some thinking into it, but most of the other bits and pieces were still not there, so we kept pushing back, delaying the implementation. It was not the right time. There were too many things to do.

Fast forward to 2020, and after completing the acquisition by Unity and after we finished deploying our new cloud infrastructure and a vastly improved Unity plugin, we thought it was the time. No more excuses, let's make it happen.

So, for the last few months, one of the teams has been working on making plasticfs and Dynamic Workspaces a reality.

It is one of the hardest and riskiest projects we've ever done

since we changed the underlying repository structure in the jump from Plastic 3 to Plastic 4 back in 2011.

The great challenge is that every single byte read and written by any application using a Dynamic Workspace will go through the plasticfs code.

Just to give you an example: One of the main advantages of Dynamic Workspaces is that Plastic doesn't have to “guess” anymore if you've moved a file.

Suppose you do mv tmpRender.cs src/render/Render.cs.

You'd make this move in your disk, and then Plastic would find that tmpRender.cs was not there anymore, but now a very similar file named Render.cs exists... it knows most likely, it was a move.

This is different with plasticfs. Plastic knows you actually performed that move. This is a gigantic leap ahead in terms of refactoring detection.

But, the challenge is that today (and we foresee this will change with virtual file systems becoming more widely adopted), many applications do not behave very politely with the filesystem.

This is what happens when you do something as simple as vim foo.cs:

• The file foo.cs.tmp is created (the actual name might differ).
• vim copies data from fooc.s into foo.cs.tmp (itself, in memory, not using the disk).
• Then when you save, foo.cs is deleted, and foo.cs.tmp is renamed to foo.cs

So, instead of a simple "save", applications tend to do many changes instead, including moves and renames, that plasticfs needs to track and understand. t really is just a file save that the application wants to do.

Pablo Santos

I'm the CTO and Founder at Códice.
I've been leading Plastic SCM since 2005. My passion is helping teams work better through version control.
I had the opportunity to see teams from many different industries at work while I helped them improving their version control practices.
I really enjoy teaching (I've been a University professor for 6+ years) and sharing my experience in talks and articles.
And I love simple code. You can reach me at @psluaces.