Maybe I have reading comprehension problems but this article sounds so conflicting and presumptuous. Is it a sales pitch or not? If it is, even when it says it isn't, I would start by looking at Arcan's bus factor, and what refactoring and architectural changes were necessary along the years.
Having no expertise in this field I have no patience for people who write in this style but never got involved in the process. But if they did then I would be more interested in reading why their proposal was better and what can be done now in the living code. Also IIUC protocols are exactly that, places to experiment and learn from mistakes until they get stable and anyone can propose them.
There are many developers who could write a good technical solution but it's the ones who can deal with the ugly politics that make a real difference. There's a phrase about that there but as always if you don't send code your opinion won't do any good.
I was feeling the same! I came away from this article even more confused as to what's been going on.
I used to be a maintainer on Xfce (though it's been a decade since I was active), and I've been worried (as a user) about the "upgrade path". Porting (if you can call it that) from X11 desktop environment to Wayland compositor (etc.) is a huge step to take, it seems, and Xfce's current core team doesn't seem interested in going in that direction (I don't blame them).
As a user, I want to just keep using Xfce until the heat death of the universe. As someone who might get involved in Xfce development again, I worry that any work I'd be doing on the X11-based things might be wasted time in the long run.
I wouldn't do that. Only move to Wayland when there is a real path for what you want and need. And it does it without hacks.
About porting I would look at what MATE is doing or at the code of Wayland shells, protocols or things like Waybar at GitHub. You don't have to write an entire compositor if you don't want to. Even better get in touch with the people who know what they are talking about, I think they have a wayland IRC channel.
There are a lot of unmaintained WMs that work well and are amusing for historical value or retro computing, and probably some of them have a small number of real world users too. I like the idea of a compatibilty layer for those.
Maybe not for a big project with ongoing development. But as a last resort to run some cool stuff.
It seems it would be difficult to do that without combining both protocols into one server. (This is not quite what xwayland, xweston, xwlnest do. They run the server X out of process and can't manage wayland windows because they don't know anything about the internal wayland state)
To that point, you would have to decide if it's more worth it to fork xwayland and merge with an existing wayland compositor, or to fork xfree86 and add wayland.
I bet you could achieve it with three layers. First, run a "top level" Wayland compositor. Then, use XWayland to run an X session (with your favorite window manager). Then, use Weston's X client mode to run Wayland applications inside the X session.
As silly as that sounds, it might actually be a good solution. Red Hat gets to push their shiny new display server protocol, and I still get to make screen recordings and use my favorite window manager. I wonder how well it works in practice...
That would sort of work too, but it comes with its own set of issues. Keep in mind that there is very little reason you would actually want to do it that way when either way you're dealing with the same impedance mismatch. In other words, you will have to solve the same issues in the display server you're running into now with screen recordings and strange window manager behavior. It's not something that you'll just get for free. If it was, the other Wayland implementations would have it fixed by now. There's no magic solutions to this without any brokenness or workarounds—the crux of the problem is that you have two protocols you're trying to match up that are compatible in some ways and incompatible in others.
Also I think it's a stretch to say the push behind Wayland is coming from Red Hat. Maybe within GNOME, but there are other implementations with no connection to Red Hat.
I don't think the impedance mismatch is the same either way. The Wayland protocol is much more restrictive than X's -- things like screen capture and window managers don't work because the APIs they need aren't present. Conversely, there are X APIs for everything a Wayland application is able to do, so a Wayland-app-in-X shim would probably work a lot better.
The outermost layer, where you run the X session inside a Wayland compositor, would probably work alright too because it just needs to send pixels to the screen and accept mouse/keyboard input. Wayland is able to do both of those things!
>Wayland protocol is much more restrictive than X's -- things like screen capture and window managers don't work because the APIs they need aren't present
This is not quite true. There are protocols for these things in some implementations. You'd want to keep compatibility with these because otherwise you'd have no reason to be supporting Wayland. Keep in mind either way you're talking about using a specific compositor—it doesn't matter if the APIs aren't present in the protocol. In this situation you'd re-use one from another implementation or add another private protocol extension. That's always been what individual compositors are encouraged to do. It's no different here.
> This is not quite true. There are protocols for these things but they live in multiple places.
Every Wayland window-manager-compositor-hybrid invents their own protocols for these things. This basically means that you can't write a program that does these things -- you need three different versions of every API call you make, and that's just to support the popular desktop environments that people happen to be using this week. So no one has, so the only Wayland protocol features you need to support are the basic ones to open a window, draw into it, and get mouse and keyboard input.
> Keep in mind either way you're talking about using a custom compositor
No, I suspect the "top-level compositor" could be any off-the-shelf Wayland compositor that supports XWayland and can make a window full-screen.
Here is the thing: it's trivial to map those basic features back to X. That's already mostly been done. If you continue with that solution you'll have the same problem in reverse—assuming X support ever gets removed from the toolkits, then in that situation you'll get Wayland apps that depend on some Wayland functionality that doesn't work. See what I mean?
(IMO we're pretty far off from X support being removed if it ever even happens, so I wouldn't worry too much about that right now)
And I actually don't think it's a big deal that there are separate implementations. You can absolutely still write programs that do these things, and people do. Usually you start with targeting just one implementation and then you port it to the others. All you need is an API to abstract the custom protocols. A lot of those differences are already abstracted in the portals API which has those three separate implementations so you may not even need to do any porting if that fits. Worst case, if something isn't in the portals API then you have some three clause switch statements when you make protocol requests. Then after you stabilize it, move those switch statements into a library and then other people can use it as an abstraction layer, and nobody needs to write the switch statements again. Yeah it's work, but for an app developer it's actually easier to do that than getting all the upstreams to standardize everything.
Here's the thing about Xwayland: Xwayland is Xorg. If you look in the xorg code base, you will find that it's separated into components called DDX and DIX. DIX (Device-Independent X) handles protocol and server state; DDX (Device-Dependent X) does the drawing. One of the back ends in the DDX code is for Wayland; when you compile Xorg with just this back end, you get Xwayland.
So if Xorg gets abandoned -- oops! There goes Xwayland as well. I do believe that this was the plan all along: keep Xwayland around as a solution for "legacy applications" for a while, then stop supporting X altogether to make it less attractive as a back end for new applications and toolkits.
Yes, I am familiar with the Xorg code base :) The point I was getting at is that if you want to keep using an old protocol far into the future, someone from the community has to step up and maintain something. That's true of any old technology faced with waning influence over time.
It is a compromise for how Xorg and Wayland can have a shared graphics backend that improves both sides and the part that the Xorg maintainer wants gone can go away.
Arcan has had its own fork of Xorg for many years, as well as a Wayland implementation. Nothing here will change that trajectory.
The library mentioned is about ~2kloc, decoupled from the rest of the project (100+kloc) and the relevant subset has not changed significantly for several years.
The core protocol was designed to be much simpler and more representative of modern graphics hardware (basically just a way for applications to get a buffer they can throw their pixels onto, which is then composited onto the screen, and passing of keyboard and mouse events to the application only when it's actually focused). Partly the issue has been getting good support for this in drivers and applications, and partly the issue is the base protocol doesn't allow for a bunch of features by default (a lot of which in X were essentially massive security holes, since any X client can take screenshots, intercept keyboard/mouse events globally, etc). These features are being added, but it seems as custom extensions in each compositor without widespread support.
It's created with a different core approach and architecture, so you can't just change API calls in some software to support it. Whole new pieces of software have to be written using a different approach (mainly around keyboard event, clipboard, remote access and screen recording features).
> There are strong differences that will not get smoothed over regardless of how many 'protocols' you define; Wayland is Policy over Mechanism, X11 is Mechanism over Policy.
Why yes, that's the whole point.
I don't know if you've noticed, but "policy over mechanism" has thoroughly won in the desktop space. The dominant desktop is Windows; even macOS has several times the mind/market share that Linux has.
This is pretty much the "Emacs modernization" debate for the desktop as a whole. There is a growing contingent of Emacs users who saw their chosen editor get its lunch eaten by Visual Studio Code, and now wish it was a bit more modern and a bit less... well, less Emacs. Similarly, Wayland was pushed by a large contingent of users who want to see the Linux desktop resemble a successful desktop used by millions to get their work done.
Back when GUIs were brand new and the Unix community was populated by neckbeards, creativity and experimentation in the desktop space was a good thing. Today, those neckbeards have grown older. Time for them is shorter. So even they don't want to experiment anymore; like everybody else they just want to get their fucking work done. Millions of people have gotten work done with the successful model of Windows and Mac, and Linux users want their desktop to resemble the most successful desktops of all time -- not something that is a large pile of hack upon hack with seams constantly showing.
Pushing back against the direction the ecosystem is going will only breed resentment and ill will, because you are committing time and brainpower to a dead end when you could have committed it to making Wayland better.
> I don't know if you've noticed, but "policy over mechanism" has thoroughly won in the desktop space. The dominant desktop is Windows; even macOS has several times the mind/market share that Linux has.
Windows exposes lots of WM mechanisms to a developer, hence why something like vrdesktop.net could be built in the first place, or previous WMs like LiteStep for that matter.
It is also why WINE does not, and cannot, have a working Wayland backend while sticking to the protocol. The same is true for X, but there XWayland requires the compositor developer to implement a sideband window manager to hack around the entire Wayland protocol to get X clients to work.
I haven't used a Microsoft OS in a long time but somebody told me a few years ago that alternative shells like Litestep became severely limited after the UI changes in Windows 8/10. Not sure if it's true or not or why that is.
I think you missed the mark on this though. I never saw Wayland as being any less about mechanism over policy. In theory X does have more mechanisms to work with but in practice the well-behaved programs that get used a lot tend use toolkits that follow ICCCM/EWMH strictly, and won't do strange things that are out of the ordinary with the policies of the popular desktops.
Compared to Wayland where the idea seems to be that WM implementers actually got more freedom to add any policies they want because they also implement the display server. From a client perspective it's still the same as X. Clients get what they need to function in isolation and don't really have to care about a WM's policies, and when they want to care they check for a specific policy. (In X they would check for an atom) I guess that probably doesn't work so well for Arcan because it also doesn't set anything in the way of policy. Maybe it would be possible for Arcan users to write policies as wayland extensions in lua?
The mechanisms in Windows changed for doing some parts of it - as far as I recall it was due to the access controls (UAC) and DRM around the Window Manager (DWM) but it was a while ago, some vendors have figured out parts of working around DWM again - see the VR desktop mentioned and Stardock WindowFX.
I am not saying that the ones that X have are great or even that good to begin with. A better set could have likely avoided the ICCCM etc. but you can do a lot with just reparent+position - WMs are mostly normal clients after all.
Compare that to say the practically obligatory 'xdg_wm_base' specification, you are much more locked down, and the objects to work with are strongly interconnected. If you want to work around that as a 3rd party developer it's a political process + waiting for n*m compositors and toolkits to adapt.
People tend to miss that Arcan is a Wayland compositor as well, but those clients gets a special path where the WM has to follow extra sets of rules because of it.
I see it as a gradient. Neither is entirely Green (Mechanism) or Red (Policy) but if you look at the sets of both sides and position, I doubt you'll assign the same colors.
There's no political process. In GNOME if you want to do strange things with moving and reparenting windows from other processes then you write a GNOME extension. Same with KDE and kwin scripts. No touching of Wayland necessary. Fundamentally it doesn't seem that different from writing Arcan WMs. If all that's not enough then you write a new Wayland compositor with the policy you want, maybe using wlroots to save time. Overall that really hasn't changed much from writing X WMs, I don't get what the concern here is. At worst, it is a change that can be adapted to just like those changes you describe in DWM.
>you are much more locked down
Is this not what a policy is? The window manager locking down clients and enforcing that they can only do certain things? I'm aware that Arcan got wayland support recently and that actually gives me more hope that we're not headed towards an apocalypse of protocol incompatibilities.
Need some shuteye (9am 'into the night'), but I think we are running into a similar misunderstanding that I spotted you having in the other Xorg thread.
Is what I mean with political process. Patching the compositor is the wrong side of the equation, it doesn't work with networked clients and it's the client that needs most of the features. It's the client that needs to have a more expressive surface to work with. You can do these things safely and securely without the overbearing constraints and complexity of the policies that we are practically stuck with. See the trayicon/stream-deck articles.
> I'm aware that Arcan got wayland support recently and that actually gives me more hope that we're not headed towards an apocalypse of protocol incompatibilities.
I don't know if 4 years is recently in that span (see the dating my X post), but Xwayland support was initially decided as off limits. That was added recently because, as it turns out, those clients are still the ones that behave the most predictably and are the most robust. Even for applications that should handle both and I need to choose explicitly (arcan-wayland -exec-x11 the_thing or arcan-wayland -exec the_thing) I really only use the x11 side. Even for gtk and Qt. That's the real tragedy.
I would say the babel-split is practically here because of all the sidebands and ad-hoc protocols that have been introduced specifically as a reaction to the strictness and limitations of the initial designs and the poor underlying primitives (as in the wire format and its de facto implementation).
The wire format and the initial implementation are definitely flawed, but if you fix them you end up in the same situation again where you break everything every release and you go through all this politics trying to get everyone to use yet another new protocol and implementation. I see various complaints about that and it always comes back to the same thing. We have alternative implementations of the protocol but nobody uses them because of this. Consequently this is the same reason that it took so long to fix issues in X.
I see what you mean with that document, but if you have been watching then none of that stuff is new or unique to Wayland. What you may have missed is that this document is a reaction to the politics that already were around in the X days. (I was alluding to this earlier when I was talking about WM atoms, the popular desktops already have their own sets of policies there that are necessary and incompatible and in some ways do limit the expressive power of clients)
Not working with networked clients is a minor detail. It's trivial to have a user upload a compositor script to a different machine. If it becomes annoying to do it manually then a client that needs more features simply needs an automated way to do this. Still no need to touch the Wayland portions there either. I actually do appreciate the efforts to fix these issues with the whole shmif setup and I sympathize, but it's the same uphill political battle if you want to push that out to the rest of the ecosystem.
Part of it is that you either go to extreme lengths (like arcan-wayland) or it embeds itself so deep in the code base that it will be hard to get anywhere. If it doesn't get fixed, well, you don't have to be a reliability expert to be worried about this one:
I have probably chased ghosts deep inside GTK from this one alone for multiple full time weeks, to no avail. There will eventually appear a rather harsh teardown post of some of the major mistakes in the implementation (and that alone, no xml business), wayland-server might not improve, but future designers should be able to stumble upon it.
I think to get further with the discussion we need a sort of list of what mechanisms we consider mechanisms, and how policies project over them. Not to be disrespectful, but my budget for engagement is thin, pandemics, sorry :-/
The politics stuff - so I have some logs that precedes the formation that are quite incendiary and outright vile, but there is enough shit flinging to suffice to stop at there being good reason why I wouldn't consider getting closer than bayonet range, and why the post was obtuse enough to discourage most #metoo and discard me as an idiot who knows nothing. The tech stuff matters though and that's only what I want to see. A protocol and not marriage as if from 'the war of the roses'(1989).
There is a fourth option that might materialise, but that's for then.
X was really bad for me and Wayland has been very reliable. Good enough for my work.
Your posts are very political stuff too. If you want to be strictly technical just skip all the meta/drama and the boogeyman talk. In the post and in the comment section. It doesn't put you in a good light even if your statements turn out to be true.
Your projects look very interesting. A suggestion for your next blog post would be talk about what you want to achieve, your real intentions, and how you plan to do that. If it's just a showcase make that clear. It's unfortunate that you don't want to work with what they have now but I hope you can find a compromise. It would be great if your contributions could make Wayland even better.
"Good enough for your work" until issues like the one mentioned appears; You plug in a kvm, close the lid or something at the wrong time, come back and see your work gone. You then blame the client, the gpu or hardware, then the toolkit, then the compositor and everyone says "can't reproduce" because the infrastructure guys went EWONTFIX because 'difficult'.
It is a design blog, it is inherently political. There's a high number of tech posts on the main site. The roadmap, the intentions, everything has been spelled out, and visible for half a decade.
I hear what you're saying and I also can't stand the namecalling and political infighting that tends to happen in some open source communities. Unfortunately I doubt a display protocol is ever going to fix that.
On that bug, that problem seems to go deeper than Wayland and I don't know if there is any way to fix that in all cases on Linux due to the nature of SIGSTOP. I know you may not answer this, but wouldn't that also affect X too? Or does it have some other way it deals with it?
Xorg is better, but not by much. On write-fail they setup an internal buffer ~16k and if that one is saturated it is closed (that + whatever the kernel has). The protocol/packing is not that busy in the server to client direction so it lasts for quite a while.
The way the disconnect happens is interesting though, I am not certain enough about the threaded-IO implementation in regards to hotplug but a quick glance looks like if a hotplug occurs at the same time a client is disconnected the sparse allocation requirement of open() could get an input device confused as a client as it is cleaned up. The odds of that happening though :D
In the wayland code it looks like they tried to have a buffer strategy at some point, but then got too clever for their own good (epoll designs + asynch-OO projected over C is... yeah) and somewhere in 'closure/write/buffering to lower #syscalls/callbacks' landed in a long chain of error propagation and immediate kill.
The best way I found (hence using it) is to have memory mapped ring buffers and just have an atomic head-tail indicator. This makes file-descriptor transfers (DuplicateHandle from win32 fame is just so much prettier) messier, but it means that you can have a recovery strategy and see the backpressure buildup and rate-limit. Since it is mostly input events or 'drag-resize' (and those can be merged) in that direction, silently dropping them and have mouse cursor 'jank' is better than losing your work. If all else fails, the watchdog handle is pulled and the client is set to migrate to a fallback server (if set).
Having no expertise in this field I have no patience for people who write in this style but never got involved in the process. But if they did then I would be more interested in reading why their proposal was better and what can be done now in the living code. Also IIUC protocols are exactly that, places to experiment and learn from mistakes until they get stable and anyone can propose them.
There are many developers who could write a good technical solution but it's the ones who can deal with the ugly politics that make a real difference. There's a phrase about that there but as always if you don't send code your opinion won't do any good.